Hi folks, I am using elastic-mapreduce in the Amazon EC2 eco-system and would like to upgrade from Hive 0.5 on Hadoop 0.20 to Hive 0.7.1 on Hadoop 0.20.205. I created a new metastore (on S3) to support testing the latter and have run into some problems. I have about 15000 partitions in S3 and in the old version of Hive/Hadoop, I have no problem creating a table, recovering the partitions, and then dropping the table. In the new version of Hive/Hadoop, the first two steps are successful, but I run into a "java.lang.OutOfMemoryError: Java heap space" error when I try to drop the table.
When I look at the output of "set;" from the hive prompt, I see several environment variables related to heap size. I was able to augment HADOOP_DATANODE_HEAPSIZE and HADOOP_NAMENODE_HEAPSIZE each to 4096 (2048 is sufficient in 0.5/0.20), but I see other parameters including HADOOP_HEAPSIZE that I cannot seem to change. To be fair, I'm just shooting in the dark here and unable to decipher from the error message *which* heap is too small. If this is already documented somewhere (neither basic tutorials nor google searches helped), I would be grateful for a reference and happy to summarize what I learn here. Or, if you simply have an answer ... well, any help would be most appreciated! Sincerely, Matt Vonkip
