Hi, there,

What's the local heap size of Hadoop? I have tried to load a local cache file which is composed of 500,000 short phrase, but the task failed. The output of Hadoop looks like(com.aliasi.dict.ExactDictionaryChunker is a third-party jar package, and the records is organized as a trie struction):

java.lang.OutOfMemoryError: Java heap space
       at java.util.HashMap.addEntry(HashMap.java:753)
       at java.util.HashMap.put(HashMap.java:385)
at com.aliasi.dict.ExactDictionaryChunker$TrieNode.getOrCreateDaughter(Ex
actDictionaryChunker.java:476)
at com.aliasi.dict.ExactDictionaryChunker$TrieNode.add(ExactDictionaryChu
nker.java:484)

When I reduce the total record number to 30,000. My mapreduce job became succeed. So, I have a question, What's the local heap size of Hadoop's Java Virtual Machine? How to increase it?

Best,
Jasmine

Reply via email to