Makoto Yui created HIVEMALL-73:
----------------------------------
Summary: Reduce memory usage of each_top_k
Key: HIVEMALL-73
URL: https://issues.apache.org/jira/browse/HIVEMALL-73
Project: Hivemall
Issue Type: Improvement
Reporter: Makoto Yui
Assignee: Makoto Yui
Priority: Minor
Better to reduce memory usage in drainQueue().
{code}
2017-02-16 05:56:22,378 FATAL [Thread-4] org.apache.hadoop.mapred.YarnChild:
Error running child : java.lang.OutOfMemoryError: Java heap space
at hivemall.tools.EachTopKUDTF.drainQueue(EachTopKUDTF.java:182)
at hivemall.tools.EachTopKUDTF.close(EachTopKUDTF.java:215)
at
org.apache.hadoop.hive.ql.exec.UDTFOperator.closeOp(UDTFOperator.java:143)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:577)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:588)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:588)
at org.apache.hadoop.hive.ql.exec.mr.ExecReducer.close(ExecReducer.java:318)
at org.apache.hadoop.io.IOUtils.cleanup(IOUtils.java:237)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:459)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
