I am running a transform script that parses through a bunch of binary data. In 99% of the cases it runs, it runs fine, but on certain files I get a failure (as seen below). Funny thing is, I can run a job with "only" the problem source file, and it will work fine, but when as a group of files, I get these warnings. I guess what I am asking here is this: Where is the heap error? Is this occurring on the nodes themselves or, since this is where the script is emitting records (and potentially large ones at that) and in this case my hive server running the job may be memory light, could the issue actually be due to heap on the hive server itself? My setup is 1 Hive node (that is woefully underpowered, under memoried, and under disk I/Oed) and 4 beefy hadoop nodes. I guess, my question is the heap issue on the sender or the receiver :)
13-01-29 08:20:24,107 INFO org.apache.hadoop.hive.ql.io.CodecPool: Got brand-new compressor 2013-01-29 08:20:24,107 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 12 forwarding 1 rows 2013-01-29 08:20:24,410 INFO org.apache.hadoop.hive.ql.exec.ScriptOperator: 3 forwarding 10 rows 2013-01-29 08:20:24,410 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 4 forwarding 10 rows 2013-01-29 08:20:24,411 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 5 forwarding 10 rows 2013-01-29 08:20:24,411 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 6 forwarding 10 rows 2013-01-29 08:20:24,411 INFO org.apache.hadoop.hive.ql.exec.FilterOperator: 8 forwarding 10 rows 2013-01-29 08:20:24,411 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 9 forwarding 10 rows 2013-01-29 08:20:24,411 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 10 forwarding 10 rows 2013-01-29 08:20:24,412 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 12 forwarding 10 rows 2013-01-29 08:20:27,170 INFO org.apache.hadoop.hive.ql.exec.ScriptOperator: 3 forwarding 100 rows 2013-01-29 08:20:27,170 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 4 forwarding 100 rows 2013-01-29 08:20:27,170 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 5 forwarding 100 rows 2013-01-29 08:20:27,171 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 6 forwarding 100 rows 2013-01-29 08:20:27,171 INFO org.apache.hadoop.hive.ql.exec.FilterOperator: 8 forwarding 100 rows 2013-01-29 08:20:27,171 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 9 forwarding 100 rows 2013-01-29 08:20:27,171 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 10 forwarding 100 rows 2013-01-29 08:20:27,171 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 12 forwarding 100 rows 2013-01-29 08:21:16,247 INFO org.apache.hadoop.hive.ql.exec.ScriptOperator: 3 forwarding 1000 rows 2013-01-29 08:21:16,247 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 4 forwarding 1000 rows 2013-01-29 08:21:16,247 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 5 forwarding 1000 rows 2013-01-29 08:21:16,247 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 6 forwarding 1000 rows 2013-01-29 08:21:16,248 INFO org.apache.hadoop.hive.ql.exec.FilterOperator: 8 forwarding 1000 rows 2013-01-29 08:21:16,248 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 9 forwarding 1000 rows 2013-01-29 08:21:16,248 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 10 forwarding 1000 rows 2013-01-29 08:21:16,248 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 12 forwarding 1000 rows 2013-01-29 08:25:47,532 INFO org.apache.hadoop.hive.ql.exec.ScriptOperator: 3 forwarding 10000 rows 2013-01-29 08:25:47,532 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 4 forwarding 10000 rows 2013-01-29 08:25:47,532 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 5 forwarding 10000 rows 2013-01-29 08:25:47,532 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 6 forwarding 10000 rows 2013-01-29 08:25:47,532 INFO org.apache.hadoop.hive.ql.exec.FilterOperator: 8 forwarding 10000 rows 2013-01-29 08:25:47,532 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 9 forwarding 10000 rows 2013-01-29 08:25:47,532 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 10 forwarding 10000 rows 2013-01-29 08:25:47,532 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 12 forwarding 10000 rows 2013-01-29 08:27:34,276 WARN org.apache.hadoop.hive.ql.exec.ScriptOperator: Exception in StreamThread.run(): Java heap space Cause: null 2013-01-29 08:27:34,277 WARN org.apache.hadoop.hive.ql.exec.ScriptOperator: java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOfRange(Arrays.java:3209) at java.lang.String.<init>(String.java:215) at java.nio.HeapCharBuffer.toString(HeapCharBuffer.java:542) at java.nio.CharBuffer.toString(CharBuffer.java:1157) at org.apache.hadoop.io.Text.decode(Text.java:350) at org.apache.hadoop.io.Text.decode(Text.java:327) at org.apache.hadoop.io.Text.toString(Text.java:254) at java.lang.String.valueOf(String.java:2826) at java.lang.StringBuilder.append(StringBuilder.java:115) at org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistry.java:873) at org.apache.hadoop.hive.ql.udf.generic.GenericUDFBridge.evaluate(GenericUDFBridge.java:181) at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.evaluate(ExprNodeGenericFuncEvaluator.java:163) at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:76) at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:762) at org.apache.hadoop.hive.ql.exec.ScriptOperator$OutputStreamProcessor.processLine(ScriptOperator.java:477) at org.apache.hadoop.hive.ql.exec.ScriptOperator$StreamThread.run(ScriptOperator.java:563) 2013-01-29 08:27:34,306 INFO org.apache.hadoop.hive.ql.exec.ScriptOperator: ErrorStreamProcessor calling reporter.progress() 2013-01-29 08:27:34,307 INFO org.apache.hadoop.hive.ql.exec.ScriptOperator: StreamThread ErrorProcessor done 2013-01-29 08:27:34,307 ERROR org.apache.hadoop.hive.ql.exec.ScriptOperator: Script failed with code 1
