Hi,
I got a query that's producing about 3000 partitions which we load
dynamically (On Hive 2.3.5).
At the end of this query (running on M/R which runs fine) the M/R job is
finished and we see this on the hive cli:
Loading data to table my_db.temp__v1_2019_12_03_182627 partition
(c_date=null, c_hour=null, c_b=null, c_p=null)
Time taken to load dynamic partitions: 540.025 seconds
Time taken for adding to write entity : 0.329 seconds
#
# java.lang.OutOfMemoryError: Java heap space
# -XX:OnOutOfMemoryError="kill -9 %p"
# Executing /bin/sh -c "kill -9 19644"...
os::fork_and_exec failed: Cannot allocate memory (12)
MapReduce Jobs Launched:
Stage-Stage-1: Map: 387 Reduce: 486 Cumulative CPU: 110521.05 sec HDFS
Read: 533411354 HDFS Write: 262054898296 SUCCESS
Stage-Stage-2: Map: 973 Reduce: 1009 Cumulative CPU: 48710.45 sec HDFS
Read: 262126094987 HDFS Write: 70666472011 SUCCESS
Total MapReduce CPU Time Spent: 1 days 20 hours 13 minutes 51 seconds 500
msec
OK
Where is this OutOfMemoryError coming from which heap space am I supposed
to increase. We've tried increasing 'yarn.app.mapreduce.am.resource.mb' but
that didn't seem to help.
I know we should probably not have this many partitions but this is a one
off would like this to just work.
Thanks for any pointers,
Patrick
