Sounds like you guys are on the right track, this is purely FYI because
I haven't seen it posted, just responding to the line in the original
post that your data structure should fit in memory.
OK two more disclaimers "FWIW" and "maybe this is not relevant or
already covered" OK here goes...
@Jim-
I'm wondering if those docs are outdated as its my understanding (please
correct if I'm wrong), that we should never be seeing OOMs as 1.5/Tungsten not
only improved (reduced) the memory footprint of our data, but also introduced
better task level - and even key level - external spilling
Hi Andy,
Could you change logging level to INFO and post some here? There will be some
logging about the memory usage of a task when OOM.
In 1.6, the memory for a task is : (HeapSize - 300M) * 0.75 / number of tasks.
Is it possible that the heap is too small?
Davies
--
Davies Liu
