Hi all I've made some memory optimizations on the reduce task and I would like to compare the old reducer vs new reducer in terms of maximum memory consumption.
I have a question regarding the description of the following counter: PHYSICAL_MEMORY_BYTES | Physical memory (bytes) snapshot | Total physical memory used by all tasks including spilled data. I'm assuming this means the aggregate of memory used throughout the entire reduce task (if viewing at the reduce task-level). Please correct me if I'm wrong on this assumption (the description seems pretty straightforward). Is there a way to get the maximum (not total) memory used by a reduce task from the default counters? Thanks!
