[ https://issues.apache.org/jira/browse/TEZ-3809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16111794#comment-16111794 ]
Muhammad Samir Khan edited comment on TEZ-3809 at 8/2/17 10:04 PM: ------------------------------------------------------------------- Took a heap dump on ordered word count before final merge. In the after case, one of the outputs was written to disk instead of kept in memory and that is why it has 37 entries. Before: Class Name | Shallow Heap | Retained Heap | Percentage ----------------------------------------------------------------------------------------------------------------------------------------------------------- java.lang.Thread @ 0x5d2c473f8 ShuffleAndMergeRunner {Tokenizer} Thread | 120 | 2,229,207,992 | 96.48% |- java.util.ArrayList @ 0x73f978f10 | 24 | 2,229,206,760 | 96.48% | '- java.lang.Object[38] @ 0x73f979130 | 168 | 2,229,206,736 | 96.48% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x5e4a88898| 32 | 68,078,192 | 2.95% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x631e0b260| 32 | 67,839,520 | 2.94% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x5e4a888b8| 32 | 67,700,608 | 2.93% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x73f9db168| 32 | 67,500,816 | 2.92% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x60ab36218| 32 | 67,408,704 | 2.92% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x631deed28| 32 | 67,367,424 | 2.92% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x743b86ee0| 32 | 67,337,936 | 2.91% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x60af3a698| 32 | 67,300,896 | 2.91% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x631e0c5b8| 32 | 67,282,464 | 2.91% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x60ab33140| 32 | 67,264,304 | 2.91% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x5e4a88878| 32 | 67,127,368 | 2.91% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x631e0b218| 32 | 67,098,216 | 2.90% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x631e0c6c8| 32 | 67,064,504 | 2.90% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x5d239a6c8| 32 | 67,003,776 | 2.90% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x5d23b7e10| 32 | 66,965,296 | 2.90% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x631def2b8| 32 | 66,928,032 | 2.90% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x60ab351d0| 32 | 66,916,896 | 2.90% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x74805dfb8| 32 | 66,886,272 | 2.89% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x60af39598| 32 | 66,718,800 | 2.89% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x73fb0fb78| 32 | 66,688,296 | 2.89% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x631e0c4b0| 32 | 66,656,312 | 2.88% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x60af39578| 32 | 66,629,936 | 2.88% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x631deec30| 32 | 66,584,576 | 2.88% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x631e0c680| 32 | 66,537,624 | 2.88% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x60af3a620| 32 | 66,529,416 | 2.88% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x631e0c570| 32 | 66,229,848 | 2.87% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x73f9d8f08| 32 | 65,616,336 | 2.84% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x60af39500| 32 | 63,794,840 | 2.76% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x5e4a7ed98| 32 | 60,026,520 | 2.60% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x60ab36280| 32 | 50,731,856 | 2.20% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x5d23c0da8| 32 | 48,164,288 | 2.08% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x5d23b85b8| 32 | 46,712,064 | 2.02% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x5d23b8310| 32 | 46,564,520 | 2.02% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x5e4a7e528| 32 | 45,990,432 | 1.99% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x60aec1cf0| 32 | 45,642,312 | 1.98% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x5d23c0a48| 32 | 8,122,976 | 0.35% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x60ab362a0| 32 | 3,712,608 | 0.16% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x60aec1ca8| 32 | 481,784 | 0.02% | '- Total: 38 entries | | | ----------------------------------------------------------------------------------------------------------------------------------------------------------- After: Class Name | Shallow Heap | Retained Heap | Percentage ----------------------------------------------------------------------------------------------------------------------------------------------------------- java.lang.Thread @ 0x5ce7a24e8 ShuffleAndMergeRunner {Tokenizer} Thread | 120 | 2,182,643,840 | 96.38% |- java.util.ArrayList @ 0x7343b6ae8 | 24 | 2,182,642,112 | 96.38% | '- java.lang.Object[37] @ 0x7343b6c18 | 168 | 2,182,642,088 | 96.38% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x767263f70| 32 | 68,078,192 | 3.01% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x719c6dbe8| 32 | 67,839,512 | 3.00% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x7672080b0| 32 | 67,700,608 | 2.99% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x767204a18| 32 | 67,500,816 | 2.98% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x5e3138258| 32 | 67,408,704 | 2.98% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x73cc9bb50| 32 | 67,367,416 | 2.97% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x5e3140588| 32 | 67,337,928 | 2.97% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x767262eb8| 32 | 67,300,888 | 2.97% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x7155fa180| 32 | 67,282,464 | 2.97% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x767232980| 32 | 67,264,296 | 2.97% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x7632001d8| 32 | 67,127,360 | 2.96% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x738be08c8| 32 | 67,098,208 | 2.96% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x767204c50| 32 | 67,064,496 | 2.96% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x76725fb38| 32 | 67,003,776 | 2.96% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x5e31303e8| 32 | 66,965,296 | 2.96% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x763200058| 32 | 66,928,024 | 2.96% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x5e3140160| 32 | 66,916,888 | 2.95% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x715866b18| 32 | 66,886,272 | 2.95% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x721d7d950| 32 | 66,718,792 | 2.95% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x719c6b988| 32 | 66,688,296 | 2.94% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x734290ec8| 32 | 66,656,312 | 2.94% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x5ce73f150| 32 | 66,629,936 | 2.94% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x767261d48| 32 | 66,584,576 | 2.94% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x721d7b6f0| 32 | 66,537,624 | 2.94% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x7632000d0| 32 | 66,529,416 | 2.94% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x767204dc0| 32 | 66,229,848 | 2.92% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x7349ccf30| 32 | 65,616,336 | 2.90% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x767260ca8| 32 | 63,794,832 | 2.82% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x76725f9f0| 32 | 60,026,512 | 2.65% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x5e31405c8| 32 | 50,731,848 | 2.24% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x5ce73a870| 32 | 48,164,288 | 2.13% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x5ce7525d8| 32 | 46,712,056 | 2.06% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x5e053c7d0| 32 | 45,990,432 | 2.03% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x5ce74ae60| 32 | 45,642,312 | 2.02% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x5e317aba0| 32 | 8,122,976 | 0.36% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x5e316e3f8| 32 | 3,712,600 | 0.16% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x5e3177a48| 32 | 481,784 | 0.02% | '- Total: 37 entries | | | ----------------------------------------------------------------------------------------------------------------------------------------------------------- was (Author: samkhan): Took a heap dump on ordered word count before final merge. In the after case, one of the outputs was written to disk instead of kept in memory and that is why it has 37 entries. Before: Class Name | Shallow Heap | Retained Heap | Percentage ----------------------------------------------------------------------------------------------------------------------------------------------------------- java.lang.Thread @ 0x5d2c473f8 ShuffleAndMergeRunner {Tokenizer} Thread | 120 | 2,229,207,992 | 96.48% |- java.util.ArrayList @ 0x73f978f10 | 24 | 2,229,206,760 | 96.48% | '- java.lang.Object[38] @ 0x73f979130 | 168 | 2,229,206,736 | 96.48% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x5e4a88898| 32 | 68,078,192 | 2.95% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x631e0b260| 32 | 67,839,520 | 2.94% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x5e4a888b8| 32 | 67,700,608 | 2.93% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x73f9db168| 32 | 67,500,816 | 2.92% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x60ab36218| 32 | 67,408,704 | 2.92% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x631deed28| 32 | 67,367,424 | 2.92% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x743b86ee0| 32 | 67,337,936 | 2.91% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x60af3a698| 32 | 67,300,896 | 2.91% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x631e0c5b8| 32 | 67,282,464 | 2.91% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x60ab33140| 32 | 67,264,304 | 2.91% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x5e4a88878| 32 | 67,127,368 | 2.91% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x631e0b218| 32 | 67,098,216 | 2.90% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x631e0c6c8| 32 | 67,064,504 | 2.90% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x5d239a6c8| 32 | 67,003,776 | 2.90% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x5d23b7e10| 32 | 66,965,296 | 2.90% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x631def2b8| 32 | 66,928,032 | 2.90% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x60ab351d0| 32 | 66,916,896 | 2.90% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x74805dfb8| 32 | 66,886,272 | 2.89% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x60af39598| 32 | 66,718,800 | 2.89% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x73fb0fb78| 32 | 66,688,296 | 2.89% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x631e0c4b0| 32 | 66,656,312 | 2.88% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x60af39578| 32 | 66,629,936 | 2.88% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x631deec30| 32 | 66,584,576 | 2.88% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x631e0c680| 32 | 66,537,624 | 2.88% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x60af3a620| 32 | 66,529,416 | 2.88% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x631e0c570| 32 | 66,229,848 | 2.87% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x73f9d8f08| 32 | 65,616,336 | 2.84% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x60af39500| 32 | 63,794,840 | 2.76% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x5e4a7ed98| 32 | 60,026,520 | 2.60% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x60ab36280| 32 | 50,731,856 | 2.20% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x5d23c0da8| 32 | 48,164,288 | 2.08% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x5d23b85b8| 32 | 46,712,064 | 2.02% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x5d23b8310| 32 | 46,564,520 | 2.02% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x5e4a7e528| 32 | 45,990,432 | 1.99% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x60aec1cf0| 32 | 45,642,312 | 1.98% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x5d23c0a48| 32 | 8,122,976 | 0.35% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x60ab362a0| 32 | 3,712,608 | 0.16% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x60aec1ca8| 32 | 481,784 | 0.02% | '- Total: 38 entries | | | ----------------------------------------------------------------------------------------------------------------------------------------------------------- After: Class Name | Shallow Heap | Retained Heap | Percentage ----------------------------------------------------------------------------------------------------------------------------------------------------------- java.lang.Thread @ 0x5ce7a24e8 ShuffleAndMergeRunner {Tokenizer} Thread | 120 | 2,182,643,840 | 96.38% |- java.util.ArrayList @ 0x7343b6ae8 | 24 | 2,182,642,112 | 96.38% | '- java.lang.Object[37] @ 0x7343b6c18 | 168 | 2,182,642,088 | 96.38% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x767263f70| 32 | 68,078,192 | 3.01% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x719c6dbe8| 32 | 67,839,512 | 3.00% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x7672080b0| 32 | 67,700,608 | 2.99% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x767204a18| 32 | 67,500,816 | 2.98% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x5e3138258| 32 | 67,408,704 | 2.98% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x73cc9bb50| 32 | 67,367,416 | 2.97% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x5e3140588| 32 | 67,337,928 | 2.97% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x767262eb8| 32 | 67,300,888 | 2.97% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x7155fa180| 32 | 67,282,464 | 2.97% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x767232980| 32 | 67,264,296 | 2.97% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x7632001d8| 32 | 67,127,360 | 2.96% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x738be08c8| 32 | 67,098,208 | 2.96% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x767204c50| 32 | 67,064,496 | 2.96% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x76725fb38| 32 | 67,003,776 | 2.96% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x5e31303e8| 32 | 66,965,296 | 2.96% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x763200058| 32 | 66,928,024 | 2.96% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x5e3140160| 32 | 66,916,888 | 2.95% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x715866b18| 32 | 66,886,272 | 2.95% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x721d7d950| 32 | 66,718,792 | 2.95% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x719c6b988| 32 | 66,688,296 | 2.94% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x734290ec8| 32 | 66,656,312 | 2.94% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x5ce73f150| 32 | 66,629,936 | 2.94% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x767261d48| 32 | 66,584,576 | 2.94% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x721d7b6f0| 32 | 66,537,624 | 2.94% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x7632000d0| 32 | 66,529,416 | 2.94% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x767204dc0| 32 | 66,229,848 | 2.92% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x7349ccf30| 32 | 65,616,336 | 2.90% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x767260ca8| 32 | 63,794,832 | 2.82% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x76725f9f0| 32 | 60,026,512 | 2.65% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x5e31405c8| 32 | 50,731,848 | 2.24% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x5ce73a870| 32 | 48,164,288 | 2.13% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x5ce7525d8| 32 | 46,712,056 | 2.06% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x5e053c7d0| 32 | 45,990,432 | 2.03% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x5ce74ae60| 32 | 45,642,312 | 2.02% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x5e317aba0| 32 | 8,122,976 | 0.36% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x5e316e3f8| 32 | 3,712,600 | 0.16% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x5e3177a48| 32 | 481,784 | 0.02% | '- Total: 37 entries | | | ----------------------------------------------------------------------------------------------------------------------------------------------------------- > The buffer size allocated for InMemoryMapOutput can be optimized > ---------------------------------------------------------------- > > Key: TEZ-3809 > URL: https://issues.apache.org/jira/browse/TEZ-3809 > Project: Apache Tez > Issue Type: Bug > Reporter: Muhammad Samir Khan > Assignee: Muhammad Samir Khan > Attachments: TEZ-3809.001.patch > > > Related jiras: TEZ-3752 and TEZ-3732. > -When shuffling input to memory, the decompressed length is used to create > the InMemoryMapOutput object. However, IFile.Reader's readToMemory reads 4 > bytes less (the IFile header). These 4 bytes can optimized and, in an extreme > case of 10,000,000 fetches, can save ~38 MB (TEZ-3732). > -Memory-to-memory merge sums up the sizes of input InMemoryMapOutput buffers > to allocate the new InMemoryMapOutput. However, each input has two > EOF_MARKERs while only two are needed at the end. > -InMemoryWriter wraps the output BoundedByteArrayOutputStream in > IFileOutputStream which will write checksum at close. This creates an > inconsistency between the primary input buffers which don't have checksum and > the merged buffers which do. IFileOutputStream wrap can be removed to save 4 > bytes per merged buffers. > -InMemoryWriter does not account for two EOF_MARKERs written at close() in > its accounting so that the getRawLength() method is off by two bytes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)