He Tianyi created MAPREDUCE-6488: ------------------------------------ Summary: Make buffer size in PipeMapRed configurable Key: MAPREDUCE-6488 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6488 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: He Tianyi Assignee: He Tianyi
Default value of buffer size is 128K in {{PipeMapRed}}. When mapper input record is large enough that it won't fit in buffer, {{MapRunner}} blocks until written. If child process and input reader are both slow (due to calculation and decompress), then process of decoding and reading will rarely overlap with each other, hurting performance. I suppose we should make the buffer size configurable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)