deniskuzZ opened a new pull request, #544: URL: https://github.com/apache/tez/pull/544
`IFile.Writer` writes every record as several small writes (the key and value lengths as vints, then the key and value bytes) directly into the codec stream, or into the checksum stream when there is no codec. Each call is a full trip into the stream: with Snappy, a synchronized `Compressor.setInput` and friends per call; without a codec, the checksum update per call. The patch puts an unsynchronized 64 KB write buffer behind a `NonSyncDataOutputStream` between the writer and those streams, so the per-record writes land in one array and reach the codec in large chunks. `close()` drains the buffer before it finishes the codec and checksum streams. A Writer is confined to one thread, so the buffer takes no lock (`BufferedOutputStream` locks on every call). Microbenchmark (JDK 21, 6 runs each, median CPU per record; base = master): | Path | Rows | Snappy | No codec | |---|---|---|---| | IFile.Writer | default | -12.5 % | -67.6 % | | IFile.Writer | small | -33 % | -72 % | | UnorderedPartitionedKVWriter, 24 partitions | default / small | -11.1 % / -11.8 % | | | UnorderedPartitionedKVWriter, 1,009 partitions | default / small | -5.8 % / -12.6 % | | Context: in Hive on Tez (LLAP), TPC-DS 10 TB, the shuffle write path (ReduceSink serialization, then UnorderedPartitionedKVWriter, IFile and Snappy) is 46-72 % of task CPU in the shuffle-heavy queries. Tests: tez-runtime-library TestIFile, TestUnorderedPartitionedKVWriter, TestPipelinedSorter, TestDefaultSorter, TestMergeManager, TestTezMerger: 455 run, 0 failures. Removing the drain in `close()` fails 179 of them, so the existing round-trip tests cover the new path. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
