peterxcli commented on PR #5807: URL: https://github.com/apache/datafusion-comet/pull/5807#issuecomment-5606302188
Numbers for the round trip this removes, which the description was missing. The work deleted per map task is: create the index file, write `numPartitions + 1` little-endian offsets, close, `readAllBytes`, convert offsets to lengths through `grouped(8).drop(1).map(ByteBuffer...)`, delete. I measured that sequence directly, and separately measured the parse with the bytes already in memory so the file syscalls and the allocation cost can be told apart. 2000 iterations after 300 warmup, each partition count measured twice and in both orders: | partitions | round trip (two runs) | parse alone | file syscalls | | --- | --- | --- | --- | | 200 | 111.8 us / 214.1 us | 4.1 us | ~110-210 us | | 2000 | 209.7 us / 181.5 us | 38.7 us | ~145-170 us | | 16000 | 588.7 us / 694.8 us | 292.0 us | ~300-400 us | So the saving is a roughly constant file-syscall floor of ~150-200 us per map task, plus a parse component that scales linearly at about 18 ns per partition. At the default 200 shuffle partitions that is ~110-210 us per map task; at 16000 partitions, ~0.6-0.7 ms. What replaces it is one JNI call returning a `long[]`, and the JVM allocated that array before as well. Caveats, since this is a per-task overhead measurement rather than a throughput one: - macOS on APFS, on a laptop with other load. The syscall half is the part most sensitive to the filesystem, and it is the part that would grow on EBS. - The write half is measured from the JVM here, while in production the native writer emits the file. The syscall cost should be comparable but this is an approximation of that half. - The 200-partition row is the noisiest, 111 us against 214 us across repeats, so treat it as a range rather than a point. - This removes fixed per-map-task overhead. It does not make the shuffle write itself faster, so the effect on a job is the saving multiplied by the number of map tasks: order of a second or two of aggregate task time for a 10,000-task stage at 200 partitions. The first measurement I took reported 401 us at 200 partitions, higher than at 2000, which was a warmup artifact from measuring that count first. The numbers above are from runs that measure each count twice in opposite orders. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
