viirya commented on PR #12046: URL: https://github.com/apache/arrow-rs/pull/12046#issuecomment-6772514094
Thanks @sunchao, both points are right. I reproduced the empty value case with a `ParquetRecordBatchReader` scan of 65,536 rows in 1,024-row batches: 2.1x slower than `main` on an M4 (macOS) and 2.7x in a Linux container on the same machine. The early return in `copy_value` fixes it (+5% / +8%), but the same scan turned up a related case it doesn't fix. When a dictionary's values add up to fewer than 16 bytes, such as `"Y"` / `"N"`, no value ever has 16 readable bytes, so every value falls back to `memcpy` and also pays for the extra checks. That case is 7% slower on macOS and 14-21% slower on Linux. The same end-to-end scan also shows little or no gain on Linux for the cases this PR targets: -4% for 8 byte values and +9% for 19 byte values (the container is noisy, so treat these as roughly neutral). That's well short of what the `arrow_reader` benchmark suggested. The large improvement is mostly on macOS, where short `memcpy` calls are expensive. Each review round has found another case where the hand-written copy is slower than `main` or needs another invariant. With little gain on Linux, I don't think the extra unsafe code is worth maintaining, so I'm closing this PR. The measurements are in #12045 for whoever looks at this path next, for example the fixed-width dictionary specialization @jhorstmann suggested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
