viirya commented on PR #12046:
URL: https://github.com/apache/arrow-rs/pull/12046#issuecomment-6772514094

   Thanks @sunchao, both points are right. I reproduced the empty value case 
with a `ParquetRecordBatchReader` scan of 65,536 rows in 1,024-row batches: 
2.1x slower than `main` on an M4 (macOS) and 2.7x in a Linux container on the 
same machine. The early return in `copy_value` fixes it (+5% / +8%), but the 
same scan turned up a related case it doesn't fix. When a dictionary's values 
add up to fewer than 16 bytes, such as `"Y"` / `"N"`, no value ever has 16 
readable bytes, so every value falls back to `memcpy` and also pays for the 
extra checks. That case is 7% slower on macOS and 14-21% slower on Linux.
   
   The same end-to-end scan also shows little or no gain on Linux for the cases 
this PR targets: -4% for 8 byte values and +9% for 19 byte values (the 
container is noisy, so treat these as roughly neutral). That's well short of 
what the `arrow_reader` benchmark suggested. The large improvement is mostly on 
macOS, where short `memcpy` calls are expensive.
   
   Each review round has found another case where the hand-written copy is 
slower than `main` or needs another invariant. With little gain on Linux, I 
don't think the extra unsafe code is worth maintaining, so I'm closing this PR. 
The measurements are in #12045 for whoever looks at this path next, for example 
the fixed-width dictionary specialization @jhorstmann suggested.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to