andygrove opened a new issue, #5535:
URL: https://github.com/apache/datafusion-comet/issues/5535
Discovered during post-merge review of #5531.
**Component**: `native/shuffle/src/ipc.rs` (`read_single_batch`),
`native/core/src/execution/jni_api.rs:1265`
(`decodeShuffleBlockWithValidation`)
`read_ipc_compressed_validated` drops `with_skip_validation(true)`, so every
remotely fetched batch
pays full Arrow buffer and offset validation. That is the right call for
untrusted bytes, but the
cost is significant and is not recorded anywhere. Since the justification
for a native Celeborn
reader is that it beats Celeborn's own row decoder, this number should be
measured before 8/n
enables the path end to end.
## Measurement
Release build, one `ShuffleBlockWriter` frame of 20 columns (10 `Int32`, 10
`Utf8`) x 8192 rows,
300 iterations after 20 warm-up, Apple silicon. "local" is
`read_ipc_compressed`; "validated" is
`read_ipc_compressed_validated` + `validate_remote_schema`, i.e. what the
reducer runs.
| codec | frame bytes | local (us/blk) | validated (us/blk) | ratio |
| -------- | ----------- | -------------- | ------------------ | ----- |
| None | 1,570,252 | 38.8 | 175.2 | 4.52x |
| Lz4Frame | 752,215 | 225.8 | 356.6 | 1.58x |
| Snappy | 821,025 | 558.5 | 696.5 | 1.25x |
| Zstd(1) | 235,724 | 636.9 | 773.1 | 1.21x |
For reference, the local path itself did not regress in #5531: against the
pre-PR implementation it
measured -8.8% (None), +0.8% (Lz4), +1.1% (Snappy), -1.8% (Zstd). The table
above is purely the
price of the new remote path.
## Two hoistable costs on the same path
- `decodeShuffleBlockWithValidation` calls `ShuffleScan::decode` and
rebuilds `expected_types` via
`to_arrow_datatype` on **every block**. Both are constant per reducer.
- `env.convert_byte_array(expected_schema)` copies the serialized schema
into a fresh `Vec` per
block.
Caching the decoded `Vec<DataType>` once per reducer, e.g. behind a native
handle, would remove
both.
## Ask
Publish a benchmark for the remote read path in the 8/n description, and
decide whether the
validation cost is acceptable or should be made configurable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
