comphead opened a new pull request, #7636:
URL: https://github.com/apache/datafusion-comet/pull/7636
## Which issue does this PR close?
Closes #7474.
## Rationale for this change
Hash, range, and hash round robin partitioning record one `(batch, row)`
pair per row and build each output chunk with `interleave_record_batch`. Arrow
copies primitive and string columns there one row at a time
(`interleave_primitive`, `interleave_bytes`) and merges contiguous rows only in
`interleave_fallback`, which Boolean columns use. So a chunk whose rows are
already one span of a buffered batch is still gathered row by row.
That is the shape an Iceberg write gets when Spark rebalances on a partition
column that holds one value per run, such as the `ds` of an hourly or daily
ETL. Every row hashes to one partition, and each buffered batch is a single
run. In the production job described in #7474, partition interleaving was 54%
of the native shuffle writer time in the map stage.
## What changes are included in this PR?
- `RowIterator` (`partitioned_batch_iterator.rs`) splits each output chunk
into runs of consecutive rows of one buffered batch with `chunk_by`. When the
runs average at least 64 rows, it slices them out of the buffered batches and
concatenates the slices, and a chunk that is a single run reaches the writer as
a zero-copy slice. Otherwise it gathers as before. Counting stops one run past
the limit, so a scattered chunk costs only a short scan.
- The partition index is unchanged, so placement, memory accounting, and
spilling are untouched. This answers the open question in #7474 with a choice
made per output chunk at flush, rather than a mixed index of rows and runs.
- A schema with a view type keeps the gather, because a sliced view array
serializes every data buffer it shares. This is the rule
`partitioning_for_schema` already applies to positional round robin, so
`contains_view_type` becomes `pub(crate)`.
- `RunIterator` and `RowIterator` share the single-slice passthrough through
a new `concat_slices`.
- A `shuffle_hash_contiguous` bench group in `benches/shuffle_writer.rs`,
with a constant and a per-row hash key over 8192-row and 1000-row input batches.
Range partitioning over clustered input takes the same path.
The threshold comes from a sweep with arrow 59.3.2 on Apple Silicon, over
chunks of 8192 rows drawn from 16 batches. Slicing and concatenating beats
`interleave_record_batch` from an average run of about 32 rows, is about 2x
faster at 64, and is 22x to 117x faster for a whole batch, on 4-column and
16-column flat schemas. I did not measure x86.
## How are these changes tested?
New unit tests in `partitioned_batch_iterator.rs`:
- `contiguous_rows_are_sliced_not_gathered`: chunks that are a whole batch,
two runs, and part of a batch come out equal to `interleave_record_batch`, a
scattered partition next to them is still gathered, and the whole-batch chunk
shares its source buffers, which shows the slice path ran.
- `view_columns_are_gathered_even_when_contiguous`: a contiguous `Utf8View`
chunk is still gathered.
The existing `boolean_interleave_matches_arrow_for_sliced_nullable_states`
also takes the new path for its contiguous 79-row selection over sliced inputs.
Benchmarks ran on an 18-core Apple Silicon machine that was shared with
other work, so the before and after binaries ran alternately and the numbers
below are means over the rounds.
`shuffle_hash_contiguous`, place and gather of 65,536 rows into 200
partitions, three rounds:
| Hash key | Rows per batch | Before | After |
| --- | --- | --- | --- |
| constant | 8192 | 693 µs | 333 µs |
| constant | 1000 | 702 µs | 408 µs |
| per row | 8192 | 981 µs | 997 µs |
| per row | 1000 | 1.369 ms | 1.329 ms |
The per-row cases only add the scan. Their differences have mixed signs
within 3%, which is run-to-run noise.
`shuffle_bench` over a generated 4M-row Parquet file with 16 columns (8
strings, 7 longs, 1 int) and a constant `ds`, 1600 partitions, zstd(1), two
rounds:
| Hash key | Write time | `interleave time` | Encode time | Output |
| --- | --- | --- | --- | --- |
| constant `ds`, before | 1.301 s | 0.151 s | 0.94 s | 335 MiB |
| constant `ds`, after | 1.161 s | 0.016 s | 0.94 s | 335 MiB |
| per-row `seq`, before | 2.317 s | 0.913 s | 1.18 s | 451 MiB |
| per-row `seq`, after | 2.304 s | 0.886 s | 1.17 s | 451 MiB |
Encode time does not change, so the IPC writer does not pay more for slices.
zstd encoding takes most of the write time on this data, so the end-to-end gain
here is smaller than the production share in #7474 suggests.
<details>
<summary>Input and commands</summary>
The input was generated with DuckDB:
```sql
COPY (
SELECT
'2026-11-03' AS ds,
md5(i::VARCHAR) AS record_id,
sha256(i::VARCHAR) AS payload_hash,
CASE WHEN i % 20 = 0 THEN NULL ELSE 'device-' || (i % 5000)::VARCHAR END
AS device,
['US', 'DE', 'FR', 'JP', 'IN', 'BR', 'GB', 'CN'][(i % 8) + 1] AS country,
(i % 256)::VARCHAR || '.' || ((i // 256) % 256)::VARCHAR || '.'
|| ((i // 65536) % 256)::VARCHAR || '.' || (i % 251)::VARCHAR AS ip,
'app/' || (i % 37)::VARCHAR || '.' || (i % 11)::VARCHAR AS app_version,
CASE WHEN i % 7 = 0 THEN NULL ELSE 'city_' || (i % 997)::VARCHAR END AS
city,
i::BIGINT AS seq,
(1730000000000000 + i * 1000)::BIGINT AS event_time_us,
(i * 2654435761 % 1000000007)::BIGINT AS user_id,
(i % 1000)::BIGINT AS bucket,
CASE WHEN i % 13 = 0 THEN NULL ELSE (i * 31)::BIGINT END AS bytes_in,
(i * 17)::BIGINT AS bytes_out,
(i % 86400)::BIGINT AS second_of_day,
(i % 100)::INTEGER AS shard
FROM range(4000000) t(i)
) TO 'constant_ds.parquet' (FORMAT parquet, ROW_GROUP_SIZE 1048576);
```
```shell
cd native
cargo bench -p datafusion-comet-shuffle --bench shuffle_writer --
shuffle_hash_contiguous
cargo build --release -p datafusion-comet-shuffle --features shuffle-bench
--bin shuffle_bench
# --hash-columns 0 is the constant ds, 8 is the per-row seq
target/release/shuffle_bench --input constant_ds.parquet --partitions 1600
--partitioning hash \
--hash-columns 0 --codec zstd --zstd-level 1 --warmup 1 --iterations 3
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]