comphead commented on PR #7636: URL: https://github.com/apache/datafusion-comet/pull/7636#issuecomment-6927290595
Thanks @andygrove, the clustered cases found a real band, and your reading of it was right. **Runs just under the threshold.** The exact count stops one run past the budget, so with runs of L rows under the threshold T it reads about L/T of the chunk and then gathers anyway. Timing the decision alone on 8192-row chunks, it costs 1.7 µs at 96-row runs with T=128, against about 55 µs for the 4-column gather it guards. Lowering T to 64 on its own moves the same cost down to runs of 32 to 63 rows. With the binaries alternated per case over four rounds, T=64 alone was 2.3% and 2.4% slower than the merge base at runs of 32 and 48 rows, and slower in every round. That is your band one octave down. On this machine the T=128 build stayed within noise at 96 rows, so I took the x86 number as the one to fix. So `has_long_runs` now starts with a lower bound. A stretch of T steps whose ends are not T rows apart in one batch has to contain a run boundary, and the stretches are disjoint, so the number of broken stretches bounds the run count from below. When every run is shorter than T, every stretch is broken and the chunk is ruled out after one probe per T rows. Below the threshold the decision now costs 0.08 to 0.10 µs whatever the run length, and scattered chunks pay about 20 ns more for the probes. The bound never changes a decision, which `long_run_bound_matches_exact_count` checks against the exact count on random mixes of short, long and out-of-order runs. With that cost gone, the threshold only decides where slicing starts, so I lowered `MIN_AVG_RUN_ROWS` to 64 as you suggested. PR time over the merge base for `place+gather` of 65,536 rows into 200 partitions, on Apple Silicon: | average run (rows) | flat | nested structs | | --- | --- | --- | | scattered | 0.99 | | | 32 | 1.01 | | | 48 | 1.00 | 1.00 | | 64 | 0.76 / 0.82 | 0.63 / 0.72 | | 96 | 0.72 / 0.79 | 0.65 / 0.66 | | 120 | 0.69 / 0.77 | 0.60 / 0.61 | | 136 | 0.71 / 0.71 | 0.57 / 0.59 | | one key per task | 0.48 / 0.51 | | Rows at or under 48 are the mean of four rounds interleaved per case, the others are two rounds. Flat is the bench's 4-column `create_batch_from` fixture and nested is its 40-column struct fixture with a key column added. **Nested types through the writer.** The slice path's only new output is a `RecordBatch::slice` of a buffered batch, since a chunk of several runs is concatenated into compact arrays, as the gather's output is. So the risk you describe is the IPC writer on sliced arrays, and an existing test already had the fixture for it. `offsets_as_lengths_round_trip_every_mapped_type` round-trips one column of every type Comet shuffles, including struct, list, large list, map, fixed size list, decimal and boolean, with nulls, empty values and inner arrays that start past offset 0. It goes through `ShuffleBlockWriter` and both readers, under every codec, with and without offsets as lengths. It now also round-trips that batch sliced from row 5, and passes. At 64 the existing `boolean_interleave_matches_arrow_for_sliced_nullable_states` also sends its contiguous selections through the slice path again. **Docs.** The Hash Partitioning section now describes how a partition's rows are written, and Range Partitioning and the `RowGroups` paragraph point at it. `shuffle_hash_contiguous` also has clustered keys over runs of 48 and 96 rows now, just under and over the threshold. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
