stuhood opened a new issue, #24597: URL: https://github.com/apache/datafusion/issues/24597
### Is your feature request related to a problem or challenge? In query plans that use range re-partitioning (for example, in order to match the range partitioning of underlying base data, or route dynamic filters in partitioned joins), range re-partitioning represents a significant fraction of total query execution time for us (20-30% of total runtime). Currently, range re-partitioning in [`BatchPartitionerState::Range`](https://github.com/apache/datafusion/blob/6fe001d4baf5c6089d24b5634a23b1345797fb14/datafusion/physical-plan/src/repartition/mod.rs#L629-L640) and range-based dynamic filter evaluation in [`RangeExpr`](https://github.com/apache/datafusion/blob/6fe001d4baf5c6089d24b5634a23b1345797fb14/datafusion/physical-plan/src/joins/hash_join/shared_bounds.rs#L180-L240) assign rows to partitions on a row-by-row basis using [`extract_row_at_idx_to_buf`](https://github.com/apache/datafusion/blob/6fe001d4baf5c6089d24b5634a23b1345797fb14/datafusion-common/src/utils/mod.rs) and dynamic [`ScalarValue`](https://github.com/apache/datafusion/blob/6fe001d4baf5c6089d24b5634a23b1345797fb14/datafusion-common/src/scalar/mod.rs) comparisons via [`compare_rows`](https://github.com/apache/datafusion/blob/6fe001d4baf5c6089d24b5634a23b1345797fb14/datafusion-common/src/utils/mod.rs). This incurs several performance penalties: 1. High per-row heap allocation: * Every row in an incoming `RecordBatch` extracts temporary `ScalarValue` vectors, allocating memory for variable-length types (strings, binary) and dynamic enum discriminants. 2. Repeated type-dispatch in the hot loop: * Binary search across split points performs dynamic `match` dispatch on `ScalarValue` types for every split point comparison for every row. 3. Significant performance gap compared to Hash partitioning: * For standard primary/foreign key integer columns, range re-partitioning is 15x–30x slower than hash re-partitioning. ### Describe the solution you'd like Optimize range re-partitioning and range routing expressions: profiling shows that allocation and type dispatch are likely culprits. ### Describe alternatives you've considered - Hash re-partitioning: - While hash re-partitioning is fast, if data is already Range partitioned in storage, re-partitioning an operator output to match it avoids one shuffle. ### Additional context Related to [EPIC: Implement Range Partitioning #22395](https://github.com/apache/datafusion/issues/22395). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
