sunchao commented on code in PR #7316:
URL: https://github.com/apache/datafusion-comet/pull/7316#discussion_r4633430066
##########
native/core/src/execution/planner.rs:
##########
@@ -1853,19 +1856,21 @@ impl PhysicalPlanner {
let (scans, shuffle_scans, child) =
self.create_plan(&children[0], inputs, partition_count)?;
- let exprs: Result<Vec<PhysicalSortExpr>, ExecutionError> = sort
+ let exprs: Vec<PhysicalSortExpr> = sort
.sort_orders
.iter()
.map(|expr| self.create_sort_expr(expr, child.schema()))
- .collect();
+ .collect::<Result<_, ExecutionError>>()?;
let fetch = sort.fetch.map(|num| num as usize);
- let sort_plan = SortExec::new(
- LexOrdering::new(exprs?).unwrap(),
+ // The sort evaluates its keys on each batch it receives.
+ let input = self.bound_scalar_expansion(
Review Comment:
[P2] Avoid exposing ordinary sorts to the shared-buffer reservation bug
Could the ordinary-sort path avoid handing `ExternalSorter` zero-copy slices
of one large allocation until its shared-buffer accounting is fixed? `SELECT
id, cume_dist() OVER (ORDER BY id) AS cd FROM range(1048576) SORT BY
concat(CAST(id AS STRING), '!')` reaches a full-partition `WindowAggExec`
followed by this sort. Even the one-byte `'!'` installs the splitter at 8192
rows, producing 128 slices of the same roughly 16 MiB buffers. In the pinned
DataFusion 55.2.0, `insert_batch` reserves `get_record_batch_memory_size(batch)
+ batch.get_sliced_size()` for each slice independently. That attempts about 2
GiB plus 16 MiB of reservations, versus about 32 MiB for the unsplit batch, and
spills when the pool refuses them. The physical input buffers remain shared.
The underlying bug already exists and is tracked in [DataFusion #25804,
finding 5](https://github.com/apache/datafusion/issues/25804), with
[#25820](https://github.com/apache/datafusion/pull/25820) providing the
upstream fix. The regression here is newly exposing a previously unsplit
window-to-sort path to it, including tiny scalars with no offset-overflow risk.
Please avoid that new exposure, compact/account the shared inputs correctly, or
sequence this path after the dependency fix, with coverage for a wide producer
and a small scalar sort key.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]