zhangfengcdt opened a new issue, #7689:
URL: https://github.com/apache/datafusion-comet/issues/7689

   ### Describe the bug
   
   #7316 bounds the rows per batch that reach an operator expanding a large 
string or binary scalar, by putting a `CometBatchSplitExec` under it. It covers 
projection, filter, hash aggregate, sort with a limit (TopK), and the hash and 
range shuffle writers. Other native operators evaluate expressions over a batch 
and expand a scalar the same way, but they get no split, so a large scalar can 
still fail with `byte array offset overflow`.
   
   These were raised in the review of #7316 and are not yet covered:
   
   - **Join filter.** The residual join condition is built as a DataFusion 
`JoinFilter` in `native/core/src/execution/planner/join.rs`, and nothing bounds 
it. It is evaluated over the matched index pairs, so the bound is the batch 
size, not the input width. A residual that references both sides stays a join 
condition, for example:
   
     ```sql
     SELECT count(*) FROM a JOIN b
       ON a.k = b.k AND length(concat(repeat('x', 262144), a.s || b.s)) > 0
     ```
   
     The same query in a plain DataFusion session (default 8192-row batch, a 
hash join with one matching key) fails with `Execution error: byte array offset 
overflow`. It has not yet been run end to end through Comet.
   
   - **`ExpandExec`.** `native/operators/src/expand.rs` calls 
`into_array(batch.num_rows())` on each projection expression, so a large scalar 
in a grouping-sets projection is expanded to the full batch length. Not yet 
reproduced.
   
   - **Window partition and order expressions.** These are evaluated over the 
batch the window receives, which can be a whole partition. Not yet reproduced.
   
   - **Sort without a limit.** #7316 puts the split under every sort, but the 
external sorter concatenates its in-memory input into larger runs before 
evaluating the keys, so the split under it does not bound what the key 
expressions see. Not yet reproduced.
   
   ### Steps to reproduce
   
   See the queries above. For comparison, the cases #7316 fixed are in 
`spark/src/test/resources/sql-tests/operators/large_scalar_string.sql`.
   
   ### Expected behavior
   
   Each query returns the same result as Spark, as the operators covered by 
#7316 now do.
   
   ### Additional context
   
   Follow-up to #7247 and #7316. Part of #7267.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to