comphead commented on code in PR #7316:
URL: https://github.com/apache/datafusion-comet/pull/7316#discussion_r4637037645


##########
native/core/src/execution/planner/scalar_expansion.rs:
##########
@@ -0,0 +1,408 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Bounds the rows per batch under an operator that expands a large string 
scalar.
+//!
+//! An aggregate, a hash partitioner and most string kernels expand a scalar 
argument to the
+//! batch's length: a 256 KiB string over the default 8192-row batch needs 2 
GiB of value bytes,
+//! past the 32-bit offsets of Arrow's `Utf8` and `Binary` arrays. Spark 
evaluates such scalars
+//! row by row and never materializes the column. See
+//! <https://github.com/apache/datafusion-comet/issues/7247>.
+//!
+//! The bound is enforced right under the operator that evaluates the 
expressions, with a
+//! [`BatchSplitExec`], because the operators below do not keep the batch size 
they were
+//! handed: a filter coalesces its output back up to the session's batch size, 
a window emits
+//! its whole partition and a broadcast arrives as a few wide batches. No 
width can be assumed,
+//! so the split is installed whenever the operator expands a string or binary 
scalar at all.
+
+use std::sync::Arc;
+
+use arrow::datatypes::{DataType, Schema};
+use arrow::record_batch::{RecordBatch, RecordBatchOptions};
+use datafusion::common::ScalarValue;
+use datafusion::logical_expr::{ColumnarValue, Operator, Volatility};
+use datafusion::physical_expr::expressions::{BinaryExpr, Column, InListExpr};
+use datafusion::physical_expr::{PhysicalExpr, ScalarFunctionExpr};
+use datafusion::physical_plan::ExecutionPlan;
+use datafusion_comet_operators::BatchSplitExec;
+use datafusion_comet_spark_expr::jvm_udf::JvmScalarUdfExpr;
+use 
datafusion_comet_spark_expr::monotonically_increasing_id::MonotonicallyIncreasingId;
+use datafusion_comet_spark_expr::{
+    RandExpr, RandStrExpr, RandnExpr, ShuffleExpr, Subquery, UuidExpr,
+};
+
+use super::{PhysicalPlanner, TEST_EXEC_CONTEXT_ID};
+
+/// The most value bytes the string and binary scalars of one operator may 
take once expanded to
+/// a batch's length. Arrow's 32-bit offsets make 2 GiB the hard limit; half 
of it leaves room for
+/// kernels that append to the expanded value, such as `concat`, and matches 
what an Arrow Java
+/// vector can hold before its reallocation fails.
+const SCALAR_EXPANSION_BYTES: usize = 1 << 30;
+
+/// The most rows per batch at which `scalar_bytes`, expanded to the batch's 
length, stay within
+/// `SCALAR_EXPANSION_BYTES`; unbounded when there is nothing to expand.
+fn scalar_rows_cap(scalar_bytes: usize) -> usize {
+    if scalar_bytes == 0 {
+        return usize::MAX;
+    }
+    (SCALAR_EXPANSION_BYTES / scalar_bytes).max(1)
+}
+
+impl PhysicalPlanner {
+    /// Wraps `input` so that the batches it emits keep the string and binary 
scalars in `exprs`,
+    /// expanded to a batch's length, within `SCALAR_EXPANSION_BYTES` and the 
session's batch
+    /// size. Returns `input` as is when `exprs` expand no scalar.
+    pub(crate) fn bound_scalar_expansion<'a>(
+        &self,
+        input: Arc<dyn ExecutionPlan>,
+        exprs: impl IntoIterator<Item = &'a Arc<dyn PhysicalExpr>>,
+    ) -> Arc<dyn ExecutionPlan> {
+        let schema = input.schema();
+        let bytes: usize = exprs
+            .into_iter()
+            .map(|expr| self.expanded_scalar_bytes(expr, &schema))
+            .sum();
+        if bytes == 0 {
+            return input;
+        }
+        let batch_size = self.session_ctx.copied_config().batch_size();
+        Arc::new(BatchSplitExec::new(
+            input,
+            scalar_rows_cap(bytes).min(batch_size),

Review Comment:
   The sort-key split is gone, but the exposure @sunchao raised at `e8eae446e8` 
still reaches a full sort through a projection. A one-byte scalar gives `bytes 
== 1`, so this installs a split at the session batch size, and `ProjectionExec` 
passes its column references through as zero-copy slices. I expect `SELECT id, 
cd, concat(CAST(id AS STRING), '!') AS k FROM (SELECT id, cume_dist() OVER 
(ORDER BY id) AS cd FROM range(1048576)) SORT BY k` to hand `ExternalSorter` 
128 slices of the window's `id` and `cd` buffers, which `insert_batch` reserves 
one by one as in the earlier report. I have not run it. Splitting at 
`scalar_rows_cap(bytes)` alone would still keep every expansion within 1 GiB, 
and `BatchSplitStream` passes a batch at or under that size straight through, 
so a scalar of ordinary size would never slice. The unit tests that assert the 
session batch size here would change with it.



##########
native/core/src/execution/jni_api.rs:
##########
@@ -1641,10 +1641,10 @@ fn execute_plan(
             exec_context.stream = Some(stream);
         }
         exec_context.root_op = Some(root_op);
-    } else {
-        // Pull input batches
-        pull_input_batches(exec_context)?;
     }
+    // JVM input is pulled only when the stream waits for it, in `next_batch` 
below. An operator
+    // above a scan, such as a `BatchSplitExec`, can still hold rows of the 
last batch it took,
+    // and a limit can finish on them without the JVM producing another batch.

Review Comment:
   The doc comment on `pull_input_batches` still says "we pull input batches 
here and insert them into scan operators before polling the stream". After this 
change it only runs from the pending branch of `next_batch`, after a poll, so 
that comment should change with this one. It also still fills every JVM-fed 
scan whose slot is empty, not only the one the plan waits on, so with two JVM 
inputs a pending poll on one side still reads the other side ahead. That part 
is unchanged from `main`, but "only when the stream waits for it" reads 
stronger than what happens.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to