nat-openai opened a new pull request, #6939: URL: https://github.com/apache/datafusion-comet/pull/6939
## Which issue does this PR close? Addresses the native allocation-ceiling portion of #5712, following the per-invocation proposal in #6819. ## Rationale for this change A native integral `sequence` invocation can allocate a large output outside Comet's memory pool. Spark's per-row collection limit and Arrow's batch element limit allow allocations large enough to exhaust an executor before either limit is reached. This change checks the logical output size before allocating the child values and list offsets. The default comes from the task's configured share of Comet's pool. The decision depends on the invocation's output and its fixed ceiling, so retaining earlier results or changing the downstream plan does not consume its allowance. A per-invocation ceiling bounds one logical output without tracking shared reservation lifetimes across arrays, slices or the JVM boundary. It makes no pool reservation; downstream operators retain their existing responsibility to reserve for buffered data. It is not an executor-wide or RSS bound. ## What changes are included in this PR? - Add `spark.comet.exec.sequence.maxBytesPerInvocation`, accepting byte units. When unset in off-heap mode, `getMemoryConfig` divides the configured pool limit by `max(1, floor(executor cores / task CPUs))`. Local mode uses its thread count; resource-profile allocations are respected. Local-cluster uses its worker cores when executor cores are unset. If standalone executor cores are unknown, the documented fallback is the whole configured pool limit; users can set executor cores or an explicit ceiling. - Leave on-heap test mode unbounded by default. An explicit override applies in either mode; `-1` disables the ceiling and `0` permits no output bytes. - Carry the resolved ceiling through the existing configuration map into an immutable sequence UDF. Count child values, list offsets and logical validity bytes, using checked arithmetic. Refuse oversized output with a resource-exhaustion error and tuning guidance, while preserving existing Spark row errors and Arrow batch-limit errors. - Preserve the existing sequence generation loop, scalar return behaviour and short-sequence optimisations. All-scalar invocations check one list; later generic scalar expansion and copies of folded literals are outside this ceiling. Scratch space, shared backing bytes outside slices and allocator padding are also excluded. - Update the memory inventory and tuning guide to explain the limit and its boundaries. Add four finite-ceiling benchmark shapes: scalar, singleton column, mixed-null and all-null inputs. ## How are these changes tested? - Scala compilation and all 26 tests in `CometConfSuite` pass with the Spark 4.1 profile. The configuration tests cover byte units, validation, task shares, resource-profile task CPUs, pool fractions, local and cluster modes, standalone fallback, on-heap defaults, explicit overrides and legacy-switch precedence. - Added native tests cover all integral widths, exact byte boundaries, null and sliced inputs, empty output, large-output refusal, arithmetic overflow, error precedence, retained results, scalar behaviour, UDF identity and configuration parsing. - Added JVM integration tests cover configuration serialisation, refusal and numeric/legacy opt-outs, smaller batches, Spark result parity and a downstream sort that buffers results. They also cover folded literals, which do not invoke the native sequence kernel. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
