andygrove opened a new issue, #7498: URL: https://github.com/apache/datafusion-comet/issues/7498
Part of #7267. ## Problem The JVM codegen dispatcher evaluates an expression over a whole Arrow batch and writes a string or binary result into one vector per batch with `setSafe` (`spark/src/main/scala/org/apache/comet/codegen/CometBatchKernelCodegenOutput.scala:258-283`). When the batch's output passes `Integer.MAX_VALUE` bytes, Arrow throws `OversizedAllocationException` and the task fails. Spark evaluates one row at a time and succeeds. The dispatcher is on by default for `ScalaUDF` (`spark.comet.exec.scalaUDF.codegen.enabled`) and for the regex family (`rlike`, `regexp_replace`, `split`, `regexp_extract`, `regexp_extract_all`, `regexp_instr`). A UDF that returns serialized or JSON-encoded values of about 300 KB per row overflows at 8192 rows, as does a `regexp_replace` that grows large text. This is the JVM side of the limit in #7247: one expression's result for one batch has to fit in 2 GiB. Line references are for main at 4e6b12290f. ## Workaround Lower `spark.comet.batchSize`, or set `spark.comet.exec.scalaUDF.codegen.enabled=false` to run the enclosing operator in Spark. ## Proposed fix Have the kernel stop before the write that would pass the limit and report how many rows it wrote, so the native side can split the input batch at that row and evaluate the rest separately. That evaluates each row once, which matters for nondeterministic UDFs. It needs the projection to emit more than one batch per input batch, the same constraint as #7247. ## Tests A real overflow needs 2 GiB of output. Test the stop-and-resume path with a small budget injected for the test. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
