andygrove opened a new issue, #7109: URL: https://github.com/apache/datafusion-comet/issues/7109
### Describe the bug `NativeUtil.exportBatch` hands each column to `Utils.getFieldVector(valueVector, "export")`, which accepts a fixed list of Arrow vector types that excludes `LargeVarCharVector` and `LargeVarBinaryVector`. A batch carrying a top-level `large_string` or `large_binary` column therefore fails at the JVM-to-native boundary with `Unsupported Arrow Vector for export`. Comet otherwise produces and reads those vectors on purpose: `CometPlainVector` tracks 64-bit variable-width offsets for PyArrow UDF output, and `CometMapInBatchExec` keeps the vectors a Python worker returns. The serialization boundary has the same gap, tracked in #5488, with #6371 narrowing top-level columns to 32-bit offsets and #6933 covering nested children. The export boundary is the other consumer of the same vectors, and #6371 does not touch it. The case that reaches it is a native Comet operator whose child is `CometMapInBatchExec`: the child's batches reach the native plan through `exportBatch`. I have not checked whether the planner can build that shape today, so this may be latent, like #5488 and #6933. ### Steps to reproduce Not run end to end. At the `NativeUtil` level, wrap a `LargeVarCharVector` holding two short strings in a `CometPlainVector`, put it in a `ColumnarBatch`, and call `exportBatch` with a schema and array address for one column. A `VarCharVector` exports on the same path. ### Expected behavior A batch Comet can produce and read can be exported to a native plan. Two directions: 1. Narrow the column to 32-bit offsets before exporting, as #6371 does for serialization. `Utils.narrowOffsets` is private there and could be shared. The copy would have to outlive the export and be released with the batch, and a column holding more than 32-bit offsets can address would still be refused. 2. Normalize once where `CometArrowPythonRunnerBase` wraps the worker's vectors, which would also cover nested children. That adds a copy per batch on the row-transition path, which today reads 64-bit offsets without copying. ### Additional context Raised by @viirya in review of #6371 (https://github.com/apache/datafusion-comet/pull/6371#pullrequestreview-5564430281), who pointed out that a native operator above `CometMapInBatchExec` is a likelier consumer than a direct mapInBatch to broadcast edge. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
