FelixYBW commented on PR #59062: URL: https://github.com/apache/spark/pull/59062#issuecomment-5852150680
For context on where this is heading: we're going to submit a couple of PRs to implement [SPARK-57468](https://issues.apache.org/jira/browse/SPARK-57468) (extend the SQL query plan API to support custom `ColumnarBatch` types). In the meantime, we'll revise #34396 ([SPARK-37124](https://issues.apache.org/jira/browse/SPARK-37124), originally by @xuechendi), parts of which are already implemented in Spark today (`ArrowWriter`, `ArrowColumnVector`, and the row-to-Arrow path of `ArrowCachedBatchSerializer`). The work list: 1. **SPARK-57468 (convention API):** `ArrowBatchType` as a batch type that operators declare. 2. **Revive SPARK-37124** as `RowToArrowColumnarExec`, built on `ArrowWriter` and registered as `ArrowBatchType.fromRow`. Without it there is no planner path from rows to Arrow, and consumers keep their row-by-row fallbacks. 3. **A shared public utility** extracted from #55120's private helpers in `ColumnarArrowPythonInput` (`isArrowBacked` + layout check, and unwrapping a batch into a `VectorSchemaRoot` / record batch). Gluten, the Arrow cache and the UDTF path then stop duplicating them. 4. **An `ArrowBatchType` contract** covering: - **Layout:** canonical Arrow layout for the plan's schema. Other layouts (e.g. the Arrow cache's internal encodings) declare their own type with a normalizing transition. - **Ownership:** a batch is valid until the next one is requested; to keep it, transfer the vectors. 5. **Optional:** - A direct vanilla-columnar → Arrow transition, instead of `ColumnarToRow` + SPARK-37124. - Closing the Python reader's allocator at task end, which removes the copy of UDF results. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
