Copilot commented on code in PR #13374: URL: https://github.com/apache/gluten/pull/13374#discussion_r4322437940
########## docs/developers/ArrowColumnarConversion.md: ########## @@ -0,0 +1,487 @@ +--- +layout: page +title: Arrow Columnar Conversion +nav_order: 21 +parent: /developer-overview/ +--- + +# Arrow Columnar Conversion in Gluten + +This document describes how Gluten converts data between rows, Spark columnar batches, Java Arrow +batches, native Arrow batches and Velox batches today, and how to add a new operator that consumes +or produces Arrow batches, using the Lance scan offload +([#12958](https://github.com/apache/gluten/pull/12958)) as the example. + +## 1. Conventions + +Every physical plan node in a Gluten query has a *convention*: the row type and the batch type it +outputs. Gluten never asks a node to convert its output itself. Instead, each type registers the +transitions to and from other types, and the planner inserts the transition operators between a +child and a parent whose conventions don't match. + + + + +| Concept | Code | Meaning | +|---|---|---| +| `Convention.RowType` | `gluten-core/.../transition/Convention.scala` | A row format. `VanillaRowType` is Spark's `InternalRow`; `None` means the node outputs no rows | +| `Convention.BatchType` | same | A columnar batch format. `VanillaBatchType` is Spark's `ColumnarBatch`; `None` means no batches | Review Comment: These tables start rows with `||`, which Markdown interprets as an empty first column (rendering an extra blank column). Use a single leading `|` consistently (and match the header separator row) to ensure the tables render with the intended number of columns. ########## docs/developers/ArrowColumnarConversion.md: ########## @@ -0,0 +1,487 @@ +--- +layout: page +title: Arrow Columnar Conversion +nav_order: 21 +parent: /developer-overview/ +--- + +# Arrow Columnar Conversion in Gluten + +This document describes how Gluten converts data between rows, Spark columnar batches, Java Arrow +batches, native Arrow batches and Velox batches today, and how to add a new operator that consumes +or produces Arrow batches, using the Lance scan offload +([#12958](https://github.com/apache/gluten/pull/12958)) as the example. + +## 1. Conventions + +Every physical plan node in a Gluten query has a *convention*: the row type and the batch type it +outputs. Gluten never asks a node to convert its output itself. Instead, each type registers the +transitions to and from other types, and the planner inserts the transition operators between a +child and a parent whose conventions don't match. + + + + +| Concept | Code | Meaning | +|---|---|---| +| `Convention.RowType` | `gluten-core/.../transition/Convention.scala` | A row format. `VanillaRowType` is Spark's `InternalRow`; `None` means the node outputs no rows | +| `Convention.BatchType` | same | A columnar batch format. `VanillaBatchType` is Spark's `ColumnarBatch`; `None` means no batches | +| `ConventionReq` | `ConventionReq.scala` | What a parent requires from each child: `Any` or `Is(type)`, for rows and for batches | +| `Transition` | `Transition.scala`, `TransitionGraph.scala` | An edge of the graph: a function `SparkPlan => SparkPlan` that wraps the child with a converting operator, or `Transition.empty` when the data is already compatible | +| `ConventionFunc` | `ConventionFunc.scala` | Derives the convention of any plan, including vanilla Spark plans and plans through `QueryStageExec` / `ReusedExchangeExec` | + +A type registers its transitions in `registerTransitions()` with `fromRow`, `toRow`, `fromBatch` and +`toBatch`. All edges form one graph, and the planner uses the shortest path between two types +(`FloydWarshallGraph`), so a conversion over several steps is assembled automatically. + +How `ConventionFunc` derives a convention: + +- A `GlutenPlan` declares it: `batchType()` and `rowType()`. `supportsColumnar` and + `supportsRowBased` are derived from them and are `final`. +- A vanilla Spark plan gets `VanillaBatchType` if `supportsColumnar`, and `VanillaRowType` if it + supports rows. +- Components can override both through `Component.convFuncOverride()`. + +How the planner requires a convention from children (`conventionReqOf`): + +- A `GlutenPlan` declares `requiredChildConvention()`. By default, children must have the parent's + own convention; a node that supports both rows and batches requires only its batch type. +- `ColumnarToRowExec` requires `VanillaBatchType`; `RowToColumnarExec` and `UnionExec` require + `VanillaRowType`. + +`InsertTransitions` removes existing transitions, then for each node compares +`conventionReqOf(node)` with `conventionOf(child)` and applies the transition path found in the +graph. + +## 2. Types and how batches look + +### Types registered with the Velox backend + +| Type | Defined in | Physical form | Notes | +|---|---|---|---| +| `VanillaRowType` | `gluten-core` | Spark `InternalRow` | | +| `VanillaBatchType` | `gluten-core` | Spark `ColumnarBatch` of any `ColumnVector` | Transitions are Spark's `RowToColumnarExec` / `ColumnarToRowExec` | +| `ArrowJavaBatchType` | `gluten-arrow` `ArrowBatchTypes` | *Heavy* batch: every column is an `ArrowWritableColumnVector`, whose data is in Java Arrow vectors | Readable from the JVM through the `ColumnVector` getters | +| `ArrowNativeBatchType` | `gluten-arrow` `ArrowBatchTypes` | *Light* batch: column 0 is an `IndicatorVector` holding the handle of a native `ColumnarBatch` (e.g. `ArrowCStructColumnarBatch`), the other columns are `PlaceholderVector`s | Not readable from the JVM | +| `VeloxBatchType` | `backends-velox` `VeloxBatchType` | Light batch whose native object is a `VeloxColumnarBatch` (a Velox `RowVector`) | | +| `VeloxCarrierRowType` | `backends-velox` `VeloxCarrierRowType` | Rows that carry a Velox batch | Used for row-based outputs that keep Velox data, e.g. for writes | + +`ColumnarBatches` (`gluten-arrow/.../columnarbatch/ColumnarBatches.java`) tells heavy and light +batches apart, and holds the reference counts of both: `IndicatorVector` for light batches and +`ArrowWritableColumnVector` for heavy ones. + +### Transition graph + + + +- **`ArrowJavaBatchType` → `VanillaBatchType` is empty** because `ArrowWritableColumnVector` is a + Spark `ColumnVector`. A vanilla consumer such as `ColumnarToRowExec` reads Gluten's Java Arrow + batches directly. +- **`VeloxBatchType` → `ArrowNativeBatchType` is empty** because a `VeloxColumnarBatch` is a native + `ColumnarBatch` that can export itself as Arrow C data, which is all that `ArrowNativeBatchType` + promises. + +Composite paths that the planner builds from these edges: + +| From → To | Operators inserted | +|---|---| +| Velox → Java Arrow | `LoadArrowDataExec` (Velox → Arrow native is empty) | +| Java Arrow → Velox | `OffloadArrowDataExec` → `ArrowColumnarToVeloxColumnarExec` | +| Java Arrow → rows | `ColumnarToRowExec` (Java Arrow → vanilla batch is empty) | +| Rows → Java Arrow | `RowToVeloxColumnarExec` → `LoadArrowDataExec` | +| Velox → rows | `VeloxColumnarToRowExec` | + +## 3. What each transition does + +| Transition | Implementation | What happens to the data | +|---|---|---| +| `LoadArrowDataExec` (native → Java Arrow) | `ColumnarBatches.load` | The native batch exports Arrow C structs (`ColumnarBatchJniWrapper.exportToArrow`). For a Velox batch, `VeloxColumnarBatch::exportArrowArray` first flattens dictionary and constant vectors (`ensureFlattened`), then calls `velox::exportToArrow`. Java imports the arrays into `ArrowWritableColumnVector`s (`ArrowAbiUtil.importToSparkColumnarBatch`) | +| `OffloadArrowDataExec` (Java Arrow → native) | `ColumnarBatches.offload` | Java exports the vectors as Arrow C structs (`ArrowAbiUtil.exportFromSparkColumnarBatch`), and native wraps them as an `ArrowCStructColumnarBatch` (`createWithArrowArray`). No copy | +| `ArrowColumnarToVeloxColumnarExec` (native Arrow → Velox) | `VeloxColumnarBatches.toVeloxBatch` → `VeloxColumnarBatch::from` | `velox::importFromArrowAsOwner` builds a Velox `RowVector` that owns the Arrow buffers | +| `RowToVeloxColumnarExec` / `VeloxColumnarToRowExec` | `backends-velox` | Converts between rows and Velox vectors (always copies) | +| `ColumnarToRowExec` over Java Arrow | Spark | Reads rows through `ArrowWritableColumnVector`'s getters | + +### Copies on the Arrow path + +| Step | Fixed-width columns | Strings | Dictionary / constant vectors | +|---|---|---|---| +| Velox → Java Arrow (`load`) | No copy | Copied: Velox's `StringView` layout is exported as an Arrow string array, since Arrow string views aren't enabled | Flattened (copied) | +| Java Arrow → native Arrow (`offload`) | No copy | No copy | Not used | +| Native Arrow → Velox | No copy | A new `StringView` buffer is built (strings of 12 bytes or less are inlined) | Not used | + +### Batch lifecycle + +- `load`, `offload` and `toVeloxBatch` convert **in place**: the new vectors are moved into the + input `ColumnarBatch` object (`SparkColumnarBatchUtil.transferVectors`), and the output keeps the + input's reference count. Code that holds the input object sees the converted batch. +- A batch is released with `ColumnarBatches.release` or `ColumnarBatch.close()`, which decrements + the reference count of its vectors. An operator that needs a batch after its child has moved on + must retain it (`ColumnarBatches.retain`) and release it later, as `ColumnarArrowEvalPythonExec` + does. +- `ColumnarToColumnarExec` closes the batches it emits on the next call only when + `needRecyclePayload` is true (e.g. `VeloxResizeBatchesExec`). The Arrow transitions keep the + default, `false`. +- Java Arrow memory comes from `ArrowBufferAllocators.contextInstance()`, Gluten's per-task + allocator, whose listener reports to Spark's memory manager. + +## 4. Example: a Velox column through an Arrow UDF + +This follows one fixed-width column, a `BIGINT` column `x`, through +`Velox op → LoadArrowDataExec → ColumnarArrowEvalPythonExec → OffloadArrowDataExec → +ArrowColumnarToVeloxColumnarExec → Velox op`, for a query such as `SELECT x, f(x)` with a pandas or +Arrow UDF `f`. . Review Comment: There’s an extra period in `Arrow UDF `f`. .` which reads as a typo. Remove the trailing extra `.` so the sentence ends cleanly. ########## docs/developers/ArrowColumnarConversion.md: ########## @@ -0,0 +1,487 @@ +--- +layout: page +title: Arrow Columnar Conversion +nav_order: 21 +parent: /developer-overview/ +--- Review Comment: This front matter appears to be Jekyll-style metadata (`layout`, `nav_order`, `parent`). In MkDocs, navigation is driven by `mkdocs.yml`, so these keys likely don’t do anything (and can be confusing for future maintainers). Consider removing non-MkDocs metadata or aligning it with whatever MkDocs metadata extension the site actually uses. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
