Copilot commented on code in PR #13374: URL: https://github.com/apache/gluten/pull/13374#discussion_r4322561582
########## docs/developers/ArrowColumnarConversion.md: ########## @@ -0,0 +1,487 @@ +--- +layout: page +title: Arrow Columnar Conversion +nav_order: 21 +parent: /developer-overview/ +--- + +# Arrow Columnar Conversion in Gluten + +This document describes how Gluten converts data between rows, Spark columnar batches, Java Arrow +batches, native Arrow batches and Velox batches today, and how to add a new operator that consumes +or produces Arrow batches, using the Lance scan offload +([#12958](https://github.com/apache/gluten/pull/12958)) as the example. + +## 1. Conventions + +Every physical plan node in a Gluten query has a *convention*: the row type and the batch type it +outputs. Gluten never asks a node to convert its output itself. Instead, each type registers the +transitions to and from other types, and the planner inserts the transition operators between a +child and a parent whose conventions don't match. + + + + +| Concept | Code | Meaning | +|---|---|---| +| `Convention.RowType` | `gluten-core/.../transition/Convention.scala` | A row format. `VanillaRowType` is Spark's `InternalRow`; `None` means the node outputs no rows | +| `Convention.BatchType` | same | A columnar batch format. `VanillaBatchType` is Spark's `ColumnarBatch`; `None` means no batches | +| `ConventionReq` | `ConventionReq.scala` | What a parent requires from each child: `Any` or `Is(type)`, for rows and for batches | +| `Transition` | `Transition.scala`, `TransitionGraph.scala` | An edge of the graph: a function `SparkPlan => SparkPlan` that wraps the child with a converting operator, or `Transition.empty` when the data is already compatible | +| `ConventionFunc` | `ConventionFunc.scala` | Derives the convention of any plan, including vanilla Spark plans and plans through `QueryStageExec` / `ReusedExchangeExec` | Review Comment: The tables use `||` at the start of each row (and `||---|...` separators), which renders as an extra empty column in most Markdown renderers (including typical MkDocs + Python-Markdown). Replace the leading double-pipes with a single leading `|` for proper table formatting. This pattern appears in multiple tables throughout the doc. ########## docs/developers/ArrowColumnarConversion.md: ########## @@ -0,0 +1,487 @@ +--- +layout: page +title: Arrow Columnar Conversion +nav_order: 21 +parent: /developer-overview/ +--- + +# Arrow Columnar Conversion in Gluten + +This document describes how Gluten converts data between rows, Spark columnar batches, Java Arrow +batches, native Arrow batches and Velox batches today, and how to add a new operator that consumes +or produces Arrow batches, using the Lance scan offload +([#12958](https://github.com/apache/gluten/pull/12958)) as the example. + +## 1. Conventions + +Every physical plan node in a Gluten query has a *convention*: the row type and the batch type it +outputs. Gluten never asks a node to convert its output itself. Instead, each type registers the +transitions to and from other types, and the planner inserts the transition operators between a +child and a parent whose conventions don't match. + + + + +| Concept | Code | Meaning | +|---|---|---| +| `Convention.RowType` | `gluten-core/.../transition/Convention.scala` | A row format. `VanillaRowType` is Spark's `InternalRow`; `None` means the node outputs no rows | +| `Convention.BatchType` | same | A columnar batch format. `VanillaBatchType` is Spark's `ColumnarBatch`; `None` means no batches | +| `ConventionReq` | `ConventionReq.scala` | What a parent requires from each child: `Any` or `Is(type)`, for rows and for batches | +| `Transition` | `Transition.scala`, `TransitionGraph.scala` | An edge of the graph: a function `SparkPlan => SparkPlan` that wraps the child with a converting operator, or `Transition.empty` when the data is already compatible | +| `ConventionFunc` | `ConventionFunc.scala` | Derives the convention of any plan, including vanilla Spark plans and plans through `QueryStageExec` / `ReusedExchangeExec` | + +A type registers its transitions in `registerTransitions()` with `fromRow`, `toRow`, `fromBatch` and +`toBatch`. All edges form one graph, and the planner uses the shortest path between two types +(`FloydWarshallGraph`), so a conversion over several steps is assembled automatically. + +How `ConventionFunc` derives a convention: + +- A `GlutenPlan` declares it: `batchType()` and `rowType()`. `supportsColumnar` and + `supportsRowBased` are derived from them and are `final`. +- A vanilla Spark plan gets `VanillaBatchType` if `supportsColumnar`, and `VanillaRowType` if it + supports rows. +- Components can override both through `Component.convFuncOverride()`. + +How the planner requires a convention from children (`conventionReqOf`): + +- A `GlutenPlan` declares `requiredChildConvention()`. By default, children must have the parent's + own convention; a node that supports both rows and batches requires only its batch type. +- `ColumnarToRowExec` requires `VanillaBatchType`; `RowToColumnarExec` and `UnionExec` require + `VanillaRowType`. + +`InsertTransitions` removes existing transitions, then for each node compares +`conventionReqOf(node)` with `conventionOf(child)` and applies the transition path found in the +graph. + +## 2. Types and how batches look + +### Types registered with the Velox backend + +| Type | Defined in | Physical form | Notes | +|---|---|---|---| +| `VanillaRowType` | `gluten-core` | Spark `InternalRow` | | +| `VanillaBatchType` | `gluten-core` | Spark `ColumnarBatch` of any `ColumnVector` | Transitions are Spark's `RowToColumnarExec` / `ColumnarToRowExec` | +| `ArrowJavaBatchType` | `gluten-arrow` `ArrowBatchTypes` | *Heavy* batch: every column is an `ArrowWritableColumnVector`, whose data is in Java Arrow vectors | Readable from the JVM through the `ColumnVector` getters | +| `ArrowNativeBatchType` | `gluten-arrow` `ArrowBatchTypes` | *Light* batch: column 0 is an `IndicatorVector` holding the handle of a native `ColumnarBatch` (e.g. `ArrowCStructColumnarBatch`), the other columns are `PlaceholderVector`s | Not readable from the JVM | +| `VeloxBatchType` | `backends-velox` `VeloxBatchType` | Light batch whose native object is a `VeloxColumnarBatch` (a Velox `RowVector`) | | +| `VeloxCarrierRowType` | `backends-velox` `VeloxCarrierRowType` | Rows that carry a Velox batch | Used for row-based outputs that keep Velox data, e.g. for writes | + +`ColumnarBatches` (`gluten-arrow/.../columnarbatch/ColumnarBatches.java`) tells heavy and light +batches apart, and holds the reference counts of both: `IndicatorVector` for light batches and +`ArrowWritableColumnVector` for heavy ones. + +### Transition graph + + + +- **`ArrowJavaBatchType` → `VanillaBatchType` is empty** because `ArrowWritableColumnVector` is a + Spark `ColumnVector`. A vanilla consumer such as `ColumnarToRowExec` reads Gluten's Java Arrow + batches directly. +- **`VeloxBatchType` → `ArrowNativeBatchType` is empty** because a `VeloxColumnarBatch` is a native + `ColumnarBatch` that can export itself as Arrow C data, which is all that `ArrowNativeBatchType` + promises. + +Composite paths that the planner builds from these edges: + +| From → To | Operators inserted | +|---|---| +| Velox → Java Arrow | `LoadArrowDataExec` (Velox → Arrow native is empty) | +| Java Arrow → Velox | `OffloadArrowDataExec` → `ArrowColumnarToVeloxColumnarExec` | +| Java Arrow → rows | `ColumnarToRowExec` (Java Arrow → vanilla batch is empty) | +| Rows → Java Arrow | `RowToVeloxColumnarExec` → `LoadArrowDataExec` | +| Velox → rows | `VeloxColumnarToRowExec` | + +## 3. What each transition does + +| Transition | Implementation | What happens to the data | +|---|---|---| +| `LoadArrowDataExec` (native → Java Arrow) | `ColumnarBatches.load` | The native batch exports Arrow C structs (`ColumnarBatchJniWrapper.exportToArrow`). For a Velox batch, `VeloxColumnarBatch::exportArrowArray` first flattens dictionary and constant vectors (`ensureFlattened`), then calls `velox::exportToArrow`. Java imports the arrays into `ArrowWritableColumnVector`s (`ArrowAbiUtil.importToSparkColumnarBatch`) | +| `OffloadArrowDataExec` (Java Arrow → native) | `ColumnarBatches.offload` | Java exports the vectors as Arrow C structs (`ArrowAbiUtil.exportFromSparkColumnarBatch`), and native wraps them as an `ArrowCStructColumnarBatch` (`createWithArrowArray`). No copy | +| `ArrowColumnarToVeloxColumnarExec` (native Arrow → Velox) | `VeloxColumnarBatches.toVeloxBatch` → `VeloxColumnarBatch::from` | `velox::importFromArrowAsOwner` builds a Velox `RowVector` that owns the Arrow buffers | +| `RowToVeloxColumnarExec` / `VeloxColumnarToRowExec` | `backends-velox` | Converts between rows and Velox vectors (always copies) | +| `ColumnarToRowExec` over Java Arrow | Spark | Reads rows through `ArrowWritableColumnVector`'s getters | + +### Copies on the Arrow path + +| Step | Fixed-width columns | Strings | Dictionary / constant vectors | +|---|---|---|---| +| Velox → Java Arrow (`load`) | No copy | Copied: Velox's `StringView` layout is exported as an Arrow string array, since Arrow string views aren't enabled | Flattened (copied) | +| Java Arrow → native Arrow (`offload`) | No copy | No copy | Not used | +| Native Arrow → Velox | No copy | A new `StringView` buffer is built (strings of 12 bytes or less are inlined) | Not used | + +### Batch lifecycle + +- `load`, `offload` and `toVeloxBatch` convert **in place**: the new vectors are moved into the + input `ColumnarBatch` object (`SparkColumnarBatchUtil.transferVectors`), and the output keeps the + input's reference count. Code that holds the input object sees the converted batch. +- A batch is released with `ColumnarBatches.release` or `ColumnarBatch.close()`, which decrements + the reference count of its vectors. An operator that needs a batch after its child has moved on + must retain it (`ColumnarBatches.retain`) and release it later, as `ColumnarArrowEvalPythonExec` + does. +- `ColumnarToColumnarExec` closes the batches it emits on the next call only when + `needRecyclePayload` is true (e.g. `VeloxResizeBatchesExec`). The Arrow transitions keep the + default, `false`. +- Java Arrow memory comes from `ArrowBufferAllocators.contextInstance()`, Gluten's per-task + allocator, whose listener reports to Spark's memory manager. + +## 4. Example: a Velox column through an Arrow UDF + +This follows one fixed-width column, a `BIGINT` column `x`, through +`Velox op → LoadArrowDataExec → ColumnarArrowEvalPythonExec → OffloadArrowDataExec → +ArrowColumnarToVeloxColumnarExec → Velox op`, for a query such as `SELECT x, f(x)` with a pandas or +Arrow UDF `f`. . Review Comment: There is an extra period at the end of the sentence (`. .`). Remove the duplicate so the sentence ends with a single period. ########## docs/developers/ArrowColumnarConversion.md: ########## @@ -0,0 +1,487 @@ +--- +layout: page +title: Arrow Columnar Conversion +nav_order: 21 +parent: /developer-overview/ +--- + +# Arrow Columnar Conversion in Gluten + +This document describes how Gluten converts data between rows, Spark columnar batches, Java Arrow +batches, native Arrow batches and Velox batches today, and how to add a new operator that consumes +or produces Arrow batches, using the Lance scan offload +([#12958](https://github.com/apache/gluten/pull/12958)) as the example. + +## 1. Conventions + +Every physical plan node in a Gluten query has a *convention*: the row type and the batch type it +outputs. Gluten never asks a node to convert its output itself. Instead, each type registers the +transitions to and from other types, and the planner inserts the transition operators between a +child and a parent whose conventions don't match. + + + + +| Concept | Code | Meaning | +|---|---|---| +| `Convention.RowType` | `gluten-core/.../transition/Convention.scala` | A row format. `VanillaRowType` is Spark's `InternalRow`; `None` means the node outputs no rows | +| `Convention.BatchType` | same | A columnar batch format. `VanillaBatchType` is Spark's `ColumnarBatch`; `None` means no batches | +| `ConventionReq` | `ConventionReq.scala` | What a parent requires from each child: `Any` or `Is(type)`, for rows and for batches | +| `Transition` | `Transition.scala`, `TransitionGraph.scala` | An edge of the graph: a function `SparkPlan => SparkPlan` that wraps the child with a converting operator, or `Transition.empty` when the data is already compatible | +| `ConventionFunc` | `ConventionFunc.scala` | Derives the convention of any plan, including vanilla Spark plans and plans through `QueryStageExec` / `ReusedExchangeExec` | + +A type registers its transitions in `registerTransitions()` with `fromRow`, `toRow`, `fromBatch` and +`toBatch`. All edges form one graph, and the planner uses the shortest path between two types +(`FloydWarshallGraph`), so a conversion over several steps is assembled automatically. + +How `ConventionFunc` derives a convention: + +- A `GlutenPlan` declares it: `batchType()` and `rowType()`. `supportsColumnar` and + `supportsRowBased` are derived from them and are `final`. +- A vanilla Spark plan gets `VanillaBatchType` if `supportsColumnar`, and `VanillaRowType` if it + supports rows. +- Components can override both through `Component.convFuncOverride()`. + +How the planner requires a convention from children (`conventionReqOf`): + +- A `GlutenPlan` declares `requiredChildConvention()`. By default, children must have the parent's + own convention; a node that supports both rows and batches requires only its batch type. +- `ColumnarToRowExec` requires `VanillaBatchType`; `RowToColumnarExec` and `UnionExec` require + `VanillaRowType`. + +`InsertTransitions` removes existing transitions, then for each node compares +`conventionReqOf(node)` with `conventionOf(child)` and applies the transition path found in the +graph. + +## 2. Types and how batches look + +### Types registered with the Velox backend + +| Type | Defined in | Physical form | Notes | +|---|---|---|---| +| `VanillaRowType` | `gluten-core` | Spark `InternalRow` | | +| `VanillaBatchType` | `gluten-core` | Spark `ColumnarBatch` of any `ColumnVector` | Transitions are Spark's `RowToColumnarExec` / `ColumnarToRowExec` | +| `ArrowJavaBatchType` | `gluten-arrow` `ArrowBatchTypes` | *Heavy* batch: every column is an `ArrowWritableColumnVector`, whose data is in Java Arrow vectors | Readable from the JVM through the `ColumnVector` getters | +| `ArrowNativeBatchType` | `gluten-arrow` `ArrowBatchTypes` | *Light* batch: column 0 is an `IndicatorVector` holding the handle of a native `ColumnarBatch` (e.g. `ArrowCStructColumnarBatch`), the other columns are `PlaceholderVector`s | Not readable from the JVM | +| `VeloxBatchType` | `backends-velox` `VeloxBatchType` | Light batch whose native object is a `VeloxColumnarBatch` (a Velox `RowVector`) | | +| `VeloxCarrierRowType` | `backends-velox` `VeloxCarrierRowType` | Rows that carry a Velox batch | Used for row-based outputs that keep Velox data, e.g. for writes | + +`ColumnarBatches` (`gluten-arrow/.../columnarbatch/ColumnarBatches.java`) tells heavy and light +batches apart, and holds the reference counts of both: `IndicatorVector` for light batches and +`ArrowWritableColumnVector` for heavy ones. + +### Transition graph + + + +- **`ArrowJavaBatchType` → `VanillaBatchType` is empty** because `ArrowWritableColumnVector` is a + Spark `ColumnVector`. A vanilla consumer such as `ColumnarToRowExec` reads Gluten's Java Arrow + batches directly. +- **`VeloxBatchType` → `ArrowNativeBatchType` is empty** because a `VeloxColumnarBatch` is a native + `ColumnarBatch` that can export itself as Arrow C data, which is all that `ArrowNativeBatchType` + promises. + +Composite paths that the planner builds from these edges: + +| From → To | Operators inserted | +|---|---| +| Velox → Java Arrow | `LoadArrowDataExec` (Velox → Arrow native is empty) | +| Java Arrow → Velox | `OffloadArrowDataExec` → `ArrowColumnarToVeloxColumnarExec` | +| Java Arrow → rows | `ColumnarToRowExec` (Java Arrow → vanilla batch is empty) | +| Rows → Java Arrow | `RowToVeloxColumnarExec` → `LoadArrowDataExec` | +| Velox → rows | `VeloxColumnarToRowExec` | + +## 3. What each transition does + +| Transition | Implementation | What happens to the data | +|---|---|---| +| `LoadArrowDataExec` (native → Java Arrow) | `ColumnarBatches.load` | The native batch exports Arrow C structs (`ColumnarBatchJniWrapper.exportToArrow`). For a Velox batch, `VeloxColumnarBatch::exportArrowArray` first flattens dictionary and constant vectors (`ensureFlattened`), then calls `velox::exportToArrow`. Java imports the arrays into `ArrowWritableColumnVector`s (`ArrowAbiUtil.importToSparkColumnarBatch`) | +| `OffloadArrowDataExec` (Java Arrow → native) | `ColumnarBatches.offload` | Java exports the vectors as Arrow C structs (`ArrowAbiUtil.exportFromSparkColumnarBatch`), and native wraps them as an `ArrowCStructColumnarBatch` (`createWithArrowArray`). No copy | +| `ArrowColumnarToVeloxColumnarExec` (native Arrow → Velox) | `VeloxColumnarBatches.toVeloxBatch` → `VeloxColumnarBatch::from` | `velox::importFromArrowAsOwner` builds a Velox `RowVector` that owns the Arrow buffers | +| `RowToVeloxColumnarExec` / `VeloxColumnarToRowExec` | `backends-velox` | Converts between rows and Velox vectors (always copies) | +| `ColumnarToRowExec` over Java Arrow | Spark | Reads rows through `ArrowWritableColumnVector`'s getters | + +### Copies on the Arrow path + +| Step | Fixed-width columns | Strings | Dictionary / constant vectors | +|---|---|---|---| +| Velox → Java Arrow (`load`) | No copy | Copied: Velox's `StringView` layout is exported as an Arrow string array, since Arrow string views aren't enabled | Flattened (copied) | +| Java Arrow → native Arrow (`offload`) | No copy | No copy | Not used | +| Native Arrow → Velox | No copy | A new `StringView` buffer is built (strings of 12 bytes or less are inlined) | Not used | + +### Batch lifecycle + +- `load`, `offload` and `toVeloxBatch` convert **in place**: the new vectors are moved into the + input `ColumnarBatch` object (`SparkColumnarBatchUtil.transferVectors`), and the output keeps the + input's reference count. Code that holds the input object sees the converted batch. +- A batch is released with `ColumnarBatches.release` or `ColumnarBatch.close()`, which decrements + the reference count of its vectors. An operator that needs a batch after its child has moved on + must retain it (`ColumnarBatches.retain`) and release it later, as `ColumnarArrowEvalPythonExec` + does. +- `ColumnarToColumnarExec` closes the batches it emits on the next call only when + `needRecyclePayload` is true (e.g. `VeloxResizeBatchesExec`). The Arrow transitions keep the + default, `false`. +- Java Arrow memory comes from `ArrowBufferAllocators.contextInstance()`, Gluten's per-task + allocator, whose listener reports to Spark's memory manager. + +## 4. Example: a Velox column through an Arrow UDF + +This follows one fixed-width column, a `BIGINT` column `x`, through +`Velox op → LoadArrowDataExec → ColumnarArrowEvalPythonExec → OffloadArrowDataExec → +ArrowColumnarToVeloxColumnarExec → Velox op`, for a query such as `SELECT x, f(x)` with a pandas or +Arrow UDF `f`. . + + + +Three counts are involved, and none of them crosses JNI: + +- **Velox buffer references** (`BufferPtr`), held by Velox vectors and by an exported `ArrowArray`. +- **Arrow Java `ArrowBuf` references**. An imported array is released through its C `release` + callback once its last `ArrowBuf` is released. +- **Gluten's `refCnt`** on `ArrowWritableColumnVector`. It only decides when Gluten closes the Java + vector. + +The Arrow C Data interface has no reference count. Ownership of an `ArrowArray` moves once, and the +holder calls `release` exactly once. + +| Step | Velox buffer refs | `x.refCnt` | What happens | +|---|---|---|---| +| 1. Export | 2 | – | `VeloxColumnarBatch::exportArrowArray` exports `ArrowArray` A, which keeps a reference to the buffer. No copy for a flat fixed-width vector | +| 2. Import | 2 | 1 | `Data.importIntoVectorSchemaRoot` wraps the Velox memory as `ArrowBuf`s, and `loadColumns` creates `x` | +| 3. Close the Velox batch | 1 | 1 | `load` closes the input batch: `ObjectStore::release` frees the `RowVector`. A keeps the buffer alive | +| 4. Retain for the UDF | 1 | 2 | `ColumnarArrowEvalPythonExec` retains the input batch until the result comes back | +| 5. Send to Python | 1 | 2 | `ArrowStreamWriter` serializes the UDF columns into the socket. The Python worker reads its own copy; when pyarrow releases it, only Python's memory is freed | +| 6. Join the result | 1 | 1 | The output batch is `[x, r]`. `adjustRefCnt` sets every column to 1, because `offload` requires one count for the whole batch | +| 7. Export to native | 1 | 0 | `offload` exports `ArrowArray` B, which retains the `ArrowBuf`s, then closes the input batch. `x` is closed, and B keeps the memory alive | +| 8. Import into Velox | 1 | – | `importFromArrowAsOwner(B)` builds a Velox vector over the same memory. No copy | +| 9. Release | 0 | – | When the downstream Velox operator frees the vector, B's `release` (Arrow Java's export callback) drops the last `ArrowBuf`, which calls A's `release` (Velox's export callback). The buffer goes back to Velox's memory pool | + +So the column's memory is freed by the last Velox operator that uses it, through two `release` +callbacks: Velox → Arrow Java → Velox. If the UDF output goes to a vanilla Spark operator instead, +step 7 doesn't happen, and the memory is freed when `ColumnarArrowEvalPythonExec` closes its output +batch on the next call (`recyclePayload`). + +<details> +<summary>PlantUML source: Velox column through an Arrow UDF</summary> + +```plantuml +@startuml VeloxColumnArrowUdfLifecycle +skinparam shadowing false +skinparam sequenceMessageAlign left +skinparam responseMessageBelowArrow true +hide footbox + +title A Velox BIGINT column through an Arrow UDF and back (Gluten main) + +box "Native (C++)" #EEF7EE +participant "Velox buffer\n(values of x)" as Buf +participant "VeloxColumnarBatch\n(upstream op)" as VB +participant "Velox\n(downstream op)" as VD +end box +box "JVM" #E8F0FB +participant "LoadArrowDataExec\nColumnarBatches.load" as Load +participant "ArrowWritableColumnVector x\n(Arrow BigIntVector)" as X +participant "ColumnarArrowEvalPythonExec" as Py +participant "OffloadArrowDataExec\n+ ArrowColumnarToVelox" as Off +end box +participant "Python worker\n(pyarrow UDF)" as W + +== 1. Velox -> Arrow C -> Java (no copy) == +Load -> VB : JNI exportToArrow(handle) +VB -> VB : ensureFlattened()\nvelox::exportToArrow() +VB -> Buf : ArrowArray A keeps a buffer ref\n<b>Velox refs: 2</b> (vector + A) +VB --> Load : ArrowArrayMove(A) +Load -> X : Data.importIntoVectorSchemaRoot(A)\nArrowBuf wraps Velox memory\nloadColumns() <b>x.refCnt = 1</b> +Load -> VB : input.close() -> ObjectStore::release\nRowVector freed +VB -> Buf : <b>Velox refs: 1</b> (only A) + +== 2. To Python and back (IPC copy) == +Py -> X : ColumnarBatches.retain <b>x.refCnt = 2</b>\ncache input batch +Py -> W : ArrowStreamWriter.writeBatch\n(serialize x into the socket) +W -> W : pyarrow reads its own copy,\nruns the UDF, frees its copy\n(no effect on JVM memory) +W --> Py : result batch r (IPC) +Py -> X : join [x, r]\nadjustRefCnt(x, 1) <b>x.refCnt = 1</b> + +== 3. Java -> Arrow C -> Velox (no copy) == +Off -> X : Data.exportVectorSchemaRoot -> ArrowArray B\nB retains the ArrowBufs +Off -> X : offload closes the input batch\n<b>x.refCnt = 0</b> -> vector.close()\n(ArrowBufs still held by B) +Off -> VD : createWithArrowArray(B)\nimportFromArrowAsOwner(B) +note right of VD : Velox reads the same memory + +== 4. Release chain == +VD -> VD : downstream op done with the vector +VD -> X : B.release() (Arrow Java export callback)\nlast ArrowBuf released +X -> Buf : A.release() (Velox export callback)\n<b>Velox refs: 0</b>, buffer returned to the MemoryPool +@enduml +``` + +</details> + +### Buffer release + +Neither `ArrowWritableColumnVector.close()` nor `B.release()` checks where a buffer came from. +Both drop one reference on each `ArrowBuf`, and when the last one goes, the buffer's +`AllocationManager`, chosen when the buffer was created, does the rest: + +- **Allocated in Java** (`allocator.buffer(...)`, e.g. by `ColumnarRangeExec` or the Python result + reader): the memory is freed in Java. +- **Imported through C Data** (`Data.importIntoVectorSchemaRoot`): the buffer has a + `ForeignAllocationManager`, whose `release0()` calls `ReferenceCountedArrowArray.release()`. When + the last buffer of the imported array is released, it calls the array's C `release` callback, + here Velox's. It's the `release` member function of Arrow C_data interface: Review Comment: `Arrow C_data interface` appears to be a typo/inconsistent naming; elsewhere the doc uses `Arrow C Data`. Consider changing this to `Arrow C Data interface` for consistency and readability. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
