sezruby opened a new pull request, #13648:
URL: https://github.com/apache/gluten/pull/13648

   ## What changes are proposed in this pull request?
   
   Lance scans are offloaded through lance-spark's Arrow batch reads 
(lance-format/lance-spark#908) instead of `LanceScanTransformer`, as proposed 
in https://github.com/apache/gluten/pull/12958#issuecomment-6304202853.
   
   - `LanceArrowBatchScan`, a pre-transform rule, switches a Lance 
`BatchScanExec` to Arrow batches (`LanceScan.withArrowBatches()`) when 
lance-spark can emit them and Velox imports every column type of them. 
`VeloxLanceComponent.convFuncOverride` declares those batches as 
`ArrowJavaBatchType`, so Gluten plans `BatchScanExec` → `OffloadArrowDataExec` 
→ `ArrowColumnarToVeloxColumnarExec` for a Velox consumer, and only 
`ColumnarToRowExec` for a row-based one.
   - The scan stays a vanilla `BatchScanExec`, so it keeps lance-spark's 
runtime filters, output partitioning and custom metrics, which 
`LanceScanTransformer` and its RDD dropped.
   - Columns Velox doesn't import (timestamps without a time zone, null and 
interval columns, and the types lance-spark can't emit as Arrow batches) and 
scans lance-spark can't emit Arrow batches for (a pushed aggregation) stay as 
vanilla Spark reads them. This is decided at planning time from the Arrow 
schema lance-spark reports, which also covers the scans #13554 keeps on vanilla 
Spark.
   - `LanceScanTransformer`, `OffloadLanceScan` and the `FixedSizeList` rewrite 
are removed: lance-spark's Arrow batches already carry vectors as lists.
   - `OffloadArrowDataExec` now closes the batches that 
`ColumnarBatches.offload` builds from Spark `ArrowColumnVector`s. For such a 
batch `offload` returns a new batch and leaves the input to its producer, but 
nothing closed the new one, so tasks ended with "There are still unreleased 
native columnar batches during ending the task." `VeloxLanceSuite` logged it 15 
times before this change and doesn't now.
   - `lance.version` is 0.9.0, the first lance-spark release with #908. **This 
PR stays a draft until that release.**
   - Docs: `VeloxLance.md`, and `ArrowColumnarConversion.md`, which used 
`LanceScanTransformer` as its example.
   
   Without pruning, this path is slower than `LanceScanTransformer`, because 
`adopt` copies each batch: lance-spark allocates them from Spark's Arrow 
allocator, and Gluten's task allocator is a separate root. In an emulated 
benchmark the copy cost up to 21% (#12958); with Gluten's task allocator under 
Spark's root allocator it would become a move. lance-spark's runtime filters 
(lance-format/lance-spark#927, Spark 4.1+) prune fragments on this path, which 
`LanceScanTransformer` couldn't.
   
   ## How was this patch tested?
   
   - `VeloxLanceSuite`: the existing tests now check that the Lance scan reads 
Arrow batches and, for a pushed aggregation, that it doesn't; a new test checks 
that a `TIMESTAMP_NTZ` column keeps the scan on vanilla Spark. Each test 
compares with vanilla Spark through `runQueryAndCompare`.
   - `ColumnarBatchTest`: `OffloadArrowDataExec`'s iterator closes the batch it 
built from Spark `ArrowColumnVector`s when the next batch is requested and on 
close, and leaves a Gluten batch to its producer.
   
   Ran both on Spark 3.5 against lance-spark main built locally and the Velox 
native libraries of the 2026-11-08 nightly build: 12 and 8 tests pass. Compiles 
on Spark 4.0 / Scala 2.13.
   
   ## Was this patch authored or co-authored using generative AI tooling?
   
   Generated-by: Claude Opus 5.5
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to