sezruby opened a new pull request, #13648: URL: https://github.com/apache/gluten/pull/13648
## What changes are proposed in this pull request? Lance scans are offloaded through lance-spark's Arrow batch reads (lance-format/lance-spark#908) instead of `LanceScanTransformer`, as proposed in https://github.com/apache/gluten/pull/12958#issuecomment-6304202853. - `LanceArrowBatchScan`, a pre-transform rule, switches a Lance `BatchScanExec` to Arrow batches (`LanceScan.withArrowBatches()`) when lance-spark can emit them and Velox imports every column type of them. `VeloxLanceComponent.convFuncOverride` declares those batches as `ArrowJavaBatchType`, so Gluten plans `BatchScanExec` → `OffloadArrowDataExec` → `ArrowColumnarToVeloxColumnarExec` for a Velox consumer, and only `ColumnarToRowExec` for a row-based one. - The scan stays a vanilla `BatchScanExec`, so it keeps lance-spark's runtime filters, output partitioning and custom metrics, which `LanceScanTransformer` and its RDD dropped. - Columns Velox doesn't import (timestamps without a time zone, null and interval columns, and the types lance-spark can't emit as Arrow batches) and scans lance-spark can't emit Arrow batches for (a pushed aggregation) stay as vanilla Spark reads them. This is decided at planning time from the Arrow schema lance-spark reports, which also covers the scans #13554 keeps on vanilla Spark. - `LanceScanTransformer`, `OffloadLanceScan` and the `FixedSizeList` rewrite are removed: lance-spark's Arrow batches already carry vectors as lists. - `OffloadArrowDataExec` now closes the batches that `ColumnarBatches.offload` builds from Spark `ArrowColumnVector`s. For such a batch `offload` returns a new batch and leaves the input to its producer, but nothing closed the new one, so tasks ended with "There are still unreleased native columnar batches during ending the task." `VeloxLanceSuite` logged it 15 times before this change and doesn't now. - `lance.version` is 0.9.0, the first lance-spark release with #908. **This PR stays a draft until that release.** - Docs: `VeloxLance.md`, and `ArrowColumnarConversion.md`, which used `LanceScanTransformer` as its example. Without pruning, this path is slower than `LanceScanTransformer`, because `adopt` copies each batch: lance-spark allocates them from Spark's Arrow allocator, and Gluten's task allocator is a separate root. In an emulated benchmark the copy cost up to 21% (#12958); with Gluten's task allocator under Spark's root allocator it would become a move. lance-spark's runtime filters (lance-format/lance-spark#927, Spark 4.1+) prune fragments on this path, which `LanceScanTransformer` couldn't. ## How was this patch tested? - `VeloxLanceSuite`: the existing tests now check that the Lance scan reads Arrow batches and, for a pushed aggregation, that it doesn't; a new test checks that a `TIMESTAMP_NTZ` column keeps the scan on vanilla Spark. Each test compares with vanilla Spark through `runQueryAndCompare`. - `ColumnarBatchTest`: `OffloadArrowDataExec`'s iterator closes the batch it built from Spark `ArrowColumnVector`s when the next batch is requested and on close, and leaves a Gluten batch to its producer. Ran both on Spark 3.5 against lance-spark main built locally and the Velox native libraries of the 2026-11-08 nightly build: 12 and 8 tests pass. Compiles on Spark 4.0 / Scala 2.13. ## Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
