sezruby commented on PR #12958: URL: https://github.com/apache/gluten/pull/12958#issuecomment-6746629127
@FelixYBW an update on the Arrow-batch path, with measurements. - **lance-format/lance-spark#908 is merged**; it will ship in the next lance-spark release. Batches are now allocated from a process-wide child of `ArrowUtils.rootAllocator()`, capped by `LANCE_ALLOCATOR_SIZE`. - **Runtime filters.** lance-format/lance-spark#936 adds runtime filtering to `LanceScan` (`SupportsRuntimeV2Filtering`, pruning fragments by zonemap). `LanceScanTransformer` plans its partitions with `lanceScan.toBatch.planInputPartitions()` and doesn't apply `BatchScanExec.runtimeFilters`, so an offloaded Lance scan reads every fragment even when Spark plans a runtime filter. Results stay correct. On the Arrow-batch path the `BatchScanExec` stays, and it applies them. I measured this with the 2026-11-08 nightly (`gluten-velox-bundle-spark3.5_2.12-linux_amd64-1.9.0-SNAPSHOT`) on Spark 3.5: a 20M-row Lance fact table in 200 fragments, joined with a filtered Lance date dimension, best of 7. It ran in an emulated amd64 container, so compare the times only with each other; the fragment counts are exact. | Query | `LanceScanTransformer` | Arrow batches, no runtime filter | Arrow batches + runtime filter | |---|---|---|---| | join, 1 month | 641 ms, 200 fragments | 769 ms, 200 | 218 ms, 4 | | join, 2 months | 565 ms, 200 | 642 ms, 200 | 188 ms, 8 | | full scan aggregate | 442 ms | 535 ms | — | Without pruning, the Arrow-batch path is up to 21% slower here because of the copy in `ArrowColumnVectors.adopt` (issue 2 in my last comment). Gluten's `ArrowBufferAllocators` are separate `RootAllocator`s, so every batch is copied rather than transferred. The Gluten side in this run is the prototype from my last comment: a `convFuncOverride` that marks a Lance `BatchScanExec` with Arrow batch reads as `ArrowJavaBatchType`, with `OffloadLanceScan` not injected. Is your allocator change planned for main? With it, this path wouldn't need the copy. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
