sezruby opened a new pull request, #13488: URL: https://github.com/apache/gluten/pull/13488
## What changes are proposed in this pull request? Lance stores vector (embedding) columns as Arrow `FixedSizeList`. The offloaded Lance scan wraps every Arrow vector it imports from lance-spark in Spark's `ArrowColumnVector`, which rejects `FixedSizeList`, so reading a vector column with Gluten enabled fails at runtime (#13487): ``` org.apache.spark.SparkUnsupportedOperationException: [UNSUPPORTED_ARROWTYPE] Unsupported arrow type FixedSizeList(4). ``` Velox's Arrow import (`importFromArrowImpl`) has no `FixedSizeList` (`+w:N`) case either, so the type has to be rewritten before the batch reaches either of them. This PR rewrites `FixedSizeList` vectors as variable-size `List`s when `LanceFragmentIterator` emits a batch: - A `List` holds the same values and matches the `ArrayType` that lance-spark already declares for the column, so both Spark's `ArrowColumnVector` and the offload to Velox read it. - Element `i` of a `FixedSizeList` of size `n` occupies child range `[i * n, (i + 1) * n)`, also for null entries, so the offsets are `i * n` throughout. The child values are moved without copying; the only new buffer is the offsets buffer (4 bytes per row). - Nested vectors (a vector inside a struct, a list of vectors) are rewritten recursively. Columns without a `FixedSizeList` keep the existing transfer path unchanged. This covers the vector part of #13487. The other column types listed there (`FixedSizeBinary`, millisecond `Date`, JSON, blob, unsigned integers, `LargeList`, metadata columns, empty projections) still need the plan-time fallback. ## How was this patch tested? `VeloxLanceSuite`, each test checked against vanilla Spark with `runQueryAndCompare` and against expected rows: - `lance scan offloads vector columns`: float and double vectors, including null vectors. - `lance scan offloads nested vector columns`: a vector inside a struct and a list of vectors, with null structs, null vectors inside a struct, null and empty lists. - `lance vector columns feed Velox operators`: element access and aggregation over top-level and nested vectors run in Velox, which reads the scan's batches directly (`OffloadArrowDataExec` over the scan, no row conversion). - `lance vector columns pass through a Velox filter, shuffle and sort`: a filter Lance does not push down, a columnar shuffle, and a sort over rows carrying vectors. - `lance scan reads vector batches that are all null or empty`. - `lance scan reads vectors across fragments and batches`: 1000 rows written as several fragments (`max_row_per_file=300`) and read in batches of 64. `LanceFixedSizeListsSuite`, unit tests on Arrow vectors (each on a `RootAllocator` that fails on leaks): values and offsets with null rows, no-null/all-null/empty batches, values moved without copying, nested vectors, and release of partial results when the rewrite or a later column fails. Ran both suites on Spark 3.5 and Spark 4.0 against the Velox native libraries of the current nightly build. On main, the new `VeloxLanceSuite` tests fail with `UNSUPPORTED_ARROWTYPE` or a `VeloxRuntimeError`, and the existing tests still pass. ## Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
