malinjawi opened a new pull request, #13554: URL: https://github.com/apache/gluten/pull/13554
## What changes are proposed in this pull request? Part of #13487. #13488 fixed vector columns. Some other Lance scans still get offloaded even though the Arrow stream can't carry their columns, so they fail on the first batch. This PR makes them fall back to vanilla Spark at plan time. `LanceScanTransformer.isExportable` now falls back when a scan reads: - a blob, JSON or metadata column (`_rowid`, `_rowaddr`, `_fragid`, row versions, `_score`). lance-spark fixes these up itself after the native scan. - no columns at all, like `SELECT 1 FROM t`. The native scan returns `_rowid` instead. - a column Spark's `ArrowColumnVector` can't wrap: `FixedSizeBinary`, `Date(MILLISECOND)` and Float16 vectors. The check maps the column with lance-spark's `LanceArrowUtils` and rewrites vectors as lists, the same as the scan does since #13488. Only the columns a query reads count, and other vector columns stay offloaded. Not covered yet: unsigned ints, `LargeList`, time and non-microsecond timestamps. lance-spark 0.8.0's Spark schema doesn't tell these apart, so they can't be caught at plan time. That needs a change in lance-spark. ## How was this patch tested? Two new tests in `VeloxLanceSuite` cover each case above. They also check that the same table still offloads when the unsupported column isn't read. The Float16 case runs on Spark 4.0+ only, because lance-spark can't write it on 3.5. `VeloxLanceSuite` and `LanceFixedSizeListsSuite` pass on Spark 3.5 and 4.0 with the lance-spark 0.8.0 release. Without this change, the two new tests fail. ## Was this patch authored or co-authored using generative AI tooling? Generated-by: IBM Bob -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
