sezruby commented on issue #13487: URL: https://github.com/apache/gluten/issues/13487#issuecomment-6476093390
Thanks for the detailed report. I opened #13488 to support vector columns in the offloaded scan instead of falling back: it rewrites `FixedSizeList` vectors as variable-size lists when the scan emits a batch, including vectors inside structs and lists of vectors. The values are not copied; each vector column only gets an offsets buffer. Velox's Arrow import has no `FixedSizeList` either, so the rewrite happens before the batch leaves the scan. For your plan-time fallback, could you leave `FixedSizeList` columns out of the fallback list, so the two changes don't conflict? Two notes on the rest: - Float16 vectors (`FixedSizeList<HALF>`, Spark 4.0+) still need the fallback: after the rewrite they are lists of `HALF`, which Spark's `ArrowColumnVector` can't read either. - The other types in this issue are unaffected by #13488. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
