FelixYBW commented on issue #12263: URL: https://github.com/apache/gluten/issues/12263#issuecomment-4666307828
> > Before that, we either need to add it to lance-spark or copy the code from lance-spark to Gluten and modify it. > > I'd prefer to introduce Lance support as a separate module, following the same pattern we used for Iceberg, and pull in lance-spark as a dependency from there. Yes, we need a separate module. Two things I'm thinking: 1. To support iceberg or delta, we copied several files from their data source and hacked in Gluten. See if we can avoid doing this in lance-spark. Create APIs in lance-spark and call from Gluten 2. General arrow input/output support. See if we can submit new APIs in Spark to support arrow format input/output. It's something we plan to do when we support bigquery connector. In future, if data source is Arrow format, Gluten can automatically support. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
