jackylee-ch commented on issue #12263: URL: https://github.com/apache/gluten/issues/12263#issuecomment-4665824080
> Before that, we either need to add it to lance-spark or copy the code from lance-spark to Gluten and modify it. I'd prefer to introduce Lance support as a separate module, following the same pattern we used for Iceberg, and pull in lance-spark as a dependency from there. A few reasons this is the cleanest path: - It's consistent with how Gluten already integrates external data sources (Iceberg being the precedent), which lowers the maintenance and review burden and keeps the project's structure predictable for contributors. - The integration framework is engine-agnostic. Once the Velox path lands, we should be able to reuse most of this scaffolding to wire Lance up to Velox with only minor changes, rather than reworking the integration. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
