zhztheplayer opened a new issue, #7031: URL: https://github.com/apache/incubator-gluten/issues/7031
### Description Part of https://github.com/apache/incubator-gluten/issues/6920 which is the final goal. `gluten-substrait` will include Gluten's substrait definitions, and the abstract APIs of Substrait-based query plans. This could mean, most of the current plan nodes' base traits/classes that are coupled with Substrait will be moved into `gluten-substrait`. Also, some backend APIs that are related to Substrait, say `SparkPlanExecApi` may be factored out to have major of the code placed in this new module. `gluten-core` will become a comparatively thin layer than before. The module will provide Spark listeners, task life cycle managers, query planner facilities, and memory consumer, metrics, config utilities, etc. After the refactor, when a new library/backend is getting integrated into Gluten, it will not be strictly required to implement complete Gluten Substrait protocols and will have its own flexibility to inject its native computation ability into Gluten. Since developer could decide whether to use `gluten-substrait` as a parent or not. Meanwhile, `gluten-data` may be renamed to `gluten-arrow` for clearer demonstration for its purpose, which targets to provide pre-defined data sharing/transition utilities for arrow-compatible libraries. But this could be in another topic. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
