EpsilonPrime commented on PR #6835: URL: https://github.com/apache/incubator-gluten/pull/6835#issuecomment-2295715712
> @EpsilonPrime Before we decide to take effort on migrating to mainstream Substrait, we may do some study on how much it could help us on next moves. > > Thing is if we unfork Substrait, we should use it for supporting more backends in Gluten. Otherwise we get less benefit than cost. > > Do you happen to know the progress of Substrait integration of some projects, for example, DuckDB, Arrow and Datafusion, if Gluten decide to add support for these libraries, are their Substrait consumer implementations reliable enough for us to use? DuckDB has the best Substrait support of the three. Datafusion has a few issues which I'm hoping are addressed in their next release. Acero is in maintenance mode but has a working implementation but it's very strict about what it accepts. Other benefits include tools which run on Substrait (like the validator and text plan format) which aren't really being used by Gluten at the moment. The Spark proposal to move the Gluten communication logic (which may or may not have included Substrait) there was the reason I started looking into this effort. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
