milenkovicm commented on issue #1241: URL: https://github.com/apache/datafusion-ballista/issues/1241#issuecomment-2808534387
I agree that adding support for popular table formats would be a valuable addition to Ballista, but I don't believe Ballista is the main hurdle in achieving that support. Recent changes made ballista modular and easy to extend. [ballista-delta] uses exposed extension points to provide support for delta table read. Overall it was not too hard to provide delta (read) support as both delta and ballista had all required parts implemented/exposed. Not sure how hard would it be to add write support. Last time I've checked Iceberg implementations had a bit bigger gap than delta, thus from my perspective in order to get Iceberg support more work is needed Iceberg side than in ballista. Also both Iceberg implementation are under active development thus we can't underestimate size of changes releases can bring. Finally, in order to make this work in a long term we need to find a sustainable way to support table formats. Currently I really don't believe integration effort could be provided by ballista maintainer(s). Effort should be either part of table format(s) (which probably has same maintenance issues like ballista) or effort could be a "ballista/datafusio-contrib" project, with its own maintenance, release cadence ... Ballista could actively support effort by exposing/adding/improving necessary interfaces needed for table effort. [ballista-delta]:https://github.com/milenkovicm/ballista_delta -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
