RatulDawar commented on issue #1241: URL: https://github.com/apache/datafusion-ballista/issues/1241#issuecomment-3845036831
I have earlier contributed to hudi plugin in trino and I find that architecture good, there are different maven projects for each type of table/tableformat. I believe ballista will be open to integrating postgres etc (non-olap) just like spark. Here it could be in the direction of having a iceberg-ballista-plugin crate. That generated data splits(what parts of data are to be read for this table). Then the coordinator/driver distributes it. So basically 2 apis that I can namely remember for now. Generate splits + get data from splits. And this split will be a generic abstraction for al types of table format. Though I will look into the implementation of delta to check this. Will update after I go through the implementation and find gaps with in balista and iceberg rust. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
