NoahKusaba commented on issue #2622: URL: https://github.com/apache/datafusion-ballista/issues/2622#issuecomment-6328877810
> 3\. one note [@NoahKusaba](https://github.com/NoahKusaba) at the moment catalog support is on client side, scheduler session does not have a catalog I guess if we want to keep that design, commits can run on the client then? On the Ballista side, this needs a client-side hook that splits an INSERT into the distributed writes and a local commit, with the client reading the writes' output through the normal result path. I thought it might be better to have scheduler do commits, as it would likely be closer to the executors / data-lake in a real production setting and reduce I/O cost. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
