AngersZhuuuu commented on pull request #33828: URL: https://github.com/apache/spark/pull/33828#issuecomment-1081362106
> Is there anything written up on this commit protocol I could look at to see what could be done? Here https://github.com/apache/spark/pull/35319/files is the whole plan. And you can help to check the new added committer (SQLPathOutputCommitter)'s logic. This committer changed from FileOutputCommitter and it will write file to staging path then when commit data, it commit data to workpath. So it's file operator nums is same and can avoid the conflicts I mentioned in the pr description. Also, you can help to check if there is any problem to optimize the committer (SQLPathOutputCommitter) to use some of your idea about avoid unnecessary operations. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
