itskals edited a comment on issue #21066: [SPARK-23977][CLOUD][WIP] Add commit protocol binding to Hadoop 3.1 PathOutputCommitter mechanism URL: https://github.com/apache/spark/pull/21066#issuecomment-473780802 Hi @steveloughran and others, I had a doubt on the behaviour of `PathOutputCommitProtocol`, which throws an exception when `dynamicPartitionOverwrite` is true. I am not clear why it should throw this exception? What would happen if this code is opened. I tried to decode the comment on the parameter, " dynamic partition overwrite is not supported, so that committers for stores which do not support rename will not get confused.", but I got bit confused. There are no renames in this complete idea. Can you kindly elaborate this? As far as I understand( please correct me if I am wrong) , when DynamicOverwrite is set, the committer(Directory/partition) will delete the existing contents(in REPLACE mode). Which is exactly what is expected. Thanks. @prakharjain09
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
