steveloughran commented on pull request #2349: URL: https://github.com/apache/hadoop/pull/2349#issuecomment-859061589
Once the manifest committer #2971 is in, the abfs and gcs stores will get something faster than v1 but with its task failure semantics * all treewalking for task attempt listing will be in task commit * no dir renames in task commit, just saving of the manifest * job commit: parallel load of manifests, merge of list of directories to create, parallel set of mkdirs and then the parallelized renames. In this world, we can just leave people using v1/v2 alone, and for (spark) jobs in azure and google cloud say "use the manifest committer". Which means I can just close this as a wontfix. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
