rongshen commented on PR #3910: URL: https://github.com/apache/gobblin/pull/3910#issuecomment-2043669144
> Two questions here: > > 1. Why do we try to use taskAttempId instead of one random UUID? It's still possible that same task get retried in the same container? It is possible that same path could be reused, the orphan file removal function in the publisher will clean up the path before writing new data. This change is to fix the issue found while testing orphan file deletions that a container in the shutdown can still run even though the task has been reassigned by Helix. > 2. Why do we want to avoid removing orphan files from the previous run? Any concern related to that? Orphan file removal should be enabled together with this change. This change is prerequisite for orphan file removal because this change is to make sure taskOutput path is unique for each work unit. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
