gianm commented on PR #11194: URL: https://github.com/apache/druid/pull/11194#issuecomment-1845655509
Some additional color- this was done to fix a race condition that was suspected to be something with `s3` deep storage (using Hadoop's `s3n` adapter) that goes like: 1. Reducer A launches 2. Reducer A successfully uploads a segment to temp location 3. Reducer A starts renaming the temp segment to the final location (this is really a delete of the final location if it exists, then a copy of the temp to the final, and finally deletion of the temp) 4. Reducer A takes too long, triggering a replica Reducer B to be launched speculatively 5. While Reducer A is still copying the temp to the final, Reducer B finishes ingesting and uploads its temp file. 6. While Reducer A is still copying the temp to the final, Reducer B issues a delete for the final file being written by A, because is "renaming" its temp file 7. Reducer A finishes its rename 8. Deletion issued by Reducer B in step 6 takes effect and deletes the final segment 9. Reducer A completes, because it finished the rename 10. Reducer A finishes, so it's the "winning" speculative execution replica, and Reducer B gets terminated before it can finish moving its temp file to the final location, leaving just its temp file (is this how Hadoop speculative execution works?) The intent was to fix a problem where sometimes, instead of `index.zip` being present in the final location, it's `index.zip.1` or something like that. We do believe this fixed the problem, although the cost is that all the renames got moved to the launcher `index_hadoop` job rather than being spread throughout the various reducers. It would potentially slow down jobs that create large numbers of segments using large numbers of reducers. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
