LakshSingla commented on PR #13062: URL: https://github.com/apache/druid/pull/13062#issuecomment-1250817660
I have updated the code in the `WorkerImpl` to do the following (when the durable storage is enabled): Once the data for all the partitions have been generated it does the following things: 1. Write the data into the path "controller_task_id/stage_a/worker_b/taskId_c/part_d". (Note: there is a slight change from the existing path structure). 2. Checks the folder "controller_task_id/stage_a/worker_b" for a file named "__success". If present, then it does nothing, else it will write it's task Id in that file. While reading the data for a particular stage, worker number and partition, the worker then does the following: 1. Checks the folder "controller_task_id/stage_a/worker_b" for a file named "__success" and attempts to read the task id of the task which successfully wrote to it. 2. If unable to read it or the file is not present, the worker throws an error. 3. Else it will fetch the task id present there and then read the data from the location: "controller_task_id/stage_a/worker_b/taskId_c/part_d" where the task id was fetched as above. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
