Pankaj260100 commented on PR #15141: URL: https://github.com/apache/druid/pull/15141#issuecomment-1815854719
@abhishekagarwal87, I have 30 ingestion tasks(set 1) running in the Druid cluster. At the time of task rollover, the existing 30 tasks(set 1) start publishing and the new 30(set 2) start ingesting. I submitted the supervisor config and the new 30(set 2) also start publishing. Now, there are a couple of tasks in set 2 which face this issue where the start metadata doesn't match with Metadatastored in DB(because all tasks from set 1 haven't finished updating the metadata store). Now these tasks start retrying but they were not retrying in parallel, I looked into the logs and the retry happened in sequence. Like task A from set 2 completes all 10 retries and then task B from set 2 completes all 10 retries and then other tasks which are facing this issue. During this, the overlord is not doing anything apart from retrying and after all tasks from set 2 are finished retrying the tasks from set 1 updates the metadata store. But all this should happen in parallel, right? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
