gianm commented on issue #15239: URL: https://github.com/apache/druid/issues/15239#issuecomment-1845659757
Hi @alonblum1— Renames were part of Hadoop ingestion in 0.17.2 as well, but what changed was in 0.22.0 with this patch: https://github.com/apache/druid/pull/11194. It moved the renames to the `index_hadoop` job, where they used to be in the reducers, to fix a race condition. Check this comment for the rationale: https://github.com/apache/druid/pull/11194#issuecomment-1845655509 I am surprised to hear that moving the renames to the `index_hadoop` job makes them take 5 hours longer! I'm wondering if you can provide a bit more detail? Like, how many renames are happening, and how many reducers do you have in your job? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
