himanshug commented on issue #8061: Native parallel batch indexing with shuffle URL: https://github.com/apache/incubator-druid/issues/8061#issuecomment-510763274 thanks for the explanations. > I think it's not very complex, but will reduce the number of calls to overlord API, so it would be good. that is fine and I am guessing it will batch task status call i.e. ask status of multiple supervisor tasks in one overlord api request. > If supervisorTask killed abnormally, then parallel index task doesn't handle this case for now. will the worker tasks complete and exit eventually or they will be left running till manual intervention ? we might consider making supervisor real "supervisor" like Kakfa instead of "task" so that they get special powers to manage things better ? but I think I am remembering some comment that they are made tasks because tasks are better equipped to work with locking framework available. one option could be to let spawned worker task do the locking , since all worker tasks would be in same group so multiple worker tasks trying to obtain same lock would still work..... this is unverified wishful thinking :)
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
