ankit0811 opened a new pull request #6828: Kill Hadoop MR task on kill of ingestion task and resume ability for Hadoop ingestion tasks URL: https://github.com/apache/incubator-druid/pull/6828 KillTask from overlord UI now makes sure that it terminates the underlying MR job, thus saving unnecessary compute #6803 Run in `jobby` is now split into 2 1. submitAndGetHadoopJobId followed by 2. run submitAndGetHadoopJobId is responsible for submitting the job and returning the jobId as a string, run monitors this job for completion `JobHelper` writes this jobId in the path provided by HadoopIndexTask which in turn is provided by the ForkingTaskRunner `HadoopIndexTask` reads this path when kill task is clicked to get the jobId and fire the kill command via the yarn api. This is taken care in the stopGracefully method which is called in SingleTaskBackgroundRunner. Have enabled `canRestore` method to return `true` for HadoopIndexTask in order for the stopGracefully method to be called `Hadoop*Job` classes have been changed to incorporate the changes to `jobby`
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
