jihoonson opened a new issue #8718: Fix graceful shutdown for tasks URL: https://github.com/apache/incubator-druid/issues/8718 ### Affected Version 0.14, 0.15, 0.16 ### Description Since https://github.com/apache/incubator-druid/pull/6828, [task shutdown is always the graceful shutdown](https://github.com/apache/incubator-druid/blob/master/indexing-service/src/main/java/org/apache/druid/indexing/overlord/SingleTaskBackgroundRunner.java#L184-L186). This was for some task types to clean up their resources. For example, Hadoop task kills its Hadoop job when it's stopped (https://github.com/apache/incubator-druid/pull/6828). The parallel indexing task kills its all running sub tasks (https://github.com/apache/incubator-druid/pull/7041). However, this is different from how the stream ingestion task had been using graceful shutdown. On graceful shutdown, they immediately start persisting all segments in memory onto disk and publishing segments before they stop. This is to create a checkpoint when they stop, not for resource cleanup. As a result, https://github.com/apache/incubator-druid/pull/6828 unexpectedly changed the behavior of Kafka/Kinesis indexing service when the supervisor kills tasks. I think we need to distinguish immediate stop and graceful stop again. Immediate stop always involves resource cleanup. Graceful stop may involve extra work in addition to necessary resource cleanup.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
