Sanil15 opened a new pull request #1290: SAMZA-2468: Make Standby containers respond to shutdown request URL: https://github.com/apache/samza/pull/1290 **Issues:** Currently, standby containers don't spin up run loop and do an infinite wait after starting the container storage manager. In the event of a shutdown request from LocalApplicationRunner, these containers don't respond since the shutdown request is coordinator through runloop and it leaves us with the only option of forcefully killing it. **Changes:** Add a countdown latch to signal the main thread to make standby containers to respond to shutdown instead of infinite sleeping **Tests:** Manually tested the shutdown sequence Before (Unclean Shutdown): ``` 2020-02-25 01:23:55.389 [Samza Container Shutdown Hook Thread] ClusterBasedProcessorLifecycleListener [INFO] Attempting to shutdown container from inside shutdownHook, will wait up to 30000 ms. 2020-02-25 01:24:25.389 [Samza Container Shutdown Hook Thread] ClusterBasedProcessorLifecycleListener [ERROR] Did not shut down within 30000 ms, exiting. 2020-02-25 01:24:25.441 [Samza Container Shutdown Hook Thread] ThreadUtil [INFO] Thread dump from Samza Container Shutdown Hook. "Samza Container Shutdown Hook Thread" Id=163 RUNNABLE at sun.management.ThreadImpl.dumpThreads0(Native Method) at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:454) at org.apache.samza.util.ThreadUtil.logThreadDump(ThreadUtil.java:35) at org.apache.samza.runtime.ClusterBasedProcessorLifecycleListener$1.run(ClusterBasedProcessorLifecycleListener.java:100) ``` After (Clean Shutdown): ``` 2020-02-25 03:21:02.225 [Samza Container Shutdown Hook Thread] ClusterBasedProcessorLifecycleListener [INFO] Attempting to shutdown container from inside shutdownHook, will wait up to 30000 ms. 2020-02-25 03:21:02.227 [main] SamzaContainer [INFO] Shutting down SamzaContainer. 2020-02-25 03:21:02.227 [main] SamzaContainer [INFO] Shutting down consumer multiplexer. 2020-02-25 03:21:02.228 [main] SamzaContainer [INFO] Shutting down task instance stream tasks. 2020-02-25 03:21:02.229 [main] SamzaContainer [INFO] Shutting down timer executor 2020-02-25 03:21:02.230 [main] SamzaContainer [INFO] Shutting down task instance table manager. 2020-02-25 03:21:02.231 [main] SamzaContainer [INFO] Shutting down container storage manager. 2020-02-25 03:21:02.236 [main] ContainerStorageManager [INFO] Shutdown complete 2020-02-25 03:21:02.236 [main] SamzaContainer [INFO] Shutting down disk space monitor. 2020-02-25 03:21:02.237 [main] SamzaContainer [INFO] Shutting down host statistics monitor. 2020-02-25 03:21:02.238 [main] SamzaContainer [INFO] Shutting down producer multiplexer. 2020-02-25 03:21:02.239 [main] KafkaSystemProducer [INFO] Stopping producer for system: logging 2020-02-25 03:21:02.240 [main] KafkaProducerImpl [INFO] Shutting down KafkaProducer in 2147483647 MILLISECONDS... 2020-02-25 03:21:02.240 [main] AbstractAuditor [INFO] Closing auditor with timeout 2147483647 MILLISECONDS 2020-02-25 03:21:02.240 [main] KafkaProducer [INFO] [Producer clientId=kafka_producer-snjain_test_cp-i001] Closing the Kafka producer with timeoutMillis = 2147483647 ms. 2020-02-25 03:21:02.241 [main] KafkaProducerImpl [INFO] KafkaProducer shutdown complete in 1 millis 2020-02-25 03:21:02.247 [main] KafkaSystemProducer [INFO] Stopping producer for system: diagnosticssystem 2020-02-25 03:21:02.247 [main] KafkaProducerImpl [INFO] Shutting down KafkaProducer in 2147483647 MILLISECONDS... 2020-02-25 03:21:02.247 [main] AbstractAuditor [INFO] Closing auditor with timeout 2147483647 MILLISECONDS 2020-02-25 03:21:02.248 [main] KafkaProducer [INFO] [Producer clientId=kafka_producer-snjain_test_cp-i001] Closing the Kafka producer with timeoutMillis = 2147483646 ms. 2020-02-25 03:21:02.250 [main] KafkaProducerImpl [INFO] KafkaProducer shutdown complete in 3 millis 2020-02-25 03:21:02.251 [main] KafkaSystemProducer [INFO] Stopping producer for system: aggregate-tracking 2020-02-25 03:21:02.251 [main] KafkaProducerImpl [INFO] Shutting down KafkaProducer in 2147483647 MILLISECONDS... 2020-02-25 03:21:02.251 [main] AbstractAuditor [INFO] Closing auditor with timeout 2147483647 MILLISECONDS 2020-02-25 03:21:02.251 [main] KafkaProducer [INFO] [Producer clientId=kafka_producer-snjain_test_cp-i001] Closing the Kafka producer with timeoutMillis = 2147483647 ms. 2020-02-25 03:21:02.252 [main] KafkaProducerImpl [INFO] KafkaProducer shutdown complete in 1 millis 2020-02-25 03:21:02.253 [main] KafkaSystemProducer [INFO] Stopping producer for system: samzainternalsystem 2020-02-25 03:21:02.253 [main] KafkaProducerImpl [INFO] Shutting down KafkaProducer in 2147483647 MILLISECONDS... 2020-02-25 03:21:02.253 [main] AbstractAuditor [INFO] Closing auditor with timeout 2147483647 MILLISECONDS 2020-02-25 03:21:02.253 [main] KafkaProducer [INFO] [Producer clientId=kafka_producer-snjain_test_cp-i001] Closing the Kafka producer with timeoutMillis = 2147483647 ms. 2020-02-25 03:21:02.254 [main] KafkaProducerImpl [INFO] KafkaProducer shutdown complete in 1 millis 2020-02-25 03:21:02.256 [main] KafkaSystemProducer [INFO] Stopping producer for system: samzametadatasystem 2020-02-25 03:21:02.256 [main] KafkaProducerImpl [INFO] Shutting down KafkaProducer in 2147483647 MILLISECONDS... 2020-02-25 03:21:02.256 [main] AbstractAuditor [INFO] Closing auditor with timeout 2147483647 MILLISECONDS 2020-02-25 03:21:02.256 [main] KafkaProducer [INFO] [Producer clientId=kafka_producer-snjain_test_cp-i001] Closing the Kafka producer with timeoutMillis = 2147483647 ms. 2020-02-25 03:21:02.258 [main] KafkaProducerImpl [INFO] KafkaProducer shutdown complete in 2 millis 2020-02-25 03:21:02.259 [main] SamzaContainer [INFO] Shutting down offset manager. 2020-02-25 03:21:02.261 [main] KafkaCheckpointManager [INFO] Stopping system admin. 2020-02-25 03:21:02.261 [main] KafkaConsumerImpl [INFO] Shutting down ... 2020-02-25 03:21:02.264 [main] AbstractAuditor [INFO] Closing auditor with timeout 9223372036854775807 MILLISECONDS 2020-02-25 03:21:02.264 [kafka_admin_consumer-snjain_test_cp-i001-auditor] KafkaProducer [INFO] [Producer clientId=kafka_admin_consumer-snjain_test_cp-i001-auditor] Closing the Kafka producer with timeoutMillis = 9223370454253913543 ms. 2020-02-25 03:21:02.265 [main] KafkaConsumerImpl [INFO] Shutdown complete in 4 millis 2020-02-25 03:21:02.268 [main] KafkaCheckpointManager [INFO] Stopping system producer. 2020-02-25 03:21:02.268 [main] KafkaSystemProducer [INFO] Stopping producer for system: samzametadatasystem 2020-02-25 03:21:02.268 [main] KafkaProducerImpl [INFO] Shutting down KafkaProducer in 2147483647 MILLISECONDS... 2020-02-25 03:21:02.268 [main] AbstractAuditor [INFO] Closing auditor with timeout 2147483647 MILLISECONDS 2020-02-25 03:21:02.268 [main] KafkaProducer [INFO] [Producer clientId=kafka_producer-snjain_test_cp-i001] Closing the Kafka producer with timeoutMillis = 2147483647 ms. 2020-02-25 03:21:02.269 [main] KafkaProducerImpl [INFO] KafkaProducer shutdown complete in 1 millis 2020-02-25 03:21:02.271 [main] KafkaCheckpointManager [INFO] CheckpointManager stopped. 2020-02-25 03:21:02.271 [main] StartpointManager [WARN] already stopped 2020-02-25 03:21:02.272 [main] SamzaContainer [INFO] Shutting down diagnostics manager. 2020-02-25 03:21:02.273 [main] SamzaContainer [INFO] Shutting down metrics reporters. 2020-02-25 03:21:02.273 [main] AmfReporter [INFO] Stopping reporter timer. 2020-02-25 03:21:02.274 [main] MetricsSnapshotReporter [INFO] Stopping reporter timer. 2020-02-25 03:21:02.278 [main] MetricsSnapshotReporter [INFO] Stopping producer. 2020-02-25 03:21:02.278 [main] KafkaSystemProducer [INFO] Stopping producer for system: diagnosticssystem 2020-02-25 03:21:02.278 [main] KafkaProducerImpl [INFO] Shutting down KafkaProducer in 2147483647 MILLISECONDS... 2020-02-25 03:21:02.278 [main] AbstractAuditor [INFO] Closing auditor with timeout 2147483647 MILLISECONDS 2020-02-25 03:21:02.279 [main] KafkaProducer [INFO] [Producer clientId=kafka_producer-snjain_test_cp-i001] Closing the Kafka producer with timeoutMillis = 2147483646 ms. 2020-02-25 03:21:02.302 [main] KafkaProducerImpl [INFO] KafkaProducer shutdown complete in 24 millis 2020-02-25 03:21:02.304 [main] SamzaContainer [INFO] Shutting down JVM metrics. 2020-02-25 03:21:02.305 [main] SamzaContainer [INFO] Shutting down admin multiplexer. 2020-02-25 03:21:02.305 [main] KafkaConsumerImpl [INFO] Shutting down ... 2020-02-25 03:21:02.305 [main] AbstractAuditor [INFO] Closing auditor with timeout 9223372036854775807 MILLISECONDS 2020-02-25 03:21:02.305 [kafka_admin_consumer-snjain_test_cp-i001-auditor] KafkaProducer [INFO] [Producer clientId=kafka_admin_consumer-snjain_test_cp-i001-auditor] Closing the Kafka producer with timeoutMillis = 9223370454253913502 ms. 2020-02-25 03:21:02.307 [main] KafkaConsumerImpl [INFO] Shutdown complete in 2 millis 2020-02-25 03:21:02.308 [main] KafkaConsumerImpl [INFO] Shutting down ... 2020-02-25 03:21:02.309 [main] AbstractAuditor [INFO] Closing auditor with timeout 9223372036854775807 MILLISECONDS 2020-02-25 03:21:02.309 [kafka_admin_consumer-snjain_test_cp-i001-auditor] KafkaProducer [INFO] [Producer clientId=kafka_admin_consumer-snjain_test_cp-i001-auditor] Closing the Kafka producer with timeoutMillis = 9223370454253913498 ms. 2020-02-25 03:21:02.310 [main] KafkaConsumerImpl [INFO] Shutdown complete in 2 millis 2020-02-25 03:21:02.312 [main] KafkaConsumerImpl [INFO] Shutting down ... 2020-02-25 03:21:02.312 [main] AbstractAuditor [INFO] Closing auditor with timeout 9223372036854775807 MILLISECONDS 2020-02-25 03:21:02.312 [kafka_admin_consumer-snjain_test_cp-i001-auditor] KafkaProducer [INFO] [Producer clientId=kafka_admin_consumer-snjain_test_cp-i001-auditor] Closing the Kafka producer with timeoutMillis = 9223370454253913495 ms. 2020-02-25 03:21:02.313 [main] KafkaConsumerImpl [INFO] Shutdown complete in 1 millis 2020-02-25 03:21:02.315 [main] KafkaConsumerImpl [INFO] Shutting down ... 2020-02-25 03:21:02.317 [main] AbstractAuditor [INFO] Closing auditor with timeout 9223372036854775807 MILLISECONDS 2020-02-25 03:21:02.317 [kafka_admin_consumer-snjain_test_cp-i001-auditor] KafkaProducer [INFO] [Producer clientId=kafka_admin_consumer-snjain_test_cp-i001-auditor] Closing the Kafka producer with timeoutMillis = 9223370454253913490 ms. 2020-02-25 03:21:02.318 [main] KafkaConsumerImpl [INFO] Shutdown complete in 3 millis 2020-02-25 03:21:02.320 [main] KafkaConsumerImpl [INFO] Shutting down ... 2020-02-25 03:21:02.320 [main] AbstractAuditor [INFO] Closing auditor with timeout 9223372036854775807 MILLISECONDS 2020-02-25 03:21:02.320 [kafka_admin_consumer-snjain_test_cp-i001-auditor] KafkaProducer [INFO] [Producer clientId=kafka_admin_consumer-snjain_test_cp-i001-auditor] Closing the Kafka producer with timeoutMillis = 9223370454253913487 ms. 2020-02-25 03:21:02.321 [main] KafkaConsumerImpl [INFO] Shutdown complete in 1 millis 2020-02-25 03:21:02.324 [main] SamzaContainer [INFO] Shutdown complete. 2020-02-25 03:21:02.324 [main] ClusterBasedProcessorLifecycleListener [INFO] Container Stopped 2020-02-25 03:21:02.324 [main] ContainerHeartbeatMonitor [INFO] Stopping ContainerHeartbeatMonitor 2020-02-25 03:21:02.324 [Samza Container Shutdown Hook Thread] ClusterBasedProcessorLifecycleListener [INFO] Shutdown complete 2020-02-25 03:21:02.324 [main] CoordinatorStreamStore [INFO] Stopping the coordinator stream system consumer. 2020-02-25 03:21:02.324 [main] KafkaConsumerImpl [INFO] Shutting down ... ``` **API Changes:** None **Upgrade instructions:** None **Usage instructions:** None
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
