Sanil15 commented on issue #1290: SAMZA-2468: Make Standby containers respond to shutdown request URL: https://github.com/apache/samza/pull/1290#issuecomment-591002919 Testing Results: Before (Unclean Shutdown): ``` 2020-02-25 01:23:55.389 [Samza Container Shutdown Hook Thread] ClusterBasedProcessorLifecycleListener [INFO] Attempting to shutdown container from inside shutdownHook, will wait up to 30000 ms. 2020-02-25 01:24:25.389 [Samza Container Shutdown Hook Thread] ClusterBasedProcessorLifecycleListener [ERROR] Did not shut down within 30000 ms, exiting. 2020-02-25 01:24:25.441 [Samza Container Shutdown Hook Thread] ThreadUtil [INFO] Thread dump from Samza Container Shutdown Hook. "Samza Container Shutdown Hook Thread" Id=163 RUNNABLE at sun.management.ThreadImpl.dumpThreads0(Native Method) at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:454) at org.apache.samza.util.ThreadUtil.logThreadDump(ThreadUtil.java:35) at org.apache.samza.runtime.ClusterBasedProcessorLifecycleListener$1.run(ClusterBasedProcessorLifecycleListener.java:100) ``` After (Clean Shutdown): ``` 2020-02-25 03:21:02.225 [Samza Container Shutdown Hook Thread] ClusterBasedProcessorLifecycleListener [INFO] Attempting to shutdown container from inside shutdownHook, will wait up to 30000 ms. 2020-02-25 03:21:02.227 [main] SamzaContainer [INFO] Shutting down SamzaContainer. 2020-02-25 03:21:02.227 [main] SamzaContainer [INFO] Shutting down consumer multiplexer. 2020-02-25 03:21:02.228 [main] SamzaContainer [INFO] Shutting down task instance stream tasks. 2020-02-25 03:21:02.229 [main] SamzaContainer [INFO] Shutting down timer executor 2020-02-25 03:21:02.230 [main] SamzaContainer [INFO] Shutting down task instance table manager. 2020-02-25 03:21:02.231 [main] SamzaContainer [INFO] Shutting down container storage manager. 2020-02-25 03:21:02.236 [main] ContainerStorageManager [INFO] Shutdown complete 2020-02-25 03:21:02.236 [main] SamzaContainer [INFO] Shutting down disk space monitor. 2020-02-25 03:21:02.237 [main] SamzaContainer [INFO] Shutting down host statistics monitor. 2020-02-25 03:21:02.238 [main] SamzaContainer [INFO] Shutting down producer multiplexer. 2020-02-25 03:21:02.239 [main] KafkaSystemProducer [INFO] Stopping producer for system: logging 2020-02-25 03:21:02.240 [main] KafkaProducerImpl [INFO] Shutting down KafkaProducer in 2147483647 MILLISECONDS... 2020-02-25 03:21:02.240 [main] AbstractAuditor [INFO] Closing auditor with timeout 2147483647 MILLISECONDS 2020-02-25 03:21:02.240 [main] KafkaProducer [INFO] [Producer clientId=kafka_producer-snjain_test_cp-i001] Closing the Kafka producer with timeoutMillis = 2147483647 ms. 2020-02-25 03:21:02.241 [main] KafkaProducerImpl [INFO] KafkaProducer shutdown complete in 1 millis 2020-02-25 03:21:02.247 [main] KafkaSystemProducer [INFO] Stopping producer for system: diagnosticssystem 2020-02-25 03:21:02.247 [main] KafkaProducerImpl [INFO] Shutting down KafkaProducer in 2147483647 MILLISECONDS... 2020-02-25 03:21:02.247 [main] AbstractAuditor [INFO] Closing auditor with timeout 2147483647 MILLISECONDS 2020-02-25 03:21:02.248 [main] KafkaProducer [INFO] [Producer clientId=kafka_producer-snjain_test_cp-i001] Closing the Kafka producer with timeoutMillis = 2147483646 ms. 2020-02-25 03:21:02.250 [main] KafkaProducerImpl [INFO] KafkaProducer shutdown complete in 3 millis 2020-02-25 03:21:02.251 [main] KafkaSystemProducer [INFO] Stopping producer for system: aggregate-tracking 2020-02-25 03:21:02.251 [main] KafkaProducerImpl [INFO] Shutting down KafkaProducer in 2147483647 MILLISECONDS... 2020-02-25 03:21:02.251 [main] AbstractAuditor [INFO] Closing auditor with timeout 2147483647 MILLISECONDS 2020-02-25 03:21:02.251 [main] KafkaProducer [INFO] [Producer clientId=kafka_producer-snjain_test_cp-i001] Closing the Kafka producer with timeoutMillis = 2147483647 ms. 2020-02-25 03:21:02.252 [main] KafkaProducerImpl [INFO] KafkaProducer shutdown complete in 1 millis 2020-02-25 03:21:02.253 [main] KafkaSystemProducer [INFO] Stopping producer for system: samzainternalsystem 2020-02-25 03:21:02.253 [main] KafkaProducerImpl [INFO] Shutting down KafkaProducer in 2147483647 MILLISECONDS... 2020-02-25 03:21:02.253 [main] AbstractAuditor [INFO] Closing auditor with timeout 2147483647 MILLISECONDS 2020-02-25 03:21:02.253 [main] KafkaProducer [INFO] [Producer clientId=kafka_producer-snjain_test_cp-i001] Closing the Kafka producer with timeoutMillis = 2147483647 ms. 2020-02-25 03:21:02.254 [main] KafkaProducerImpl [INFO] KafkaProducer shutdown complete in 1 millis 2020-02-25 03:21:02.256 [main] KafkaSystemProducer [INFO] Stopping producer for system: samzametadatasystem 2020-02-25 03:21:02.256 [main] KafkaProducerImpl [INFO] Shutting down KafkaProducer in 2147483647 MILLISECONDS... 2020-02-25 03:21:02.256 [main] AbstractAuditor [INFO] Closing auditor with timeout 2147483647 MILLISECONDS 2020-02-25 03:21:02.256 [main] KafkaProducer [INFO] [Producer clientId=kafka_producer-snjain_test_cp-i001] Closing the Kafka producer with timeoutMillis = 2147483647 ms. 2020-02-25 03:21:02.258 [main] KafkaProducerImpl [INFO] KafkaProducer shutdown complete in 2 millis 2020-02-25 03:21:02.259 [main] SamzaContainer [INFO] Shutting down offset manager. 2020-02-25 03:21:02.261 [main] KafkaCheckpointManager [INFO] Stopping system admin. 2020-02-25 03:21:02.261 [main] KafkaConsumerImpl [INFO] Shutting down ... 2020-02-25 03:21:02.264 [main] AbstractAuditor [INFO] Closing auditor with timeout 9223372036854775807 MILLISECONDS 2020-02-25 03:21:02.264 [kafka_admin_consumer-snjain_test_cp-i001-auditor] KafkaProducer [INFO] [Producer clientId=kafka_admin_consumer-snjain_test_cp-i001-auditor] Closing the Kafka producer with timeoutMillis = 9223370454253913543 ms. 2020-02-25 03:21:02.265 [main] KafkaConsumerImpl [INFO] Shutdown complete in 4 millis 2020-02-25 03:21:02.268 [main] KafkaCheckpointManager [INFO] Stopping system producer. 2020-02-25 03:21:02.268 [main] KafkaSystemProducer [INFO] Stopping producer for system: samzametadatasystem 2020-02-25 03:21:02.268 [main] KafkaProducerImpl [INFO] Shutting down KafkaProducer in 2147483647 MILLISECONDS... 2020-02-25 03:21:02.268 [main] AbstractAuditor [INFO] Closing auditor with timeout 2147483647 MILLISECONDS 2020-02-25 03:21:02.268 [main] KafkaProducer [INFO] [Producer clientId=kafka_producer-snjain_test_cp-i001] Closing the Kafka producer with timeoutMillis = 2147483647 ms. 2020-02-25 03:21:02.269 [main] KafkaProducerImpl [INFO] KafkaProducer shutdown complete in 1 millis 2020-02-25 03:21:02.271 [main] KafkaCheckpointManager [INFO] CheckpointManager stopped. 2020-02-25 03:21:02.271 [main] StartpointManager [WARN] already stopped 2020-02-25 03:21:02.272 [main] SamzaContainer [INFO] Shutting down diagnostics manager. 2020-02-25 03:21:02.273 [main] SamzaContainer [INFO] Shutting down metrics reporters. 2020-02-25 03:21:02.273 [main] AmfReporter [INFO] Stopping reporter timer. 2020-02-25 03:21:02.274 [main] MetricsSnapshotReporter [INFO] Stopping reporter timer. 2020-02-25 03:21:02.278 [main] MetricsSnapshotReporter [INFO] Stopping producer. 2020-02-25 03:21:02.278 [main] KafkaSystemProducer [INFO] Stopping producer for system: diagnosticssystem 2020-02-25 03:21:02.278 [main] KafkaProducerImpl [INFO] Shutting down KafkaProducer in 2147483647 MILLISECONDS... 2020-02-25 03:21:02.278 [main] AbstractAuditor [INFO] Closing auditor with timeout 2147483647 MILLISECONDS 2020-02-25 03:21:02.279 [main] KafkaProducer [INFO] [Producer clientId=kafka_producer-snjain_test_cp-i001] Closing the Kafka producer with timeoutMillis = 2147483646 ms. 2020-02-25 03:21:02.302 [main] KafkaProducerImpl [INFO] KafkaProducer shutdown complete in 24 millis 2020-02-25 03:21:02.304 [main] SamzaContainer [INFO] Shutting down JVM metrics. 2020-02-25 03:21:02.305 [main] SamzaContainer [INFO] Shutting down admin multiplexer. 2020-02-25 03:21:02.305 [main] KafkaConsumerImpl [INFO] Shutting down ... 2020-02-25 03:21:02.305 [main] AbstractAuditor [INFO] Closing auditor with timeout 9223372036854775807 MILLISECONDS 2020-02-25 03:21:02.305 [kafka_admin_consumer-snjain_test_cp-i001-auditor] KafkaProducer [INFO] [Producer clientId=kafka_admin_consumer-snjain_test_cp-i001-auditor] Closing the Kafka producer with timeoutMillis = 9223370454253913502 ms. 2020-02-25 03:21:02.307 [main] KafkaConsumerImpl [INFO] Shutdown complete in 2 millis 2020-02-25 03:21:02.308 [main] KafkaConsumerImpl [INFO] Shutting down ... 2020-02-25 03:21:02.309 [main] AbstractAuditor [INFO] Closing auditor with timeout 9223372036854775807 MILLISECONDS 2020-02-25 03:21:02.309 [kafka_admin_consumer-snjain_test_cp-i001-auditor] KafkaProducer [INFO] [Producer clientId=kafka_admin_consumer-snjain_test_cp-i001-auditor] Closing the Kafka producer with timeoutMillis = 9223370454253913498 ms. 2020-02-25 03:21:02.310 [main] KafkaConsumerImpl [INFO] Shutdown complete in 2 millis 2020-02-25 03:21:02.312 [main] KafkaConsumerImpl [INFO] Shutting down ... 2020-02-25 03:21:02.312 [main] AbstractAuditor [INFO] Closing auditor with timeout 9223372036854775807 MILLISECONDS 2020-02-25 03:21:02.312 [kafka_admin_consumer-snjain_test_cp-i001-auditor] KafkaProducer [INFO] [Producer clientId=kafka_admin_consumer-snjain_test_cp-i001-auditor] Closing the Kafka producer with timeoutMillis = 9223370454253913495 ms. 2020-02-25 03:21:02.313 [main] KafkaConsumerImpl [INFO] Shutdown complete in 1 millis 2020-02-25 03:21:02.315 [main] KafkaConsumerImpl [INFO] Shutting down ... 2020-02-25 03:21:02.317 [main] AbstractAuditor [INFO] Closing auditor with timeout 9223372036854775807 MILLISECONDS 2020-02-25 03:21:02.317 [kafka_admin_consumer-snjain_test_cp-i001-auditor] KafkaProducer [INFO] [Producer clientId=kafka_admin_consumer-snjain_test_cp-i001-auditor] Closing the Kafka producer with timeoutMillis = 9223370454253913490 ms. 2020-02-25 03:21:02.318 [main] KafkaConsumerImpl [INFO] Shutdown complete in 3 millis 2020-02-25 03:21:02.320 [main] KafkaConsumerImpl [INFO] Shutting down ... 2020-02-25 03:21:02.320 [main] AbstractAuditor [INFO] Closing auditor with timeout 9223372036854775807 MILLISECONDS 2020-02-25 03:21:02.320 [kafka_admin_consumer-snjain_test_cp-i001-auditor] KafkaProducer [INFO] [Producer clientId=kafka_admin_consumer-snjain_test_cp-i001-auditor] Closing the Kafka producer with timeoutMillis = 9223370454253913487 ms. 2020-02-25 03:21:02.321 [main] KafkaConsumerImpl [INFO] Shutdown complete in 1 millis 2020-02-25 03:21:02.324 [main] SamzaContainer [INFO] Shutdown complete. 2020-02-25 03:21:02.324 [main] ClusterBasedProcessorLifecycleListener [INFO] Container Stopped 2020-02-25 03:21:02.324 [main] ContainerHeartbeatMonitor [INFO] Stopping ContainerHeartbeatMonitor 2020-02-25 03:21:02.324 [Samza Container Shutdown Hook Thread] ClusterBasedProcessorLifecycleListener [INFO] Shutdown complete 2020-02-25 03:21:02.324 [main] CoordinatorStreamStore [INFO] Stopping the coordinator stream system consumer. 2020-02-25 03:21:02.324 [main] KafkaConsumerImpl [INFO] Shutting down ... ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
