[jira] [Resolved] (SAMZA-1371) Some Samza Containers get stuck at "Starting BrokerProxy for hostname:portnum" while others seem to be fine
[ https://issues.apache.org/jira/browse/SAMZA-1371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik resolved SAMZA-1371. --- Resolution: Fixed > Some Samza Containers get stuck at "Starting BrokerProxy for > hostname:portnum" while others seem to be fine > --- > > Key: SAMZA-1371 > URL: https://issues.apache.org/jira/browse/SAMZA-1371 > Project: Samza > Issue Type: Bug > Components: container >Affects Versions: 0.11.0, 0.12.0 > Environment: Samza version: 0.11, 0.12 > Kafka version: 0.11.0.0 >Reporter: Ak Ka >Assignee: Boris Shkolnik >Priority: Blocker > Fix For: 1.0 > > Attachments: stdout.log, thread_dump.txt > > > We have multiple Samza apps using local store that have this issue. Some > containers get stuck on "Starting BrokerProxy for hostname:portnum" while > others seem to work as expected. > Here is the log: > stuck: > ``` > [...] > 2017-07-25 17:11:26.546 [main] org.apache.samza.system.kafka.BrokerProxy > [INFO] Creating new SimpleConsumer for host hostname:portnum for system kafka > 2017-07-25 17:11:26.547 [main] org.apache.samza.system.kafka.GetOffset [INFO] > Validating offset 0 for topic and partition > [prod.localStateChangeLog.prod.ListingAlertsEmailOrganizerInstant_matcherValidation,2] > 2017-07-25 17:11:26.648 [main] org.apache.samza.system.kafka.GetOffset [INFO] > Able to successfully read from offset 0 for topic and partition > [prod.localStateChangeLog.prod.ListingAlertsEmailOrganizerInstant_matcherValidation,2]. > Using it to instantiate consumer. > 2017-07-25 17:11:26.649 [main] org.apache.samza.system.kafka.BrokerProxy > [INFO] Starting BrokerProxy for hostname:portnum > // it's dead, Jim > ``` > healthy: > ``` > [...] > 2017-07-25 17:11:26.920 [main] org.apache.samza.system.kafka.BrokerProxy > [INFO] Creating new SimpleConsumer for host hostname:portnum for system kafka > 2017-07-25 17:11:26.921 [main] org.apache.samza.system.kafka.GetOffset [INFO] > Validating offset 0 for topic and partition > [prod.localStateChangeLog.prod.ListingAlertsEmailOrganizerInstant_matcherValidation,1] > 2017-07-25 17:11:27.023 [main] org.apache.samza.system.kafka.GetOffset [INFO] > Able to successfully read from offset 0 for topic and partition > [prod.localStateChangeLog.prod.ListingAlertsEmailOrganizerInstant_matcherValidation,1]. > Using it to instantiate consumer. > 2017-07-25 17:11:27.023 [main] org.apache.samza.system.kafka.BrokerProxy > [INFO] Starting BrokerProxy for hostname:portnum > 2017-07-25 17:11:29.194 [main] org.apache.samza.system.kafka.BrokerProxy > [INFO] Shutting down BrokerProxy for hostname:portnum > 2017-07-25 17:11:29.194 [main] org.apache.samza.system.kafka.BrokerProxy > [INFO] closing simple consumer... > 2017-07-25 17:11:29.239 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at > hostname:portnum for client > samza_consumer-prod_ListingAlertsEmailOrganizerInstant-1] > org.apache.samza.system.kafka.DefaultFetchSimpleConsumer [INFO] Reconnect due > to socket error: java.nio.channels.ClosedChannelException > 2017-07-25 17:11:29.244 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at > hostname:portnum for client > samza_consumer-prod_ListingAlertsEmailOrganizerInstant-1] > org.apache.samza.system.kafka.BrokerProxy [WARN] Restarting consumer due to > java.nio.channels.ClosedChannelException. Releasing ownership of all > partitions, and restarting consumer. Turn on debugging to get a full stack > trace. > 2017-07-25 17:11:29.247 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at > hostname:portnum for client > samza_consumer-prod_ListingAlertsEmailOrganizerInstant-1] > org.apache.samza.system.kafka.KafkaSystemConsumer [INFO] Abdicating for > [prod.localStateChangeLog.prod.ListingAlertsEmailOrganizerInstant_alertSetting,1] > 2017-07-25 17:11:29.247 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at > hostname:portnum for client > samza_consumer-prod_ListingAlertsEmailOrganizerInstant-1] > org.apache.samza.system.kafka.KafkaSystemConsumer [INFO] Refreshing brokers > for: > Map([prod.localStateChangeLog.prod.ListingAlertsEmailOrganizerInstant_alertSetting,1] > -> 13572) > 2017-07-25 17:11:29.247 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at > hostname:portnum for client > samza_consumer-prod_ListingAlertsEmailOrganizerInstant-1] > org.apache.samza.system.kafka.BrokerProxy [INFO] Shutting down due to > interrupt. > 2017-07-25 17:11:29.247 [main] org.apache.samza.system.kafka.BrokerProxy > [INFO] Shutting down BrokerProxy for hostname:portnum > 2017-07-25 17:11:29.248 [main] org.apache.samza.system.kafka.BrokerProxy > [INFO] closing simple consumer... > 2017-07-25 17:11:29.265 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at >
[jira] [Resolved] (SAMZA-1371) Some Samza Containers get stuck at "Starting BrokerProxy for hostname:portnum" while others seem to be fine
[ https://issues.apache.org/jira/browse/SAMZA-1371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Lin resolved SAMZA-1371. - Resolution: Won't Fix > Some Samza Containers get stuck at "Starting BrokerProxy for > hostname:portnum" while others seem to be fine > --- > > Key: SAMZA-1371 > URL: https://issues.apache.org/jira/browse/SAMZA-1371 > Project: Samza > Issue Type: Bug > Components: container >Affects Versions: 0.11.0, 0.12.0 > Environment: Samza version: 0.11, 0.12 > Kafka version: 0.11.0.0 >Reporter: Ak Ka >Assignee: Yi Pan (Data Infrastructure) >Priority: Blocker > Attachments: thread_dump.txt > > > We have multiple Samza apps using local store that have this issue. Some > containers get stuck on "Starting BrokerProxy for hostname:portnum" while > others seem to work as expected. > Here is the log: > stuck: > ``` > [...] > 2017-07-25 17:11:26.546 [main] org.apache.samza.system.kafka.BrokerProxy > [INFO] Creating new SimpleConsumer for host hostname:portnum for system kafka > 2017-07-25 17:11:26.547 [main] org.apache.samza.system.kafka.GetOffset [INFO] > Validating offset 0 for topic and partition > [prod.localStateChangeLog.prod.AlertsOrganizerInstant_matcherValidation,2] > 2017-07-25 17:11:26.648 [main] org.apache.samza.system.kafka.GetOffset [INFO] > Able to successfully read from offset 0 for topic and partition > [prod.localStateChangeLog.prod.AlertsOrganizerInstant_matcherValidation,2]. > Using it to instantiate consumer. > 2017-07-25 17:11:26.649 [main] org.apache.samza.system.kafka.BrokerProxy > [INFO] Starting BrokerProxy for hostname:portnum > // it's dead, Jim > ``` > healthy: > ``` > [...] > 2017-07-25 17:11:26.920 [main] org.apache.samza.system.kafka.BrokerProxy > [INFO] Creating new SimpleConsumer for host hostname:portnum for system kafka > 2017-07-25 17:11:26.921 [main] org.apache.samza.system.kafka.GetOffset [INFO] > Validating offset 0 for topic and partition > [prod.localStateChangeLog.prod.AlertsOrganizerInstant_matcherValidation,1] > 2017-07-25 17:11:27.023 [main] org.apache.samza.system.kafka.GetOffset [INFO] > Able to successfully read from offset 0 for topic and partition > [prod.localStateChangeLog.prod.AlertsOrganizerInstant_matcherValidation,1]. > Using it to instantiate consumer. > 2017-07-25 17:11:27.023 [main] org.apache.samza.system.kafka.BrokerProxy > [INFO] Starting BrokerProxy for hostname:portnum > 2017-07-25 17:11:29.194 [main] org.apache.samza.system.kafka.BrokerProxy > [INFO] Shutting down BrokerProxy for hostname:portnum > 2017-07-25 17:11:29.194 [main] org.apache.samza.system.kafka.BrokerProxy > [INFO] closing simple consumer... > 2017-07-25 17:11:29.239 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at > hostname:portnum for client samza_consumer-prod_AlertsOrganizerInstant-1] > org.apache.samza.system.kafka.DefaultFetchSimpleConsumer [INFO] Reconnect due > to socket error: java.nio.channels.ClosedChannelException > 2017-07-25 17:11:29.244 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at > hostname:portnum for client samza_consumer-prod_AlertsOrganizerInstant-1] > org.apache.samza.system.kafka.BrokerProxy [WARN] Restarting consumer due to > java.nio.channels.ClosedChannelException. Releasing ownership of all > partitions, and restarting consumer. Turn on debugging to get a full stack > trace. > 2017-07-25 17:11:29.247 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at > hostname:portnum for client samza_consumer-prod_AlertsOrganizerInstant-1] > org.apache.samza.system.kafka.KafkaSystemConsumer [INFO] Abdicating for > [prod.localStateChangeLog.prod.AlertsOrganizerInstant_alertSetting,1] > 2017-07-25 17:11:29.247 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at > hostname:portnum for client samza_consumer-prod_AlertsOrganizerInstant-1] > org.apache.samza.system.kafka.KafkaSystemConsumer [INFO] Refreshing brokers > for: > Map([prod.localStateChangeLog.prod.AlertsOrganizerInstant_alertSetting,1] -> > 13572) > 2017-07-25 17:11:29.247 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at > hostname:portnum for client samza_consumer-prod_AlertsOrganizerInstant-1] > org.apache.samza.system.kafka.BrokerProxy [INFO] Shutting down due to > interrupt. > 2017-07-25 17:11:29.247 [main] org.apache.samza.system.kafka.BrokerProxy > [INFO] Shutting down BrokerProxy for hostname:portnum > 2017-07-25 17:11:29.248 [main] org.apache.samza.system.kafka.BrokerProxy > [INFO] closing simple consumer... > 2017-07-25 17:11:29.265 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at > hostname:portnum for client samza_consumer-prod_AlertsOrganizerInstant-1] > org.apache.samza.system.kafka.BrokerProxy [INFO] Shutting down due to > interrupt. > 2017-07-25
