Re: Streamtasks not instantiating consumers on startup
Thanks Yi. We recovered this time around by processing all of the checkpoints manually, deleting and recreating the checkpoint topic, then re-populating the checkpoint topic with the saved offsets using the checkpoint tool,. If it happens again, I'll turn up logging to try and get more information. Fwiw, we're using Kafka 0.11.0.2 with Samza 0.14.1; my understanding is that there should be version compatibility between Kafka 0.11.0.x-2.x. If you have any other ideas, I'd be interested in hearing them. Cheers, Malcolm McFarland Cavulus On Fri, Apr 10, 2020 at 8:36 PM Yi Pan wrote: > Hi, Malcolm, > > Samza 0.14.1 is pretty old and if you have already upgraded Kafka to 2.2.1, > I would highly recommend you to migrate to latest Samza version, which has > Kafka 2.0 client. Depending on how you configure your broker, a 2.0 Kafka > broker can be incompatible with older client like 0.8.6, since there have > been wire-protocol changes in Kafka since 0.11. > > P.S. a more detailed log + configuration would be required to debug your > issue (probably, turn on debug log for Kafka consumer lib), if you still > choose to stay with Samza 0.14.1. > > Best, > > -Yi > > On Thu, Apr 9, 2020 at 12:50 PM Malcolm McFarland > wrote: > > > Hey folks, > > > > We're occasionally seeing an issue when starting Samza containers where > > none of the streamtasks for a job will instantiate consumers. We'll see > the > > beginnings of an attempt to read the checkpoint stream and nothing > further > > (including no errors). All of these streamtasks seem to be creating > > producers -- there are plenty of "Registering > > TaskName-SystemStreamPartition [kafka, , ] with > > producer." messages. We're not seeing any of the usual messages about > > instantiating consumers; all we're seeing is this: > > > > 2020-04-09T19:24:14.968Z Validating offset 0 for topic and partition > > [__samza_checkpoint_ver_1_for__1,0] > > 2020-04-09T19:24:14.969Z Able to successfully read from offset 0 for > topic > > and partition [__samza_checkpoint_ver_1_for__1,0]. Using it to > > instantiate consumer. > > 2020-04-09T19:24:14.974Z Reading checkpoint for taskName > > SystemStreamPartition [kafka, , ] > > > > ...and that's as far as it will go. I am also noticing that when using > > different versions of librdkafka, I *can* read the checkpoint stream with > > librdkafka 1.3.0, but *not* with librdkafka 0.8.6. Could there be a > version > > incompatibility with how the data is being stored on the kafka server? > > We're running Samza 0.14.1 and using AWS MSK which is running version > Kafka > > 2.2.1. > > > > > > Thanks so much, > > Malcolm McFarland > > Cavulus > > >
Re: Streamtasks not instantiating consumers on startup
Hi, Malcolm, Samza 0.14.1 is pretty old and if you have already upgraded Kafka to 2.2.1, I would highly recommend you to migrate to latest Samza version, which has Kafka 2.0 client. Depending on how you configure your broker, a 2.0 Kafka broker can be incompatible with older client like 0.8.6, since there have been wire-protocol changes in Kafka since 0.11. P.S. a more detailed log + configuration would be required to debug your issue (probably, turn on debug log for Kafka consumer lib), if you still choose to stay with Samza 0.14.1. Best, -Yi On Thu, Apr 9, 2020 at 12:50 PM Malcolm McFarland wrote: > Hey folks, > > We're occasionally seeing an issue when starting Samza containers where > none of the streamtasks for a job will instantiate consumers. We'll see the > beginnings of an attempt to read the checkpoint stream and nothing further > (including no errors). All of these streamtasks seem to be creating > producers -- there are plenty of "Registering > TaskName-SystemStreamPartition [kafka, , ] with > producer." messages. We're not seeing any of the usual messages about > instantiating consumers; all we're seeing is this: > > 2020-04-09T19:24:14.968Z Validating offset 0 for topic and partition > [__samza_checkpoint_ver_1_for__1,0] > 2020-04-09T19:24:14.969Z Able to successfully read from offset 0 for topic > and partition [__samza_checkpoint_ver_1_for__1,0]. Using it to > instantiate consumer. > 2020-04-09T19:24:14.974Z Reading checkpoint for taskName > SystemStreamPartition [kafka, , ] > > ...and that's as far as it will go. I am also noticing that when using > different versions of librdkafka, I *can* read the checkpoint stream with > librdkafka 1.3.0, but *not* with librdkafka 0.8.6. Could there be a version > incompatibility with how the data is being stored on the kafka server? > We're running Samza 0.14.1 and using AWS MSK which is running version Kafka > 2.2.1. > > > Thanks so much, > Malcolm McFarland > Cavulus >
Streamtasks not instantiating consumers on startup
Hey folks, We're occasionally seeing an issue when starting Samza containers where none of the streamtasks for a job will instantiate consumers. We'll see the beginnings of an attempt to read the checkpoint stream and nothing further (including no errors). All of these streamtasks seem to be creating producers -- there are plenty of "Registering TaskName-SystemStreamPartition [kafka, , ] with producer." messages. We're not seeing any of the usual messages about instantiating consumers; all we're seeing is this: 2020-04-09T19:24:14.968Z Validating offset 0 for topic and partition [__samza_checkpoint_ver_1_for__1,0] 2020-04-09T19:24:14.969Z Able to successfully read from offset 0 for topic and partition [__samza_checkpoint_ver_1_for__1,0]. Using it to instantiate consumer. 2020-04-09T19:24:14.974Z Reading checkpoint for taskName SystemStreamPartition [kafka, , ] ...and that's as far as it will go. I am also noticing that when using different versions of librdkafka, I *can* read the checkpoint stream with librdkafka 1.3.0, but *not* with librdkafka 0.8.6. Could there be a version incompatibility with how the data is being stored on the kafka server? We're running Samza 0.14.1 and using AWS MSK which is running version Kafka 2.2.1. Thanks so much, Malcolm McFarland Cavulus
