Hi, Malcolm, Samza 0.14.1 is pretty old and if you have already upgraded Kafka to 2.2.1, I would highly recommend you to migrate to latest Samza version, which has Kafka 2.0 client. Depending on how you configure your broker, a 2.0 Kafka broker can be incompatible with older client like 0.8.6, since there have been wire-protocol changes in Kafka since 0.11.
P.S. a more detailed log + configuration would be required to debug your issue (probably, turn on debug log for Kafka consumer lib), if you still choose to stay with Samza 0.14.1. Best, -Yi On Thu, Apr 9, 2020 at 12:50 PM Malcolm McFarland <mmcfarl...@cavulus.com> wrote: > Hey folks, > > We're occasionally seeing an issue when starting Samza containers where > none of the streamtasks for a job will instantiate consumers. We'll see the > beginnings of an attempt to read the checkpoint stream and nothing further > (including no errors). All of these streamtasks seem to be creating > producers -- there are plenty of "Registering > TaskName-SystemStreamPartition [kafka, <topic>, <partition>] with > producer." messages. We're not seeing any of the usual messages about > instantiating consumers; all we're seeing is this: > > 2020-04-09T19:24:14.968Z Validating offset 0 for topic and partition > [__samza_checkpoint_ver_1_for_<job_name>_1,0] > 2020-04-09T19:24:14.969Z Able to successfully read from offset 0 for topic > and partition [__samza_checkpoint_ver_1_for_<job_name>_1,0]. Using it to > instantiate consumer. > 2020-04-09T19:24:14.974Z Reading checkpoint for taskName > SystemStreamPartition [kafka, <topic>, <partition>] > > ...and that's as far as it will go. I am also noticing that when using > different versions of librdkafka, I *can* read the checkpoint stream with > librdkafka 1.3.0, but *not* with librdkafka 0.8.6. Could there be a version > incompatibility with how the data is being stored on the kafka server? > We're running Samza 0.14.1 and using AWS MSK which is running version Kafka > 2.2.1. > > > Thanks so much, > Malcolm McFarland > Cavulus >
