Hey folks, We're occasionally seeing an issue when starting Samza containers where none of the streamtasks for a job will instantiate consumers. We'll see the beginnings of an attempt to read the checkpoint stream and nothing further (including no errors). All of these streamtasks seem to be creating producers -- there are plenty of "Registering TaskName-SystemStreamPartition [kafka, <topic>, <partition>] with producer." messages. We're not seeing any of the usual messages about instantiating consumers; all we're seeing is this:
2020-04-09T19:24:14.968Z Validating offset 0 for topic and partition [__samza_checkpoint_ver_1_for_<job_name>_1,0] 2020-04-09T19:24:14.969Z Able to successfully read from offset 0 for topic and partition [__samza_checkpoint_ver_1_for_<job_name>_1,0]. Using it to instantiate consumer. 2020-04-09T19:24:14.974Z Reading checkpoint for taskName SystemStreamPartition [kafka, <topic>, <partition>] ...and that's as far as it will go. I am also noticing that when using different versions of librdkafka, I *can* read the checkpoint stream with librdkafka 1.3.0, but *not* with librdkafka 0.8.6. Could there be a version incompatibility with how the data is being stored on the kafka server? We're running Samza 0.14.1 and using AWS MSK which is running version Kafka 2.2.1. Thanks so much, Malcolm McFarland Cavulus
