Hi, I am working on updating our infrastructure by upgrading Samza, Kafka and Hadoop(our container manager for samza).
I am upgrading from/to these versions: Samza: from 2.11-0.12.0 to 2.12-1.4.0 Kafka: from 2.11-0.10.0.1 to 2.12-2.5.0 Hadoop: from 2.7.2 to 2.9.2 After updating the versions on starting the Samza application I am getting the following error: Exception in thread "main" java.lang.NoClassDefFoundError: kafka/common/TopicAndPartition at org.apache.samza.system.kafka.KafkaSystemConsumer.toTopicAndPartition(KafkaSystemConsumer.java:317) at org.apache.samza.system.kafka.KafkaSystemConsumer.register(KafkaSystemConsumer.java:284) at org.apache.samza.coordinator.stream.CoordinatorStreamSystemConsumer.register(CoordinatorStreamSystemConsumer.java:130) at org.apache.samza.util.CoordinatorStreamUtil$.writeConfigToCoordinatorStream(CoordinatorStreamUtil.scala:159) at org.apache.samza.job.JobRunner.run(JobRunner.scala:80) at org.apache.samza.job.JobRunner$.doOperation(JobRunner.scala:52) at org.apache.samza.job.JobRunner$.main(JobRunner.scala:47) at org.apache.samza.job.JobRunner.main(JobRunner.scala) Caused by: java.lang.ClassNotFoundException: kafka.common.TopicAndPartition at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) ... 8 more On further checking I found that ‘KafkaSystemConsumer’ is part of samza-kafka_2.12-1.4.0.jar. And I looked into our deploy jars and I am finding only one copy of this class. The reference class ‘TopicAndPartition’ is not present anywhere in the deploy jars or kafka_2.12-2.5.0 jar in particular. I tried to find a Samza-Kafka version compatibility documentation. But I didn’t find any. On investigating our import of samza-kafka_2.12-1.4.0.jar, I found that it is including the kafka_2.12-2.5.0 dependency, which leads me to think that Samza should be compatible the Kafka version I am referencing. I am also including all the required maven dependencies such as kafka, kafka-clients, samza-core, samza-api of these latest versions and made sure no other version of the same is present in my deploy jars. Please let me know If I am missing something. And what can do to resolve this issue. Thanks in Advance, Suraj
