Hi Matt, >> We’re trying to connect to a cluster that has SSL enabled, and we’re getting some errors that are indicative of SSL connectivity failing.
Can you share some more details on the errors - eg: log files, stack-traces? Do you get the error when Samza attempts to create the topic? >> Is it possible that Samza 0.14.1 doesn’t support SSL for Kafka when creating its checkpoint topics? Samza 0.14.1 uses AdminUtils to do admin operations on Kafka, while Samza 1.0 uses the AdminClient instead. Can you write a small test and confirm if you can create a topic in your Kafka cluster using AdminUtils.createTopic <https://github.com/apache/samza/blob/0.14.1/samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemAdmin.scala#L451>? Please provide the topic-name, zkClient, # partitions, replication factor and additional SSL properties. On Mon, Mar 25, 2019 at 9:47 AM Prateek Maheshwari <prateek...@gmail.com> wrote: > Forwarding again. Original email did not show up on the OSS mailing list. > > ---------- Forwarded message --------- > From: Deshpande, Omkar <omkar_deshpa...@intuit.com> > Date: Fri, Mar 22, 2019 at 5:08 PM > Subject: Fwd: SSL with Samza 0.14.1? > To: prateek...@gmail.com <prateek...@gmail.com> > > > ++Prateek gmail > ------------------------------ > *From:* LeVeck, Matt > *Sent:* Thursday, March 21, 2019 10:33:11 PM > *To:* dev@samza.apache.org; pmaheshw...@linkedin.com; Deshpande, Omkar; > Audo, Nicholas > *Subject:* SSL with Samza 0.14.1? > > > Prateek, Samza dev team, > > This is Matt from Intuit. We met briefly at the beginning of this > week’s meetup. I’m wondering if you could help give us some guidance on > Kafka SSL with Samza. Here, I’m talking about the Kafka cluster that Samza > uses to store checkpoints, etc. We’re trying to connect to a cluster that > has SSL enabled, and we’re getting some errors that are indicative of SSL > connectivity failing. It might just be that our properties file isn’t > correct. But we’re a wondering if there is another possibility. This > indicates that Samza 0.14.1 uses Kafka 0.11 which should have SSL support. > But Samza 0.14.1 also requires access to zookeeper for its consumer client, > which is indicative of older clients (see > > https://samza.apache.org/learn/documentation/0.14/jobs/configuration-table.html#kafka > ). > Is it possible that Samza 0.14.1 doesn’t support SSL for Kafka when > creating its checkpoint topics? > > Anyways, I’m hoping that’s not the case, and either our config is wrong or > we’re doing something else wrong. Here is our properties snippet in case > we’ve messed up the config key names. Any guidance is appreciated. > > > # Kafka System > > systems.kafka.zookeeper.connect= > sppzookeeper.data-lake-dev.a.intuit.com:2181 > ,sppzookeeper.data-lake-dev.a.intuit.com:2182, > sppzookeeper.data-lake-dev.a.intuit.com:2183 > > systems.kafka.security.protocol=SSL > > systems.kafka.ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1 > > systems.kafka.ssl.truststore.type=JKS > > systems.kafka.ssl.truststore.location=/home/appuser/spp/kabini.jks > > systems.kafka.ssl.truststore.password=Intuit01 > > systems.kafka.bootstrap.servers=sppkafka.data-lake-dev.a.intuit.com:19701, > sppkafka.data-lake-dev.a.intuit.com:19801, > sppkafka.data-lake-dev.a.intuit.com:19901 > > > systems.kafka.samza.factory=org.apache.samza.system.kafka.KafkaSystemFactory > > > > We’ve also tried adding producer and consumer specific entries: > > > > systems.kafka.producer.security.protocol=SSL > > systems.kafka.producer.ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1 > > systems.kafka.producer.ssl.truststore.type=JKS > > systems.kafka.producer.ssl.truststore.location=/home/appuser/spp/kabini.jks > > systems.kafka.producer.ssl.truststore.password=Intuit01 > > systems.kafka.producer.bootstrap.servers= > sppkafka.data-lake-dev.a.intuit.com:19701, > sppkafka.data-lake-dev.a.intuit.com:19801, > sppkafka.data-lake-dev.a.intuit.com:19901 > > systems.kafka.consumer.zookeeper.connect= > sppzookeeper.data-lake-dev.a.intuit.com:2181, > sppzookeeper.data-lake-dev.a.intuit.com:2182, > sppzookeeper.data-lake-dev.a.intuit.com:2183 > > systems.kafka.consumer.security.protocol=SSL > > systems.kafka.consumer.ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1 > > systems.kafka.consumer.ssl.truststore.type=JKS > > systems.kafka.consumer.ssl.truststore.location=/home/appuser/spp/kabini.jks > > systems.kafka.consumer.ssl.truststore.password=Intuit01 > > systems.kafka.consumer.bootstrap.servers= > sppkafka.data-lake-dev.a.intuit.com:19701, > sppkafka.data-lake-dev.a.intuit.com:19801, > sppkafka.data-lake-dev.a.intuit.com:19901 > > systems.kafka.zookeeper.connect= > sppzookeeper.data-lake-dev.a.intuit.com:2181 > ,sppzookeeper.data-lake-dev.a.intuit.com:2182, > sppzookeeper.data-lake-dev.a.intuit.com:2183 > > systems.kafka.security.protocol=SSL > > systems.kafka.ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1 > > systems.kafka.ssl.truststore.type=JKS > > systems.kafka.ssl.truststore.location=/home/appuser/spp/kabini.jks > > systems.kafka.ssl.truststore.password=Intuit01 > > systems.kafka.bootstrap.servers=sppkafka.data-lake-dev.a.intuit.com:19701, > sppkafka.data-lake-dev.a.intuit.com:19801, > sppkafka.data-lake-dev.a.intuit.com:19901 > > Thanks, > > Matt > -- Jagadish V, Graduate Student, Department of Computer Science, Stanford University
