I guess you are missing the KafkaClient section in
*${STORM_CONF_DIR}/storm_jaas.conf* in UI node.On Mon, Jun 3, 2019 at 8:32 PM Stig Rohde Døssing <[email protected]> wrote: > There was a recent PR to partially address this at > https://github.com/apache/storm/pull/3016, it probably doesn't fix your > exact case, but fixing your case would probably require doing something > similar. > > The reason this is broken is that storm-kafka-monitor is a workaround to > us not having generic metrics delivery from spouts to Storm UI. The way > storm-kafka-monitor works, is that your spout reports some of its > configuration to Nimbus. When someone requests a Storm UI page, Nimbus then > boots up a new JVM running storm-kafka-monitor, which just starts a > KafkaConsumer with the config from Nimbus, checks the current consumer > position, and exits. > > We want to get rid of storm-kafka-monitor eventually. I've been looking at > it in https://issues.apache.org/jira/browse/STORM-3202, but it will > probably take a while to get done. If you'd like to take a look at > implementing that issue, I'd be happy to share the code (it does not > currently compile). > > Den man. 3. jun. 2019 kl. 10.52 skrev Alexandre Vermeerbergen < > [email protected]>: > >> Hello, >> >> I am trying to switch my Storm topologies based on Kafka spout from >> authenticated & unencrypted Kafka brokers to SSL + Authenticated Kafka >> Brokers. >> >> So I have changed my Kafka spouts build code to something like this >> (actial user & password have been changed to 'foo' and 'bar'): >> >> kafkaSpoutConfig = >> KafkaSpoutConfig.builder(supConfig.kafka_broker_hosts_str, "myTopic") >> .setProp(ConsumerConfig.GROUP_ID_CONFIG, "myConsumer") >> .setProp("security.protocol", "SASL_SSL") >> .setProp("sasl.jaas.config", >> "org.apache.kafka.common.security.scram.ScramLoginModule required >> username=\"foo\" password=\"bar\";") >> .setProp("sasl.mechanism", "SCRAM-SHA-512") >> >> .setProp(SslConfigs.SSL_ENDPOINT_IDENTIFICATION_ALGORITHM_CONFIG, "") >> >> .setFirstPollOffsetStrategy(FirstPollOffsetStrategy.LATEST) >> .setRecordTranslator(new >> PodSyncPodsKafkaRecordTranslator()) >> .build(); >> >> >> My problem is that in Storm UI, I see that the usual Kafka Lag >> monitoring fails with these errors: >> >> Topology spouts lag error >> >> IdTypeMessage >> eventFromAdminTopicKAFKAUnable to get offset lags for kafka. Reason: >> org.apache.kafka.shaded.common.KafkaException: Failed to construct >> kafka consumer at >> >> org.apache.kafka.shaded.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:703) >> at >> org.apache.kafka.shaded.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:584) >> at >> org.apache.kafka.shaded.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:566) >> at >> org.apache.storm.kafka.monitor.KafkaOffsetLagUtil.getOffsetLags(KafkaOffsetLagUtil.java:230) >> at >> org.apache.storm.kafka.monitor.KafkaOffsetLagUtil.main(KafkaOffsetLagUtil.java:144) >> Caused by: org.apache.kafka.shaded.common.KafkaException: >> java.lang.IllegalArgumentException: Could not find a 'KafkaClient' >> entry in the JAAS configuration. System property >> 'java.security.auth.login.config' is not set at >> >> org.apache.kafka.shaded.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:86) >> at >> org.apache.kafka.shaded.common.network.ChannelBuilders.create(ChannelBuilders.java:70) >> at >> org.apache.kafka.shaded.clients.ClientUtils.createChannelBuilder(ClientUtils.java:85) >> at >> org.apache.kafka.shaded.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:644) >> ... 4 more Caused by: java.lang.IllegalArgumentException: Could not >> find a 'KafkaClient' entry in the JAAS configuration. System property >> 'java.security.auth.login.config' is not set at >> >> org.apache.kafka.shaded.common.security.authenticator.AbstractLogin.login(AbstractLogin.java:65) >> at >> org.apache.kafka.shaded.common.security.authenticator.LoginManager.<init>(LoginManager.java:46) >> at >> org.apache.kafka.shaded.common.security.authenticator.LoginManager.acquireLoginManager(LoginManager.java:68) >> at >> org.apache.kafka.shaded.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:78) >> ... 7 more >> >> Question: >> >> I guess that this issue comes from the fact that storm-kafka-monitor >> is not aware of the security setup which have been applied to the >> Spout. >> => How can I configure storm-kafka-monitor to pass it the Kafka >> consumer properties that will allow it to query the Kafka lag without >> failing on Kafka authentication? >> >> I have not found a clue about this in documentation... >> >> Also, is it normal that Kafka spouts is not automatically inherited by >> storm-kafka-monitor? >> >> Kind regards, >> Alexandre Vermeerbergen >> > -- * Regards* * Sandeep Nemuri*
