There was a recent PR to partially address this at https://github.com/apache/storm/pull/3016, it probably doesn't fix your exact case, but fixing your case would probably require doing something similar.
The reason this is broken is that storm-kafka-monitor is a workaround to us not having generic metrics delivery from spouts to Storm UI. The way storm-kafka-monitor works, is that your spout reports some of its configuration to Nimbus. When someone requests a Storm UI page, Nimbus then boots up a new JVM running storm-kafka-monitor, which just starts a KafkaConsumer with the config from Nimbus, checks the current consumer position, and exits. We want to get rid of storm-kafka-monitor eventually. I've been looking at it in https://issues.apache.org/jira/browse/STORM-3202, but it will probably take a while to get done. If you'd like to take a look at implementing that issue, I'd be happy to share the code (it does not currently compile). Den man. 3. jun. 2019 kl. 10.52 skrev Alexandre Vermeerbergen < avermeerber...@gmail.com>: > Hello, > > I am trying to switch my Storm topologies based on Kafka spout from > authenticated & unencrypted Kafka brokers to SSL + Authenticated Kafka > Brokers. > > So I have changed my Kafka spouts build code to something like this > (actial user & password have been changed to 'foo' and 'bar'): > > kafkaSpoutConfig = > KafkaSpoutConfig.builder(supConfig.kafka_broker_hosts_str, "myTopic") > .setProp(ConsumerConfig.GROUP_ID_CONFIG, "myConsumer") > .setProp("security.protocol", "SASL_SSL") > .setProp("sasl.jaas.config", > "org.apache.kafka.common.security.scram.ScramLoginModule required > username=\"foo\" password=\"bar\";") > .setProp("sasl.mechanism", "SCRAM-SHA-512") > > .setProp(SslConfigs.SSL_ENDPOINT_IDENTIFICATION_ALGORITHM_CONFIG, "") > > .setFirstPollOffsetStrategy(FirstPollOffsetStrategy.LATEST) > .setRecordTranslator(new > PodSyncPodsKafkaRecordTranslator()) > .build(); > > > My problem is that in Storm UI, I see that the usual Kafka Lag > monitoring fails with these errors: > > Topology spouts lag error > > IdTypeMessage > eventFromAdminTopicKAFKAUnable to get offset lags for kafka. Reason: > org.apache.kafka.shaded.common.KafkaException: Failed to construct > kafka consumer at > > org.apache.kafka.shaded.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:703) > at > org.apache.kafka.shaded.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:584) > at > org.apache.kafka.shaded.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:566) > at > org.apache.storm.kafka.monitor.KafkaOffsetLagUtil.getOffsetLags(KafkaOffsetLagUtil.java:230) > at > org.apache.storm.kafka.monitor.KafkaOffsetLagUtil.main(KafkaOffsetLagUtil.java:144) > Caused by: org.apache.kafka.shaded.common.KafkaException: > java.lang.IllegalArgumentException: Could not find a 'KafkaClient' > entry in the JAAS configuration. System property > 'java.security.auth.login.config' is not set at > > org.apache.kafka.shaded.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:86) > at > org.apache.kafka.shaded.common.network.ChannelBuilders.create(ChannelBuilders.java:70) > at > org.apache.kafka.shaded.clients.ClientUtils.createChannelBuilder(ClientUtils.java:85) > at > org.apache.kafka.shaded.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:644) > ... 4 more Caused by: java.lang.IllegalArgumentException: Could not > find a 'KafkaClient' entry in the JAAS configuration. System property > 'java.security.auth.login.config' is not set at > > org.apache.kafka.shaded.common.security.authenticator.AbstractLogin.login(AbstractLogin.java:65) > at > org.apache.kafka.shaded.common.security.authenticator.LoginManager.<init>(LoginManager.java:46) > at > org.apache.kafka.shaded.common.security.authenticator.LoginManager.acquireLoginManager(LoginManager.java:68) > at > org.apache.kafka.shaded.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:78) > ... 7 more > > Question: > > I guess that this issue comes from the fact that storm-kafka-monitor > is not aware of the security setup which have been applied to the > Spout. > => How can I configure storm-kafka-monitor to pass it the Kafka > consumer properties that will allow it to query the Kafka lag without > failing on Kafka authentication? > > I have not found a clue about this in documentation... > > Also, is it normal that Kafka spouts is not automatically inherited by > storm-kafka-monitor? > > Kind regards, > Alexandre Vermeerbergen >
