There has been some discussion about getting Kafka 0.10.x working on BEAM-307[1].
As an immediate way to unblock yourself, modify your local copy of the KafkaIO source to include setting the system property in a static block before the class is loaded or before the Kafka client is instantiated and used. Also consider contributing to the Kafka connector to getting 0.10.x working. 1: https://issues.apache.org/jira/browse/BEAM-307 On Mon, Oct 30, 2017 at 8:14 AM, Shen Li <[email protected]> wrote: > Hi, > > To use KafkaIO in secure mode, I need to set > -Djava.security.auth.login.config to point to a JAAS configuration file. > It > works fine for local execution. But how can I configure the > "java.security.auth.login.config" property in the Beam app when the > pipeline is submitted to a cluster/cloud-service? Even if I use a ParDo to > set the system property, there is no guarantee that the ParDo will run on > the same server with the KafkaIO source. > > For this specific problem, it would be helpful to upgrade to Kafka Client > 0.10.2.0, which provides a "sasl.jaas.config" property that can be updated > programmatically. Or is there any other work around? > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-85%3A+Dynamic+JAAS+ > configuration+for+Kafka+clients > > Thanks, > Shen >
