Hm I would expect that to work. Can you tell what container Flink is using if it's not using the one you specified? +Chamikara Jayalath <[email protected]> may have some insight here
Brian On Tue, Feb 2, 2021 at 3:27 AM Paul Nimusiima <[email protected]> wrote: > Hello Beam community, > > I am wondering whether it is possible to connect to a secure Kafka broker > with the python beam SDK. So for example, in the code below, how would I > make the "ssl.truststore.location" and "ssl.keystore.location" accessible > inside the Java SDK harness which runs the code. > > ReadFromKafka( > consumer_config={ > "bootstrap.servers": "bootstrap-server:17032", > "security.protocol": "SSL", > "ssl.truststore.location": "/opt/keys/client.truststore.jks", # > how do I make this available to the Java SDK harness > "ssl.truststore.password": "password", > "ssl.keystore.type": "PKCS12", > "ssl.keystore.location": "/opt/keys/client.keystore.p12", # how > do I make this available to the Java SDK harness > "ssl.keystore.password": "password", > "group.id": "group", > "basic.auth.credentials.source": "USER_INFO", > "schema.registry.basic.auth.user.info": "user:password" > }, > topics=["topic"], > max_num_records=2, > # expansion_service="localhost:56938" > ) > > I tried building a custom image that has pulls credentials into the > container based on "apache/beam_java11_sdk:2.27.0" and using the > "--sdk_harness_container_image_overrides=.*java.*,{image}:{tag}" > argument, but it does not appear to be used when the pipeline runs on > Flink. > > Thank you for the great tool and any help or advice is greatly appreciated. > > -- > paul >
