[
https://issues.apache.org/jira/browse/BEAM-11102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17293802#comment-17293802
]
Michael Wizner commented on BEAM-11102:
---------------------------------------
Hey, thanks everyone for chipping in.
{quote}one option might be to predefine a Java function for this purpose in
cross-language builder and configure it appropriately through parameters
received from Python side (for example, the location of the local credentials
file). Will this work ?
{quote}
Yeah, that sounds reasonable to me (y)
[~chamikara], are there any examples in the codebase of something like that
done for other cross-language transforms? (Is that a well established pattern?)
I'm not much familiar with the internals, but perhaps if someone could point me
in the right direction I might have a go at this one day when I feel brave
enough :)
{quote}Alternatively props python factory function executed on a worker might
do the job nicely.
{quote}
Hey Maxim, from my understanding that wouldn't be enough, because the Python
API is just a wrapper around the external Java transform, so without the
ability to pass the config to it you'd not be able to configure the Java
consumer/producer correctly.
> Unable to use KafkaIO with SSL via Python API
> ---------------------------------------------
>
> Key: BEAM-11102
> URL: https://issues.apache.org/jira/browse/BEAM-11102
> Project: Beam
> Issue Type: Improvement
> Components: cross-language, io-py-kafka
> Reporter: Michael Wizner
> Priority: P2
>
> I've come across the same issue described in BEAM-9627 while trying to use
> KafkaIO with SSL via the Python API.
> While the solution in the Java API is to use withConsumerFactoryFn or
> withProducerFactoryFn and provide a custom function to download or generate
> key/trust stores before the consumer or producer gets created, there is no
> such functionality exposed via in the Python API since it's just a wrapper
> around the external transform.
> Is there currently any other way to work around this issue in Python or would
> it be possible to expose a similar functionality in the API? (I'm using the
> Dataflow runner, but hopefully it shouldn't matter.)
> As far as my understanding goes, alternatively one could write an external
> transform that's a thin wrapper around the Java's KafkaIO API that handles
> key/trust stores under the hood and make it available via an external
> expansion service, but that's definitely way less ideal than being able to
> achieve the same thing via the built-in API.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
