[
https://issues.apache.org/jira/browse/BEAM-11102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Wizner updated BEAM-11102:
----------------------------------
Description:
I've come across the same issue described in BEAM-9627 while trying to use
KafkaIO with SSL via the Python API.
While the solution in the Java API is to use withConsumerFactoryFn or
withProducerFactoryFn and provide a custom function to download or generate
key/trust stores before the consumer or producer gets created, there is no such
functionality exposed via in the Python API since it's just a wrapper around
the external transform.
Is there currently any other way to work around this issue in Python or would
it be possible to expose a similar functionality in the API? (I'm using the
Dataflow runner, but hopefully it shouldn't matter.)
As far as my understanding goes, alternatively one could write an external
transform that's a thin wrapper around the Java's KafkaIO API and make it
available via an external expansion service, but that's definitely way less
ideal than being able to achieve the same thing via the built-in API.
was:
I've come across the same issue described in BEAM-9627 while trying to use SSL
with KafkaIO via the Python API.
While the solution in the Java API is to use withConsumerFactoryFn or
withProducerFactoryFn and provide a custom function to download or generate
key/trust stores before the consumer or producer gets created, there is no such
functionality exposed via in the Python API since it's just a wrapper around
the external transform.
Is there currently any other way to work around this issue in Python or would
it be possible to expose a similar functionality in the API? (I'm using the
Dataflow runner, but hopefully it shouldn't matter.)
As far as my understanding goes, alternatively one could write an external
transform that's a thin wrapper around the Java's KafkaIO API and make it
available via an external expansion service, but that's definitely way less
ideal than being able to achieve the same thing via the built-in API.
> Unable to use SSL with KafkaIO via Python API
> ---------------------------------------------
>
> Key: BEAM-11102
> URL: https://issues.apache.org/jira/browse/BEAM-11102
> Project: Beam
> Issue Type: Bug
> Components: io-py-kafka
> Affects Versions: 2.24.0
> Reporter: Michael Wizner
> Priority: P2
>
> I've come across the same issue described in BEAM-9627 while trying to use
> KafkaIO with SSL via the Python API.
> While the solution in the Java API is to use withConsumerFactoryFn or
> withProducerFactoryFn and provide a custom function to download or generate
> key/trust stores before the consumer or producer gets created, there is no
> such functionality exposed via in the Python API since it's just a wrapper
> around the external transform.
> Is there currently any other way to work around this issue in Python or would
> it be possible to expose a similar functionality in the API? (I'm using the
> Dataflow runner, but hopefully it shouldn't matter.)
> As far as my understanding goes, alternatively one could write an external
> transform that's a thin wrapper around the Java's KafkaIO API and make it
> available via an external expansion service, but that's definitely way less
> ideal than being able to achieve the same thing via the built-in API.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
