[
https://issues.apache.org/jira/browse/BEAM-11102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17293852#comment-17293852
]
Chamikara Madhusanka Jayalath commented on BEAM-11102:
------------------------------------------------------
Yeah, pre-defined functions is a perfectly fine things to do. It's needed till
we have support for true multi-language UDFs and might be preferable even after
that for efficiency.
An example would be ReadFromKafka timestamp_policy parameter which currently
have three possible values.
[https://github.com/apache/beam/blob/release-2.28.0/sdks/python/apache_beam/io/kafka.py#L100]
[https://github.com/apache/beam/blob/release-2.28.0/sdks/python/apache_beam/io/kafka.py#L120]
Thanks for considering fixing this :)
> Unable to use KafkaIO with SSL via Python API
> ---------------------------------------------
>
> Key: BEAM-11102
> URL: https://issues.apache.org/jira/browse/BEAM-11102
> Project: Beam
> Issue Type: Improvement
> Components: cross-language, io-py-kafka
> Reporter: Michael Wizner
> Priority: P2
>
> I've come across the same issue described in BEAM-9627 while trying to use
> KafkaIO with SSL via the Python API.
> While the solution in the Java API is to use withConsumerFactoryFn or
> withProducerFactoryFn and provide a custom function to download or generate
> key/trust stores before the consumer or producer gets created, there is no
> such functionality exposed via in the Python API since it's just a wrapper
> around the external transform.
> Is there currently any other way to work around this issue in Python or would
> it be possible to expose a similar functionality in the API? (I'm using the
> Dataflow runner, but hopefully it shouldn't matter.)
> As far as my understanding goes, alternatively one could write an external
> transform that's a thin wrapper around the Java's KafkaIO API that handles
> key/trust stores under the hood and make it available via an external
> expansion service, but that's definitely way less ideal than being able to
> achieve the same thing via the built-in API.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
