Creating your own jar with updated dependencies is the right solution,
to ensure it gets used you may need to set Python's --beam_services
pipeline option, e.g.
--beam_services='{"sdks:java:io:expansion-service:shadowJar":
"/path/to/your.jar"}'
See
https://github.com/apache/beam/blob/release-2.46.0/sdks/python/apache_beam/options/pipeline_options.py#L512
Equivalently, you can pass a manually created expansion service into
your ReadFromKafka constructor.
Long term the nicest solution is to update the version used by Beam.
Pull requests welcome!
- Robert
On Wed, Mar 26, 2025 at 8:02 AM Kaur Jaanson <kjaan...@gmail.com> wrote:
>
> Hello!
>
> Hoping someone can point me in the right direction. I am trying to use PEM
> certificates for authenticating to Kafka topic using Python and Java
> Expansion Service. However I am running into an error:
>
>> Caused by: java.security.NoSuchAlgorithmException: PEM KeyStore not
>> available
>
>
> I am using FlinkRunner and starting expansion service myself with the
> expansion service jar taken from Maven. I suspect that the issue is older
> version of Java-s kafka-client that has been packaged into expansion service
> jar (and job server?). From Apache Beam docs page on Java SDK dependencies,
> it seems that default libraries that get used during building jars are set in
> https://github.com/apache/beam/blob/867d70c1689bc4b16141043616f912ce0b10b74c/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L626
> and kafka-client version there is quite old (2.4.1). PEM support is supposed
> to be there after 2.7.
>
> With that I wonder what would be a good way to specify Java dependencies for
> multi language pipeline, eg using KafkaIO with newer kafka-client from Python?
>
> PS: I tried updating the kafka-client version in BeamModulePlugin.groovy and
> creating expansion-service jar and packaging into my python beam image,
> however it seems that the error still remained. It might be skill issue, I am
> not very familiar with java build tooling (maven, gradle) and might have
> missed something. Beam version I am using is 2.61.0.
>
> Thank you,
> Kaur Jaanson
