[
https://issues.apache.org/jira/browse/BEAM-9449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17423748#comment-17423748
]
Eddie Wang commented on BEAM-9449:
----------------------------------
not stale.
i've spent the past few weekends trying to compile a custom beam container to
support a k8s deployment of the java harness. the main use-case here being to
leverage the Pubsub/KafkaIO libraries from the Java sdk inside of the Python
environment.
i was able to customize the default environment in the ExpansionService.java
file to use an `EXTERNAL` environment by default, but was unsure how to setup
the java sdk worker pool... python workers have a worker pool flag, but there
is not such option for the java beam sdk.
after that, i modified the default environment to `EMBEDDED` in a final attempt
to get the PubsubIO library working on Flink. Unfortunately, this is leading to
class not found issues: `Caused by: java.lang.NoClassDefFoundError: Could not
initialize class org.apache.beam.runners.core.construction.CoderTranslation`
from the Flink taskmanager.
IO is a such a fundamental primitive of any data-pipeline, and unfortunately...
most of the transforms in Python require the Java expansion service in order to
work.
> Consider passing pipeline options for expansion service.
> --------------------------------------------------------
>
> Key: BEAM-9449
> URL: https://issues.apache.org/jira/browse/BEAM-9449
> Project: Beam
> Issue Type: New Feature
> Components: beam-model, cross-language
> Reporter: Robert Bradshaw
> Priority: P2
> Labels: stale-P2
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
