[
https://issues.apache.org/jira/browse/BEAM-10039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17218517#comment-17218517
]
Valentyn Tymofieiev edited comment on BEAM-10039 at 12/14/20, 6:09 PM:
-----------------------------------------------------------------------
Currently, a possible workaround can be to manually add required worker
initialization steps in the SDK worker[1], and supply a custom-built Beam SDK
from sources, via --sdk_location flag.
[1]
https://github.com/apache/beam/blob/7ad4c4c8e601e39573aae7b4d778be2e908a0868/sdks/python/apache_beam/runners/worker/sdk_worker_main.py#L85
was (Author: tvalentyn):
Currently, a possible workaround can be to manually add required worker
initialization steps in the SDK worker[1], and supply a custom-built Beam SDK
from sources, and supply it via --sdk_location flag.
[1]
https://github.com/apache/beam/blob/7ad4c4c8e601e39573aae7b4d778be2e908a0868/sdks/python/apache_beam/runners/worker/sdk_worker_main.py#L85
> Create a mechanism to run a custom worker intialization code on Python
> workers
> -------------------------------------------------------------------------------
>
> Key: BEAM-10039
> URL: https://issues.apache.org/jira/browse/BEAM-10039
> Project: Beam
> Issue Type: New Feature
> Components: sdk-py-core
> Reporter: Valentyn Tymofieiev
> Priority: P3
>
> A couple of Beam users mentioned a usecase where some initialization code
> needs to run on Python workers before pipeline processing starts.
> Such code needs to be executed run early in the main() method of python
> worker[1].
> Java SDK has provides this capability via JvmInitializer [2], BEAM-6872.
> Let's add such capability for Python users as well.
> [1]
> https://github.com/apache/beam/blob/7ad4c4c8e601e39573aae7b4d778be2e908a0868/sdks/python/apache_beam/runners/worker/sdk_worker_main.py#L85
> [2]
> https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/harness/JvmInitializer.java
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
