ASF GitHub Bot logged work on BEAM-7973:

                Author: ASF GitHub Bot
            Created on: 27/Aug/19 23:16
            Start Date: 27/Aug/19 23:16
    Worklog Time Spent: 10m 
      Work Description: ibzib commented on issue #9340: [BEAM-7973] py: shut 
down Flink job server automatically
URL: https://github.com/apache/beam/pull/9340#issuecomment-525519383
   > Note that if there are multiple pipelines, also multiple instances of the 
job server will be started, which is not really necessary.
   It is possible (but unlikely) the user could even construct different 
pipelines within the same python script that specify different options, meaning 
different job servers would be necessary. For the more likely case of multiple 
pipelines run on the same job server, it would probably make more sense to 
start a single job server the "old-fashioned" way.
   > It would be even better if we shut the job server down after the pipeline 
execution finishes.
   In most cases, I assume the script will exit immediately or shortly after 
pipeline execution finishes. However, I agree that a better solution to 
managing the job server lifecycle in the Python SDK might be to put it 
somewhere like `pipeline_result._cleanup` instead. I will merge this bandaid 
fix and file [BEAM-8103](https://issues.apache.org/jira/browse/BEAM-8103) for 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:

Issue Time Tracking

    Worklog Id:     (was: 302456)
    Time Spent: 50m  (was: 40m)

> Python doesn't shut down Flink job server properly
> --------------------------------------------------
>                 Key: BEAM-7973
>                 URL: https://issues.apache.org/jira/browse/BEAM-7973
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-py-harness
>            Reporter: Kyle Weaver
>            Priority: Major
>              Labels: portability-flink
>          Time Spent: 50m
>  Remaining Estimate: 0h
> Using the new Python FlinkRunner [1], a new job server is created and the job 
> succeeds, but seemingly not being shut down properly when the Python command 
> exits. Specifically, the java -jar command that started the job server is 
> still running in the background, eating up memory.
> Relevant args:
> python ...
>  --runner FlinkRunner \ 
>  --flink_job_server_jar $FLINK_JOB_SERVER_JAR ...
> [1] [https://github.com/apache/beam/pull/9043]

This message was sent by Atlassian Jira

Reply via email to