[jira] [Work logged] (BEAM-7973) Python doesn't shut down Flink job server properly
[ https://issues.apache.org/jira/browse/BEAM-7973?focusedWorklogId=302457=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-302457 ] ASF GitHub Bot logged work on BEAM-7973: Author: ASF GitHub Bot Created on: 27/Aug/19 23:16 Start Date: 27/Aug/19 23:16 Worklog Time Spent: 10m Work Description: ibzib commented on pull request #9340: [BEAM-7973] py: shut down Flink job server automatically URL: https://github.com/apache/beam/pull/9340 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 302457) Time Spent: 1h (was: 50m) > Python doesn't shut down Flink job server properly > -- > > Key: BEAM-7973 > URL: https://issues.apache.org/jira/browse/BEAM-7973 > Project: Beam > Issue Type: Bug > Components: sdk-py-harness >Reporter: Kyle Weaver >Priority: Major > Labels: portability-flink > Time Spent: 1h > Remaining Estimate: 0h > > Using the new Python FlinkRunner [1], a new job server is created and the job > succeeds, but seemingly not being shut down properly when the Python command > exits. Specifically, the java -jar command that started the job server is > still running in the background, eating up memory. > Relevant args: > python ... > --runner FlinkRunner \ > --flink_job_server_jar $FLINK_JOB_SERVER_JAR ... > [1] [https://github.com/apache/beam/pull/9043] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-7973) Python doesn't shut down Flink job server properly
[ https://issues.apache.org/jira/browse/BEAM-7973?focusedWorklogId=302456=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-302456 ] ASF GitHub Bot logged work on BEAM-7973: Author: ASF GitHub Bot Created on: 27/Aug/19 23:16 Start Date: 27/Aug/19 23:16 Worklog Time Spent: 10m Work Description: ibzib commented on issue #9340: [BEAM-7973] py: shut down Flink job server automatically URL: https://github.com/apache/beam/pull/9340#issuecomment-525519383 > Note that if there are multiple pipelines, also multiple instances of the job server will be started, which is not really necessary. It is possible (but unlikely) the user could even construct different pipelines within the same python script that specify different options, meaning different job servers would be necessary. For the more likely case of multiple pipelines run on the same job server, it would probably make more sense to start a single job server the "old-fashioned" way. > It would be even better if we shut the job server down after the pipeline execution finishes. In most cases, I assume the script will exit immediately or shortly after pipeline execution finishes. However, I agree that a better solution to managing the job server lifecycle in the Python SDK might be to put it somewhere like `pipeline_result._cleanup` instead. I will merge this bandaid fix and file [BEAM-8103](https://issues.apache.org/jira/browse/BEAM-8103) for later. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 302456) Time Spent: 50m (was: 40m) > Python doesn't shut down Flink job server properly > -- > > Key: BEAM-7973 > URL: https://issues.apache.org/jira/browse/BEAM-7973 > Project: Beam > Issue Type: Bug > Components: sdk-py-harness >Reporter: Kyle Weaver >Priority: Major > Labels: portability-flink > Time Spent: 50m > Remaining Estimate: 0h > > Using the new Python FlinkRunner [1], a new job server is created and the job > succeeds, but seemingly not being shut down properly when the Python command > exits. Specifically, the java -jar command that started the job server is > still running in the background, eating up memory. > Relevant args: > python ... > --runner FlinkRunner \ > --flink_job_server_jar $FLINK_JOB_SERVER_JAR ... > [1] [https://github.com/apache/beam/pull/9043] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-7973) Python doesn't shut down Flink job server properly
[ https://issues.apache.org/jira/browse/BEAM-7973?focusedWorklogId=300591=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-300591 ] ASF GitHub Bot logged work on BEAM-7973: Author: ASF GitHub Bot Created on: 23/Aug/19 22:46 Start Date: 23/Aug/19 22:46 Worklog Time Spent: 10m Work Description: ibzib commented on issue #9340: [BEAM-7973] py: shut down Flink job server automatically URL: https://github.com/apache/beam/pull/9340#issuecomment-524484739 I'm fine with exit. If the user doesn't want the job server hogging resources, they can just not call `wait_until_finish`, right? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 300591) Time Spent: 40m (was: 0.5h) > Python doesn't shut down Flink job server properly > -- > > Key: BEAM-7973 > URL: https://issues.apache.org/jira/browse/BEAM-7973 > Project: Beam > Issue Type: Bug > Components: sdk-py-harness >Reporter: Kyle Weaver >Priority: Major > Labels: portability-flink > Time Spent: 40m > Remaining Estimate: 0h > > Using the new Python FlinkRunner [1], a new job server is created and the job > succeeds, but seemingly not being shut down properly when the Python command > exits. Specifically, the java -jar command that started the job server is > still running in the background, eating up memory. > Relevant args: > python ... > --runner FlinkRunner \ > --flink_job_server_jar $FLINK_JOB_SERVER_JAR ... > [1] [https://github.com/apache/beam/pull/9043] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-7973) Python doesn't shut down Flink job server properly
[ https://issues.apache.org/jira/browse/BEAM-7973?focusedWorklogId=300435=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-300435 ] ASF GitHub Bot logged work on BEAM-7973: Author: ASF GitHub Bot Created on: 23/Aug/19 18:34 Start Date: 23/Aug/19 18:34 Worklog Time Spent: 10m Work Description: ibzib commented on issue #9340: [BEAM-7973] py: shut down Flink job server automatically URL: https://github.com/apache/beam/pull/9340#issuecomment-524418714 R: @mxm This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 300435) Time Spent: 0.5h (was: 20m) > Python doesn't shut down Flink job server properly > -- > > Key: BEAM-7973 > URL: https://issues.apache.org/jira/browse/BEAM-7973 > Project: Beam > Issue Type: Bug > Components: sdk-py-harness >Reporter: Kyle Weaver >Priority: Major > Labels: portability-flink > Time Spent: 0.5h > Remaining Estimate: 0h > > Using the new Python FlinkRunner [1], a new job server is created and the job > succeeds, but seemingly not being shut down properly when the Python command > exits. Specifically, the java -jar command that started the job server is > still running in the background, eating up memory. > Relevant args: > python ... > --runner FlinkRunner \ > --flink_job_server_jar $FLINK_JOB_SERVER_JAR ... > [1] [https://github.com/apache/beam/pull/9043] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-7973) Python doesn't shut down Flink job server properly
[ https://issues.apache.org/jira/browse/BEAM-7973?focusedWorklogId=298285=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-298285 ] ASF GitHub Bot logged work on BEAM-7973: Author: ASF GitHub Bot Created on: 20/Aug/19 22:49 Start Date: 20/Aug/19 22:49 Worklog Time Spent: 10m Work Description: ibzib commented on issue #9340: [BEAM-7973] py: shut down Flink job server automatically URL: https://github.com/apache/beam/pull/9340#issuecomment-523224904 ping @robertwb This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 298285) Time Spent: 20m (was: 10m) > Python doesn't shut down Flink job server properly > -- > > Key: BEAM-7973 > URL: https://issues.apache.org/jira/browse/BEAM-7973 > Project: Beam > Issue Type: Bug > Components: sdk-py-harness >Reporter: Kyle Weaver >Priority: Major > Labels: portability-flink > Time Spent: 20m > Remaining Estimate: 0h > > Using the new Python FlinkRunner [1], a new job server is created and the job > succeeds, but seemingly not being shut down properly when the Python command > exits. Specifically, the java -jar command that started the job server is > still running in the background, eating up memory. > Relevant args: > python ... > --runner FlinkRunner \ > --flink_job_server_jar $FLINK_JOB_SERVER_JAR ... > [1] [https://github.com/apache/beam/pull/9043] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-7973) Python doesn't shut down Flink job server properly
[ https://issues.apache.org/jira/browse/BEAM-7973?focusedWorklogId=294906=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-294906 ] ASF GitHub Bot logged work on BEAM-7973: Author: ASF GitHub Bot Created on: 14/Aug/19 18:10 Start Date: 14/Aug/19 18:10 Worklog Time Spent: 10m Work Description: ibzib commented on pull request #9340: [BEAM-7973] py: shut down Flink job server automatically URL: https://github.com/apache/beam/pull/9340 R: @robertwb Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/) | --- | --- | [