[jira] [Work logged] (BEAM-8166) Support Graceful shutdown of worker harness.
[ https://issues.apache.org/jira/browse/BEAM-8166?focusedWorklogId=308148=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-308148 ] ASF GitHub Bot logged work on BEAM-8166: Author: ASF GitHub Bot Created on: 06/Sep/19 20:46 Start Date: 06/Sep/19 20:46 Worklog Time Spent: 10m Work Description: lostluck commented on pull request #9504: [BEAM-8166] Allow in process workers to exit without killing the process. URL: https://github.com/apache/beam/pull/9504 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 308148) Time Spent: 50m (was: 40m) > Support Graceful shutdown of worker harness. > > > Key: BEAM-8166 > URL: https://issues.apache.org/jira/browse/BEAM-8166 > Project: Beam > Issue Type: Improvement > Components: runner-core, sdk-go >Reporter: Robert Burke >Priority: Minor > Time Spent: 50m > Remaining Estimate: 0h > > Ideally there should be a clear Shutdown control RPC a runner can send a > worker harness to trigger an orderly shutdown. > Absent that, errors on the runner side shouldn't manifest as SDK worker > harness errors. SDKs should log, and gracefully shutdown from GRPC errors. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-8166) Support Graceful shutdown of worker harness.
[ https://issues.apache.org/jira/browse/BEAM-8166?focusedWorklogId=308145=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-308145 ] ASF GitHub Bot logged work on BEAM-8166: Author: ASF GitHub Bot Created on: 06/Sep/19 20:45 Start Date: 06/Sep/19 20:45 Worklog Time Spent: 10m Work Description: lostluck commented on issue #9504: [BEAM-8166] Allow in process workers to exit without killing the process. URL: https://github.com/apache/beam/pull/9504#issuecomment-529008251 Excellent question, and you're correct the new mode isn't presently available for any public runners at this time. Flink, Spark, Dataflow all rely on containers, which don't run into this. However, native Go runners might choose to make different calls, or runners where the runner half a worker is linked into the same binary run into this, such as the internal Google runner I work on uses. It has an inprocess mode that can take advantage of this as well. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 308145) Time Spent: 40m (was: 0.5h) > Support Graceful shutdown of worker harness. > > > Key: BEAM-8166 > URL: https://issues.apache.org/jira/browse/BEAM-8166 > Project: Beam > Issue Type: Improvement > Components: runner-core, sdk-go >Reporter: Robert Burke >Priority: Minor > Time Spent: 40m > Remaining Estimate: 0h > > Ideally there should be a clear Shutdown control RPC a runner can send a > worker harness to trigger an orderly shutdown. > Absent that, errors on the runner side shouldn't manifest as SDK worker > harness errors. SDKs should log, and gracefully shutdown from GRPC errors. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-8166) Support Graceful shutdown of worker harness.
[ https://issues.apache.org/jira/browse/BEAM-8166?focusedWorklogId=308131=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-308131 ] ASF GitHub Bot logged work on BEAM-8166: Author: ASF GitHub Bot Created on: 06/Sep/19 20:16 Start Date: 06/Sep/19 20:16 Worklog Time Spent: 10m Work Description: lostluck commented on issue #9504: [BEAM-8166] Allow in process workers to exit without killing the process. URL: https://github.com/apache/beam/pull/9504#issuecomment-528999334 Run Go PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 308131) Time Spent: 0.5h (was: 20m) > Support Graceful shutdown of worker harness. > > > Key: BEAM-8166 > URL: https://issues.apache.org/jira/browse/BEAM-8166 > Project: Beam > Issue Type: Improvement > Components: runner-core, sdk-go >Reporter: Robert Burke >Priority: Minor > Time Spent: 0.5h > Remaining Estimate: 0h > > Ideally there should be a clear Shutdown control RPC a runner can send a > worker harness to trigger an orderly shutdown. > Absent that, errors on the runner side shouldn't manifest as SDK worker > harness errors. SDKs should log, and gracefully shutdown from GRPC errors. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-8166) Support Graceful shutdown of worker harness.
[ https://issues.apache.org/jira/browse/BEAM-8166?focusedWorklogId=308129=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-308129 ] ASF GitHub Bot logged work on BEAM-8166: Author: ASF GitHub Bot Created on: 06/Sep/19 20:14 Start Date: 06/Sep/19 20:14 Worklog Time Spent: 10m Work Description: lostluck commented on issue #9504: [BEAM-8166] Allow in process workers to exit without killing the process. URL: https://github.com/apache/beam/pull/9504#issuecomment-528998600 R: @youngoli This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 308129) Time Spent: 20m (was: 10m) > Support Graceful shutdown of worker harness. > > > Key: BEAM-8166 > URL: https://issues.apache.org/jira/browse/BEAM-8166 > Project: Beam > Issue Type: Improvement > Components: runner-core, sdk-go >Reporter: Robert Burke >Priority: Minor > Time Spent: 20m > Remaining Estimate: 0h > > Ideally there should be a clear Shutdown control RPC a runner can send a > worker harness to trigger an orderly shutdown. > Absent that, errors on the runner side shouldn't manifest as SDK worker > harness errors. SDKs should log, and gracefully shutdown from GRPC errors. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-8166) Support Graceful shutdown of worker harness.
[ https://issues.apache.org/jira/browse/BEAM-8166?focusedWorklogId=308128=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-308128 ] ASF GitHub Bot logged work on BEAM-8166: Author: ASF GitHub Bot Created on: 06/Sep/19 20:13 Start Date: 06/Sep/19 20:13 Worklog Time Spent: 10m Work Description: lostluck commented on pull request #9504: [BEAM-8166] Allow in process workers to exit without killing the process. URL: https://github.com/apache/beam/pull/9504 Allow for runners that do not spawn distinct processes for workers. InProcess runners can be useful for testing, or to support for any future native go runner. When running multiple pipelines simultaneously, it's desireable to allow for some measure of independance as well. Responding fatally on GRPC connection terminations prevents these behaviours. Further, improve the error messages to include additional context when there are GRPC issues. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[