[
https://issues.apache.org/jira/browse/BEAM-6777?focusedWorklogId=251335&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-251335
]
ASF GitHub Bot logged work on BEAM-6777:
----------------------------------------
Author: ASF GitHub Bot
Created on: 30/May/19 21:32
Start Date: 30/May/19 21:32
Worklog Time Spent: 10m
Work Description: angoenka commented on pull request #8681: [BEAM-6777]
Add enable_health_checker flag for Dataflow FnAPI worker (Python)
URL: https://github.com/apache/beam/pull/8681#discussion_r289180657
##########
File path: sdks/python/apache_beam/runners/dataflow/internal/apiclient.py
##########
@@ -177,20 +177,27 @@ def __init__(self, packages, options,
environment_version, pipeline_url):
key='major', value=to_json_value(environment_version))])
# TODO: Use enumerated type instead of strings for job types.
if job_type.startswith('FNAPI_'):
+ self.debug_options.experiments = self.debug_options.experiments or []
+ debug_options_experiments = self.debug_options.experiments
runner_harness_override = (
get_runner_harness_container_image())
- self.debug_options.experiments = self.debug_options.experiments or []
if runner_harness_override:
- self.debug_options.experiments.append(
+ debug_options_experiments.append(
'runner_harness_container_image=' + runner_harness_override)
- # Add use_multiple_sdk_containers flag if its not already present. Do not
+ # Add use_multiple_sdk_containers flag if it's not already present. Do
not
# add the flag if 'no_use_multiple_sdk_containers' is present.
# TODO: Cleanup use_multiple_sdk_containers once we deprecate Python SDK
# till version 2.4.
- debug_options_experiments = self.debug_options.experiments
if ('use_multiple_sdk_containers' not in debug_options_experiments and
'no_use_multiple_sdk_containers' not in debug_options_experiments):
- self.debug_options.experiments.append('use_multiple_sdk_containers')
+ debug_options_experiments.append('use_multiple_sdk_containers')
+ # Add enable_health_checker flag if it's not already present. Do not
+ # add the flag if 'disable_health_checker' is present.
+ # TODO: Cleanup enable_health_checker once we deprecate Python SDK
+ # till version 2.14.
Review comment:
Shall we also file a jira for the cleanup.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 251335)
Time Spent: 2h 50m (was: 2h 40m)
> SDK Harness Resilience
> ----------------------
>
> Key: BEAM-6777
> URL: https://issues.apache.org/jira/browse/BEAM-6777
> Project: Beam
> Issue Type: Improvement
> Components: runner-dataflow
> Reporter: Sam Rohde
> Assignee: Yueyang Qiu
> Priority: Major
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> If the Python SDK Harness crashes in any way (user code exception, OOM, etc)
> the job will hang and waste resources. The fix is to add a daemon in the SDK
> Harness and Runner Harness to communicate with Dataflow to restart the VM
> when stuckness is detected.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
