[jira] [Commented] (BEAM-3284) Python SDK, dataflow runner, the method modify_job_status is calling the wrong API endpoint
[
https://issues.apache.org/jira/browse/BEAM-3284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16286815#comment-16286815
]
ASF GitHub Bot commented on BEAM-3284:
--
aaltay closed pull request #4240: [BEAM-3284] Use the correct endpoint for
modify job state
URL: https://github.com/apache/beam/pull/4240
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py
b/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py
index 6253c80f83b..38c6df1ab12 100644
--- a/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py
+++ b/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py
@@ -936,7 +936,7 @@ def _is_in_terminal_state(self):
return self._job.currentState in [
values_enum.JOB_STATE_STOPPED, values_enum.JOB_STATE_DONE,
values_enum.JOB_STATE_FAILED, values_enum.JOB_STATE_CANCELLED,
-values_enum.JOB_STATE_DRAINED]
+values_enum.JOB_STATE_UPDATED, values_enum.JOB_STATE_DRAINED]
def wait_until_finish(self, duration=None):
if not self._is_in_terminal_state():
@@ -957,7 +957,7 @@ def wait_until_finish(self, duration=None):
# TODO: Merge the termination code in poll_for_job_completion and
# _is_in_terminal_state.
- terminated = (str(self._job.currentState) != 'JOB_STATE_RUNNING')
+ terminated = self._is_in_terminal_state()
assert duration or terminated, (
'Job did not reach to a terminal state after waiting indefinitely.')
diff --git a/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py
b/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py
index 64c4ac98ac2..dd6bf95706e 100644
--- a/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py
+++ b/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py
@@ -569,7 +569,7 @@ def modify_job_state(self, job_id, new_state):
request.location = self.google_cloud_options.region
request.job = dataflow.Job(requestedState=new_state)
-self._client.projects_jobs.Update(request)
+self._client.projects_locations_jobs.Update(request)
return True
@retry.with_exponential_backoff() # Using retry defaults from utils/retry.py
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Python SDK, dataflow runner, the method modify_job_status is calling the
> wrong API endpoint
> ---
>
> Key: BEAM-3284
> URL: https://issues.apache.org/jira/browse/BEAM-3284
> Project: Beam
> Issue Type: Bug
> Components: runner-dataflow
>Affects Versions: 2.1.0, 2.2.0
>Reporter: Ron Mahoney
>Assignee: Ahmet Altay
>Priority: Minor
>
> In the Python SDK, for dataflow runner, the method modify_job_status is
> calling the wrong API endpoint. Discovered while trying to cancel a job by
> setting the status to JOB_STATE_CANCELLED, received the following error:
> {noformat}
> WARNING:root:Retry with exponential backoff: waiting for 154.109699453
> seconds before retrying modify_job_state because we caught exception:
> TypecheckError: Type of arg is " 'apache_beam.runners.dataflow.internal.clients.dataflow.dataflow_v1b3_messages.DataflowProjectsLocationsJobsUpdateRequest'>",
> not " 'apache_beam.runners.dataflow.internal.clients.dataflow.dataflow_v1b3_messages.DataflowProjectsJobsUpdateRequest'>"
> {noformat}
> The following change fixes this:
> {noformat}
> diff --git a/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py
> b/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py
> index edac9d7d5..1124ee182 100644
> --- a/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py
> +++ b/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py
> @@ -512,7 +512,7 @@ class DataflowApplicationClient(object):
># Other states could only be set by the service.
>return False
> -request = dataflow.DataflowProjectsLocationsJobsUpdateRequest()
> +request = dataflow.DataflowProjectsJobsUpdateRequest()
> request.jobId = job_id
> request.projectId = self.google_cloud_options.project
> request.location = self.google_cloud_options.region
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (BEAM-3284) Python SDK, dataflow runner, the method modify_job_status is calling the wrong API endpoint
[
https://issues.apache.org/jira/browse/BEAM-3284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16284409#comment-16284409
]
Ahmet Altay commented on BEAM-3284:
---
Thank you for reporting this. I confirmed the issue and prepared a PR to fix it.
> Python SDK, dataflow runner, the method modify_job_status is calling the
> wrong API endpoint
> ---
>
> Key: BEAM-3284
> URL: https://issues.apache.org/jira/browse/BEAM-3284
> Project: Beam
> Issue Type: Bug
> Components: runner-dataflow
>Affects Versions: 2.1.0, 2.2.0
>Reporter: Ron Mahoney
>Assignee: Ahmet Altay
>Priority: Minor
>
> In the Python SDK, for dataflow runner, the method modify_job_status is
> calling the wrong API endpoint. Discovered while trying to cancel a job by
> setting the status to JOB_STATE_CANCELLED, received the following error:
> {noformat}
> WARNING:root:Retry with exponential backoff: waiting for 154.109699453
> seconds before retrying modify_job_state because we caught exception:
> TypecheckError: Type of arg is " 'apache_beam.runners.dataflow.internal.clients.dataflow.dataflow_v1b3_messages.DataflowProjectsLocationsJobsUpdateRequest'>",
> not " 'apache_beam.runners.dataflow.internal.clients.dataflow.dataflow_v1b3_messages.DataflowProjectsJobsUpdateRequest'>"
> {noformat}
> The following change fixes this:
> {noformat}
> diff --git a/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py
> b/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py
> index edac9d7d5..1124ee182 100644
> --- a/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py
> +++ b/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py
> @@ -512,7 +512,7 @@ class DataflowApplicationClient(object):
># Other states could only be set by the service.
>return False
> -request = dataflow.DataflowProjectsLocationsJobsUpdateRequest()
> +request = dataflow.DataflowProjectsJobsUpdateRequest()
> request.jobId = job_id
> request.projectId = self.google_cloud_options.project
> request.location = self.google_cloud_options.region
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (BEAM-3284) Python SDK, dataflow runner, the method modify_job_status is calling the wrong API endpoint
[
https://issues.apache.org/jira/browse/BEAM-3284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16284407#comment-16284407
]
ASF GitHub Bot commented on BEAM-3284:
--
aaltay opened a new pull request #4240: [BEAM-3284] Use the correct endpoint
for modify job state
URL: https://github.com/apache/beam/pull/4240
R: @udim
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Python SDK, dataflow runner, the method modify_job_status is calling the
> wrong API endpoint
> ---
>
> Key: BEAM-3284
> URL: https://issues.apache.org/jira/browse/BEAM-3284
> Project: Beam
> Issue Type: Bug
> Components: runner-dataflow
>Affects Versions: 2.1.0, 2.2.0
>Reporter: Ron Mahoney
>Assignee: Ahmet Altay
>Priority: Minor
>
> In the Python SDK, for dataflow runner, the method modify_job_status is
> calling the wrong API endpoint. Discovered while trying to cancel a job by
> setting the status to JOB_STATE_CANCELLED, received the following error:
> {noformat}
> WARNING:root:Retry with exponential backoff: waiting for 154.109699453
> seconds before retrying modify_job_state because we caught exception:
> TypecheckError: Type of arg is " 'apache_beam.runners.dataflow.internal.clients.dataflow.dataflow_v1b3_messages.DataflowProjectsLocationsJobsUpdateRequest'>",
> not " 'apache_beam.runners.dataflow.internal.clients.dataflow.dataflow_v1b3_messages.DataflowProjectsJobsUpdateRequest'>"
> {noformat}
> The following change fixes this:
> {noformat}
> diff --git a/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py
> b/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py
> index edac9d7d5..1124ee182 100644
> --- a/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py
> +++ b/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py
> @@ -512,7 +512,7 @@ class DataflowApplicationClient(object):
># Other states could only be set by the service.
>return False
> -request = dataflow.DataflowProjectsLocationsJobsUpdateRequest()
> +request = dataflow.DataflowProjectsJobsUpdateRequest()
> request.jobId = job_id
> request.projectId = self.google_cloud_options.project
> request.location = self.google_cloud_options.region
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
