Wajid Khattak created AIRFLOW-4004:
--------------------------------------
Summary: Dataflow jobs only launched in us-central1 region
Key: AIRFLOW-4004
URL: https://issues.apache.org/jira/browse/AIRFLOW-4004
Project: Apache Airflow
Issue Type: Bug
Components: Dataflow, hooks
Affects Versions: 1.10.2
Environment: OS = Windows 7 Enterprise, Python = 2.7.12
Reporter: Wajid Khattak
Dataflow jobs can only be launched in us-central1 region. Raeson seems to be
that for launching jobs the REST endpoint
"https://cloud.google.com/dataflow/docs/reference/rest/v1b3/projects.templates/launch";
is called as below in gcp_dataflow_hook.py:
{code:java}
...
request = service.projects().locations().templates().launch(
projectId=variables['project'],
location=variables['region'],
gcsPath=dataflow_template,
body=body
)
...
{code}
However, for checking the progress of the launched job, the REST endpoint
"https://cloud.google.com/dataflow/docs/reference/rest/v1b3/projects.locations.jobs/get";
is called as below in gcp_dataflow_hook.py:
{code:java}
...
def _get_job(self):
if self._job_id:
job = self._dataflow.projects().locations().jobs().get(
projectId=self._project_number,
location=self._job_location,
jobId=self._job_id).execute(num_retries=5)
elif self._job_name:
job = self._get_job_id_from_name()
else:
raise Exception('Missing both dataflow job ID and name.')
...{code}
The simple fix is to use the correct REST endpoint for launching jobs i.e
"https://cloud.google.com/dataflow/docs/reference/rest/v1b3/projects.locations.templates/launch";
so that the job is launched in the correct region as specified in the launch
parameters.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
