DanSedov commented on a change in pull request #5269: [AIRFLOW-4492] Rebase
Dataproc Cluster operators to poll Operations
URL: https://github.com/apache/airflow/pull/5269#discussion_r283922932
##########
File path: airflow/contrib/operators/dataproc_operator.py
##########
@@ -1467,6 +1322,6 @@ def start(self):
self.hook.get_conn().projects().regions().workflowTemplates()
.instantiateInline(
parent='projects/%s/regions/%s' % (self.project_id,
self.region),
- instanceId=str(uuid.uuid4()),
+ requestId=str(uuid.uuid4()),
Review comment:
1 - we're replacing instanceId with requestId. They mean the same thing
2 - you're right that its intended for idempotency. Instead of protecting
against Airflow restarts, it guards against first request resulting in HTTP 500
(but succeeding) and then the client retrying and resulting in 2 concurrent
dataproc workflows running.
Does Airflow provide any internal id that we could use here instead?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
