tgmof commented on code in PR #72406:
URL: https://github.com/apache/airflow/pull/72406#discussion_r3999790438
##########
providers/google/src/airflow/providers/google/cloud/hooks/datafusion.py:
##########
@@ -133,8 +133,11 @@ def wait_for_pipeline_state(
namespace=namespace,
)
current_state = workflow["status"]
- except KeyError:
- pass # Because the pipeline may not be visible in system yet
+ except (HTTPError, KeyError):
+ # A 404 is raised as HTTPError by
_check_response_status_and_data, and a
+ # missing "status" key raises KeyError. Both mean the run is
not visible
+ # in the system yet, so keep polling instead of failing the
task.
+ pass
Review Comment:
Probably not because showing any kind of error when it's the "expected
behaviour" would be quite misleading. The fundamental issue might be an
software/logic design issue:
- How come is there a noticeable "gap" between pipeline creation and status
tracking? The pipeline creation call should not return before some "in
progress" status is stored (and thus this gap should never be possible if
'creation' and 'status check' are made sequentially).
I think making a DataFusion software/logic design flaw visible would be more
confusing than helpful for anyone looking at the logs and seeing a "404 error
that happens 30% of the time" if that "error" is expected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
