josh-fell commented on a change in pull request #17885:
URL: https://github.com/apache/airflow/pull/17885#discussion_r704591128
##########
File path: airflow/providers/microsoft/azure/hooks/data_factory.py
##########
@@ -583,6 +584,77 @@ def get_pipeline_run(
"""
return self.get_conn().pipeline_runs.get(resource_group_name,
factory_name, run_id, **config)
+ def check_pipeline_run_status(
Review comment:
For now I chose to have two functions:
`get_pipeline_run_status` - Simply retrieves the current status of the
pipeline run
`wait_for_pipeline_run_status` - Periodically checks if the pipeline run
status matches a terminal status or any input status(es)
After iterating a few times, splitting the functionality like we thought
always led to instances of having unnecessary extra connections to ADF. (e.g.
at the end of run a connection is made to ADF to wait for termination (via
`wait_for_pipeline_run_termination`, pipeline reaches a terminal status, and
then connect again to check if the status was expected (via
`check_pipeline_run_status`) even though the logic had just done this task.
Felt like the solution was too isolated functionally. The
`wait_for_pipeline_run_status` function seems aligned with some other similar
implementations but let me know what you think.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
