dabla commented on code in PR #77162:
URL: https://github.com/apache/airflow/pull/77162#discussion_r4589934548


##########
providers/microsoft/azure/src/airflow/providers/microsoft/azure/operators/data_factory.py:
##########
@@ -115,8 +120,13 @@ class AzureDataFactoryRunPipelineOperator(BaseOperator):
     :param check_interval: Time in seconds to check on a pipeline run's status 
for non-asynchronous waits.
         Used only if ``wait_for_termination`` is True.
     :param deferrable: Run operator in deferrable mode.
+    :param durable: Keep the pipeline run id in the task state store, so that 
a retry reconnects to the

Review Comment:
   **[warning]** The new default changes what a retry and a Dag run clear do, 
and the provider changelog does not say so.
   
   On Airflow 3.3+ every existing `AzureDataFactoryRunPipelineOperator` becomes 
durable with this release. The part users will notice is the one the docs page 
describes: on 3.3 and 3.4, clearing a whole Dag run keeps the stored run id, so 
a task whose pipeline run succeeded returns without running the pipeline again. 
For someone who clears a Dag run to re-run the pipeline, that is a silent no-op.
   
   The Power BI change got a `.. note::` under the Changelog header for the 
same default 
([15.3.0](https://github.com/apache/airflow/blob/e6146e2bac63569c61491b429e831c58451dc560/providers/microsoft/azure/docs/changelog.rst#L57-L65)).
 Could this one get the same, naming the Dag run clear case and `durable=False`?



##########
providers/microsoft/azure/src/airflow/providers/microsoft/azure/operators/data_factory.py:
##########
@@ -181,11 +199,28 @@ def execute(self, context: Context) -> None:
             start_from_failure=self.start_from_failure,
             parameters=self.parameters,
         )
-        self.run_id = response.run_id
-        # Push the ``run_id`` value to XCom regardless of what happens during 
execution. This allows for
-        # retrieval the executed pipeline's ``run_id`` for downstream tasks 
especially if performing an
-        # asynchronous wait.
-        context["ti"].xcom_push(key="run_id", value=self.run_id)
+        self._record_run_id(response.run_id, context)
+        return response.run_id
+
+    def get_job_status(self, external_id: JsonValue, context: Context) -> str:
+        try:
+            return self.hook.get_pipeline_run_status(
+                str(external_id), self.resource_group_name, self.factory_name
+            )
+        except ResourceNotFoundError:
+            return "NotFound"
+
+    def is_job_active(self, status: str) -> bool:
+        return self.wait_for_termination and status in 
AzureDataFactoryPipelineRunStatus.INTERMEDIATE_STATES

Review Comment:
   **[warning]** A stored run in `Canceling` is reconnected to, so the retry 
waits for a run that can only end `Cancelled`, and fails.
   
   `INTERMEDIATE_STATES` includes `Canceling`. The retry then waits on it, 
`wait_for_pipeline_run_status` returns `False` and the try raises "has failed 
or has been cancelled". Only the try after that sees `Cancelled` and starts a 
new run, so one cancelled run costs two tries, and with `retries=1` the task 
fails without the pipeline being started again.
   
   This is reachable in deferrable mode without the user doing anything: the 
trigger cancels the pipeline run on any exception while polling 
([triggers/data_factory.py#L246-L258](https://github.com/apache/airflow/blob/e6146e2bac63569c61491b429e831c58451dc560/providers/microsoft/azure/src/airflow/providers/microsoft/azure/triggers/data_factory.py#L246-L258)).
 A transient polling error therefore still ends the run, and the retry has 
nothing to reconnect to. The Power BI operator turned that off for the durable 
case with `cancel_on_error=not self.durable` 
([powerbi.py#L230](https://github.com/apache/airflow/blob/e6146e2bac63569c61491b429e831c58451dc560/providers/microsoft/azure/src/airflow/providers/microsoft/azure/operators/powerbi.py#L230));
 `AzureDataFactoryTrigger` has no such switch yet.
   
   Two things would close it:
   
   - treat `Canceling` like `Cancelled` on the retry (start a new run), or wait 
for it to settle and then start one, instead of reconnecting;
   - let the trigger keep the run alive on a polling error when the operator is 
durable, as Power BI does.
   
   If the second is out of scope here, the docs section should say that a 
trigger error in deferrable mode still cancels the run.



##########
providers/microsoft/azure/src/airflow/providers/microsoft/azure/operators/data_factory.py:
##########
@@ -163,13 +177,17 @@ def __init__(
         self.timeout = timeout
         self.check_interval = check_interval
         self.deferrable = deferrable
+        self.run_id: str | None = None

Review Comment:
   **[blocker]** `self.run_id: str | None` makes "MyPy providers checks" fail 
with three errors.
   
   `poll_until_complete` passes `self.run_id` to `wait_for_pipeline_run_status` 
(line 230), `get_pipeline_run_status` (line 245) and 
`AzureDataFactoryTrigger(run_id=...)` (line 252), which all take `str`. mypy 
cannot narrow the attribute through `_record_run_id`.
   
   A local in `poll_until_complete` settles all three and keeps the attribute 
for `on_kill`:
   
   ```python
   run_id = str(external_id)
   self._record_run_id(run_id, context)
   ```
   
   then `run_id` instead of `self.run_id` in the rest of the method.



##########
providers/microsoft/azure/src/airflow/providers/microsoft/azure/operators/data_factory.py:
##########
@@ -181,11 +199,28 @@ def execute(self, context: Context) -> None:
             start_from_failure=self.start_from_failure,
             parameters=self.parameters,
         )
-        self.run_id = response.run_id
-        # Push the ``run_id`` value to XCom regardless of what happens during 
execution. This allows for
-        # retrieval the executed pipeline's ``run_id`` for downstream tasks 
especially if performing an
-        # asynchronous wait.
-        context["ti"].xcom_push(key="run_id", value=self.run_id)
+        self._record_run_id(response.run_id, context)

Review Comment:
   **[nit]** The XCom push sits between `run_pipeline` and the write to the 
task state store.
   
   `execute_resumable` stores the id only after `submit_job` returns, and its 
docstring names that gap as the window in which a retry still duplicates the 
run. `_record_run_id` here adds a supervisor round trip to the window: if the 
push fails, the run is started and never stored.
   
   `poll_until_complete` calls `_record_run_id` right after the store write on 
every path (fresh, reconnect, `durable=False`), so `submit_job` only needs 
`self.run_id = response.run_id` for `on_kill`, with the push left to the later 
call.



##########
providers/microsoft/azure/src/airflow/providers/microsoft/azure/operators/data_factory.py:
##########
@@ -181,11 +199,28 @@ def execute(self, context: Context) -> None:
             start_from_failure=self.start_from_failure,
             parameters=self.parameters,
         )
-        self.run_id = response.run_id
-        # Push the ``run_id`` value to XCom regardless of what happens during 
execution. This allows for
-        # retrieval the executed pipeline's ``run_id`` for downstream tasks 
especially if performing an
-        # asynchronous wait.
-        context["ti"].xcom_push(key="run_id", value=self.run_id)
+        self._record_run_id(response.run_id, context)
+        return response.run_id
+
+    def get_job_status(self, external_id: JsonValue, context: Context) -> str:
+        try:
+            return self.hook.get_pipeline_run_status(
+                str(external_id), self.resource_group_name, self.factory_name
+            )
+        except ResourceNotFoundError:
+            return "NotFound"
+
+    def is_job_active(self, status: str) -> bool:
+        return self.wait_for_termination and status in 
AzureDataFactoryPipelineRunStatus.INTERMEDIATE_STATES
+
+    def is_job_succeeded(self, status: str) -> bool:
+        # Without wait_for_termination a retry does not wait for a run in 
progress either.
+        return status == AzureDataFactoryPipelineRunStatus.SUCCEEDED or (
+            not self.wait_for_termination and status in 
AzureDataFactoryPipelineRunStatus.INTERMEDIATE_STATES

Review Comment:
   **[nit]** `is_job_succeeded` answers `True` for a run in progress when 
`wait_for_termination=False`.
   
   The mixin then logs "Job already completed successfully, skipping 
resubmission" and counts `resumable_job.already_succeeded` for a run that is 
`Queued` or `InProgress`.
   
   `poll_until_complete` already returns without waiting when 
`wait_for_termination` is `False`, so both predicates can say what the status 
is:
   
   ```python
   def is_job_active(self, status: str) -> bool:
       return status in AzureDataFactoryPipelineRunStatus.INTERMEDIATE_STATES
   
   def is_job_succeeded(self, status: str) -> bool:
       return status == AzureDataFactoryPipelineRunStatus.SUCCEEDED
   ```
   
   The retry takes the reconnect branch, pushes the XCom and returns, with the 
right log line and metric, and it also emits the `deferrable` has no effect 
warning a first try gives. 
`test_execute_without_waiting_returns_for_run_in_progress` passes unchanged.



##########
providers/microsoft/azure/tests/unit/microsoft/azure/operators/test_data_factory.py:
##########
@@ -496,3 +501,140 @@ def 
test_azure_data_factory_run_pipeline_operator_async_execute_complete_fail(se
                 context={},
                 event={"status": "error", "message": "error", "run_id": 
AZ_PIPELINE_RUN_ID},
             )
+
+
[email protected](not AIRFLOW_V_3_3_PLUS, reason="The task state store 
arrived in Airflow 3.3")
+class TestAzureDataFactoryRunPipelineOperatorDurable:
+    CONFIG = {
+        "task_id": TASK_ID,
+        "azure_data_factory_conn_id": AZURE_DATA_FACTORY_CONN_ID,
+        "pipeline_name": PIPELINE_NAME,
+        "resource_group_name": "resource-group-name",
+        "factory_name": "factory-name",
+        "check_interval": 1,
+        "timeout": 3,
+    }
+
+    @pytest.fixture(autouse=True)
+    def setup_mocks(self):
+        with (
+            patch.object(
+                AzureDataFactoryHook,
+                "run_pipeline",
+                autospec=True,
+                return_value=MagicMock(run_id=NEW_RUN_ID),

Review Comment:
   **[nit]** Two mocks without a spec: `MagicMock(run_id=NEW_RUN_ID)` here and 
`{"ti": MagicMock()}` in `make_context` (line 536).
   
   `AGENTS.md` asks for `spec`/`autospec`. `spec=CreateRunResponse` for the 
response and a `spec` of the task instance for `ti` would make a typo in 
`xcom_push` fail the test.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to