ephraimbuddy commented on code in PR #74794:
URL: https://github.com/apache/airflow/pull/74794#discussion_r4536135627


##########
airflow-core/src/airflow/dag_processing/processor.py:
##########
@@ -183,6 +193,21 @@ class DagFileParsingResult(BaseModel):
 ]
 
 
[email protected]()
+class _DagParseCommsDecoder(CommsDecoder[ToDagProcessor, ToManager]):
+    api_request_failed: bool = attrs.field(default=False, init=False)
+    """Whether the Dag processor could not answer a request of this process 
through the Execution API."""
+
+    def _from_frame(self, frame) -> ToDagProcessor | None:

Review Comment:
   Unanswered requests now count as API failures for both `send` and `asend`, 
and only their matching responses clear them. An import timeout therefore 
retries the callback batch when none of its targets loaded, within the existing 
retry deadline. Regression tests cover interrupted imports and response-ID 
matching.



##########
airflow-core/src/airflow/dag_processing/manager.py:
##########
@@ -2716,10 +2853,15 @@ def terminate(self):
             processor.kill(signal.SIGTERM, escalation_delay=5.0)
 
     def end(self):
-        """Kill all child processes on exit since we don't want to leave them 
as orphaned."""
+        """Kill child processes so none are orphaned, then acknowledge work 
that already finished."""
         pids_to_kill = [p.pid for p in self._processors.values()]
         if pids_to_kill:
             kill_child_processes_by_pids(pids_to_kill)
+        # Claims the server releases after this Job ends are delivered again, 
repeating their callbacks.
+        try:
+            self.work_source.flush(deadline=math.inf)
+        except Exception:
+            self.log.exception("Unable to acknowledge requested work before 
exiting")

Review Comment:
   Retired registrations now produce a warning without a traceback during 
shutdown. Unexpected errors still retain their traceback. The test checks both 
paths; mutation testing confirmed that replacing `log.exception` with 
`log.error` fails the traceback assertion.



##########
airflow-core/src/airflow/config_templates/config.yml:
##########
@@ -3314,6 +3314,32 @@ dag_processor:
          - how often to refresh DAGs from those sources
          - how many files to parse concurrently
   options:
+    execution_api_token_file:

Review Comment:
   Added `[dag_processor] use_execution_api`, defaulting to `False`, 
independently of `execution_api_token_file`. A token file alone keeps database 
mode. Future credential sources can use the same switch, and #74796 now uses it 
for probes, rollout strategy and replica validation.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to