Arunodoy18 commented on PR #61024: URL: https://github.com/apache/airflow/pull/61024#issuecomment-3797086271
@potiuk Thank you for the detailed review. You're absolutely right - I apologize for the confusion in my earlier explanation. Clarification on what this PR fixes This PR specifically addresses DAG-level callbacks (not task-level callbacks). The issue is: DAG-level callbacks have [on_failure_callback](vscode-file://vscode-app/c:/Users/aruno/AppData/Local/Programs/Microsoft%20VS%20Code/resources/app/out/vs/code/electron-browser/workbench/workbench.html) at the DAG definition level These callbacks run during DAG file parsing when the DAG fails Because they run outside any TaskInstance context, there are no log handlers attached Result: All [logger.info()](vscode-file://vscode-app/c:/Users/aruno/AppData/Local/Programs/Microsoft%20VS%20Code/resources/app/out/vs/code/electron-browser/workbench/workbench.html), [logger.warning()](vscode-file://vscode-app/c:/Users/aruno/AppData/Local/Programs/Microsoft%20VS%20Code/resources/app/out/vs/code/electron-browser/workbench/workbench.html), [logger.error()](vscode-file://vscode-app/c:/Users/aruno/AppData/Local/Programs/Microsoft%20VS%20Code/resources/app/out/vs/code/electron-browser/workbench/workbench.html) calls inside the callback are silently dropped Updated implementation (commit 2e73bb2) Following your architecture guidance, I've corrected the implementation: ✅ Now routes logs to DAG file parsing log (via structlog) ❌ NOT to scheduler logs (as they're separate in distributed deployments) Changes: [_redirect_callback_logs_to_dag_parsing_log(structlog_logger)](vscode-file://vscode-app/c:/Users/aruno/AppData/Local/Programs/Microsoft%20VS%20Code/resources/app/out/vs/code/electron-browser/workbench/workbench.html) - forwards to the structlog logger used by DAG processor Callbacks during DAG file parsing → DAG file parsing log Callbacks during [dag.test()](vscode-file://vscode-app/c:/Users/aruno/AppData/Local/Programs/Microsoft%20VS%20Code/resources/app/out/vs/code/electron-browser/workbench/workbench.html) → Handled separately in dagrun.py (existing implementation) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
