Arunodoy18 commented on PR #61024:
URL: https://github.com/apache/airflow/pull/61024#issuecomment-3797087077
@sarth-akvaish mentioned that callback logs appear in
logs/dag_processor/latest by default now. I believe this might be:
Task-level callbacks (which do have log handlers), OR
A misunderstanding - I couldn't find any existing code that captures
DAG-level callback logs
Would you like me to provide a reproducible test case? I've created
[test_dag_callback_logs.py](vscode-file://vscode-app/c:/Users/aruno/AppData/Local/Programs/Microsoft%20VS%20Code/resources/app/out/vs/code/electron-browser/workbench/workbench.html)
that demonstrates the issue:
def dag_failure_callback(context):
logger.info("=" * 80)
logger.info("DAG FAILURE CALLBACK TRIGGERED") # ← These logs are
currently lost
logger.error("This is an ERROR from dag_failure_callback")
logger.info("=" * 80)
with DAG(
dag_id="test_dag_callback_logs",
on_failure_callback=dag_failure_callback, # ← DAG-level callback
...
)
Without this PR: Searching grep -r "DAG FAILURE CALLBACK" ~/airflow/logs/
returns nothing
With this PR: Logs appear in DAG file parsing log with [[DAG
Callback]](vscode-file://vscode-app/c:/Users/aruno/AppData/Local/Programs/Microsoft%20VS%20Code/resources/app/out/vs/code/electron-browser/workbench/workbench.html)
prefix
Verification
The unit tests in this PR verify:
test_execute_dag_callbacks_logs_captured - Tests
[processor.py](vscode-file://vscode-app/c:/Users/aruno/AppData/Local/Programs/Microsoft%20VS%20Code/resources/app/out/vs/code/electron-browser/workbench/workbench.html)
path (DAG file parsing)
test_handle_dag_callback_logs_captured - Tests dagrun.py path (dag.test())
Would you like me to:
Provide a step-by-step reproduction of the missing logs issue?
Show specific grep commands demonstrating logs are currently lost?
Clarify any other aspect of the implementation?
Thank you for maintaining architectural correctness - the distinction
between scheduler and DAG parsing logs in distributed deployments is critical.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
