potiuk commented on issue #27345:
URL: https://github.com/apache/airflow/issues/27345#issuecomment-1295253404
> So based on your comment, I assume these are probably lines from DAG
parsing then?
Not really - there are lines from execution of tasks. "Top-level" Logs from
parsing are generated when the file is picked up by airflow - for example if
you have import error, or print directly at the top level of Python code:
```
print('x') # <- that's top level code
with DAG(...) as dag:
...
```
So my wild guess was wrong but possibly not that much. I think the change
might come frrom this change https://github.com/apache/airflow/pull/26779
I believe that your duplicate logs come from logs coming from two sources:
1) Airflow Cloud Watch logger
2) Fargate instance
When you have REMOTE_LOGGING turned on and cloudwatch configured, airflow
will use CloudWatch to log entries:
https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/logging/cloud-watch-task-handlers.html
However I think your fargate container might also send logs to cloudwatch
from the CLOUDWATCH instance or w
One of the things that changed in #26779 was setting of task logger to
propagate logs to higher-level logger:
```
'airflow.task': {
'handlers': ['task'],
'level': LOG_LEVEL,
# Set to true here (and reset via set_context) so that if no
file is configured we still get logs!
'propagate': True, ##### <- This was previously False
'filters': ['mask_secrets'],
},
```
Which sounds quite plausible explanation why it changed when you migrated.
Do you have any custom logging configured? like that :
https://airflow.apache.org/docs/apache-airflow/stable/logging-monitoring/logging-tasks.html#advanced-configuration
? Can you explain what you have if you do ?
If no, can you follow the instruction and overwirit "airflow.task" in the
dictionary and set False instead of True and see if it helps ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
