ghostbody edited a comment on issue #19767:
URL: https://github.com/apache/airflow/issues/19767#issuecomment-989709988
I had a look at the airflow sheduler logs and found this:
```
./logs/dag_processor_manager/dag_processor_manager.log:[2021-12-08
22:15:52,278] {manager.py:1092} ERROR - Processor for some_dag.py with PID 250
started at 2021-12-08T14:15:01.525929+00:00 has timed out, killing it.
./logs/dag_processor_manager/dag_processor_manager.log:[2021-12-08
22:16:43,632] {manager.py:1092} ERROR - Processor for some_dag.py with PID 581
started at 2021-12-08T14:15:53.491981+00:00 has timed out, killing it.
./logs/dag_processor_manager/dag_processor_manager.log:[2021-12-08
22:17:35,850] {manager.py:1092} ERROR - Processor for some_dag.py with PID 898
started at 2021-12-08T14:16:45.152570+00:00 has timed out, killing it.
./logs/dag_processor_manager/dag_processor_manager.log:[2021-12-08
22:18:28,167] {manager.py:1092} ERROR - Processor for some_dag.py with PID 1114
started at 2021-12-08T14:17:37.449065+00:00 has timed out, killing it.
```
While dag callbacks are executed by the processor as the source code
indicated(in the process_file function), this timeout error could lead to
callbacks missed.
I set up this
[config](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#dag-file-processor-timeout)
(to 100 seconds or more) and this problem seems to be fixed.
I think it's a not a good idea to execute callback while file procesing.
Maybe to have a specialized process to handle callback is better. (for
airflowers :) )
@jpradass
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
