PApostol edited a comment on issue #17507:
URL: https://github.com/apache/airflow/issues/17507#issuecomment-960612281
I'm seeing the same issue on 2.1.4 with 2 schedulers, even after patching
with #18975. See scheduler logs below - note the ~8 minute gap between the
first 2 lines of each log.
Scheduler 1 logs:
```
[2021-11-03 20:04:53,236] {scheduler_job.py:619} INFO - Executor reports
execution of FOO_BAR execution_date=2021-11-03 00:00:00+00:00 exited with
status success for try_number 80
[2021-11-03 20:13:11,965] {processor.py:243} WARNING - Killing
DAGFileProcessorProcess (PID=11211)
[2021-11-03 20:13:11,966] {processor.py:243} WARNING - Killing
DAGFileProcessorProcess (PID=11215)
[2021-11-03 20:13:11,967] {processor.py:243} WARNING - Killing
DAGFileProcessorProcess (PID=11223)
Process ForkProcess-49:
Traceback (most recent call last):
File "/usr/lib64/python3.7/multiprocessing/process.py", line 297, in
_bootstrap
self.run()
File "/usr/lib64/python3.7/multiprocessing/process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File
"/home/ec2-user/venv/lib/python3.7/site-packages/airflow/dag_processing/manager.py",
line 370, in _run_processor_manager
processor_manager.start()
File
"/home/ec2-user/venv/lib/python3.7/site-packages/airflow/dag_processing/manager.py",
line 610, in start
return self._run_parsing_loop()
File
"/home/ec2-user/venv/lib/python3.7/site-packages/airflow/dag_processing/manager.py",
line 700, in _run_parsing_loop
self.collect_results()
File
"/home/ec2-user/venv/lib/python3.7/site-packages/airflow/dag_processing/manager.py",
line 1009, in collect_results
self._collect_results_from_processor(processor)
File
"/home/ec2-user/venv/lib/python3.7/site-packages/airflow/dag_processing/manager.py",
line 981, in _collect_results_from_processor
if processor.result is not None:
File
"/home/ec2-user/venv/lib/python3.7/site-packages/airflow/dag_processing/processor.py",
line 321, in result
if not self.done:
File
"/home/ec2-user/venv/lib/python3.7/site-packages/airflow/dag_processing/processor.py",
line 286, in done
if self._parent_channel.poll():
File "/usr/lib64/python3.7/multiprocessing/connection.py", line 255, in
poll
self._check_closed()
File "/usr/lib64/python3.7/multiprocessing/connection.py", line 136, in
_check_closed
raise OSError("handle is closed")
OSError: handle is closed
```
Scheduler 2 log:
```
[2021-11-03 20:04:51,417] {dagrun.py:477} INFO - Marking run <DagRun FOO @
2021-11-03 00:00:00+00:00: manual__2021-11-03T00:00:00+00:00, externally
triggered: True> successful
[2021-11-03 20:13:11,924] {dagrun.py:477} INFO - Marking run <DagRun BAR @
2021-11-03 00:00:00+00:00: manual__2021-11-03T00:00:00+00:00, externally
triggered: True> successful
[2021-11-03 20:13:11,959] {processor.py:243} WARNING - Killing
DAGFileProcessorProcess (PID=10834)
[2021-11-03 20:13:11,961] {processor.py:243} WARNING - Killing
DAGFileProcessorProcess (PID=10840)
Process ForkProcess-48:
Traceback (most recent call last):
File "/usr/lib64/python3.7/multiprocessing/process.py", line 297, in
_bootstrap
self.run()
File "/usr/lib64/python3.7/multiprocessing/process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File
"/home/ec2-user/venv/lib/python3.7/site-packages/airflow/dag_processing/manager.py",
line 370, in _run_processor_manager
processor_manager.start()
File
"/home/ec2-user/venv/lib/python3.7/site-packages/airflow/dag_processing/manager.py",
line 610, in start
return self._run_parsing_loop()
File
"/home/ec2-user/venv/lib/python3.7/site-packages/airflow/dag_processing/manager.py",
line 700, in _run_parsing_loop
self.collect_results()
File
"/home/ec2-user/venv/lib/python3.7/site-packages/airflow/dag_processing/manager.py",
line 1009, in collect_results
self._collect_results_from_processor(processor)
File
"/home/ec2-user/venv/lib/python3.7/site-packages/airflow/dag_processing/manager.py",
line 981, in _collect_results_from_processor
if processor.result is not None:
File
"/home/ec2-user/venv/lib/python3.7/site-packages/airflow/dag_processing/processor.py",
line 321, in result
if not self.done:
File
"/home/ec2-user/venv/lib/python3.7/site-packages/airflow/dag_processing/processor.py",
line 286, in done
if self._parent_channel.poll():
File "/usr/lib64/python3.7/multiprocessing/connection.py", line 255, in
poll
self._check_closed()
File "/usr/lib64/python3.7/multiprocessing/connection.py", line 136, in
_check_closed
raise OSError("handle is closed")
OSError: handle is closed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
