nanaones commented on issue #18989: URL: https://github.com/apache/airflow/issues/18989#issuecomment-1047505092
> @nanaones @rmakoto-ze @hamedhsn @rtudo > > Can you please provide the exact steps to reproduce the problem you have? What exactly you do, what logs and behaviours you see? I would like to take a close look at that but I need a little bit more information to start looking at it. ENV: Airflow on EKS 1.21 Helm 1.3.0 Airflow 2.2.3 This is the screen seen from the [Lens](https://github.com/lensapp/lens). It's constantly restarting. I have no problems using it now, but I think it could be a problem.  Airflow scheduler keeps restarting with the following error: ```shell [2022-02-22 01:34:22,995] {process_utils.py:70} INFO - Process psutil.Process(pid=2304, status='terminated', started='01:34:21') (2304) terminated with exit code None [2022-02-22 01:34:22,995] {process_utils.py:70} INFO - Process psutil.Process(pid=35, status='terminated', exitcode=0, started='01:23:31') (35) terminated with exit code 0 [2022-02-22 01:34:22,995] {kubernetes_executor.py:790} INFO - Shutting down Kubernetes executor [2022-02-22 01:34:22,996] {scheduler_job.py:650} ERROR - Exception when executing Executor.end Traceback (most recent call last): File "/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 628, in _execute self._run_scheduler_loop() File "/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 730, in _run_scheduler_loop time.sleep(min(self._scheduler_idle_sleep_time, next_event)) File "/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 164, in _exit_gracefully sys.exit(os.EX_OK) SystemExit: 0 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 648, in _execute self.executor.end() File "/home/airflow/.local/lib/python3.7/site-packages/airflow/executors/kubernetes_executor.py", line 792, in end self._flush_task_queue() File "/home/airflow/.local/lib/python3.7/site-packages/airflow/executors/kubernetes_executor.py", line 745, in _flush_task_queue self.log.debug('Executor shutting down, task_queue approximate size=%d', self.task_queue.qsize()) File "<string>", line 2, in qsize File "/usr/local/lib/python3.7/multiprocessing/managers.py", line 818, in _callmethod conn.send((self._id, methodname, args, kwds)) File "/usr/local/lib/python3.7/multiprocessing/connection.py", line 206, in send self._send_bytes(_ForkingPickler.dumps(obj)) File "/usr/local/lib/python3.7/multiprocessing/connection.py", line 404, in _send_bytes self._send(header + buf) File "/usr/local/lib/python3.7/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe [2022-02-22 01:34:22,997] {process_utils.py:124} INFO - Sending Signals.SIGTERM to group 35. PIDs of all processes in the group: [] [2022-02-22 01:34:22,997] {process_utils.py:75} INFO - Sending the signal Signals.SIGTERM to group 35 [2022-02-22 01:34:22,998] {process_utils.py:90} INFO - Sending the signal Signals.SIGTERM to process 35 as process group is missing. [2022-02-22 01:34:22,998] {scheduler_job.py:655} INFO - Exited execute loop ____________ _____________ ____ |__( )_________ __/__ /________ __ ____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / / ___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ / _/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/ [2022-02-22 01:34:30,903] {scheduler_job.py:596} INFO - Starting the scheduler ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
