mtraynham edited a comment on issue #13824: URL: https://github.com/apache/airflow/issues/13824#issuecomment-766093578
One thing to note with this issue, I believe this is preventing workers from also consuming more work via a 60 second sleep at the end of every task. We noticed that there was a ~30-40 second delay between 2 workers running tasks, as if all workers were busy doing something, but nothing was actually happening. It seemed work had been queued, but was not being performed. After looking at the logs within for a given task in CloudWatch (which has the added benefit of including times in the log), there's a 60 sleep after SIGTERM to when the process is finally killed with SIGKILL. ``` 2021-01-23T15:00:02.682Z | Marking task as SUCCESS. dag_id=xxxxxxxx, task_id=xxxxxxx, execution_date=20210123T140000, start_date=20210123T150001, end_date=20210123T150002 -- | -- | 2021-01-23T15:00:02.715Z | 0 downstream tasks scheduled from follow-on schedule check | 2021-01-23T15:00:06.606Z | State of this instance has been externally set to success. Terminating instance. | 2021-01-23T15:00:06.607Z | Sending Signals.SIGTERM to GPID 155 | 2021-01-23T15:01:06.617Z | process psutil.Process(pid=155, name='airflow task ru', status='sleeping', started='15:00:01') did not respond to SIGTERM. Trying SIGKILL | 2021-01-23T15:01:06.625Z | Process psutil.Process(pid=155, name='airflow task ru', status='terminated', exitcode=<Negsignal.SIGKILL: -9>, started='15:00:01') (155) terminated with exit code Negsignal.SIGKILL | 2021-01-23T15:01:06.625Z | Task exited with return code Negsignal.SIGKILL ``` ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
