[
https://issues.apache.org/jira/browse/AIRFLOW-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Barker updated AIRFLOW-1885:
---------------------------------
Description:
If one of the gunicorn workers happens to become a zombie between
{{children()}} and {{cmdline()}} calls to psutil in
{{get_num_ready_workers_running}} will raise an IndexError:
{code}
Traceback (most recent call last):
File "/usr/local/bin/airflow", line 28, in <module>
args.func(args)
File "/usr/local/lib/python3.5/dist-packages/airflow/bin/cli.py", line 803,
in webserver
restart_workers(gunicorn_master_proc, num_workers)
File "/usr/local/lib/python3.5/dist-packages/airflow/bin/cli.py", line 687,
in restart_workers
num_ready_workers_running =
get_num_ready_workers_running(gunicorn_master_proc)
File "/usr/local/lib/python3.5/dist-packages/airflow/bin/cli.py", line 663,
in get_num_ready_workers_running
proc for proc in workers
File "/usr/local/lib/python3.5/dist-packages/airflow/bin/cli.py", line 664,
in <listcomp>
if settings.GUNICORN_WORKER_READY_PREFIX in proc.cmdline()[0]
IndexError: list index out of range
{code}
In version 4.2 of psutil, {{cmdline}} can return an empty array if the process
is zombied:
https://github.com/giampaolo/psutil/blob/release-4.2.0/psutil/_pslinux.py#L1007
so one must ensure that an array is returned with at least one item from
{{cmdline}} before doing the {{in}} check.
was:
If one of the gunicorn workers happens to become a zombie between
{{children()}} and {{cmdline()}} calls to psutil in
{{get_num_ready_workers_running}} will raise an IndexError:
{code}
Traceback (most recent call last):
File "/usr/local/bin/airflow", line 28, in <module>
args.func(args)
File "/usr/local/lib/python3.5/dist-packages/airflow/bin/cli.py", line 803,
in webserver
restart_workers(gunicorn_master_proc, num_workers)
File "/usr/local/lib/python3.5/dist-packages/airflow/bin/cli.py", line 687,
in restart_workers
num_ready_workers_running =
get_num_ready_workers_running(gunicorn_master_proc)
File "/usr/local/lib/python3.5/dist-packages/airflow/bin/cli.py", line 663,
in get_num_ready_workers_running
proc for proc in workers
File "/usr/local/lib/python3.5/dist-packages/airflow/bin/cli.py", line 664,
in <listcomp>
if settings.GUNICORN_WORKER_READY_PREFIX in proc.cmdline()[0]
IndexError: list index out of range
{code}
In version 4.2 of psutil, {{cmdline}} can return an empty array if the process
is zombied:
https://github.com/giampaolo/psutil/blob/release-4.2.0/psutil/_pslinux.py#L1007
so ensure that an array is actually returned from {{cmdline}} before doing the
{{in}} check.
> IndexError when polling ready workers and a gunicorn worker becomes a zombie
> ----------------------------------------------------------------------------
>
> Key: AIRFLOW-1885
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1885
> Project: Apache Airflow
> Issue Type: Bug
> Affects Versions: Airflow 1.8
> Reporter: John Barker
> Assignee: John Barker
>
> If one of the gunicorn workers happens to become a zombie between
> {{children()}} and {{cmdline()}} calls to psutil in
> {{get_num_ready_workers_running}} will raise an IndexError:
> {code}
> Traceback (most recent call last):
> File "/usr/local/bin/airflow", line 28, in <module>
> args.func(args)
> File "/usr/local/lib/python3.5/dist-packages/airflow/bin/cli.py", line 803,
> in webserver
> restart_workers(gunicorn_master_proc, num_workers)
> File "/usr/local/lib/python3.5/dist-packages/airflow/bin/cli.py", line 687,
> in restart_workers
> num_ready_workers_running =
> get_num_ready_workers_running(gunicorn_master_proc)
> File "/usr/local/lib/python3.5/dist-packages/airflow/bin/cli.py", line 663,
> in get_num_ready_workers_running
> proc for proc in workers
> File "/usr/local/lib/python3.5/dist-packages/airflow/bin/cli.py", line 664,
> in <listcomp>
> if settings.GUNICORN_WORKER_READY_PREFIX in proc.cmdline()[0]
> IndexError: list index out of range
> {code}
> In version 4.2 of psutil, {{cmdline}} can return an empty array if the
> process is zombied:
> https://github.com/giampaolo/psutil/blob/release-4.2.0/psutil/_pslinux.py#L1007
> so one must ensure that an array is returned with at least one item from
> {{cmdline}} before doing the {{in}} check.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
