leezu removed a comment on pull request #17794: URL: https://github.com/apache/incubator-mxnet/pull/17794#issuecomment-624404875
@szha this broke with pytest xdist ``` [2020-05-05T04:38:33.811Z] gw0 I / gw1 I / gw2 I / gw3 I [2020-05-05T04:38:33.811Z] [gw0] linux Python 3.6.9 cwd: /work/mxnet [2020-05-05T04:38:33.811Z] [gw1] linux Python 3.6.9 cwd: /work/mxnet [2020-05-05T04:38:33.811Z] [gw2] linux Python 3.6.9 cwd: /work/mxnet [2020-05-05T04:38:33.811Z] [gw3] linux Python 3.6.9 cwd: /work/mxnet [2020-05-05T04:38:40.350Z] /usr/lib/python3.6/importlib/_bootstrap_external.py:426: ImportWarning: Not importing directory /usr/local/lib/python3.6/dist-packages/google: missing __init__ [2020-05-05T04:38:40.350Z] _warnings.warn(msg.format(portions[0]), ImportWarning) [2020-05-05T04:38:40.350Z] /usr/lib/python3.6/importlib/_bootstrap_external.py:426: ImportWarning: Not importing directory /usr/lib/python3/dist-packages/google: missing __init__ [2020-05-05T04:38:40.350Z] _warnings.warn(msg.format(portions[0]), ImportWarning) [2020-05-05T04:38:40.350Z] /usr/lib/python3.6/importlib/_bootstrap_external.py:426: ImportWarning: Not importing directory /usr/local/lib/python3.6/dist-packages/google: missing __init__ [2020-05-05T04:38:40.350Z] _warnings.warn(msg.format(portions[0]), ImportWarning) [2020-05-05T04:38:40.350Z] /usr/lib/python3.6/importlib/_bootstrap_external.py:426: ImportWarning: Not importing directory /usr/lib/python3/dist-packages/google: missing __init__ [2020-05-05T04:38:40.350Z] _warnings.warn(msg.format(portions[0]), ImportWarning) [2020-05-05T04:38:40.611Z] [gw2] Python 3.6.9 (default, Apr 18 2020, 01:56:04) -- [GCC 8.4.0] [2020-05-05T04:38:40.611Z] /usr/lib/python3.6/importlib/_bootstrap_external.py:426: ImportWarning: Not importing directory /usr/local/lib/python3.6/dist-packages/google: missing __init__ [2020-05-05T04:38:40.611Z] _warnings.warn(msg.format(portions[0]), ImportWarning) [2020-05-05T04:38:40.611Z] /usr/lib/python3.6/importlib/_bootstrap_external.py:426: ImportWarning: Not importing directory /usr/lib/python3/dist-packages/google: missing __init__ [2020-05-05T04:38:40.611Z] _warnings.warn(msg.format(portions[0]), ImportWarning) [2020-05-05T04:38:40.611Z] [gw0] Python 3.6.9 (default, Apr 18 2020, 01:56:04) -- [GCC 8.4.0] [2020-05-05T04:38:40.869Z] /usr/lib/python3.6/importlib/_bootstrap_external.py:426: ImportWarning: Not importing directory /usr/local/lib/python3.6/dist-packages/google: missing __init__ [2020-05-05T04:38:40.869Z] _warnings.warn(msg.format(portions[0]), ImportWarning) [2020-05-05T04:38:40.869Z] /usr/lib/python3.6/importlib/_bootstrap_external.py:426: ImportWarning: Not importing directory /usr/lib/python3/dist-packages/google: missing __init__ [2020-05-05T04:38:40.869Z] _warnings.warn(msg.format(portions[0]), ImportWarning) [2020-05-05T04:38:41.130Z] [gw1] Python 3.6.9 (default, Apr 18 2020, 01:56:04) -- [GCC 8.4.0] [2020-05-05T04:38:41.130Z] [gw3] Python 3.6.9 (default, Apr 18 2020, 01:56:04) -- [GCC 8.4.0] [2020-05-05T04:38:53.318Z] [gw1] node down: Not properly terminated [2020-05-05T04:38:53.318Z] [2020-05-05T04:38:53.318Z] replacing crashed worker gw1 [2020-05-05T04:38:53.318Z] [gw4] linux Python 3.6.9 cwd: /work/mxnet [2020-05-05T04:38:53.318Z] [gw0] node down: Not properly terminated [2020-05-05T04:38:53.318Z] [2020-05-05T04:38:53.318Z] replacing crashed worker gw0 [2020-05-05T04:38:53.318Z] [gw5] linux Python 3.6.9 cwd: /work/mxnet [2020-05-05T04:38:53.318Z] [gw3] node down: Not properly terminated [2020-05-05T04:38:53.318Z] [2020-05-05T04:38:53.318Z] replacing crashed worker gw3 [2020-05-05T04:38:53.318Z] [gw6] linux Python 3.6.9 cwd: /work/mxnet [2020-05-05T04:38:53.318Z] [gw2] node down: Not properly terminated [2020-05-05T04:38:53.318Z] [2020-05-05T04:38:53.318Z] replacing crashed worker gw2 [2020-05-05T04:38:53.318Z] [gw7] linux Python 3.6.9 cwd: /work/mxnet [2020-05-05T04:38:58.561Z] /usr/lib/python3.6/importlib/_bootstrap_external.py:426: ImportWarning: Not importing directory /usr/local/lib/python3.6/dist-packages/google: missing __init__ [2020-05-05T04:38:58.561Z] _warnings.warn(msg.format(portions[0]), ImportWarning) [2020-05-05T04:38:58.561Z] /usr/lib/python3.6/importlib/_bootstrap_external.py:426: ImportWarning: Not importing directory /usr/lib/python3/dist-packages/google: missing __init__ [2020-05-05T04:38:58.561Z] _warnings.warn(msg.format(portions[0]), ImportWarning) [2020-05-05T04:38:58.561Z] /usr/lib/python3.6/importlib/_bootstrap_external.py:426: ImportWarning: Not importing directory /usr/local/lib/python3.6/dist-packages/google: missing __init__ [2020-05-05T04:38:58.561Z] _warnings.warn(msg.format(portions[0]), ImportWarning) [2020-05-05T04:38:58.561Z] /usr/lib/python3.6/importlib/_bootstrap_external.py:426: ImportWarning: Not importing directory /usr/lib/python3/dist-packages/google: missing __init__ [2020-05-05T04:38:58.561Z] _warnings.warn(msg.format(portions[0]), ImportWarning) [2020-05-05T04:38:58.818Z] /usr/lib/python3.6/importlib/_bootstrap_external.py:426: ImportWarning: Not importing directory /usr/local/lib/python3.6/dist-packages/google: missing __init__ [2020-05-05T04:38:58.818Z] _warnings.warn(msg.format(portions[0]), ImportWarning) [2020-05-05T04:38:58.818Z] /usr/lib/python3.6/importlib/_bootstrap_external.py:426: ImportWarning: Not importing directory /usr/lib/python3/dist-packages/google: missing __init__ [2020-05-05T04:38:58.818Z] _warnings.warn(msg.format(portions[0]), ImportWarning) [2020-05-05T04:38:58.818Z] [gw7] Python 3.6.9 (default, Apr 18 2020, 01:56:04) -- [GCC 8.4.0] [2020-05-05T04:38:59.075Z] [gw4] Python 3.6.9 (default, Apr 18 2020, 01:56:04) -- [GCC 8.4.0] [2020-05-05T04:38:59.075Z] /usr/lib/python3.6/importlib/_bootstrap_external.py:426: ImportWarning: Not importing directory /usr/local/lib/python3.6/dist-packages/google: missing __init__ [2020-05-05T04:38:59.075Z] _warnings.warn(msg.format(portions[0]), ImportWarning) [2020-05-05T04:38:59.075Z] /usr/lib/python3.6/importlib/_bootstrap_external.py:426: ImportWarning: Not importing directory /usr/lib/python3/dist-packages/google: missing __init__ [2020-05-05T04:38:59.075Z] _warnings.warn(msg.format(portions[0]), ImportWarning) [2020-05-05T04:38:59.075Z] [gw5] Python 3.6.9 (default, Apr 18 2020, 01:56:04) -- [GCC 8.4.0] [2020-05-05T04:38:59.336Z] [gw6] Python 3.6.9 (default, Apr 18 2020, 01:56:04) -- [GCC 8.4.0] [2020-05-05T04:39:11.547Z] [gw5] node down: Not properly terminated [2020-05-05T04:39:11.547Z] [2020-05-05T04:39:11.547Z] replacing crashed worker gw5 [2020-05-05T04:39:11.547Z] [gw8] linux Python 3.6.9 cwd: /work/mxnet [2020-05-05T04:39:11.547Z] [gw7] node down: Not properly terminated [2020-05-05T04:39:11.547Z] [2020-05-05T04:39:11.547Z] replacing crashed worker gw7 [2020-05-05T04:39:11.547Z] [gw9] linux Python 3.6.9 cwd: /work/mxnet [2020-05-05T04:39:11.547Z] [gw4] node down: Not properly terminated [2020-05-05T04:39:11.547Z] [2020-05-05T04:39:11.547Z] replacing crashed worker gw4 [2020-05-05T04:39:11.547Z] [gw10] linux Python 3.6.9 cwd: /work/mxnet [2020-05-05T04:39:11.547Z] [gw6] node down: Not properly terminated [2020-05-05T04:39:11.547Z] [2020-05-05T04:39:11.547Z] replacing crashed worker gw6 [2020-05-05T04:39:11.547Z] [gw11] linux Python 3.6.9 cwd: /work/mxnet [2020-05-05T04:39:15.716Z] /usr/lib/python3.6/importlib/_bootstrap_external.py:426: ImportWarning: Not importing directory /usr/local/lib/python3.6/dist-packages/google: missing __init__ [2020-05-05T04:39:15.716Z] _warnings.warn(msg.format(portions[0]), ImportWarning) [2020-05-05T04:39:15.716Z] /usr/lib/python3.6/importlib/_bootstrap_external.py:426: ImportWarning: Not importing directory /usr/lib/python3/dist-packages/google: missing __init__ [2020-05-05T04:39:15.716Z] _warnings.warn(msg.format(portions[0]), ImportWarning) [2020-05-05T04:39:15.716Z] /usr/lib/python3.6/importlib/_bootstrap_external.py:426: ImportWarning: Not importing directory /usr/local/lib/python3.6/dist-packages/google: missing __init__ [2020-05-05T04:39:15.716Z] _warnings.warn(msg.format(portions[0]), ImportWarning) [2020-05-05T04:39:15.716Z] /usr/lib/python3.6/importlib/_bootstrap_external.py:426: ImportWarning: Not importing directory /usr/lib/python3/dist-packages/google: missing __init__ [2020-05-05T04:39:15.716Z] _warnings.warn(msg.format(portions[0]), ImportWarning) [2020-05-05T04:39:15.716Z] /usr/lib/python3.6/importlib/_bootstrap_external.py:426: ImportWarning: Not importing directory /usr/local/lib/python3.6/dist-packages/google: missing __init__ [2020-05-05T04:39:15.716Z] _warnings.warn(msg.format(portions[0]), ImportWarning) [2020-05-05T04:39:15.716Z] /usr/lib/python3.6/importlib/_bootstrap_external.py:426: ImportWarning: Not importing directory /usr/lib/python3/dist-packages/google: missing __init__ [2020-05-05T04:39:15.716Z] _warnings.warn(msg.format(portions[0]), ImportWarning) [2020-05-05T04:39:15.716Z] [gw8] Python 3.6.9 (default, Apr 18 2020, 01:56:04) -- [GCC 8.4.0] [2020-05-05T04:39:15.716Z] [gw10] Python 3.6.9 (default, Apr 18 2020, 01:56:04) -- [GCC 8.4.0] [2020-05-05T04:39:15.716Z] /usr/lib/python3.6/importlib/_bootstrap_external.py:426: ImportWarning: Not importing directory /usr/local/lib/python3.6/dist-packages/google: missing __init__ [2020-05-05T04:39:15.716Z] _warnings.warn(msg.format(portions[0]), ImportWarning) [2020-05-05T04:39:15.716Z] /usr/lib/python3.6/importlib/_bootstrap_external.py:426: ImportWarning: Not importing directory /usr/lib/python3/dist-packages/google: missing __init__ [2020-05-05T04:39:15.716Z] _warnings.warn(msg.format(portions[0]), ImportWarning) [2020-05-05T04:39:15.716Z] [gw11] Python 3.6.9 (default, Apr 18 2020, 01:56:04) -- [GCC 8.4.0] [2020-05-05T04:39:15.978Z] [gw9] Python 3.6.9 (default, Apr 18 2020, 01:56:04) -- [GCC 8.4.0] [2020-05-05T04:39:26.015Z] [gw8] node down: Not properly terminated [2020-05-05T04:39:26.015Z] [2020-05-05T04:39:26.015Z] replacing crashed worker gw8 [2020-05-05T04:39:26.015Z] [gw12] linux Python 3.6.9 cwd: /work/mxnet [2020-05-05T04:39:26.015Z] [gw10] node down: Not properly terminated [2020-05-05T04:39:26.015Z] [2020-05-05T04:39:26.015Z] replacing crashed worker gw10 [2020-05-05T04:39:26.015Z] [gw13] linux Python 3.6.9 cwd: /work/mxnet [2020-05-05T04:39:26.015Z] [gw11] node down: Not properly terminated [2020-05-05T04:39:26.015Z] [2020-05-05T04:39:26.015Z] replacing crashed worker gw11 [2020-05-05T04:39:26.015Z] [gw14] linux Python 3.6.9 cwd: /work/mxnet [2020-05-05T04:39:26.015Z] [gw9] node down: Not properly terminated [2020-05-05T04:39:26.015Z] [2020-05-05T04:39:26.015Z] replacing crashed worker gw9 [2020-05-05T04:39:26.015Z] [gw15] linux Python 3.6.9 cwd: /work/mxnet [2020-05-05T04:39:31.276Z] /usr/lib/python3.6/importlib/_bootstrap_external.py:426: ImportWarning: Not importing directory /usr/local/lib/python3.6/dist-packages/google: missing __init__ [2020-05-05T04:39:31.276Z] _warnings.warn(msg.format(portions[0]), ImportWarning) [2020-05-05T04:39:31.276Z] /usr/lib/python3.6/importlib/_bootstrap_external.py:426: ImportWarning: Not importing directory /usr/lib/python3/dist-packages/google: missing __init__ [2020-05-05T04:39:31.276Z] _warnings.warn(msg.format(portions[0]), ImportWarning) [2020-05-05T04:39:31.276Z] /usr/lib/python3.6/importlib/_bootstrap_external.py:426: ImportWarning: Not importing directory /usr/local/lib/python3.6/dist-packages/google: missing __init__ [2020-05-05T04:39:31.276Z] _warnings.warn(msg.format(portions[0]), ImportWarning) [2020-05-05T04:39:31.276Z] /usr/lib/python3.6/importlib/_bootstrap_external.py:426: ImportWarning: Not importing directory /usr/lib/python3/dist-packages/google: missing __init__ [2020-05-05T04:39:31.276Z] _warnings.warn(msg.format(portions[0]), ImportWarning) [2020-05-05T04:39:31.276Z] /usr/lib/python3.6/importlib/_bootstrap_external.py:426: ImportWarning: Not importing directory /usr/local/lib/python3.6/dist-packages/google: missing __init__ [2020-05-05T04:39:31.276Z] _warnings.warn(msg.format(portions[0]), ImportWarning) [2020-05-05T04:39:31.276Z] /usr/lib/python3.6/importlib/_bootstrap_external.py:426: ImportWarning: Not importing directory /usr/lib/python3/dist-packages/google: missing __init__ [2020-05-05T04:39:31.276Z] _warnings.warn(msg.format(portions[0]), ImportWarning) [2020-05-05T04:39:31.276Z] /usr/lib/python3.6/importlib/_bootstrap_external.py:426: ImportWarning: Not importing directory /usr/local/lib/python3.6/dist-packages/google: missing __init__ [2020-05-05T04:39:31.276Z] _warnings.warn(msg.format(portions[0]), ImportWarning) [2020-05-05T04:39:31.276Z] /usr/lib/python3.6/importlib/_bootstrap_external.py:426: ImportWarning: Not importing directory /usr/lib/python3/dist-packages/google: missing __init__ [2020-05-05T04:39:31.276Z] _warnings.warn(msg.format(portions[0]), ImportWarning) [2020-05-05T04:39:31.276Z] [gw12] Python 3.6.9 (default, Apr 18 2020, 01:56:04) -- [GCC 8.4.0] [2020-05-05T04:39:31.276Z] [gw15] Python 3.6.9 (default, Apr 18 2020, 01:56:04) -- [GCC 8.4.0] [2020-05-05T04:39:31.276Z] [gw13] Python 3.6.9 (default, Apr 18 2020, 01:56:04) -- [GCC 8.4.0] [2020-05-05T04:39:31.276Z] [gw14] Python 3.6.9 (default, Apr 18 2020, 01:56:04) -- [GCC 8.4.0] [2020-05-05T04:39:41.264Z] [gw14] node down: Not properly terminated [2020-05-05T04:39:41.264Z] [2020-05-05T04:39:41.264Z] replacing crashed worker gw14 [2020-05-05T04:39:41.264Z] [gw16] linux Python 3.6.9 cwd: /work/mxnet [2020-05-05T04:39:41.264Z] [gw15] node down: Not properly terminated [2020-05-05T04:39:41.264Z] [2020-05-05T04:39:41.264Z] replacing crashed worker gw15 [2020-05-05T04:39:41.264Z] [gw17] linux Python 3.6.9 cwd: /work/mxnet [2020-05-05T04:39:41.264Z] [gw12] node down: Not properly terminated [2020-05-05T04:39:41.264Z] [2020-05-05T04:39:41.264Z] replacing crashed worker gw12 [2020-05-05T04:39:41.264Z] [gw18] linux Python 3.6.9 cwd: /work/mxnet [2020-05-05T04:39:41.264Z] [gw13] node down: Not properly terminated [2020-05-05T04:39:41.264Z] [2020-05-05T04:39:41.264Z] replacing crashed worker gw13 [2020-05-05T04:39:41.264Z] [gw19] linux Python 3.6.9 cwd: /work/mxnet [2020-05-05T04:39:43.786Z] /usr/lib/python3.6/importlib/_bootstrap_external.py:426: ImportWarning: Not importing directory /usr/local/lib/python3.6/dist-packages/google: missing __init__ [2020-05-05T04:39:43.786Z] _warnings.warn(msg.format(portions[0]), ImportWarning) [2020-05-05T04:39:43.786Z] /usr/lib/python3.6/importlib/_bootstrap_external.py:426: ImportWarning: Not importing directory /usr/lib/python3/dist-packages/google: missing __init__ [2020-05-05T04:39:43.786Z] _warnings.warn(msg.format(portions[0]), ImportWarning) [2020-05-05T04:39:44.042Z] [gw16] Python 3.6.9 (default, Apr 18 2020, 01:56:04) -- [GCC 8.4.0] [2020-05-05T04:39:47.312Z] /usr/lib/python3.6/importlib/_bootstrap_external.py:426: ImportWarning: Not importing directory /usr/local/lib/python3.6/dist-packages/google: missing __init__ [2020-05-05T04:39:47.312Z] _warnings.warn(msg.format(portions[0]), ImportWarning) [2020-05-05T04:39:47.312Z] /usr/lib/python3.6/importlib/_bootstrap_external.py:426: ImportWarning: Not importing directory /usr/lib/python3/dist-packages/google: missing __init__ [2020-05-05T04:39:47.312Z] _warnings.warn(msg.format(portions[0]), ImportWarning) [2020-05-05T04:39:47.312Z] [gw18] Python 3.6.9 (default, Apr 18 2020, 01:56:04) -- [GCC 8.4.0] [2020-05-05T04:39:47.312Z] /usr/lib/python3.6/importlib/_bootstrap_external.py:426: ImportWarning: Not importing directory /usr/local/lib/python3.6/dist-packages/google: missing __init__ [2020-05-05T04:39:47.312Z] _warnings.warn(msg.format(portions[0]), ImportWarning) [2020-05-05T04:39:47.312Z] /usr/lib/python3.6/importlib/_bootstrap_external.py:426: ImportWarning: Not importing directory /usr/lib/python3/dist-packages/google: missing __init__ [2020-05-05T04:39:47.312Z] _warnings.warn(msg.format(portions[0]), ImportWarning) [2020-05-05T04:39:47.312Z] /usr/lib/python3.6/importlib/_bootstrap_external.py:426: ImportWarning: Not importing directory /usr/local/lib/python3.6/dist-packages/google: missing __init__ [2020-05-05T04:39:47.312Z] _warnings.warn(msg.format(portions[0]), ImportWarning) [2020-05-05T04:39:47.312Z] /usr/lib/python3.6/importlib/_bootstrap_external.py:426: ImportWarning: Not importing directory /usr/lib/python3/dist-packages/google: missing __init__ [2020-05-05T04:39:47.312Z] _warnings.warn(msg.format(portions[0]), ImportWarning) [2020-05-05T04:39:47.312Z] [gw17] Python 3.6.9 (default, Apr 18 2020, 01:56:04) -- [GCC 8.4.0] [2020-05-05T04:39:47.567Z] [gw19] Python 3.6.9 (default, Apr 18 2020, 01:56:04) -- [GCC 8.4.0] [2020-05-05T04:39:48.501Z] [gw16] node down: Not properly terminated [2020-05-05T04:39:48.501Z] [2020-05-05T04:39:48.501Z] maximum crashed workers reached: 16 [2020-05-05T04:39:53.790Z] [gw18] node down: Not properly terminated [2020-05-05T04:39:53.790Z] [2020-05-05T04:39:53.790Z] maximum crashed workers reached: 16 [2020-05-05T04:39:53.790Z] [gw17] node down: Not properly terminated [2020-05-05T04:39:53.790Z] [2020-05-05T04:39:53.790Z] maximum crashed workers reached: 16 [2020-05-05T04:39:54.719Z] [gw19] node down: Not properly terminated [2020-05-05T04:39:54.719Z] [2020-05-05T04:39:54.719Z] maximum crashed workers reached: 16 [2020-05-05T04:39:54.719Z] [2020-05-05T04:39:54.719Z] ================== xdist: maximum crashed workers reached: 16 ================== ``` ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
