eric-haibin-lin opened a new issue #17562: Fail to import MXNet in distributed kvstore test URL: https://github.com/apache/incubator-mxnet/issues/17562 The distributed kvstore test has been failing on the master branch: ``` [2020-02-10T20:44:37.605Z] + ../../tools/launch.py -n 4 --launcher local python dist_device_sync_kvstore.py [2020-02-10T20:44:38.537Z] Traceback (most recent call last): [2020-02-10T20:44:38.537Z] File "dist_device_sync_kvstore.py", line 22, in <module> [2020-02-10T20:44:38.537Z] import mxnet as mx [2020-02-10T20:44:38.537Z] File "../../python/mxnet/__init__.py", line 34, in <module> [2020-02-10T20:44:38.537Z] from . import contrib [2020-02-10T20:44:38.537Z] File "../../python/mxnet/contrib/__init__.py", line 30, in <module> [2020-02-10T20:44:38.537Z] from . import text [2020-02-10T20:44:38.537Z] File "../../python/mxnet/contrib/text/__init__.py", line 23, in <module> [2020-02-10T20:44:38.537Z] from . import embedding [2020-02-10T20:44:38.537Z] File "../../python/mxnet/contrib/text/embedding.py", line 39, in <module> [2020-02-10T20:44:38.537Z] from ... import numpy as _mx_np [2020-02-10T20:44:38.537Z] File "../../python/mxnet/numpy/__init__.py", line 24, in <module> [2020-02-10T20:44:38.537Z] from .multiarray import * # pylint: disable=wildcard-import [2020-02-10T20:44:38.537Z] File "../../python/mxnet/numpy/multiarray.py", line 49, in <module> [2020-02-10T20:44:38.537Z] from .fallback import * # pylint: disable=wildcard-import,unused-wildcard-import [2020-02-10T20:44:38.537Z] File "../../python/mxnet/numpy/fallback.py", line 29, in <module> [2020-02-10T20:44:38.537Z] heaviside = onp.heaviside [2020-02-10T20:44:38.537Z] AttributeError: 'module' object has no attribute 'heaviside' [2020-02-10T20:44:38.537Z] Traceback (most recent call last): [2020-02-10T20:44:38.537Z] File "dist_device_sync_kvstore.py", line 22, in <module> [2020-02-10T20:44:38.537Z] import mxnet as mx [2020-02-10T20:44:38.537Z] File "../../python/mxnet/__init__.py", line 34, in <module> [2020-02-10T20:44:38.537Z] from . import contrib [2020-02-10T20:44:38.537Z] File "../../python/mxnet/contrib/__init__.py", line 30, in <module> [2020-02-10T20:44:38.537Z] from . import text [2020-02-10T20:44:38.537Z] File "../../python/mxnet/contrib/text/__init__.py", line 23, in <module> [2020-02-10T20:44:38.537Z] from . import embedding [2020-02-10T20:44:38.537Z] File "../../python/mxnet/contrib/text/embedding.py", line 39, in <module> [2020-02-10T20:44:38.537Z] from ... import numpy as _mx_np [2020-02-10T20:44:38.537Z] File "../../python/mxnet/numpy/__init__.py", line 24, in <module> [2020-02-10T20:44:38.537Z] from .multiarray import * # pylint: disable=wildcard-import [2020-02-10T20:44:38.537Z] File "../../python/mxnet/numpy/multiarray.py", line 49, in <module> [2020-02-10T20:44:38.537Z] from .fallback import * # pylint: disable=wildcard-import,unused-wildcard-import [2020-02-10T20:44:38.537Z] File "../../python/mxnet/numpy/fallback.py", line 29, in <module> [2020-02-10T20:44:38.537Z] heaviside = onp.heaviside [2020-02-10T20:44:38.537Z] AttributeError: 'module' object has no attribute 'heaviside' [2020-02-10T20:44:38.537Z] Traceback (most recent call last): [2020-02-10T20:44:38.537Z] File "dist_device_sync_kvstore.py", line 22, in <module> [2020-02-10T20:44:38.537Z] import mxnet as mx [2020-02-10T20:44:38.537Z] File "../../python/mxnet/__init__.py", line 34, in <module> [2020-02-10T20:44:38.537Z] from . import contrib [2020-02-10T20:44:38.537Z] File "../../python/mxnet/contrib/__init__.py", line 30, in <module> [2020-02-10T20:44:38.537Z] from . import text [2020-02-10T20:44:38.537Z] File "../../python/mxnet/contrib/text/__init__.py", line 23, in <module> [2020-02-10T20:44:38.537Z] from . import embedding [2020-02-10T20:44:38.537Z] File "../../python/mxnet/contrib/text/embedding.py", line 39, in <module> [2020-02-10T20:44:38.537Z] from ... import numpy as _mx_np [2020-02-10T20:44:38.537Z] File "../../python/mxnet/numpy/__init__.py", line 24, in <module> [2020-02-10T20:44:38.537Z] from .multiarray import * # pylint: disable=wildcard-import [2020-02-10T20:44:38.537Z] File "../../python/mxnet/numpy/multiarray.py", line 49, in <module> [2020-02-10T20:44:38.537Z] from .fallback import * # pylint: disable=wildcard-import,unused-wildcard-import [2020-02-10T20:44:38.537Z] File "../../python/mxnet/numpy/fallback.py", line 29, in <module> [2020-02-10T20:44:38.537Z] heaviside = onp.heaviside [2020-02-10T20:44:38.537Z] AttributeError: 'module' object has no attribute 'heaviside' [2020-02-10T20:44:38.537Z] Traceback (most recent call last): [2020-02-10T20:44:38.537Z] File "dist_device_sync_kvstore.py", line 22, in <module> [2020-02-10T20:44:38.537Z] import mxnet as mx [2020-02-10T20:44:38.537Z] File "../../python/mxnet/__init__.py", line 34, in <module> [2020-02-10T20:44:38.537Z] from . import contrib [2020-02-10T20:44:38.537Z] File "../../python/mxnet/contrib/__init__.py", line 30, in <module> [2020-02-10T20:44:38.537Z] from . import text [2020-02-10T20:44:38.537Z] File "../../python/mxnet/contrib/text/__init__.py", line 23, in <module> [2020-02-10T20:44:38.537Z] from . import embedding [2020-02-10T20:44:38.537Z] File "../../python/mxnet/contrib/text/embedding.py", line 39, in <module> [2020-02-10T20:44:38.537Z] from ... import numpy as _mx_np [2020-02-10T20:44:38.537Z] File "../../python/mxnet/numpy/__init__.py", line 24, in <module> [2020-02-10T20:44:38.537Z] from .multiarray import * # pylint: disable=wildcard-import [2020-02-10T20:44:38.537Z] File "../../python/mxnet/numpy/multiarray.py", line 49, in <module> [2020-02-10T20:44:38.537Z] from .fallback import * # pylint: disable=wildcard-import,unused-wildcard-import [2020-02-10T20:44:38.537Z] File "../../python/mxnet/numpy/fallback.py", line 29, in <module> [2020-02-10T20:44:38.537Z] heaviside = onp.heaviside [2020-02-10T20:44:38.537Z] AttributeError: 'module' object has no attribute 'heaviside' [2020-02-10T20:44:38.537Z] Exception in thread Thread-2: [2020-02-10T20:44:38.537Z] Traceback (most recent call last): [2020-02-10T20:44:38.537Z] File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner [2020-02-10T20:44:38.537Z] self.run() [2020-02-10T20:44:38.537Z] File "/usr/lib/python2.7/threading.py", line 754, in run [2020-02-10T20:44:38.537Z] self.__target(*self.__args, **self.__kwargs) [2020-02-10T20:44:38.537Z] File "/work/mxnet/tools/../3rdparty/dmlc-core/tracker/dmlc_tracker/local.py", line 49, in exec_cmd [2020-02-10T20:44:38.537Z] raise RuntimeError('Get nonzero return code=%d on %s %s' % (ret, cmd, env)) [2020-02-10T20:44:38.537Z] RuntimeError: Get nonzero return code=1 on ['python', 'dist_device_sync_kvstore.py'] {'MXNET_SUBGRAPH_VERBOSE': '0', 'CCACHE_TEMPDIR': '/tmp/ccache', 'BUILD_TAG': 'jenkins-mxnet-validation-unix-gpu-master-1622', 'CCACHE_MAXSIZE': '500G', 'CUDA_VERSION': '10.1.105', 'HOME': '/home/jenkins_slave', 'CCACHE_LOGFILE': '/tmp/ccache.log', 'LD_LIBRARY_PATH': '/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/usr/local/cuda/compat', 'DMLC_LOG_STACK_TRACE_DEPTH': '10', 'LIBRARY_PATH': '/usr/local/cuda/lib64/stubs', 'CUDA_PKG_VERSION': '10-1=10.1.243-1', 'SHLVL': '1', 'DMLC_PS_ROOT_URI': '172.17.0.2', 'NCCL_VERSION': '2.4.8', 'NVIDIA_VISIBLE_DEVICES': 'all', 'CUDNN_VERSION': '7.6.5.32', 'DMLC_TASK_ID': '0', 'BUILD_ID': '1622', 'BUILD_NUMBER': '1622', 'PYTHONPATH': './python/', 'DMLC_PS_ROOT_PORT': '9091', 'PATH': '/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'DMLC_NUM_WORKER': '4', 'NVIDIA_DRIVER_CAPABILITIES': 'compute,utility', '_': '../../tools/launch.py', 'NVIDIA_REQUIRE_CUDA': 'cuda>=10.1 brand=tesla,driver>=384,driver<385 brand=tesla,driver>=396,driver<397 brand=tesla,driver>=410,driver<411', 'OLDPWD': '/work/mxnet', 'HOSTNAME': 'a9f40a381df5', 'MXNET_STORAGE_FALLBACK_LOG_VERBOSE': '0', 'DMLC_ROLE': 'worker', 'DMLC_JOB_CLUSTER': 'local', 'PWD': '/work/mxnet/tests/nightly', 'DMLC_NUM_SERVER': '4', 'CCACHE_DIR': '/work/ccache'} ``` CI reports test passes by mistake. MXNet actually cannot be imported due to some changed from mxnet.numpy. http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Funix-gpu/detail/master/1622/pipeline @haojin2 have you seen this?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
