Zha0q1 opened a new issue #19948: URL: https://github.com/apache/incubator-mxnet/issues/19948
This issue start to happen after we use the new ami for restricted-mxnetlinux-gpu. https://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/restricted-mxnet-cd%2Fmxnet-cd-release-job-1.x/detail/mxnet-cd-release-job-1.x/1553/pipeline I was able to reproduce by basically building the same image as in the cd pipeline, using the same g3 instance and the same ami ``` docker build -f docker/Dockerfile.build.ubuntu_gpu_cu102 --build-arg USER_ID=1001 --build-arg GROUP_ID=1001 --cache-from 021742426385.dkr.ecr.us-west-2.amazonaws.com/mxnet-ci:build.ubuntu_gpu_cu102-81dcd5660530 -t 021742426385.dkr.ecr.us-west-2.amazonaws.com/mxnet-ci:build.ubuntu_gpu_cu102-81dcd5660530 docker ``` after entering the docker container I did ``` pip3 install mxnet-cu102 ``` I was able to reproduce the exact error by running ```python >>> import mxnet >>> import mxnet as mx >>> ctx = mx.gpu(0) >>> a = mx.nd.ones((100), ctx=ctx) ``` ``` [02:55:03] src/base.cc:49: GPU context requested, but no GPUs found. Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/local/lib/python3.7/dist-packages/mxnet/ndarray/ndarray.py", line 3295, in ones return _internal._ones(shape=shape, ctx=ctx, dtype=dtype, **kwargs) File "<string>", line 39, in _ones File "/usr/local/lib/python3.7/dist-packages/mxnet/_ctypes/ndarray.py", line 91, in _imperative_invoke ctypes.byref(out_stypes))) File "/usr/local/lib/python3.7/dist-packages/mxnet/base.py", line 246, in check_call raise get_last_ffi_error() mxnet.base.MXNetError: Traceback (most recent call last): File "src/engine/threaded_engine.cc", line 331 MXNetError: Check failed: device_count_ > 0 (-1 vs. 0) : GPU usage requires at least 1 GPU ``` ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@mxnet.apache.org For additional commands, e-mail: issues-h...@mxnet.apache.org
