Hi Ozawa, Thanks for follow up. Unfortunately I didn't get time to work on this today.
However I have couple of points to mentions. 1. Looks like this backtrace has been present since long time, since this was not a test failure or build failure we never got notified about it. Here <https://builds.apache.org/view/Incubator%20Projects/job/incubator-mxnet/job/master/448/consoleFull> is the recent build log where back trace is present but build succeeded. 2. I don't think the default version of python on Ubuntu is 3.0, I logged into one of the apache slave and the default version of Python is 2.7.6 3. There has been slight change <https://github.com/apache/incubator-mxnet/pull/7658> in Jenkins file where we tried to parallelize python2 and 3 test run. I am not sure if it affects. I can probably scrub the build log and figure out if thats the case. Feel free to send the PR, if you have it ready. -Gautam On Wed, Sep 27, 2017 at 9:39 PM, Tsuyoshi Ozawa <[email protected]> wrote: > Hi Kumar, > > Thanks for looking into the issue. How is the progress of this problem? > Shouldn't we call /usr/bin/env python2 or python2.7 in following > source code instead of python since MXNet only supports python2 > currently? > I think default version of python in Ubuntu is now python3, so it can > cause the problem. > If you have not yet done the work, I can create a PR for that in this > weekend. > > ./python/mxnet/__init__.py:#!/usr/bin/env python > ./python/mxnet/log.py:#!/usr/bin/env python > ./tests/nightly/dist_lenet.py:#!/usr/bin/env python > ./tests/nightly/dist_sync_kvstore.py:#!/usr/bin/env python > ./tests/nightly/multi_lenet.py:#!/usr/bin/env python > ./tests/nightly/test_kvstore.py:#!/usr/bin/env python > ./tools/coreml/mxnet_coreml_converter.py:#!/usr/bin/env python > ./tools/ipynb2md.py:#!/usr/bin/env python > ./tools/kill-mxnet.py:#!/usr/bin/env python > ./tools/launch.py:#!/usr/bin/env python > ./tools/parse_log.py:#!/usr/bin/env python > > On Wed, Sep 27, 2017 at 5:39 PM, Sunderland, Kellen <[email protected]> > wrote: > > Many thanks Gautam. > > > > On 9/26/17, 8:37 PM, "Kumar, Gautam" <[email protected]> wrote: > > > > Hi Kellen, > > > > This issue has been happening since last 3-4 days along with few > other test failure. > > I am looking into it. > > > > -Gautam > > > > On 9/26/17, 7:45 AM, "Sunderland, Kellen" <[email protected]> wrote: > > > > I’ve been noticing in a few failed builds that the stack trace > indicates we’re actually running python 3.4 in the python 2 tests. I know > the CI folks are working hard getting everything setup, is this a known > issue for the CI team? > > > > For example: https://builds.apache.org/ > blue/organizations/jenkins/incubator-mxnet/detail/PR-8026/3/pipeline/281 > > > > Steps Python2: MKLML-CPU > > > > StackTrace: > > Stack trace returned 10 entries: > > [bt] (0) /workspace/python/mxnet/../../lib/libmxnet.so(_ > ZN4dmlc15LogMessageFatalD1Ev+0x3c) [0x7fadb8999aac] > > [bt] (1) /workspace/python/mxnet/../../lib/libmxnet.so(_ > ZN5mxnet7kvstore12KVStoreLocal12GroupKVPairsISt4pairIPNS_ > 7NDArrayES4_EZNS1_19GroupKVPairsPullRspERKSt6vectorIiSaIiEERKS7_IS6_SaIS6_ > EEPS9_PS7_ISD_SaISD_EEEUliRKS6_E_EEvSB_RKS7_IT_SaISN_EESG_PS7_ISP_SaISP_EERKT0_+0x56b) > [0x7fadba32c01b] > > [bt] (2) /workspace/python/mxnet/../../lib/libmxnet.so(_ > ZN5mxnet7kvstore12KVStoreLocal17PullRowSparseImplERKSt6vecto > rIiSaIiEERKS2_ISt4pairIPNS_7NDArrayES8_ESaISA_EEi+0xa6) [0x7fadba32c856] > > [bt] (3) > > /workspace/python/mxnet/../../lib/libmxnet.so(MXKVStorePullRowSparse+0x245) > [0x7fadba18f165] > > [bt] (4) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call_unix64+0x4c) > [0x7fadde26cadc] > > [bt] (5) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call+0x1fc) > [0x7fadde26c40c] > > [bt] (6) /usr/lib/python3.4/lib-dynload/_ctypes.cpython-34m- > x86_64-linux-gnu.so(_ctypes_callproc+0x21d) [0x7fadde47e12d] > > [bt] (7) /usr/lib/python3.4/lib-dynload/_ctypes.cpython-34m- > x86_64-linux-gnu.so(+0xf6a3) [0x7fadde47e6a3] > > [bt] (8) /usr/bin/python3(PyEval_EvalFrameEx+0x41d7) [0x48a487] > > [bt] (9) /usr/bin/python3() [0x48f2df] > > > > -Kellen > > Amazon Development Center Germany GmbH > > Berlin - Dresden - Aachen > > main office: Krausenstr. 38, 10117 Berlin > > Geschaeftsfuehrer: Dr. Ralf Herbrich, Christian Schlaeger > > Ust-ID: DE289237879 > > Eingetragen am Amtsgericht Charlottenburg HRB 149173 B > > > > > > > > > > Amazon Development Center Germany GmbH > > Berlin - Dresden - Aachen > > main office: Krausenstr. 38, 10117 Berlin > > Geschaeftsfuehrer: Dr. Ralf Herbrich, Christian Schlaeger > > Ust-ID: DE289237879 > > Eingetragen am Amtsgericht Charlottenburg HRB 149173 B > -- Best Regards, Gautam Kumar
