Hi Gautam, I see that ‘nosetests’ is the command used to run python2 tests. It looks like that’s being mapped to use python3. I’ve checked that this is the case on my Ubuntu instance. I need to use ‘nosetests-2.7’ to use python2 for the tests. Please check if this fix works in the build environment (slave/docker container) as well.
The PR you refer to only parallelized tests that were running one after the other, this command was being used even before that PR. Regards, Rahul On Wed, 27 Sep 2017 at 22:46 Gautam <[email protected]> wrote: > Hi Ozawa, > > Thanks for follow up. > Unfortunately I didn't get time to work on this today. > > However I have couple of points to mentions. > 1. Looks like this backtrace has been present since long time, since this > was not a test failure or build failure we never got notified about it. > Here > < > https://builds.apache.org/view/Incubator%20Projects/job/incubator-mxnet/job/master/448/consoleFull > > > is the recent build log where back trace is present but build succeeded. > > 2. I don't think the default version of python on Ubuntu is 3.0, I logged > into one of the apache slave and the default version of Python is 2.7.6 > > 3. There has been slight change > <https://github.com/apache/incubator-mxnet/pull/7658> in Jenkins file > where > we tried to parallelize python2 and 3 test run. I am not sure if it > affects. I can probably scrub the build log and figure out if thats the > case. > > > Feel free to send the PR, if you have it ready. > > > -Gautam > > > On Wed, Sep 27, 2017 at 9:39 PM, Tsuyoshi Ozawa <[email protected]> wrote: > > > Hi Kumar, > > > > Thanks for looking into the issue. How is the progress of this problem? > > Shouldn't we call /usr/bin/env python2 or python2.7 in following > > source code instead of python since MXNet only supports python2 > > currently? > > I think default version of python in Ubuntu is now python3, so it can > > cause the problem. > > If you have not yet done the work, I can create a PR for that in this > > weekend. > > > > ./python/mxnet/__init__.py:#!/usr/bin/env python > > ./python/mxnet/log.py:#!/usr/bin/env python > > ./tests/nightly/dist_lenet.py:#!/usr/bin/env python > > ./tests/nightly/dist_sync_kvstore.py:#!/usr/bin/env python > > ./tests/nightly/multi_lenet.py:#!/usr/bin/env python > > ./tests/nightly/test_kvstore.py:#!/usr/bin/env python > > ./tools/coreml/mxnet_coreml_converter.py:#!/usr/bin/env python > > ./tools/ipynb2md.py:#!/usr/bin/env python > > ./tools/kill-mxnet.py:#!/usr/bin/env python > > ./tools/launch.py:#!/usr/bin/env python > > ./tools/parse_log.py:#!/usr/bin/env python > > > > On Wed, Sep 27, 2017 at 5:39 PM, Sunderland, Kellen <[email protected]> > > wrote: > > > Many thanks Gautam. > > > > > > On 9/26/17, 8:37 PM, "Kumar, Gautam" <[email protected]> wrote: > > > > > > Hi Kellen, > > > > > > This issue has been happening since last 3-4 days along with few > > other test failure. > > > I am looking into it. > > > > > > -Gautam > > > > > > On 9/26/17, 7:45 AM, "Sunderland, Kellen" <[email protected]> > wrote: > > > > > > I’ve been noticing in a few failed builds that the stack trace > > indicates we’re actually running python 3.4 in the python 2 tests. I know > > the CI folks are working hard getting everything setup, is this a known > > issue for the CI team? > > > > > > For example: https://builds.apache.org/ > > blue/organizations/jenkins/incubator-mxnet/detail/PR-8026/3/pipeline/281 > > > > > > Steps Python2: MKLML-CPU > > > > > > StackTrace: > > > Stack trace returned 10 entries: > > > [bt] (0) /workspace/python/mxnet/../../lib/libmxnet.so(_ > > ZN4dmlc15LogMessageFatalD1Ev+0x3c) [0x7fadb8999aac] > > > [bt] (1) /workspace/python/mxnet/../../lib/libmxnet.so(_ > > ZN5mxnet7kvstore12KVStoreLocal12GroupKVPairsISt4pairIPNS_ > > > 7NDArrayES4_EZNS1_19GroupKVPairsPullRspERKSt6vectorIiSaIiEERKS7_IS6_SaIS6_ > > > EEPS9_PS7_ISD_SaISD_EEEUliRKS6_E_EEvSB_RKS7_IT_SaISN_EESG_PS7_ISP_SaISP_EERKT0_+0x56b) > > [0x7fadba32c01b] > > > [bt] (2) /workspace/python/mxnet/../../lib/libmxnet.so(_ > > ZN5mxnet7kvstore12KVStoreLocal17PullRowSparseImplERKSt6vecto > > rIiSaIiEERKS2_ISt4pairIPNS_7NDArrayES8_ESaISA_EEi+0xa6) [0x7fadba32c856] > > > [bt] (3) > /workspace/python/mxnet/../../lib/libmxnet.so(MXKVStorePullRowSparse+0x245) > > [0x7fadba18f165] > > > [bt] (4) > /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call_unix64+0x4c) > > [0x7fadde26cadc] > > > [bt] (5) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call+0x1fc) > > [0x7fadde26c40c] > > > [bt] (6) /usr/lib/python3.4/lib-dynload/_ctypes.cpython-34m- > > x86_64-linux-gnu.so(_ctypes_callproc+0x21d) [0x7fadde47e12d] > > > [bt] (7) /usr/lib/p > <https://maps.google.com/?q=(7)+/usr/lib/p&entry=gmail&source=g> > ython3.4/lib-dynload/_ctypes.cpython-34m- > > x86_64-linux-gnu.so(+0xf6a3) [0x7fadde47e6a3] > > > [bt] (8) /usr/bin/python3(PyEval_EvalFrameEx+0x41d7) [0x48a487] > > > [bt] (9) /usr/bin/python3() [0x48f2df] > > > > > > -Kellen > > > Amazon Development Center Germany GmbH > > > Berlin - Dresden - Aachen > > > main office: Krausenstr. 38, 10117 Berlin > > > Geschaeftsfuehrer: Dr. Ralf Herbrich, Christian Schlaeger > > > Ust-ID: DE289237879 > > > Eingetragen am Amtsgericht Charlottenburg HRB 149173 B > > > > > > > > > > > > > > > Amazon Development Center Germany GmbH > > > Berlin - Dresden - Aachen > > > main office: Krausenstr. 38, 10117 Berlin > > > Geschaeftsfuehrer: Dr. Ralf Herbrich, Christian Schlaeger > > > Ust-ID: DE289237879 > > > Eingetragen am Amtsgericht Charlottenburg HRB 149173 B > > > > > > -- > Best Regards, > Gautam Kumar >
