Hi Rahul, Thanks for sharing the information and taking on the problem! I will also try the option(using nosetests-2.7 in Jenkinsfile).
Regards, - Tsuyoshi On Thu, Sep 28, 2017 at 3:34 PM, Rahul Huilgol <[email protected]> wrote: > Hi Gautam, > > I see that ‘nosetests’ is the command used to run python2 tests. It looks > like that’s being mapped to use python3. I’ve checked that this is the case > on my Ubuntu instance. I need to use ‘nosetests-2.7’ to use python2 for the > tests. Please check if this fix works in the build environment > (slave/docker container) as well. > > The PR you refer to only parallelized tests that were running one after the > other, this command was being used even before that PR. > > Regards, > Rahul > > On Wed, 27 Sep 2017 at 22:46 Gautam <[email protected]> wrote: > >> Hi Ozawa, >> >> Thanks for follow up. >> Unfortunately I didn't get time to work on this today. >> >> However I have couple of points to mentions. >> 1. Looks like this backtrace has been present since long time, since this >> was not a test failure or build failure we never got notified about it. >> Here >> < >> https://builds.apache.org/view/Incubator%20Projects/job/incubator-mxnet/job/master/448/consoleFull >> > >> is the recent build log where back trace is present but build succeeded. >> >> 2. I don't think the default version of python on Ubuntu is 3.0, I logged >> into one of the apache slave and the default version of Python is 2.7.6 >> >> 3. There has been slight change >> <https://github.com/apache/incubator-mxnet/pull/7658> in Jenkins file >> where >> we tried to parallelize python2 and 3 test run. I am not sure if it >> affects. I can probably scrub the build log and figure out if thats the >> case. >> >> >> Feel free to send the PR, if you have it ready. >> >> >> -Gautam >> >> >> On Wed, Sep 27, 2017 at 9:39 PM, Tsuyoshi Ozawa <[email protected]> wrote: >> >> > Hi Kumar, >> > >> > Thanks for looking into the issue. How is the progress of this problem? >> > Shouldn't we call /usr/bin/env python2 or python2.7 in following >> > source code instead of python since MXNet only supports python2 >> > currently? >> > I think default version of python in Ubuntu is now python3, so it can >> > cause the problem. >> > If you have not yet done the work, I can create a PR for that in this >> > weekend. >> > >> > ./python/mxnet/__init__.py:#!/usr/bin/env python >> > ./python/mxnet/log.py:#!/usr/bin/env python >> > ./tests/nightly/dist_lenet.py:#!/usr/bin/env python >> > ./tests/nightly/dist_sync_kvstore.py:#!/usr/bin/env python >> > ./tests/nightly/multi_lenet.py:#!/usr/bin/env python >> > ./tests/nightly/test_kvstore.py:#!/usr/bin/env python >> > ./tools/coreml/mxnet_coreml_converter.py:#!/usr/bin/env python >> > ./tools/ipynb2md.py:#!/usr/bin/env python >> > ./tools/kill-mxnet.py:#!/usr/bin/env python >> > ./tools/launch.py:#!/usr/bin/env python >> > ./tools/parse_log.py:#!/usr/bin/env python >> > >> > On Wed, Sep 27, 2017 at 5:39 PM, Sunderland, Kellen <[email protected]> >> > wrote: >> > > Many thanks Gautam. >> > > >> > > On 9/26/17, 8:37 PM, "Kumar, Gautam" <[email protected]> wrote: >> > > >> > > Hi Kellen, >> > > >> > > This issue has been happening since last 3-4 days along with few >> > other test failure. >> > > I am looking into it. >> > > >> > > -Gautam >> > > >> > > On 9/26/17, 7:45 AM, "Sunderland, Kellen" <[email protected]> >> wrote: >> > > >> > > I’ve been noticing in a few failed builds that the stack trace >> > indicates we’re actually running python 3.4 in the python 2 tests. I know >> > the CI folks are working hard getting everything setup, is this a known >> > issue for the CI team? >> > > >> > > For example: https://builds.apache.org/ >> > blue/organizations/jenkins/incubator-mxnet/detail/PR-8026/3/pipeline/281 >> > > >> > > Steps Python2: MKLML-CPU >> > > >> > > StackTrace: >> > > Stack trace returned 10 entries: >> > > [bt] (0) /workspace/python/mxnet/../../lib/libmxnet.so(_ >> > ZN4dmlc15LogMessageFatalD1Ev+0x3c) [0x7fadb8999aac] >> > > [bt] (1) /workspace/python/mxnet/../../lib/libmxnet.so(_ >> > ZN5mxnet7kvstore12KVStoreLocal12GroupKVPairsISt4pairIPNS_ >> > >> 7NDArrayES4_EZNS1_19GroupKVPairsPullRspERKSt6vectorIiSaIiEERKS7_IS6_SaIS6_ >> > >> EEPS9_PS7_ISD_SaISD_EEEUliRKS6_E_EEvSB_RKS7_IT_SaISN_EESG_PS7_ISP_SaISP_EERKT0_+0x56b) >> > [0x7fadba32c01b] >> > > [bt] (2) /workspace/python/mxnet/../../lib/libmxnet.so(_ >> > ZN5mxnet7kvstore12KVStoreLocal17PullRowSparseImplERKSt6vecto >> > rIiSaIiEERKS2_ISt4pairIPNS_7NDArrayES8_ESaISA_EEi+0xa6) [0x7fadba32c856] >> > > [bt] (3) >> /workspace/python/mxnet/../../lib/libmxnet.so(MXKVStorePullRowSparse+0x245) >> > [0x7fadba18f165] >> > > [bt] (4) >> /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call_unix64+0x4c) >> > [0x7fadde26cadc] >> > > [bt] (5) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call+0x1fc) >> > [0x7fadde26c40c] >> > > [bt] (6) /usr/lib/python3.4/lib-dynload/_ctypes.cpython-34m- >> > x86_64-linux-gnu.so(_ctypes_callproc+0x21d) [0x7fadde47e12d] >> > > [bt] (7) /usr/lib/p >> <https://maps.google.com/?q=(7)+/usr/lib/p&entry=gmail&source=g> >> ython3.4/lib-dynload/_ctypes.cpython-34m- >> > x86_64-linux-gnu.so(+0xf6a3) [0x7fadde47e6a3] >> > > [bt] (8) /usr/bin/python3(PyEval_EvalFrameEx+0x41d7) [0x48a487] >> > > [bt] (9) /usr/bin/python3() [0x48f2df] >> > > >> > > -Kellen >> > > Amazon Development Center Germany GmbH >> > > Berlin - Dresden - Aachen >> > > main office: Krausenstr. 38, 10117 Berlin >> > > Geschaeftsfuehrer: Dr. Ralf Herbrich, Christian Schlaeger >> > > Ust-ID: DE289237879 >> > > Eingetragen am Amtsgericht Charlottenburg HRB 149173 B >> > > >> > > >> > > >> > > >> > > Amazon Development Center Germany GmbH >> > > Berlin - Dresden - Aachen >> > > main office: Krausenstr. 38, 10117 Berlin >> > > Geschaeftsfuehrer: Dr. Ralf Herbrich, Christian Schlaeger >> > > Ust-ID: DE289237879 >> > > Eingetragen am Amtsgericht Charlottenburg HRB 149173 B >> > >> >> >> >> -- >> Best Regards, >> Gautam Kumar >>
