[
https://issues.apache.org/jira/browse/AURORA-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15434791#comment-15434791
]
Stephan Erb commented on AURORA-1717:
-------------------------------------
I have seen the issue a couple of times over the last fewdays and have started
to investigate once more:
Our tests are started via `./pants test.pytest` and are then calling `./pants
binary` within some test setup routines. Looks like that the `PYTHONPATH` can
be tainted for the second run. Unsetting it seems to prevent the test failures.
Specifically, `PYTHONPATH` is not set when calling pants for the first time,
but filled with the following when the second invocation is performed by the
tests:
{code}
PYTHONPATH=/home/jenkins/jenkins-slave/workspace/AuroraDebug/src/test/python/apache/thermos/common:/home/jenkins/jenkins-slave/workspace/AuroraDebug/src/test/python/apache/aurora/common:/home/jenkins/jenkins-slave/workspace/AuroraDebug/src/test/python/apache:/home/jenkins/jenkins-slave/workspace/AuroraDebug/src/test/python/apache/thermos/observer:/home/jenkins/jenkins-slave/workspace/AuroraDebug/src/test/python/apache/aurora/executor:/home/jenkins/jenkins-slave/workspace/AuroraDebug/src/test/python/apache:/home/jenkins/jenkins-slave/workspace/AuroraDebug/src/test/python/apache/thermos/cli:/home/jenkins/jenkins-slave/workspace/AuroraDebug/src/test/python/apache/thermos/cli/commands:/home/jenkins/jenkins-slave/workspace/AuroraDebug/src/test/python/apache/aurora/common:/home/jenkins/jenkins-slave/workspace/AuroraDebug/src/test/python/apache/aurora/executor:/home/jenkins/jenkins-slave/workspace/AuroraDebug/src/test/python/apache/thermos/config:/home/jenkins/jenkins-slave/workspace/AuroraDebug/src/test/python/apache/aurora/executor/bin:/home/jenkins/jenkins-slave/workspace/AuroraDebug/src/test/python/apache/aurora/config:/home/jenkins/jenkins-slave/workspace/AuroraDebug/src/test/python/apache/aurora/client/hooks:/home/jenkins/jenkins-slave/workspace/AuroraDebug/src/test/python/apache/thermos/monitoring:/home/jenkins/jenkins-slave/workspace/AuroraDebug/src/test/python/apache:/home/jenkins/jenkins-slave/workspace/AuroraDebug/src/test/python/apache/aurora/tools:/home/jenkins/jenkins-slave/workspace/AuroraDebug/src/test/python/apache/thermos/observer/http:/home/jenkins/jenkins-slave/workspace/AuroraDebug/src/test/python/apache/thermos/core:/home/jenkins/jenkins-slave/workspace/AuroraDebug/.pants.d/python-setup/chroots/965c8509f886522de50b650745d54c840b4e86c4/.bootstrap:/home/jenkins/jenkins-slave/workspace/AuroraDebug/.pants.d/python-setup/chroots/965c8509f886522de50b650745d54c840b4e86c4:/usr/lib/python2.7:/usr/lib/python2.7/plat-x86_64-linux-gnu:/usr/lib/python2.7/lib-tk:/usr/lib/python2.7/lib-old:/usr/lib/python2.7/lib-dynload:/home/jenkins/jenkins-slave/workspace/AuroraDebug/.pants.d/python-setup/chroots/965c8509f886522de50b650745d54c840b4e86c4/.deps/twitter.common.quantity-0.3.3-py2-none-any.whl:/home/jenkins/jenkins-slave/workspace/AuroraDebug/.pants.d/python-setup/chroots/965c8509f886522de50b650745d54c840b4e86c4/.deps/twitter.common.string-0.3.3-py2-none-any.whl:/home/jenkins/jenkins-slave/workspace/AuroraDebug/.pants.d/python-setup/chroots/965c8509f886522de50b650745d54c840b4e86c4/.deps/api.src.main.thrift.org.apache.aurora.gen._test_e269e4ad-0.0.0-py2-none-any.whl:/home/jenkins/jenkins-slave/workspace/AuroraDebug/.pants.d/python-setup/chroots/965c8509f886522de50b650745d54c840b4e86c4/.deps/bottle-0.11.6-py2.py3-none-any.whl:/home/jenkins/jenkins-slave/workspace/AuroraDebug/.pants.d/python-setup/chroots/965c8509f886522de50b650745d54c840b4e86c4/.deps/pystachio-0.8.1-py2.py3-none-any.whl:/home/jenkins/jenkins-slave/workspace/AuroraDebug/.pants.d/python-setup/chroots/965c8509f886522de50b650745d54c840b4e86c4/.deps/twitter.common.exceptions-0.3.3-py2-none-any.whl:/home/jenkins/jenkins-slave/workspace/AuroraDebug/.pants.d/python-setup/chroots/965c8509f886522de50b650745d54c840b4e86c4/.deps/traceback2-1.4.0-py2.py3-none-any.whl:/home/jenkins/jenkins-slave/workspace/AuroraDebug/.pants.d/python-setup/chroots/965c8509f886522de50b650745d54c840b4e86c4/.deps/zope.interface-4.2.0-cp27-none-linux_x86_64.whl:/home/jenkins/jenkins-slave/workspace/AuroraDebug/.pants.d/python-setup/chroots/965c8509f886522de50b650745d54c840b4e86c4/.deps/pytest_cov-1.8.1-py2-none-any.whl:/home/jenkins/jenkins-slave/workspace/AuroraDebug/.pants.d/python-setup/chroots/965c8509f886522de50b650745d54c840b4e86c4/.deps/pytest-2.6.4-py2-none-any.whl:/home/jenkins/jenkins-slave/workspace/AuroraDebug/.pants.d/python-setup/chroots/965c8509f886522de50b650745d54c840b4e86c4/.deps/api.src.main.thrift.org.apache.thermos.thermos_c09aec64-0.0.0-py2-none-any.whl:/home/jenkins/jenkins-slave/workspace/AuroraDebug/.pants.d/python-setup/chroots/965c8509f886522de50b650745d54c840b4e86c4/.deps/unittest2-1.1.0-py2.py3-none-any.whl:/home/jenkins/jenkins-slave/workspace/AuroraDebug/.pants.d/python-setup/chroots/965c8509f886522de50b650745d54c840b4e86c4/.deps/requests-2.7.0-py2.py3-none-any.whl:/home/jenkins/jenkins-slave/workspace/AuroraDebug/.pants.d/python-setup/chroots/965c8509f886522de50b650745d54c840b4e86c4/.deps/Mako-0.4.0-py2-none-any.whl:/home/jenkins/jenkins-slave/workspace/AuroraDebug/.pants.d/python-setup/chroots/965c8509f886522de50b650745d54c840b4e86c4/.deps/coverage-3.7.1-cp27-none-linux_x86_64.whl:/home/jenkins/jenkins-slave/workspace/AuroraDebug/.pants.d/python-setup/chroots/965c8509f886522de50b650745d54c840b4e86c4/.deps/protobuf-2.6.1-py2.7.egg:/home/jenkins/jenkins-slave/workspace/AuroraDebug/.pants.d/python-setup/chroots/965c8509f886522de50b650745d54c840b4e86c4/.deps/CherryPy-3.6.0-py2-none-any.whl:/home/jenkins/jenkins-slave/workspace/AuroraDebug/.pants.d/python-setup/chroots/965c8509f886522de50b650745d54c840b4e86c4/.deps/api.src.main.thrift.org.apache.aurora.gen._api_776615e2-0.0.0-py2-none-any.whl:/home/jenkins/jenkins-slave/workspace/AuroraDebug/.pants.d/python-setup/chroots/965c8509f886522de50b650745d54c840b4e86c4/.deps/twitter.common.options-0.3.3-py2-none-any.whl:/home/jenkins/jenkins-slave/workspace/AuroraDebug/.pants.d/python-setup/chroots/965c8509f886522de50b650745d54c840b4e86c4/.deps/google_common-0.0.1-py2-none-any.whl:/home/jenkins/jenkins-slave/workspace/AuroraDebug/.pants.d/python-setup/chroots/965c8509f886522de50b650745d54c840b4e86c4/.deps/six-1.10.0-py2.py3-none-any.whl:/home/jenkins/jenkins-slave/workspace/AuroraDebug/.pants.d/python-setup/chroots/965c8509f886522de50b650745d54c840b4e86c4/.deps/api.src.main.thrift.org.apache.aurora.gen._storage_35ba8045-0.0.0-py2-none-any.whl:/home/jenkins/jenkins-slave/workspace/AuroraDebug/.pants.d/python-setup/chroots/965c8509f886522de50b650745d54c840b4e86c4/.deps/twitter.common.process-0.3.3-py2-none-any.whl:/home/jenkins/jenkins-slave/workspace/AuroraDebug/.pants.d/python-setup/chroots/965c8509f886522de50b650745d54c840b4e86c4/.deps/pex-1.1.2-py2.py3-none-any.whl:/home/jenkins/jenkins-slave/workspace/AuroraDebug/.pants.d/python-setup/chroots/965c8509f886522de50b650745d54c840b4e86c4/.deps/psutil-3.2.2-cp27-none-linux_x86_64.whl:/home/jenkins/jenkins-slave/workspace/AuroraDebug/.pants.d/python-setup/chroots/965c8509f886522de50b650745d54c840b4e86c4/.deps/twitter.common.zookeeper-0.3.3-py2-none-any.whl:/home/jenkins/jenkins-slave/workspace/AuroraDebug/.pants.d/python-setup/chroots/965c8509f886522de50b650745d54c840b4e86c4/.deps/twitter.common.util-0.3.3-py2-none-any.whl:/home/jenkins/jenkins-slave/workspace/AuroraDebug/.pants.d/python-setup/chroots/965c8509f886522de50b650745d54c840b4e86c4/.deps/twitter.common.lang-0.3.3-py2-none-any.whl:/home/jenkins/jenkins-slave/workspace/AuroraDebug/.pants.d/python-setup/chroots/965c8509f886522de50b650745d54c840b4e86c4/.deps/enum34-1.1.6-py2-none-any.whl:/home/jenkins/jenkins-slave/workspace/AuroraDebug/.pants.d/python-setup/chroots/965c8509f886522de50b650745d54c840b4e86c4/.deps/mock-1.0.1-py2-none-any.whl:/home/jenkins/jenkins-slave/workspace/AuroraDebug/.pants.d/python-setup/chroots/965c8509f886522de50b650745d54c840b4e86c4/.deps/py-1.4.31-py2.py3-none-any.whl:/home/jenkins/jenkins-slave/workspace/AuroraDebug/.pants.d/python-setup/chroots/965c8509f886522de50b650745d54c840b4e86c4/.deps/twitter.common.concurrent-0.3.3-py2-none-any.whl:/home/jenkins/jenkins-slave/workspace/AuroraDebug/.pants.d/python-setup/chroots/965c8509f886522de50b650745d54c840b4e86c4/.deps/twitter.common.testing-0.3.3-py2-none-any.whl:/home/jenkins/jenkins-slave/workspace/AuroraDebug/.pants.d/python-setup/chroots/965c8509f886522de50b650745d54c840b4e86c4/.deps/twitter.common.http-0.3.3-py2-none-any.whl:/home/jenkins/jenkins-slave/workspace/AuroraDebug/.pants.d/python-setup/chroots/965c8509f886522de50b650745d54c840b4e86c4/.deps/thrift-0.9.1-cp27-none-linux_x86_64.whl:/home/jenkins/jenkins-slave/workspace/AuroraDebug/.pants.d/python-setup/chroots/965c8509f886522de50b650745d54c840b4e86c4/.deps/futures-2.1.6-py2.py3-none-any.whl:/home/jenkins/jenkins-slave/workspace/AuroraDebug/.pants.d/python-setup/chroots/965c8509f886522de50b650745d54c840b4e86c4/.deps/argparse-1.4.0-py2.py3-none-any.whl:/home/jenkins/jenkins-slave/workspace/AuroraDebug/.pants.d/python-setup/chroots/965c8509f886522de50b650745d54c840b4e86c4/.deps/twitter.common.decorators-0.3.3-py2-none-any.whl:/home/jenkins/jenkins-slave/workspace/AuroraDebug/.pants.d/python-setup/chroots/965c8509f886522de50b650745d54c840b4e86c4/.deps/setuptools-19.7-py2.py3-none-any.whl:/home/jenkins/jenkins-slave/workspace/AuroraDebug/.pants.d/python-setup/chroots/965c8509f886522de50b650745d54c840b4e86c4/.deps/mox-0.5.3-py2-none-any.whl:/home/jenkins/jenkins-slave/workspace/AuroraDebug/.pants.d/python-setup/chroots/965c8509f886522de50b650745d54c840b4e86c4/.deps/twitter.common.dirutil-0.3.3-py2-none-any.whl:/home/jenkins/jenkins-slave/workspace/AuroraDebug/.pants.d/python-setup/chroots/965c8509f886522de50b650745d54c840b4e86c4/.deps/pytest_timeout-0.5-py2.py3-none-any.whl:/home/jenkins/jenkins-slave/workspace/AuroraDebug/.pants.d/python-setup/chroots/965c8509f886522de50b650745d54c840b4e86c4/.deps/twitter.common.recordio-0.3.3-py2-none-any.whl:/home/jenkins/jenkins-slave/workspace/AuroraDebug/.pants.d/python-setup/chroots/965c8509f886522de50b650745d54c840b4e86c4/.deps/mesos.interface-1.0.0-py2-none-any.whl:/home/jenkins/jenkins-slave/workspace/AuroraDebug/.pants.d/python-setup/chroots/965c8509f886522de50b650745d54c840b4e86c4/.deps/twitter.common.app-0.3.3-py2-none-any.whl:/home/jenkins/jenkins-slave/workspace/AuroraDebug/.pants.d/python-setup/chroots/965c8509f886522de50b650745d54c840b4e86c4/.deps/MarkupSafe-0.23-cp27-none-linux_x86_64.whl:/home/jenkins/jenkins-slave/workspace/AuroraDebug/.pants.d/python-setup/chroots/965c8509f886522de50b650745d54c840b4e86c4/.deps/twitter.common.metrics-0.3.3-py2-none-any.whl:/home/jenkins/jenkins-slave/workspace/AuroraDebug/.pants.d/python-setup/chroots/965c8509f886522de50b650745d54c840b4e86c4/.deps/subprocess32-3.2.7-cp27-none-linux_x86_64.whl:/home/jenkins/jenkins-slave/workspace/AuroraDebug/.pants.d/python-setup/chroots/965c8509f886522de50b650745d54c840b4e86c4/.deps/twitter.common.contextutil-0.3.3-py2-none-any.whl:/home/jenkins/jenkins-slave/workspace/AuroraDebug/.pants.d/python-setup/chroots/965c8509f886522de50b650745d54c840b4e86c4/.deps/linecache2-1.0.0-py2.py3-none-any.whl:/home/jenkins/jenkins-slave/workspace/AuroraDebug/.pants.d/python-setup/chroots/965c8509f886522de50b650745d54c840b4e86c4/.deps/twitter.common.log-0.3.3-py2-none-any.whl:/home/jenkins/jenkins-slave/workspace/AuroraDebug/.pants.d/python-setup/chroots/965c8509f886522de50b650745d54c840b4e86c4/.deps/twitter.common.collections-0.3.3-py2-none-any.whl:/home/jenkins/jenkins-slave/workspace/AuroraDebug/.pants.d/python-setup/chroots/965c8509f886522de50b650745d54c840b4e86c4/.deps/cov_core-1.15.0-py2-none-any.whl:/home/jenkins/jenkins-slave/workspace/AuroraDebug/.pants.d/python-setup/chroots/965c8509f886522de50b650745d54c840b4e86c4/.deps/kazoo-1.3.1-py2-none-any.whl
{code}
For my jenkins test job I have 6 failures in 8 builds before the fix, and 0
failures in 7 builds afterwards. https://builds.apache.org/job/AuroraDebug/
Review request: https://reviews.apache.org/r/51366/
> pants fails to run in integration tests (flaky)
> -----------------------------------------------
>
> Key: AURORA-1717
> URL: https://issues.apache.org/jira/browse/AURORA-1717
> Project: Aurora
> Issue Type: Bug
> Components: Build
> Reporter: John Sirois
> Assignee: John Sirois
> Attachments: AURORA-1717-XDG_CACHE_HOME.edited.png,
> Aurora1555_H10_ws_.home_.cache_pants_plugins.zip,
> Aurora1555_H10_ws_.home_.cache_pants_setup_bootstrap-Linux-x86_64_1.1.0-pre6_lib_python2.7_site-packages_pants.zip,
> builds.apache.org_job_Aurora_1552_consoleText.txt, pip.zip, pip2.zip
>
>
> As noted by [~StephanErb] in
> [IRC|http://wilderness.apache.org/channels/?f=aurora/2016-06-15#1466009185],
> python tests occasionally fail like so under AuroraBot:
> {noformat}
> ...
> [1m with temporary_dir() as checkpoint_root:[0m
> [1m te = AuroraExecutor([0m
> [1m> runner_provider=make_provider(checkpoint_root),[0m
> [1m sandbox_provider=DefaultTestSandboxProvider())[0m
> src/test/python/apache/aurora/executor/test_thermos_executor.py:580:
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> src/test/python/apache/aurora/executor/test_thermos_executor.py:193: in
> make_provider
> [1m pex_location=thermos_runner_path(),[0m
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> build = True
> [1m def thermos_runner_path(build=True):[0m
> [1m if not build:[0m
> [1m return getattr(thermos_runner_path, 'value', None)[0m
> [1m [0m
> [1m if not hasattr(thermos_runner_path, 'value'):[0m
> [1m pex_dir = safe_mkdtemp()[0m
> [1m> assert subprocess.call(["./pants", "--pants-distdir=%s" %
> pex_dir, "binary",[0m
> [1m "src/main/python/apache/thermos/runner:thermos_runner"]) ==
> 0[0m
> [1m[31mE assert 1 == 0[0m
> [1m[31mE + where 1 = <function call at 0x7f39ef905938>(['./pants',
> '--pants-distdir=/tmp/user/2396/tmpU8sLiq', 'binary',
> 'src/main/python/apache/thermos/runner:thermos_runner'])[0m
> [1m[31mE + where <function call at 0x7f39ef905938> =
> subprocess.call[0m
> src/test/python/apache/aurora/executor/test_thermos_executor.py:185:
> AssertionError
> -------------- Captured stderr call --------------
> Traceback (most recent call last):
> File
> "/home/jenkins/.cache/pants/setup/bootstrap-Linux-x86_64/0.0.80/bin/pants",
> line 7, in <module>
> from pants.bin.pants_exe import main
> ImportError: No module named pants.bin.pants_exe
> generated xml file:
> /home/jenkins/jenkins-slave/workspace/AuroraBot/dist/test-results/415337499eb72578eab327a6487c1f5c9452b3d6.xml
>
> [1m[31m 16 failed, 638 passed, 6 skipped, 1 warnings, 8 error in 127.24
> seconds [0m
> ...
> {noformat}
> This has all the signs of a race in pants bootstrapping; ie: the {{\[version
> symlink]/bin/pants}} script is installed but the interpreter it points to in
> its shebang does not have pants fully installed yet. That said, a review of
> the pants bootstrap script seems to indicate correct atomicity.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
