[jira] [Comment Edited] (BEAM-3120) Jenkins postcommit test suite triggered for a release branch uses code from master branch.
[
https://issues.apache.org/jira/browse/BEAM-3120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267260#comment-16267260
]
Valentyn Tymofieiev edited comment on BEAM-3120 at 11/27/17 6:59 PM:
-
After worksbeam_PostCommit_Python_Verify, the builds is failing with "bash:
sdks/python/run_postcommit.sh: No such file or directory".
It appears that in https://github.com/apache/beam/pull/3976 we changed the
source directory, into which the code is checked in, but several build configs
were not updated.
The following builds are affected:
job_beam_PostCommit_Python_Verify
job_beam_PerformanceTests_Python
job_beam_PostCommit_Python_ValidatesRunner_Dataflow
These builds don't depend on maven and use shell scripts to run the test
workflow. Before we cleaned the workspace, these builds were using left-over
source code in the workspace, that was checked out from master branch. This
explains why it appeared as if the postcommit suite triggered for the release
branch were using the code from Master.
I'll send a PR to update build configs.
We should also clean the workspace on Jenkins slaves before executing builds:
https://issues.apache.org/jira/browse/BEAM-3258.
was (Author: tvalentyn):
After worksbeam_PostCommit_Python_Verify, the builds is failing with "bash:
sdks/python/run_postcommit.sh: No such file or directory".
It appears that in https://github.com/apache/beam/pull/3976 we changed the
source directory, into which the code is checked in, but several presubmit
configs were not updated.
The following builds are affected:
job_beam_PostCommit_Python_Verify
job_beam_PerformanceTests_Python
job_beam_PostCommit_Python_ValidatesRunner_Dataflow
These builds don't depend on maven and use shell scripts to run the test
workflow. Before we cleaned the workspace, these builds were using left-over
source code in the workspace, that was checked out from master branch. This
explains why it appeared as if the postcommit suite triggered for the release
branch were using the code from Master.
I'll send a PR to update build configs.
We should also clean the workspace on Jenkins slaves before executing builds:
https://issues.apache.org/jira/browse/BEAM-3258.
> Jenkins postcommit test suite triggered for a release branch uses code from
> master branch.
> --
>
> Key: BEAM-3120
> URL: https://issues.apache.org/jira/browse/BEAM-3120
> Project: Beam
> Issue Type: Bug
> Components: testing
>Reporter: Valentyn Tymofieiev
>Assignee: Jason Kuster
>
> From
> https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Python_Verify/3452/
> we can see the build was triggered on Revision:
> 576d22a67ffeb29adf26f8adc09aaa6078099cf6
> origin/release-2.2.0. However, looking at test logs we can see that 2.3.0.dev
> version of the Python SDK is being installed, and tested:
> # Tox runs unit tests in a virtual environment
> ${LOCAL_PATH}/tox -e ALL -c sdks/python/tox.ini
> GLOB sdist-make:
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/setup.py
> docs create:
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/target/.tox/docs
> docs installdeps: nose==1.3.7, grpcio-tools==1.3.5, Sphinx==1.5.5,
> sphinx_rtd_theme==0.2.4
> docs inst:
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/target/.tox/dist/apache-beam-2.3.0.dev.zip
> docs installed:
> alabaster==0.7.10,apache-beam==2.3.0.dev0,avro==1.8.2,Babel==2.5.1,certifi==2017.7.27.1,chardet==3.0.4,crcmod==1.7,dill==0.2.6,docutils==0.14,enum34==1.1.6,funcsigs==1.0.2,futures==3.1.1,grpcio==1.7.0,grpcio-tools==1.3.5,httplib2==0.9.2,idna==2.6,imagesize==0.7.1,Jinja2==2.9.6,MarkupSafe==1.0,mock==2.0.0,nose==1.3.7,oauth2client==3.0.0,pbr==3.1.1,protobuf==3.3.0,pyasn1==0.3.7,pyasn1-modules==0.1.5,Pygments==2.2.0,pytz==2017.3,PyYAML==3.12,requests==2.18.4,rsa==3.4.2,six==1.10.0,snowballstemmer==1.2.1,Sphinx==1.5.5,sphinx-rtd-theme==0.2.4,typing==3.6.2,urllib3==1.22
> Looking further.
> cc [~altay]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (BEAM-3120) Jenkins postcommit test suite triggered for a release branch uses code from master branch.
[
https://issues.apache.org/jira/browse/BEAM-3120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263637#comment-16263637
]
Valentyn Tymofieiev edited comment on BEAM-3120 at 11/23/17 1:13 AM:
-
One of first line in Jenkins logs is
> git rev-parse --is-inside-work-tree # timeout=10
I suspect Git-SCM plugin is trying to find it's way into a Git repository. The
workspace is dirty in the sense that it has two repositories, in workspace root
and in /src, and the plugin picks the repository in workspace root, which does
not have recent commits. I'll file an INFRA ticket to clean the workspace on
the slaves per [~jasonkuster]'s suggestion.
was (Author: tvalentyn):
One of first line in Jenkins logs is
> git rev-parse --is-inside-work-tree # timeout=10
I suspect Git-SCM plugin is trying to find it's way into a Git repository. The
workspace is dirty in the sense that it has two repositories, in workspace root
and in /src, and the plugin picks the Workspace. I'll file an INFRA ticket to
clean the workspace on the slaves per [~jasonkuster]'s suggestion.
> Jenkins postcommit test suite triggered for a release branch uses code from
> master branch.
> --
>
> Key: BEAM-3120
> URL: https://issues.apache.org/jira/browse/BEAM-3120
> Project: Beam
> Issue Type: Bug
> Components: testing
>Reporter: Valentyn Tymofieiev
>Assignee: Jason Kuster
>
> From
> https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Python_Verify/3452/
> we can see the build was triggered on Revision:
> 576d22a67ffeb29adf26f8adc09aaa6078099cf6
> origin/release-2.2.0. However, looking at test logs we can see that 2.3.0.dev
> version of the Python SDK is being installed, and tested:
> # Tox runs unit tests in a virtual environment
> ${LOCAL_PATH}/tox -e ALL -c sdks/python/tox.ini
> GLOB sdist-make:
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/setup.py
> docs create:
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/target/.tox/docs
> docs installdeps: nose==1.3.7, grpcio-tools==1.3.5, Sphinx==1.5.5,
> sphinx_rtd_theme==0.2.4
> docs inst:
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/target/.tox/dist/apache-beam-2.3.0.dev.zip
> docs installed:
> alabaster==0.7.10,apache-beam==2.3.0.dev0,avro==1.8.2,Babel==2.5.1,certifi==2017.7.27.1,chardet==3.0.4,crcmod==1.7,dill==0.2.6,docutils==0.14,enum34==1.1.6,funcsigs==1.0.2,futures==3.1.1,grpcio==1.7.0,grpcio-tools==1.3.5,httplib2==0.9.2,idna==2.6,imagesize==0.7.1,Jinja2==2.9.6,MarkupSafe==1.0,mock==2.0.0,nose==1.3.7,oauth2client==3.0.0,pbr==3.1.1,protobuf==3.3.0,pyasn1==0.3.7,pyasn1-modules==0.1.5,Pygments==2.2.0,pytz==2017.3,PyYAML==3.12,requests==2.18.4,rsa==3.4.2,six==1.10.0,snowballstemmer==1.2.1,Sphinx==1.5.5,sphinx-rtd-theme==0.2.4,typing==3.6.2,urllib3==1.22
> Looking further.
> cc [~altay]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (BEAM-3120) Jenkins postcommit test suite triggered for a release branch uses code from master branch.
[
https://issues.apache.org/jira/browse/BEAM-3120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263250#comment-16263250
]
Valentyn Tymofieiev edited comment on BEAM-3120 at 11/22/17 7:56 PM:
-
I looked at the staged version of the SDK - it includes changes from
https://github.com/apache/beam/commit/3b24ce8d649c44fb4fa867b9706014852eeb41da.
Which means it's no longer that 7 days old, but still old since
https://github.com/apache/beam/commit/fc57082ce76d0821b0b4979fe3d7935e2f8e64ae
is not picked up.
was (Author: tvalentyn):
I looked at the staged version of the SDK - it includes changes from
https://github.com/apache/beam/commit/3b24ce8d649c44fb4fa867b9706014852eeb41da.
Which means it's no longer that 7 days old, but still more than 1 day old since
https://github.com/apache/beam/commit/fc57082ce76d0821b0b4979fe3d7935e2f8e64ae
is not picked up.
> Jenkins postcommit test suite triggered for a release branch uses code from
> master branch.
> --
>
> Key: BEAM-3120
> URL: https://issues.apache.org/jira/browse/BEAM-3120
> Project: Beam
> Issue Type: Bug
> Components: testing
>Reporter: Valentyn Tymofieiev
>Assignee: Jason Kuster
>
> From
> https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Python_Verify/3452/
> we can see the build was triggered on Revision:
> 576d22a67ffeb29adf26f8adc09aaa6078099cf6
> origin/release-2.2.0. However, looking at test logs we can see that 2.3.0.dev
> version of the Python SDK is being installed, and tested:
> # Tox runs unit tests in a virtual environment
> ${LOCAL_PATH}/tox -e ALL -c sdks/python/tox.ini
> GLOB sdist-make:
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/setup.py
> docs create:
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/target/.tox/docs
> docs installdeps: nose==1.3.7, grpcio-tools==1.3.5, Sphinx==1.5.5,
> sphinx_rtd_theme==0.2.4
> docs inst:
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/target/.tox/dist/apache-beam-2.3.0.dev.zip
> docs installed:
> alabaster==0.7.10,apache-beam==2.3.0.dev0,avro==1.8.2,Babel==2.5.1,certifi==2017.7.27.1,chardet==3.0.4,crcmod==1.7,dill==0.2.6,docutils==0.14,enum34==1.1.6,funcsigs==1.0.2,futures==3.1.1,grpcio==1.7.0,grpcio-tools==1.3.5,httplib2==0.9.2,idna==2.6,imagesize==0.7.1,Jinja2==2.9.6,MarkupSafe==1.0,mock==2.0.0,nose==1.3.7,oauth2client==3.0.0,pbr==3.1.1,protobuf==3.3.0,pyasn1==0.3.7,pyasn1-modules==0.1.5,Pygments==2.2.0,pytz==2017.3,PyYAML==3.12,requests==2.18.4,rsa==3.4.2,six==1.10.0,snowballstemmer==1.2.1,Sphinx==1.5.5,sphinx-rtd-theme==0.2.4,typing==3.6.2,urllib3==1.22
> Looking further.
> cc [~altay]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (BEAM-3120) Jenkins postcommit test suite triggered for a release branch uses code from master branch.
[
https://issues.apache.org/jira/browse/BEAM-3120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263025#comment-16263025
]
Valentyn Tymofieiev edited comment on BEAM-3120 at 11/22/17 6:57 PM:
-
I am no longer sure which code Jenkins Python postcommit tests suite is using
at all.
Let's look at https://builds.apache.org/job/beam_PostCommit_Python_Verify/3595/
that was triggered for the master branch.
Jenkins says:
Revision: a40bdfd756e5b187a6c7fea86c253efd0feb1701
origin/master
We have a failing integration test, test_wordcount_fnapi_it
(apache_beam.examples.wordcount_it_test.WordCountIT) ... ERROR
Job logs:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2017-11-22_07_29_57-11055969376023337206?project=apache-beam-testing
Job logs show that we are using Container {Name:sdk
Image:dataflow.gcr.io/v1beta3/python-fnapi:beam-2.1.0-20170621 ... }
Looking at git show
a40bdfd756e5b187a6c7fea86c253efd0feb1701:./apache_beam/runners/dataflow/internal/dependency.py
| grep BEAM_FNAPI_CONTAINER_VERSION
we can see BEAM_FNAPI_CONTAINER_VERSION = 'beam-2.3.0-20171121' . It's not
clear which revision Jenkins is using...
was (Author: tvalentyn):
I am no longer sure which code Jenkins Python postcommit tests suite is using
at all.
Let's look at https://builds.apache.org/job/beam_PostCommit_Python_Verify/3595/
Jenkins says:
Revision: a40bdfd756e5b187a6c7fea86c253efd0feb1701
origin/master
We have a failing integration test, test_wordcount_fnapi_it
(apache_beam.examples.wordcount_it_test.WordCountIT) ... ERROR
Job logs:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2017-11-22_07_29_57-11055969376023337206?project=apache-beam-testing
Job logs show that we are using Container {Name:sdk
Image:dataflow.gcr.io/v1beta3/python-fnapi:beam-2.1.0-20170621 ... }
Looking at git show
a40bdfd756e5b187a6c7fea86c253efd0feb1701:./apache_beam/runners/dataflow/internal/dependency.py
| grep BEAM_FNAPI_CONTAINER_VERSION
we can see BEAM_FNAPI_CONTAINER_VERSION = 'beam-2.3.0-20171121' . It's not
clear which revision Jenkins is using...
> Jenkins postcommit test suite triggered for a release branch uses code from
> master branch.
> --
>
> Key: BEAM-3120
> URL: https://issues.apache.org/jira/browse/BEAM-3120
> Project: Beam
> Issue Type: Bug
> Components: testing
>Reporter: Valentyn Tymofieiev
>Assignee: Jason Kuster
>
> From
> https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Python_Verify/3452/
> we can see the build was triggered on Revision:
> 576d22a67ffeb29adf26f8adc09aaa6078099cf6
> origin/release-2.2.0. However, looking at test logs we can see that 2.3.0.dev
> version of the Python SDK is being installed, and tested:
> # Tox runs unit tests in a virtual environment
> ${LOCAL_PATH}/tox -e ALL -c sdks/python/tox.ini
> GLOB sdist-make:
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/setup.py
> docs create:
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/target/.tox/docs
> docs installdeps: nose==1.3.7, grpcio-tools==1.3.5, Sphinx==1.5.5,
> sphinx_rtd_theme==0.2.4
> docs inst:
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/target/.tox/dist/apache-beam-2.3.0.dev.zip
> docs installed:
> alabaster==0.7.10,apache-beam==2.3.0.dev0,avro==1.8.2,Babel==2.5.1,certifi==2017.7.27.1,chardet==3.0.4,crcmod==1.7,dill==0.2.6,docutils==0.14,enum34==1.1.6,funcsigs==1.0.2,futures==3.1.1,grpcio==1.7.0,grpcio-tools==1.3.5,httplib2==0.9.2,idna==2.6,imagesize==0.7.1,Jinja2==2.9.6,MarkupSafe==1.0,mock==2.0.0,nose==1.3.7,oauth2client==3.0.0,pbr==3.1.1,protobuf==3.3.0,pyasn1==0.3.7,pyasn1-modules==0.1.5,Pygments==2.2.0,pytz==2017.3,PyYAML==3.12,requests==2.18.4,rsa==3.4.2,six==1.10.0,snowballstemmer==1.2.1,Sphinx==1.5.5,sphinx-rtd-theme==0.2.4,typing==3.6.2,urllib3==1.22
> Looking further.
> cc [~altay]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
