[
https://issues.apache.org/jira/browse/BEAM-8481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16995289#comment-16995289
]
Valentyn Tymofieiev edited comment on BEAM-8481 at 12/13/19 2:26 AM:
---------------------------------------------------------------------
By broken I meant that protobuf stubs in generated tarball are not futurized.
This is what happens.
There is a race between tasks :sdks:python:sdist and
:some:python:project:installGcpTest task. installGcpTest is a gradle-generated
task that is added to build files that call applyPythonNature() and is defined
in beamModulePlugin[2]
sdist command runs python setup.py -q sdist [3].
installGcpTest runs activates a new virtual environment and runs
python -e {pathToSdkInJeniknsWorkspace}[gcp,test] [4]
There seems to be an intent [5] that installGcpTest should run after sdist,
after tarball is generated, however this is not happening, I have verified that
installGcpTest occasionally starts running before sdist.
installGcpTest and sdist commands introduce side effects to the only copy sdk
of sources, which is cloned into Jenkins workspace. These commands call python
setup.py egg_info, which generates [6][7] protocol buffer stubs, and futurizes
[8] them. There is some caching[9] in proto generation logic: if stubs are
already generated, re-generation may be skipped.
In the following scenario, sdist produces a tarball where stubs are are not
futurized:
[installGcpTest called]
[proto stubs are generated as part of installGcpTest, but not yet futurized]
[sdist is called]
[proto stubs are already generated, so regeneration is skipped]
[sdist finishes and prouduces sdk tarball, with stubs not futurized]
[proto stubs are futurized as part of installGcpTest]
[installGcpTest finishes]
Following may be possible fixes of the issue:
- Establish dependency relationship: 'installGcpTest' must-run-after
':sdks:python:sdist' that actually works
- Don't install sdk in a develop mode (that points virtualenv directory to a
shared source folder), instead install sdk into virtualenv from a tarball.
[1]
https://github.com/apache/beam/blob/c7f7da602adc146806f04eb9f6b2070bab745d3d/sdks/python/test-suites/dataflow/py37/build.gradle#L20
[2]
https://github.com/apache/beam/blob/c7f7da602adc146806f04eb9f6b2070bab745d3d/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L1808
[3]
https://github.com/apache/beam/blob/c7f7da602adc146806f04eb9f6b2070bab745d3d/sdks/python/build.gradle#L48
[4]
https://github.com/apache/beam/blob/c7f7da602adc146806f04eb9f6b2070bab745d3d/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L1812
[5]
https://github.com/apache/beam/blob/c7f7da602adc146806f04eb9f6b2070bab745d3d/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L1816
[6]
https://github.com/apache/beam/blob/c7f7da602adc146806f04eb9f6b2070bab745d3d/sdks/python/setup.py#L293
[7]
https://github.com/apache/beam/blob/c7f7da602adc146806f04eb9f6b2070bab745d3d/sdks/python/gen_protos.py#L140
[8]
https://github.com/apache/beam/blob/c7f7da602adc146806f04eb9f6b2070bab745d3d/sdks/python/gen_protos.py#L155
[9]
https://github.com/apache/beam/blob/c7f7da602adc146806f04eb9f6b2070bab745d3d/sdks/python/gen_protos.py#L102
was (Author: tvalentyn):
By broken I meant that protobuf stubs in generated tarball are not futurized.
This is what happens.
There is a race between tasks :sdks:python:sdist and
:some:python:project:installGcpTest task. installGcpTest is a gradle-generated
task that is added to build files that call applyPythonNature() and is defined
in beamModulePlugin[2]
sdist command runs python setup.py -q sdist [3].
installGcpTest runs activates a new virtual environment and runs python -e
{pathToSdkInJeniknsWorkspace}[gcp,test] [4]
There seems to be an intent [5] that installGcpTest should run after sdist,
after tarball is generated, however this is not happening, I have verified that
installGcpTest occasionally starts running before sdist.
installGcpTest and sdist commands introduce side effects to the only copy sdk
of sources, which is cloned into Jenkins workspace. These commands call python
setup.py egg_info, which generates [6][7] protocol buffer stubs, and futurizes
[8] them. There is some caching[9] in proto generation logic: if stubs are
already generated, re-generation may be skipped.
In the following scenario, sdist produces a tarball where stubs are are not
futurized:
[installGcpTest called]
[proto stubs are generated as part of installGcpTest, but not yet futurized]
[sdist is called]
[proto stubs are already generated, so regeneration is skipped]
[sdist finishes and prouduces sdk tarball, with stubs not futurized]
[proto stubs are futurized as part of installGcpTest]
[installGcpTest finishes]
Following may be possible fixes of the issue:
- Establish dependency relationship: 'installGcpTest' must-run-after
':sdks:python:sdist' that actually works
- Don't install sdk in a develop mode (that points virtualenv directory to a
shared source folder), instead install sdk into virtualenv from a tarball.
[1]
https://github.com/apache/beam/blob/c7f7da602adc146806f04eb9f6b2070bab745d3d/sdks/python/test-suites/dataflow/py37/build.gradle#L20
[2]
https://github.com/apache/beam/blob/c7f7da602adc146806f04eb9f6b2070bab745d3d/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L1808
[3]
https://github.com/apache/beam/blob/c7f7da602adc146806f04eb9f6b2070bab745d3d/sdks/python/build.gradle#L48
[4]
https://github.com/apache/beam/blob/c7f7da602adc146806f04eb9f6b2070bab745d3d/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L1812
[5]
https://github.com/apache/beam/blob/c7f7da602adc146806f04eb9f6b2070bab745d3d/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L1816
[6]
https://github.com/apache/beam/blob/c7f7da602adc146806f04eb9f6b2070bab745d3d/sdks/python/setup.py#L293
[7]
https://github.com/apache/beam/blob/c7f7da602adc146806f04eb9f6b2070bab745d3d/sdks/python/gen_protos.py#L140
[8]
https://github.com/apache/beam/blob/c7f7da602adc146806f04eb9f6b2070bab745d3d/sdks/python/gen_protos.py#L155
[9]
https://github.com/apache/beam/blob/c7f7da602adc146806f04eb9f6b2070bab745d3d/sdks/python/gen_protos.py#L102
> Python 3.7 Postcommit test -- frequent timeouts
> -----------------------------------------------
>
> Key: BEAM-8481
> URL: https://issues.apache.org/jira/browse/BEAM-8481
> Project: Beam
> Issue Type: Bug
> Components: test-failures
> Reporter: Ahmet Altay
> Assignee: Valentyn Tymofieiev
> Priority: Critical
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> [https://builds.apache.org/job/beam_PostCommit_Python37/] – this suite
> seemingly frequently timing out. Other suites are not affected by these
> timeouts. From the history, the issues started before Oct 10 and we cannot
> pinpoint because history is lost.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
