[jira] [Commented] (BEAM-8229) Python postcommit 2.7 tests are failing since we try to run a test with Python 3.6+ syntax on Python 2.
[ https://issues.apache.org/jira/browse/BEAM-8229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930749#comment-16930749 ] Chad Dombrova commented on BEAM-8229: - this is resolved > Python postcommit 2.7 tests are failing since we try to run a test with > Python 3.6+ syntax on Python 2. > --- > > Key: BEAM-8229 > URL: https://issues.apache.org/jira/browse/BEAM-8229 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Ahmet Altay >Assignee: Udi Meiri >Priority: Blocker > Time Spent: 1h 50m > Remaining Estimate: 0h > > Root cause: https://github.com/apache/beam/pull/9098 > 20:12:14 Traceback (most recent call last): > 20:12:14 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python2_PR/src/build/gradleenv/-194514014/local/lib/python2.7/site-packages/nose/loader.py", > line 418, in loadTestsFromName > 20:12:14 addr.filename, addr.module) > 20:12:14 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python2_PR/src/build/gradleenv/-194514014/local/lib/python2.7/site-packages/nose/importer.py", > line 47, in importFromPath > 20:12:14 return self.importFromDir(dir_path, fqname) > 20:12:14 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python2_PR/src/build/gradleenv/-194514014/local/lib/python2.7/site-packages/nose/importer.py", > line 94, in importFromDir > 20:12:14 mod = load_module(part_fqname, fh, filename, desc) > 20:12:14 SyntaxError: invalid syntax (external_test_py37.py, line 46) -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (BEAM-8229) Python postcommit 2.7 tests are failing since we try to run a test with Python 3.6+ syntax on Python 2.
[ https://issues.apache.org/jira/browse/BEAM-8229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16929412#comment-16929412 ] Chad Dombrova commented on BEAM-8229: - Based on feedback from Robert, I'm going to split my PR into two: the first which excludes the python3.6-only feature, and the other which includes it and the additional test filtering features. > Python postcommit 2.7 tests are failing since we try to run a test with > Python 3.6+ syntax on Python 2. > --- > > Key: BEAM-8229 > URL: https://issues.apache.org/jira/browse/BEAM-8229 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Ahmet Altay >Assignee: Udi Meiri >Priority: Blocker > > Root cause: https://github.com/apache/beam/pull/9098 > 20:12:14 Traceback (most recent call last): > 20:12:14 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python2_PR/src/build/gradleenv/-194514014/local/lib/python2.7/site-packages/nose/loader.py", > line 418, in loadTestsFromName > 20:12:14 addr.filename, addr.module) > 20:12:14 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python2_PR/src/build/gradleenv/-194514014/local/lib/python2.7/site-packages/nose/importer.py", > line 47, in importFromPath > 20:12:14 return self.importFromDir(dir_path, fqname) > 20:12:14 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python2_PR/src/build/gradleenv/-194514014/local/lib/python2.7/site-packages/nose/importer.py", > line 94, in importFromDir > 20:12:14 mod = load_module(part_fqname, fh, filename, desc) > 20:12:14 SyntaxError: invalid syntax (external_test_py37.py, line 46) -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (BEAM-8229) Python postcommit 2.7 tests are failing since we try to run a test with Python 3.6+ syntax on Python 2.
[ https://issues.apache.org/jira/browse/BEAM-8229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16929378#comment-16929378 ] Valentyn Tymofieiev commented on BEAM-8229: --- As Udi said, there aren't any Py3-only integration tests, as we run all ITs across Py2.7, 3.5-3.7. We don't plan to keep our codebase Py2 compatible once we cut Beam 2.18.0, so py3-only exclusions won't be necessary in integration_test.sh in early 2020, however if at some time we will have a Python3.7 IT that we don't want to run on Python 3.6, then we will need better exclusion mechanism for minor versions. At the moment Chad's proposal for extending regex SGTM. > Python postcommit 2.7 tests are failing since we try to run a test with > Python 3.6+ syntax on Python 2. > --- > > Key: BEAM-8229 > URL: https://issues.apache.org/jira/browse/BEAM-8229 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Ahmet Altay >Assignee: Udi Meiri >Priority: Blocker > > Root cause: https://github.com/apache/beam/pull/9098 > 20:12:14 Traceback (most recent call last): > 20:12:14 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python2_PR/src/build/gradleenv/-194514014/local/lib/python2.7/site-packages/nose/loader.py", > line 418, in loadTestsFromName > 20:12:14 addr.filename, addr.module) > 20:12:14 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python2_PR/src/build/gradleenv/-194514014/local/lib/python2.7/site-packages/nose/importer.py", > line 47, in importFromPath > 20:12:14 return self.importFromDir(dir_path, fqname) > 20:12:14 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python2_PR/src/build/gradleenv/-194514014/local/lib/python2.7/site-packages/nose/importer.py", > line 94, in importFromDir > 20:12:14 mod = load_module(part_fqname, fh, filename, desc) > 20:12:14 SyntaxError: invalid syntax (external_test_py37.py, line 46) -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (BEAM-8229) Python postcommit 2.7 tests are failing since we try to run a test with Python 3.6+ syntax on Python 2.
[ https://issues.apache.org/jira/browse/BEAM-8229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16929370#comment-16929370 ] Ahmet Altay commented on BEAM-8229: --- For the immediate problem, If the change in run_integration_tests.sh will unblock postcommit tests let's go ahead with that. > Python postcommit 2.7 tests are failing since we try to run a test with > Python 3.6+ syntax on Python 2. > --- > > Key: BEAM-8229 > URL: https://issues.apache.org/jira/browse/BEAM-8229 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Ahmet Altay >Assignee: Udi Meiri >Priority: Blocker > > Root cause: https://github.com/apache/beam/pull/9098 > 20:12:14 Traceback (most recent call last): > 20:12:14 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python2_PR/src/build/gradleenv/-194514014/local/lib/python2.7/site-packages/nose/loader.py", > line 418, in loadTestsFromName > 20:12:14 addr.filename, addr.module) > 20:12:14 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python2_PR/src/build/gradleenv/-194514014/local/lib/python2.7/site-packages/nose/importer.py", > line 47, in importFromPath > 20:12:14 return self.importFromDir(dir_path, fqname) > 20:12:14 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python2_PR/src/build/gradleenv/-194514014/local/lib/python2.7/site-packages/nose/importer.py", > line 94, in importFromDir > 20:12:14 mod = load_module(part_fqname, fh, filename, desc) > 20:12:14 SyntaxError: invalid syntax (external_test_py37.py, line 46) -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (BEAM-8229) Python postcommit 2.7 tests are failing since we try to run a test with Python 3.6+ syntax on Python 2.
[ https://issues.apache.org/jira/browse/BEAM-8229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16929368#comment-16929368 ] Udi Meiri commented on BEAM-8229: - Yes Chad, you're right that we don't run py3-only ITs (there aren't any currently). I'm in favor of simplifying our test infrastructure. > Python postcommit 2.7 tests are failing since we try to run a test with > Python 3.6+ syntax on Python 2. > --- > > Key: BEAM-8229 > URL: https://issues.apache.org/jira/browse/BEAM-8229 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Ahmet Altay >Assignee: Udi Meiri >Priority: Blocker > > Root cause: https://github.com/apache/beam/pull/9098 > 20:12:14 Traceback (most recent call last): > 20:12:14 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python2_PR/src/build/gradleenv/-194514014/local/lib/python2.7/site-packages/nose/loader.py", > line 418, in loadTestsFromName > 20:12:14 addr.filename, addr.module) > 20:12:14 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python2_PR/src/build/gradleenv/-194514014/local/lib/python2.7/site-packages/nose/importer.py", > line 47, in importFromPath > 20:12:14 return self.importFromDir(dir_path, fqname) > 20:12:14 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python2_PR/src/build/gradleenv/-194514014/local/lib/python2.7/site-packages/nose/importer.py", > line 94, in importFromDir > 20:12:14 mod = load_module(part_fqname, fh, filename, desc) > 20:12:14 SyntaxError: invalid syntax (external_test_py37.py, line 46) -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (BEAM-8229) Python postcommit 2.7 tests are failing since we try to run a test with Python 3.6+ syntax on Python 2.
[
https://issues.apache.org/jira/browse/BEAM-8229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16929363#comment-16929363
]
Chad Dombrova commented on BEAM-8229:
-
{quote}
I am not familiar with pytest, but suspected it may have some solutions for
this problem. I thought that you might know if this is the case.
{quote}
I've used pytest quite a bit and I like it better than nose. With pytest you
can create a conftest.py file which can setup exclusions programmatically (e.g.
based on the current sys.executable), which would help consolidate the
exclusions by moving them from the nosetests cli args currently in various
shell scripts and tox.ini, into the test code itself. It's possible to do this
with nose as well, but IIRC it takes a bit more boilerplate.
> Python postcommit 2.7 tests are failing since we try to run a test with
> Python 3.6+ syntax on Python 2.
> ---
>
> Key: BEAM-8229
> URL: https://issues.apache.org/jira/browse/BEAM-8229
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Ahmet Altay
>Assignee: Udi Meiri
>Priority: Blocker
>
> Root cause: https://github.com/apache/beam/pull/9098
> 20:12:14 Traceback (most recent call last):
> 20:12:14 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python2_PR/src/build/gradleenv/-194514014/local/lib/python2.7/site-packages/nose/loader.py",
> line 418, in loadTestsFromName
> 20:12:14 addr.filename, addr.module)
> 20:12:14 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python2_PR/src/build/gradleenv/-194514014/local/lib/python2.7/site-packages/nose/importer.py",
> line 47, in importFromPath
> 20:12:14 return self.importFromDir(dir_path, fqname)
> 20:12:14 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python2_PR/src/build/gradleenv/-194514014/local/lib/python2.7/site-packages/nose/importer.py",
> line 94, in importFromDir
> 20:12:14 mod = load_module(part_fqname, fh, filename, desc)
> 20:12:14 SyntaxError: invalid syntax (external_test_py37.py, line 46)
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (BEAM-8229) Python postcommit 2.7 tests are failing since we try to run a test with Python 3.6+ syntax on Python 2.
[
https://issues.apache.org/jira/browse/BEAM-8229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16929360#comment-16929360
]
Chad Dombrova commented on BEAM-8229:
-
Back with more insight.
The integration tests don't go through tox, which means logic regarding
viritualenv setup and test exclusion is spread around the project a bit.
Outside of tox.ini test exclusion is performed in build.gradle,
run_integration_tests.sh, run_validatescontainer.sh.
Currently it looks like the integration tests don't run the python3-specific
tests at all:
Here's run_integration_tests.sh:
{code:java}
python setup.py nosetests \
--test-pipeline-options="$PIPELINE_OPTS" \
--with-xunitmp --xunitmp-file=$XUNIT_FILE \
--ignore-files '.*py3.py$' \
$TEST_OPTS
{code}
Since this is already a bit of a sub-par situation (not running the
py3-specific tests), I think the most reasonable solution is to extend this
regex to py3[0-9]*, which will exclude my py37 tests as well. (Also, note
that since this is a regex and not a glob, the period should have been escaped:
as it is it means "any character").
I think the ideal long term solution is to make all tests pass through tox,
which will increase their discoverability, runability, and standardization.
> Python postcommit 2.7 tests are failing since we try to run a test with
> Python 3.6+ syntax on Python 2.
> ---
>
> Key: BEAM-8229
> URL: https://issues.apache.org/jira/browse/BEAM-8229
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Ahmet Altay
>Assignee: Udi Meiri
>Priority: Blocker
>
> Root cause: https://github.com/apache/beam/pull/9098
> 20:12:14 Traceback (most recent call last):
> 20:12:14 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python2_PR/src/build/gradleenv/-194514014/local/lib/python2.7/site-packages/nose/loader.py",
> line 418, in loadTestsFromName
> 20:12:14 addr.filename, addr.module)
> 20:12:14 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python2_PR/src/build/gradleenv/-194514014/local/lib/python2.7/site-packages/nose/importer.py",
> line 47, in importFromPath
> 20:12:14 return self.importFromDir(dir_path, fqname)
> 20:12:14 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python2_PR/src/build/gradleenv/-194514014/local/lib/python2.7/site-packages/nose/importer.py",
> line 94, in importFromDir
> 20:12:14 mod = load_module(part_fqname, fh, filename, desc)
> 20:12:14 SyntaxError: invalid syntax (external_test_py37.py, line 46)
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
