[
https://issues.apache.org/jira/browse/BEAM-6527?focusedWorklogId=211320&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-211320
]
ASF GitHub Bot logged work on BEAM-6527:
----------------------------------------
Author: ASF GitHub Bot
Created on: 11/Mar/19 21:39
Start Date: 11/Mar/19 21:39
Worklog Time Spent: 10m
Work Description: markflyhigh commented on pull request #7675:
[BEAM-6527] Use Gradle to parallel Python tox tests
URL: https://github.com/apache/beam/pull/7675#discussion_r264442150
##########
File path:
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
##########
@@ -1611,18 +1611,48 @@ class BeamModulePlugin implements Plugin<Project> {
outputs.dirs(project.ext.envdir)
}
+ def pythonSdkDeps = project.files(
+ project.fileTree(dir: "${project.rootDir}", include: [
+ 'model/**',
+ 'sdks/python/apache_beam/**/*.py',
+ 'sdks/python/apache_beam/**/*.pyx',
+ 'sdks/python/apache_beam/**/*.pxd',
+ 'sdks/python/apache_beam/testing/data/**',
+ 'sdks/python/apache_beam/scripts/**',
+ 'sdks/python/.pylintrc',
+ 'sdks/python/MANIFEST.in',
+ 'sdks/python/gen_protos.py',
+ 'sdks/python/setup.cfg',
+ 'sdks/python/setup.py',
+ 'sdks/python/test_config.py',
+ 'sdks/python/tox.ini',
+ ])
+ )
+
project.configurations { distConfig }
project.task('sdist', dependsOn: 'setupVirtualenv') {
doLast {
+ // Copy sdk sources to isolate directory
+ def copiedSrcDir = "${project.buildDir}/srcs"
+ project.copy {
+ from pythonSdkDeps
+ into copiedSrcDir
+ }
+
+ // Build artifact
project.exec {
executable 'sh'
- args '-c', ". ${project.ext.envdir}/bin/activate && cd
${pythonRootDir} && python setup.py sdist --keep-temp --formats zip,gztar
--dist-dir ${project.buildDir}"
+ args '-c', ". ${project.ext.envdir}/bin/activate && cd
${copiedSrcDir}/sdks/python && python setup.py sdist --formats zip,gztar
--dist-dir ${project.buildDir}"
Review comment:
I added this flag to fix the parallel failure in integration tests since the
temp directory is shared between different build processes and by default it's
deleted after a build finish. However, it never works for tox tests.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 211320)
Time Spent: 2h 10m (was: 2h)
> Parallel tox (unit) tests run on Jenkins
> ----------------------------------------
>
> Key: BEAM-6527
> URL: https://issues.apache.org/jira/browse/BEAM-6527
> Project: Beam
> Issue Type: Sub-task
> Components: testing
> Reporter: Mark Liu
> Assignee: Mark Liu
> Priority: Major
> Labels: triaged
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> Existing tox unit test suite (basic, gcp and cython) will be enabled in
> multiple version of Python 3, which will significantly increase runtime of
> Pre/PostCommit build. A parallel is wanted in tox or Gradle invocation to
> control the time in a reasonable range (<30mins for PreCommit is desired).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
