[
https://issues.apache.org/jira/browse/BEAM-5058?focusedWorklogId=171254&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-171254
]
ASF GitHub Bot logged work on BEAM-5058:
----------------------------------------
Author: ASF GitHub Bot
Created on: 01/Dec/18 00:30
Start Date: 01/Dec/18 00:30
Worklog Time Spent: 10m
Work Description: markflyhigh commented on a change in pull request
#7163: [BEAM-5058] Run basic ITs in Python Precommit in parallel
URL: https://github.com/apache/beam/pull/7163#discussion_r238040508
##########
File path:
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
##########
@@ -1502,5 +1502,89 @@ artifactId=${project.name}
dependsOn ':beam-sdks-java-container:docker'
}
}
+
+ /**
***********************************************************************************************/
+
+ def randomGenerator = { int length ->
+ String charSet = (('A'..'Z')+('0'..'9')).join()
+ new Random().with{
+ (1..length).collect { charSet[ nextInt(charSet.length()) ] }.join()
+ }
+ }
+
+ project.ext.applyPythonNature = {
+
+ // Define common lifecycle tasks and artifact types
+ project.apply plugin: "base"
+
+ // For some reason base doesn't define a test task so we define it
below and make
+ // check depend on it. This makes the Python project similar to the task
layout like
+ // Java projects, see
https://docs.gradle.org/4.2.1/userguide/img/javaPluginTasks.png
+ project.task('test', type: Test) {}
+ project.check.dependsOn project.test
+
+
project.evaluationDependsOn(":beam-runners-google-cloud-dataflow-java-fn-api-worker")
+
+ // Due to Beam-4256, we need to limit the length of virtualenv path to
make the
+ // virtualenv activated properly. So instead of using build directory of
the
+ // subproject, we put all in build directory of root with fixed length
of random
+ // string as path suffix.
+ project.ext.envdir =
"${project.rootProject.buildDir}/gradleenv/${randomGenerator(15)}"
Review comment:
Good point. A consistent env path make build directory more cleaner. Will
change to hash value.
As for clean up, we can consider call `cleanupPython` (line 1578) in Jenkins
before and after each build. However, I feel that Jenkins starts a clean
directory before each build but I don't have evidence yet.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 171254)
Time Spent: 5h 20m (was: 5h 10m)
> Python precommits should run E2E tests
> --------------------------------------
>
> Key: BEAM-5058
> URL: https://issues.apache.org/jira/browse/BEAM-5058
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, testing
> Reporter: Udi Meiri
> Assignee: Mark Liu
> Priority: Major
> Time Spent: 5h 20m
> Remaining Estimate: 0h
>
> According to [https://beam.apache.org/contribute/testing/] (which I'm working
> on), end-to-end tests should be run in precommit on each combination of
> \{batch, streaming}x\{SDK language}x\{supported runner}.
> At least 2 tests need to be added to Python's precommit: wordcount and
> wordcount_streaming on Dataflow, and possibly on other supported runners
> (direct runner and new runners plz).
> These tests should be configured to run from a Gradle sub-project, so that
> they're run in parallel to the unit tests.
> Example that parallelizes Java precommit integration tests:
> [https://github.com/apache/beam/pull/5731]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
