[
https://issues.apache.org/jira/browse/BEAM-5703?focusedWorklogId=159279&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-159279
]
ASF GitHub Bot logged work on BEAM-5703:
----------------------------------------
Author: ASF GitHub Bot
Created on: 26/Oct/18 18:06
Start Date: 26/Oct/18 18:06
Worklog Time Spent: 10m
Work Description: charlesccychen closed pull request #6817:
[BEAM-5703]Migrate py integration test to using customized worker jar
URL: https://github.com/apache/beam/pull/6817
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/sdks/python/build.gradle b/sdks/python/build.gradle
index 59fb73ed957..ffc13e7fa2e 100644
--- a/sdks/python/build.gradle
+++ b/sdks/python/build.gradle
@@ -25,6 +25,8 @@ apply plugin: "base"
task test {}
check.dependsOn test
+evaluationDependsOn(":beam-runners-google-cloud-dataflow-java-fn-api-worker")
+
def envdir = "${project.buildDir}/gradleenv"
def tox_opts = "-c tox.ini --recreate"
@@ -322,11 +324,19 @@ task validatesRunnerBatchTests(dependsOn:
'installGcpTest') {
}
task validatesRunnerStreamingTests(dependsOn: 'installGcpTest') {
+ dependsOn ":beam-runners-google-cloud-dataflow-java-fn-api-worker:shadowJar"
+
+ def dataflowProject = project.findProperty('dataflowProject') ?:
'apache-beam-testing'
+ def dataflowTempRoot = project.findProperty('dataflowTempRoot') ?:
'gs://temp-storage-for-end-to-end-tests'
+ def dataflowWorkerJar = project.findProperty('dataflowWorkerJar') ?:
project(":beam-runners-google-cloud-dataflow-java-fn-api-worker").shadowJar.archivePath
+
doLast {
exec {
executable 'sh'
// TODO(BEAM-3544,BEAM-5025): Disable tests with sickbay-streaming tag.
- args '-c', ". ${envdir}/bin/activate && ./scripts/run_postcommit.sh
ValidatesRunner,'!sickbay-streaming' streaming"
+ args '-c', ". ${envdir}/bin/activate && ./scripts/run_postcommit.sh
ValidatesRunner,'!sickbay-streaming' " +
+ "streaming TestDataflowRunner " +
+ "${dataflowTempRoot} ${dataflowProject} ${dataflowWorkerJar} "
}
}
}
diff --git a/sdks/python/scripts/run_postcommit.sh
b/sdks/python/scripts/run_postcommit.sh
index 2bd1ca41889..f14b77206b8 100755
--- a/sdks/python/scripts/run_postcommit.sh
+++ b/sdks/python/scripts/run_postcommit.sh
@@ -30,12 +30,14 @@
# Usage check.
if (( $# < 2 )); then
- printf "Usage: \n$> ./scripts/run_postcommit.sh <test_type> <pipeline_type>
<runner_type> [gcp_location] [gcp_project]"
+ printf "Usage: \n$> ./scripts/run_postcommit.sh <test_type> <pipeline_type>
<runner_type> "
+ printf " [gcp_location] [gcp_project] [dataflow_worker_jar]"
printf "\n\ttest_type: [required] ValidatesRunner or IT"
printf "\n\tpipeline_type: [required] streaming or batch"
printf "\n\trunner_type: [optional] TestDataflowRunner or TestDirectRunner"
printf "\n\tgcp_location: [optional] A gs:// path to stage artifacts and
output results"
- printf "\n\tgcp_project: [optional] A GCP project to run Dataflow
pipelines\n"
+ printf "\n\tgcp_project: [optional] A GCP project to run Dataflow pipelines"
+ printf "\n\tdataflow_worker_jar: [optional] Customized worker jar for
dataflow runner.\n"
exit 1
fi
@@ -64,6 +66,9 @@ GCS_LOCATION=${4:-gs://temp-storage-for-end-to-end-tests}
PROJECT=${5:-apache-beam-testing}
+# Path to customized worker jar file.
+DATAFLOW_WORKER_JAR=${6:-}
+
# Create a tarball
python setup.py -q sdist
@@ -87,10 +92,20 @@ PIPELINE_OPTIONS=(
"--sleep_secs=20"
)
+# Validation: For testing purpose, we require to use the optional feature of
dataflow worker
+# jar file, when streaming and TestDataFlow are both specified.
+if [ "$2" = "streaming" ] && [ "$3" = "TestDataflowRunner" ] && [ -z
$DATAFLOW_WORKER_JAR]; then
+ echo "Failure: Testing pipelines for streaming expect a dataflow worker jar
file."
+ exit 1
+fi
+
# Add streaming flag if specified.
if [[ "$2" = "streaming" ]]; then
echo ">>> Set test pipeline to streaming"
PIPELINE_OPTIONS+=("--streaming")
+ if [ "$3" = "TestDataflowRunner" ]; then
+ PIPELINE_OPTIONS+=("--dataflow_worker_jar $DATAFLOW_WORKER_JAR")
+ fi
else
echo ">>> Set test pipeline to batch"
fi
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 159279)
Time Spent: 1h (was: 50m)
> Migrate Python streaming and portable integration tests to use a staged
> dataflow worker jar
> -------------------------------------------------------------------------------------------
>
> Key: BEAM-5703
> URL: https://issues.apache.org/jira/browse/BEAM-5703
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
> Reporter: Henning Rohde
> Assignee: Ruoyun Huang
> Priority: Major
> Time Spent: 1h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
