[
https://issues.apache.org/jira/browse/BEAM-11415?focusedWorklogId=532175&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-532175
]
ASF GitHub Bot logged work on BEAM-11415:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 06/Jan/21 22:42
Start Date: 06/Jan/21 22:42
Worklog Time Spent: 10m
Work Description: lostluck commented on a change in pull request #13678:
URL: https://github.com/apache/beam/pull/13678#discussion_r552986389
##########
File path: sdks/go/test/run_validatesrunner_tests.sh
##########
@@ -16,39 +16,68 @@
# limitations under the License.
# This script executes ValidatesRunner tests including launching any additional
-# services needed, such as job services or expansion services. The following
-# runners are supported, and selected via a flag:
+# services needed, such as job services or expansion services. This script
+# should be executed from the root of the Beam repository.
#
+# The following runners are supported, and selected via a flag:
# --runner {portable|direct|flink} (default: portable)
# Select which runner to execute tests on. This flag also determines which
# services to start up and which tests may be skipped.
# direct - Go SDK Direct Runner
# portable - (default) Python Portable Runner (aka. Reference Runner or
FnAPI Runner)
# flink - Java Flink Runner (local mode)
# spark - Java Spark Runner (local mode)
+# dataflow - Dataflow Runner
#
-# --flink_job_server_jar -> Filepath to jar, used if runner is Flink.
-# --spark_job_server_jar -> Filepath to jar, used if runner is Spark.
-# --endpoint -> Replaces jar filepath with existing job server endpoint.
+# General flags:
+# --timeout -> Timeout for the go test command, on a per-package level.
+# --endpoint -> An endpoint for an existing job server outside the script.
+# If present, job server jar flags are ignored.
+# --expansion_service_jar -> Filepath to jar for expansion service, for
+# runners that support cross-language.
+# --expansion_addr -> An endpoint for an existing expansion service outside
+# the script. If present, expansion_service_jar is ignored.
#
-# --expansion_service_jar -> Filepath to jar for expansion service.
-# --expansion_addr -> Replaces jar filepath with existing expansion service
endpoint.
-#
-# Execute from the root of the repository. This script requires that necessary
-# services can be built from the repository.
+# Runner-specific flags:
+# Flink
+# --flink_job_server_jar -> Filepath to jar, used if runner is Flink.
+# Spark
+# --spark_job_server_jar -> Filepath to jar, used if runner is Spark.
+# Dataflow
+# --dataflow_project -> GCP project to run Dataflow jobs on.
+# --project -> Same project as dataflow-project, but in URL format, for
+# example in the format "us.gcr.io/<project>".
+# --region -> GCP region to run Dataflow jobs on.
+# --gcs_location -> GCS URL for storing temporary files for Dataflow jobs.
+# --dataflow_worker_jar -> The Dataflow worker jar to use when running jobs.
+# If not specified, the script attempts to retrieve a previously built
+# jar from the appropriate gradle module, which may not succeed.
set -e
set -v
+# Default runner.
RUNNER=portable
+# Default timeout. This timeout is applied per-package, as tests in different
+# packages are executed in parallel.
+TIMEOUT=1h
+
+# Where to store integration test outputs.
+GCS_LOCATION=gs://clouddfe-danoliveira/temp-storage-go-end-to-end-tests
Review comment:
Definitely will need to change this to something more jenkins safe. And
the defaults below.
##########
File path: sdks/go/test/run_validatesrunner_tests.sh
##########
@@ -16,39 +16,68 @@
# limitations under the License.
# This script executes ValidatesRunner tests including launching any additional
-# services needed, such as job services or expansion services. The following
-# runners are supported, and selected via a flag:
+# services needed, such as job services or expansion services. This script
+# should be executed from the root of the Beam repository.
#
+# The following runners are supported, and selected via a flag:
# --runner {portable|direct|flink} (default: portable)
# Select which runner to execute tests on. This flag also determines which
# services to start up and which tests may be skipped.
# direct - Go SDK Direct Runner
# portable - (default) Python Portable Runner (aka. Reference Runner or
FnAPI Runner)
# flink - Java Flink Runner (local mode)
# spark - Java Spark Runner (local mode)
+# dataflow - Dataflow Runner
#
-# --flink_job_server_jar -> Filepath to jar, used if runner is Flink.
-# --spark_job_server_jar -> Filepath to jar, used if runner is Spark.
-# --endpoint -> Replaces jar filepath with existing job server endpoint.
+# General flags:
+# --timeout -> Timeout for the go test command, on a per-package level.
+# --endpoint -> An endpoint for an existing job server outside the script.
+# If present, job server jar flags are ignored.
+# --expansion_service_jar -> Filepath to jar for expansion service, for
+# runners that support cross-language.
+# --expansion_addr -> An endpoint for an existing expansion service outside
+# the script. If present, expansion_service_jar is ignored.
#
-# --expansion_service_jar -> Filepath to jar for expansion service.
-# --expansion_addr -> Replaces jar filepath with existing expansion service
endpoint.
-#
-# Execute from the root of the repository. This script requires that necessary
-# services can be built from the repository.
+# Runner-specific flags:
+# Flink
+# --flink_job_server_jar -> Filepath to jar, used if runner is Flink.
+# Spark
+# --spark_job_server_jar -> Filepath to jar, used if runner is Spark.
+# Dataflow
+# --dataflow_project -> GCP project to run Dataflow jobs on.
+# --project -> Same project as dataflow-project, but in URL format, for
Review comment:
```suggestion
# --project -> Same project as --dataflow_project, but in URL format, for
```
##########
File path: sdks/go/test/run_validatesrunner_tests.sh
##########
@@ -151,11 +222,70 @@ if [[ "$RUNNER" == "flink" || "$RUNNER" == "spark" ||
"$RUNNER" == "portable" ]]
java -jar $EXPANSION_SERVICE_JAR $EXPANSION_PORT &
EXPANSION_PID=$!
fi
+fi
+
+if [[ "$RUNNER" == "dataflow" ]]; then
+ # Verify docker and gcloud commands exist
+ command -v docker
+ docker -v
+ command -v gcloud
+ gcloud --version
+
+ # ensure gcloud is version 186 or above
+ TMPDIR=$(mktemp -d)
+ gcloud_ver=$(gcloud -v | head -1 | awk '{print $4}')
+ if [[ "$gcloud_ver" < "186" ]]
+ then
+ pushd $TMPDIR
+ curl
https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-186.0.0-linux-x86_64.tar.gz
--output gcloud.tar.gz
+ tar xf gcloud.tar.gz
+ ./google-cloud-sdk/install.sh --quiet
+ . ./google-cloud-sdk/path.bash.inc
+ popd
+ gcloud components update --quiet || echo 'gcloud components update failed'
+ gcloud -v
+ fi
- ARGS="$ARGS --endpoint=$ENDPOINT --expansion_addr=$EXPANSION_ADDR"
+ # Build the container
+ TAG=$(date +%Y%m%d-%H%M%S)
+ CONTAINER=us.gcr.io/$PROJECT/$USER/beam_go_sdk
Review comment:
The $USER here is just the standard OS $USER executing the script then?
Just confirming something wasn't missed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 532175)
Time Spent: 3h 50m (was: 3h 40m)
> Create ValidatesRunner test framework for the Go SDK.
> -----------------------------------------------------
>
> Key: BEAM-11415
> URL: https://issues.apache.org/jira/browse/BEAM-11415
> Project: Beam
> Issue Type: Test
> Components: sdk-go
> Reporter: Daniel Oliveira
> Assignee: Daniel Oliveira
> Priority: P2
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
> I've actually already started working on this and it's ending up being more
> like an expansion on the existing integration test system.
> Current main improvements this will add:
> * The new approach is based on the go testing framework, meaning tests do not
> need to be manually registered with a driver. This is bug for usability and
> scalability as more tests will need to be added.
> * Will allow tests to be run with the "go test ..." command, assuming
> necessary services are running.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
