[
https://issues.apache.org/jira/browse/BEAM-3827?focusedWorklogId=104083&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104083
]
ASF GitHub Bot logged work on BEAM-3827:
----------------------------------------
Author: ASF GitHub Bot
Created on: 21/May/18 15:15
Start Date: 21/May/18 15:15
Worklog Time Spent: 10m
Work Description: kennknowles closed pull request #5409: [BEAM-3827]
Invoke Go Dataflow integration tests from post-commit
URL: https://github.com/apache/beam/pull/5409
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/build.gradle b/build.gradle
index 9cc86f067cd..88a757dfff7 100644
--- a/build.gradle
+++ b/build.gradle
@@ -157,8 +157,18 @@ task goPreCommit() {
}
task goPostCommit() {
- // Same currently as precommit, but duplicated to provide a clearer signal
of reliability.
dependsOn ":goPreCommit"
+ dependsOn ":goIntegrationTests"
+}
+
+task goIntegrationTests() {
+ doLast {
+ exec {
+ executable 'sh'
+ args '-c', './sdks/go/test/run_integration_tests.sh'
+ }
+ }
+ dependsOn ":beam-sdks-go-test:build"
}
task pythonPreCommit() {
diff --git a/sdks/go/build.gradle b/sdks/go/build.gradle
index 9cd18b17972..7a3a95792d9 100644
--- a/sdks/go/build.gradle
+++ b/sdks/go/build.gradle
@@ -31,5 +31,8 @@ golang {
// Build debugging utilities
go 'build -o ./build/bin/beamctl
github.com/apache/beam/sdks/go/cmd/beamctl'
+
+ // Build integration test driver
+ go 'build -o ./build/bin/integration
github.com/apache/beam/sdks/go/test/integration'
}
}
diff --git a/sdks/go/test/build.gradle b/sdks/go/test/build.gradle
new file mode 100644
index 00000000000..6fcd8151a83
--- /dev/null
+++ b/sdks/go/test/build.gradle
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+apply from: project(":").file("build_rules.gradle")
+applyGoNature()
+
+description = "Apache Beam :: SDKs :: Go :: Test"
+
+// Figure out why the golang plugin does not add a build dependency between
projects.
+// Without the line below, we get spurious errors about not being able to
resolve
+// "./github.com/apache/beam/sdks/go"
+resolveBuildDependencies.dependsOn ":beam-sdks-go:build"
+
+dependencies {
+ golang {
+ // TODO(herohde): use "./" prefix to prevent gogradle use base github
path, for now.
+ // TODO(herohde): get the pkg subdirectory only, if possible. We spend
mins pulling cmd/beamctl deps.
+ build name: './github.com/apache/beam/sdks/go', dir:
project(':beam-sdks-go').projectDir
+ test name: './github.com/apache/beam/sdks/go', dir:
project(':beam-sdks-go').projectDir
+ }
+}
+
+golang {
+ packagePath = 'github.com/apache/beam/sdks/go/test'
+ build {
+ // Build the linux-amd64 worker. The native version is built in the parent
to
+ // have a fixed name, which is not possible with multiple target
platforms. The
+ // script would otherwise have to figure out which arch/platform binary to
invoke.
+
+ targetPlatform = ['linux-amd64']
+ go 'build -o ./build/bin/linux-amd64/worker
github.com/apache/beam/sdks/go/test/integration'
+ }
+}
diff --git a/sdks/go/test/run_integration_tests.sh
b/sdks/go/test/run_integration_tests.sh
new file mode 100755
index 00000000000..5f9317f4b70
--- /dev/null
+++ b/sdks/go/test/run_integration_tests.sh
@@ -0,0 +1,92 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# This script will be run by Jenkins as a post commit test. In order to run
+# locally make the following changes:
+#
+# GCS_LOCATION -> Temporary location to use for service tests.
+# PROJECT -> Project name to use for docker images.
+# DATAFLOW_PROJECT -> Project name to use for dataflow.
+#
+# Execute from the root of the repository. It assumes binaries are built.
+
+set -e
+set -v
+
+# Where to store integration test outputs.
+GCS_LOCATION=gs://temp-storage-for-end-to-end-tests
+
+# Project for the container and integration test
+PROJECT=apache-beam-testing
+DATAFLOW_PROJECT=apache-beam-testing
+
+# Verify in the root of the repository
+test -d sdks/go/test
+
+# Verify docker and gcloud commands exist
+command -v docker
+command -v gcloud
+docker -v
+gcloud -v
+
+# ensure gcloud is version 186 or above
+TMPDIR=$(mktemp -d)
+gcloud_ver=$(gcloud -v | head -1 | awk '{print $4}')
+if [[ "$gcloud_ver" < "186" ]]
+then
+ pushd $TMPDIR
+ curl
https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-186.0.0-linux-x86_64.tar.gz
--output gcloud.tar.gz
+ tar xf gcloud.tar.gz
+ ./google-cloud-sdk/install.sh --quiet
+ . ./google-cloud-sdk/path.bash.inc
+ popd
+ gcloud components update --quiet || echo 'gcloud components update failed'
+ gcloud -v
+fi
+
+# Build the container
+TAG=$(date +%Y%m%d-%H%M%S)
+CONTAINER=us.gcr.io/$PROJECT/$USER/go
+echo "Using container $CONTAINER"
+./gradlew :beam-sdks-go-container:docker
-Pdocker-repository-root=us.gcr.io/$PROJECT/$USER -Pdocker-tag=$TAG
+
+# Verify it exists
+docker images | grep $TAG
+
+# Push the container
+gcloud docker -- push $CONTAINER
+
+echo ">>> RUNNING DATAFLOW INTEGRATION TESTS"
+./sdks/go/build/bin/integration \
+ --runner=dataflow \
+ --project=$DATAFLOW_PROJECT \
+ --worker_harness_container_image=$CONTAINER:$TAG \
+ --staging_location=$GCS_LOCATION/staging-validatesrunner-test \
+ --temp_location=$GCS_LOCATION/temp-validatesrunner-test \
+ --worker_binary=./sdks/go/test/build/bin/linux-amd64/worker
+
+# TODO(herohde) 5/9/2018: run other runner tests here to reuse the container
image?
+
+# Delete the container locally and remotely
+docker rmi $CONTAINER:$TAG || echo "Failed to remove container"
+gcloud --quiet container images delete $CONTAINER:$TAG || echo "Failed to
delete container"
+
+# Clean up tempdir
+rm -rf $TMPDIR
+
+echo ">>> SUCCESS"
diff --git a/settings.gradle b/settings.gradle
index f0703a32adb..ab09fad3f98 100644
--- a/settings.gradle
+++ b/settings.gradle
@@ -66,6 +66,8 @@ include "beam-sdks-go-container"
project(":beam-sdks-go-container").dir = file("sdks/go/container")
include "beam-sdks-go-examples"
project(":beam-sdks-go-examples").dir = file("sdks/go/examples")
+include "beam-sdks-go-test"
+project(":beam-sdks-go-test").dir = file("sdks/go/test")
include "beam-sdks-java-build-tools"
project(":beam-sdks-java-build-tools").dir = file("sdks/java/build-tools")
include "beam-sdks-java-container"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 104083)
Time Spent: 1h 50m (was: 1h 40m)
> Add Go SDK integration tests
> ----------------------------
>
> Key: BEAM-3827
> URL: https://issues.apache.org/jira/browse/BEAM-3827
> Project: Beam
> Issue Type: Task
> Components: sdk-go
> Reporter: Henning Rohde
> Assignee: Henning Rohde
> Priority: Major
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> We should add post-commit testing similar to ValidateRunner tests to ensure
> that the model is implemented correctly.
> Proposal:
> https://docs.google.com/document/d/1jy6EE7D4RjgfNV0FhD3rMsT1YKhnUfcHRZMAlC6ygXw/edit?usp=sharing
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
