[
https://issues.apache.org/jira/browse/BEAM-5419?focusedWorklogId=174671&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-174671
]
ASF GitHub Bot logged work on BEAM-5419:
----------------------------------------
Author: ASF GitHub Bot
Created on: 12/Dec/18 20:52
Start Date: 12/Dec/18 20:52
Worklog Time Spent: 10m
Work Description: mxm closed pull request #7229: [BEAM-5419] Add Flink
multi-version builds
URL: https://github.com/apache/beam/pull/7229
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/runners/flink/1.6/build.gradle b/runners/flink/1.6/build.gradle
new file mode 100644
index 000000000000..1792389e7dbf
--- /dev/null
+++ b/runners/flink/1.6/build.gradle
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+def basePath = '..'
+
+/* All properties required for loading the Flink build script */
+project.ext {
+ // Set the version of all Flink-related dependencies here.
+ flink_version = '1.6.2'
+ // Look for the source code in the parent module
+ main_source_dirs = ["$basePath/src/main/java"]
+ test_source_dirs = ["$basePath/src/test/java"]
+ main_resources_dirs = ["$basePath/src/main/resources"]
+ test_resources_dirs = ["$basePath/src/test/resources"]
+}
+
+// Load the main build script which contains all build logic.
+apply from: "$basePath/flink_runner.gradle"
diff --git a/runners/flink/1.6/job-server-container/build.gradle
b/runners/flink/1.6/job-server-container/build.gradle
new file mode 100644
index 000000000000..a1ae99a5d6eb
--- /dev/null
+++ b/runners/flink/1.6/job-server-container/build.gradle
@@ -0,0 +1,25 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+project.ext {
+ docker_file = file('../../job-server-container/Dockerfile')
+ startup_script = file('../../job-server-container/flink-job-server.sh')
+}
+
+// Load the main build script which contains all build logic.
+apply from: '../../job-server-container/flink_job_server_container.gradle'
diff --git a/runners/flink/1.6/job-server/build.gradle
b/runners/flink/1.6/job-server/build.gradle
new file mode 100644
index 000000000000..d5251e048213
--- /dev/null
+++ b/runners/flink/1.6/job-server/build.gradle
@@ -0,0 +1,30 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+def basePath = '../../job-server'
+
+project.ext {
+ // Look for the source code in the parent module
+ main_source_dirs = ["$basePath/src/main/java"]
+ test_source_dirs = ["$basePath/src/test/java"]
+ main_resources_dirs = ["$basePath/src/main/resources"]
+ test_resources_dirs = ["$basePath/src/test/resources"]
+}
+
+// Load the main build script which contains all build logic.
+apply from: "$basePath/flink_job_server.gradle"
diff --git a/runners/flink/build.gradle b/runners/flink/build.gradle
index 87c827d06cd1..0db72f3b3e7a 100644
--- a/runners/flink/build.gradle
+++ b/runners/flink/build.gradle
@@ -16,123 +16,16 @@
* limitations under the License.
*/
-import groovy.json.JsonOutput
-
-apply plugin: org.apache.beam.gradle.BeamModulePlugin
-applyJavaNature()
-
-
-description = "Apache Beam :: Runners :: Flink"
-
-/*
- * We need to rely on manually specifying these evaluationDependsOn to ensure
that
- * the following projects are evaluated before we evaluate this project. This
is because
- * we are attempting to reference the "sourceSets.test.output" directly.
- */
-evaluationDependsOn(":beam-sdks-java-core")
-evaluationDependsOn(":beam-runners-core-java")
-
-
-test {
- systemProperty "log4j.configuration", "log4j-test.properties"
- //systemProperty "org.slf4j.simpleLogger.defaultLogLevel", "debug"
- jvmArgs "-XX:-UseGCOverheadLimit"
- if (System.getProperty("beamSurefireArgline")) {
- jvmArgs System.getProperty("beamSurefireArgline")
- }
-}
-
-configurations {
- validatesRunner
-}
-
-def flink_version = "1.5.5"
-
-dependencies {
- compile library.java.guava
- shadow project(path: ":beam-sdks-java-core", configuration: "shadow")
- shadow project(path: ":beam-runners-core-java", configuration: "shadow")
- shadow project(path: ":beam-runners-core-construction-java", configuration:
"shadow")
- shadow project(path: ":beam-runners-java-fn-execution", configuration:
"shadow")
- shadow library.java.vendored_grpc_1_13_1
- shadow library.java.jackson_annotations
- shadow library.java.slf4j_api
- shadow library.java.joda_time
- shadow library.java.commons_compress
- shadow library.java.args4j
- shadow "org.apache.flink:flink-clients_2.11:$flink_version"
- shadow "org.apache.flink:flink-core:$flink_version"
- shadow "org.apache.flink:flink-metrics-core:$flink_version"
- shadow "org.apache.flink:flink-java:$flink_version"
- shadow "org.apache.flink:flink-runtime_2.11:$flink_version"
- shadow "org.apache.flink:flink-streaming-java_2.11:$flink_version"
- shadowTest project(path: ":beam-sdks-java-core", configuration: "shadowTest")
- // FlinkStateInternalsTest extends abstract StateInternalsTest
- shadowTest project(path: ":beam-runners-core-java", configuration:
"shadowTest")
- shadowTest library.java.commons_lang3
- shadowTest library.java.hamcrest_core
- shadowTest library.java.junit
- shadowTest library.java.mockito_core
- shadowTest library.java.google_api_services_bigquery
- shadowTest library.java.slf4j_simple
- shadowTest project(path: ":beam-sdks-java-io-google-cloud-platform",
configuration: "shadow")
- shadowTest library.java.jackson_dataformat_yaml
- shadowTest "org.apache.flink:flink-core:$flink_version:tests"
- shadowTest "org.apache.flink:flink-runtime_2.11:$flink_version:tests"
- shadowTest "org.apache.flink:flink-streaming-java_2.11:$flink_version:tests"
- shadowTest "org.apache.flink:flink-test-utils_2.11:$flink_version"
- shadowTest project(":beam-sdks-java-harness")
- validatesRunner project(path: ":beam-sdks-java-core", configuration:
"shadowTest")
- validatesRunner project(path: ":beam-runners-core-java", configuration:
"shadowTest")
- validatesRunner project(path: project.path, configuration: "shadow")
-}
-
-class ValidatesRunnerConfig {
- String name
- boolean streaming
-}
-
-def createValidatesRunnerTask(Map m) {
- def config = m as ValidatesRunnerConfig
- tasks.create(name: config.name, type: Test) {
- group = "Verification"
- def runnerType = config.streaming ? "streaming" : "batch"
- description = "Validates the ${runnerType} runner"
- def pipelineOptions = JsonOutput.toJson(
- ["--runner=TestFlinkRunner",
- "--streaming=${config.streaming}",
- // TODO This should be changed to > 1
- "--parallelism=1",
- ])
- systemProperty "beamTestPipelineOptions", pipelineOptions
- classpath = configurations.validatesRunner
- testClassesDirs =
files(project(":beam-sdks-java-core").sourceSets.test.output.classesDirs,
project(":beam-runners-core-java").sourceSets.test.output.classesDirs)
- // maxParallelForks decreased from 4 in order to avoid OOM errors
- maxParallelForks 2
- useJUnit {
- includeCategories 'org.apache.beam.sdk.testing.ValidatesRunner'
- excludeCategories
'org.apache.beam.sdk.testing.FlattenWithHeterogeneousCoders'
- excludeCategories 'org.apache.beam.sdk.testing.LargeKeys$Above100MB'
- excludeCategories 'org.apache.beam.sdk.testing.UsesCommittedMetrics'
- if (config.streaming) {
- excludeCategories 'org.apache.beam.sdk.testing.UsesImpulse'
- } else {
- excludeCategories
'org.apache.beam.sdk.testing.UsesUnboundedSplittableParDo'
- }
- excludeCategories 'org.apache.beam.sdk.testing.UsesTestStream'
- }
- }
-}
-
-createValidatesRunnerTask(name: "validatesRunnerBatch", streaming: false)
-createValidatesRunnerTask(name: "validatesRunnerStreaming", streaming: true)
-
-task validatesRunner {
- group = "Verification"
- description "Validates Flink runner"
- dependsOn validatesRunnerBatch
- dependsOn validatesRunnerStreaming
+/* All properties required for loading the Flink build script. */
+project.ext {
+ // Set the version of all Flink-related dependencies here.
+ flink_version = '1.5.5'
+ // Look for the source code in the current module
+ main_source_dirs = ['./src/main/java']
+ test_source_dirs = ['./src/test/java']
+ main_resources_dirs = ['./src/main/resources']
+ test_resources_dirs = ['./src/test/resources']
}
-// Generates :beam-runners-flink_2.11:runQuickstartJavaFlinkLocal
-createJavaExamplesArchetypeValidationTask(type: 'Quickstart', runner:
'FlinkLocal')
+// Load the main build script which contains all build logic.
+apply from: 'flink_runner.gradle'
diff --git a/runners/flink/flink_runner.gradle
b/runners/flink/flink_runner.gradle
new file mode 100644
index 000000000000..2cafd601b504
--- /dev/null
+++ b/runners/flink/flink_runner.gradle
@@ -0,0 +1,164 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/**
+ * Main Flink Runner build file shared by all of its build targets.
+ * The file needs to be parameterized by the Flink version and the source
directories.
+ *
+ * See build.gradle files for an example of how to use this script.
+ */
+
+import groovy.json.JsonOutput
+
+apply plugin: org.apache.beam.gradle.BeamModulePlugin
+applyJavaNature()
+
+description = "Apache Beam :: Runners :: Flink $flink_version"
+
+/*
+ * We need to rely on manually specifying these evaluationDependsOn to ensure
that
+ * the following projects are evaluated before we evaluate this project. This
is because
+ * we are attempting to reference the "sourceSets.test.output" directly.
+ */
+evaluationDependsOn(":beam-sdks-java-core")
+evaluationDependsOn(":beam-runners-core-java")
+
+/*
+ * We have to explicitly set all directories here to make sure each
+ * version of Flink has the correct overrides set.
+ */
+sourceSets {
+ main {
+ java {
+ srcDirs = main_source_dirs
+ }
+ resources {
+ srcDirs = main_resources_dirs
+ }
+ }
+ test {
+ java {
+ srcDirs = test_source_dirs
+ }
+ resources {
+ srcDirs = test_resources_dirs
+ }
+ }
+}
+
+test {
+ systemProperty "log4j.configuration", "log4j-test.properties"
+ //systemProperty "org.slf4j.simpleLogger.defaultLogLevel", "debug"
+ jvmArgs "-XX:-UseGCOverheadLimit"
+ if (System.getProperty("beamSurefireArgline")) {
+ jvmArgs System.getProperty("beamSurefireArgline")
+ }
+}
+
+configurations {
+ validatesRunner
+}
+
+dependencies {
+ compile library.java.guava
+ shadow project(path: ":beam-sdks-java-core", configuration: "shadow")
+ shadow project(path: ":beam-runners-core-java", configuration: "shadow")
+ shadow project(path: ":beam-runners-core-construction-java", configuration:
"shadow")
+ shadow project(path: ":beam-runners-java-fn-execution", configuration:
"shadow")
+ shadow library.java.vendored_grpc_1_13_1
+ shadow library.java.jackson_annotations
+ shadow library.java.slf4j_api
+ shadow library.java.joda_time
+ shadow library.java.commons_compress
+ shadow library.java.args4j
+ shadow "org.apache.flink:flink-clients_2.11:$flink_version"
+ shadow "org.apache.flink:flink-core:$flink_version"
+ shadow "org.apache.flink:flink-metrics-core:$flink_version"
+ shadow "org.apache.flink:flink-java:$flink_version"
+ shadow "org.apache.flink:flink-runtime_2.11:$flink_version"
+ shadow "org.apache.flink:flink-streaming-java_2.11:$flink_version"
+ shadowTest project(path: ":beam-sdks-java-core", configuration: "shadowTest")
+ // FlinkStateInternalsTest extends abstract StateInternalsTest
+ shadowTest project(path: ":beam-runners-core-java", configuration:
"shadowTest")
+ shadowTest library.java.commons_lang3
+ shadowTest library.java.hamcrest_core
+ shadowTest library.java.junit
+ shadowTest library.java.mockito_core
+ shadowTest library.java.google_api_services_bigquery
+ shadowTest library.java.slf4j_simple
+ shadowTest project(path: ":beam-sdks-java-io-google-cloud-platform",
configuration: "shadow")
+ shadowTest library.java.jackson_dataformat_yaml
+ shadowTest "org.apache.flink:flink-core:$flink_version:tests"
+ shadowTest "org.apache.flink:flink-runtime_2.11:$flink_version:tests"
+ shadowTest "org.apache.flink:flink-streaming-java_2.11:$flink_version:tests"
+ shadowTest "org.apache.flink:flink-test-utils_2.11:$flink_version"
+ shadowTest project(":beam-sdks-java-harness")
+ validatesRunner project(path: ":beam-sdks-java-core", configuration:
"shadowTest")
+ validatesRunner project(path: ":beam-runners-core-java", configuration:
"shadowTest")
+ validatesRunner project(path: project.path, configuration: "shadow")
+}
+
+class ValidatesRunnerConfig {
+ String name
+ boolean streaming
+}
+
+def createValidatesRunnerTask(Map m) {
+ def config = m as ValidatesRunnerConfig
+ tasks.create(name: config.name, type: Test) {
+ group = "Verification"
+ def runnerType = config.streaming ? "streaming" : "batch"
+ description = "Validates the ${runnerType} runner"
+ def pipelineOptions = JsonOutput.toJson(
+ ["--runner=TestFlinkRunner",
+ "--streaming=${config.streaming}",
+ // TODO This should be changed to > 1
+ "--parallelism=1",
+ ])
+ systemProperty "beamTestPipelineOptions", pipelineOptions
+ classpath = configurations.validatesRunner
+ testClassesDirs =
files(project(":beam-sdks-java-core").sourceSets.test.output.classesDirs,
project(":beam-runners-core-java").sourceSets.test.output.classesDirs)
+ // maxParallelForks decreased from 4 in order to avoid OOM errors
+ maxParallelForks 2
+ useJUnit {
+ includeCategories 'org.apache.beam.sdk.testing.ValidatesRunner'
+ excludeCategories
'org.apache.beam.sdk.testing.FlattenWithHeterogeneousCoders'
+ excludeCategories 'org.apache.beam.sdk.testing.LargeKeys$Above100MB'
+ excludeCategories 'org.apache.beam.sdk.testing.UsesCommittedMetrics'
+ if (config.streaming) {
+ excludeCategories 'org.apache.beam.sdk.testing.UsesImpulse'
+ } else {
+ excludeCategories
'org.apache.beam.sdk.testing.UsesUnboundedSplittableParDo'
+ }
+ excludeCategories 'org.apache.beam.sdk.testing.UsesTestStream'
+ }
+ }
+}
+
+createValidatesRunnerTask(name: "validatesRunnerBatch", streaming: false)
+createValidatesRunnerTask(name: "validatesRunnerStreaming", streaming: true)
+
+task validatesRunner {
+ group = "Verification"
+ description "Validates Flink runner"
+ dependsOn validatesRunnerBatch
+ dependsOn validatesRunnerStreaming
+}
+
+// Generates :beam-runners-flink_2.11:runQuickstartJavaFlinkLocal
+createJavaExamplesArchetypeValidationTask(type: 'Quickstart', runner:
'FlinkLocal')
diff --git a/runners/flink/job-server-container/build.gradle
b/runners/flink/job-server-container/build.gradle
index d2b026804d79..96d4e12117fe 100644
--- a/runners/flink/job-server-container/build.gradle
+++ b/runners/flink/job-server-container/build.gradle
@@ -16,39 +16,10 @@
* limitations under the License.
*/
-/**
- * Build a Docker image to bootstrap FlinkJobServerDriver which requires a
Java environment.
- * Alternatively, it can also be bootstrapped through
:beam-runners-flink_2.11-job-server:runShadow
- * or by directly running the generated JAR file.
- */
-
-apply plugin: org.apache.beam.gradle.BeamModulePlugin
-applyDockerNature()
-
-description = "Apache Beam :: Runners :: Flink :: Job Server :: Container"
-
-configurations {
- dockerDependency
-}
-
-dependencies {
- dockerDependency project(path: ":beam-runners-flink_2.11-job-server",
configuration: "shadow")
-}
-
-task copyDockerfileDependencies(type: Copy) {
- // Required Jars
- from configurations.dockerDependency
- rename 'beam-runners-flink_2.11-job-server.*.jar',
'beam-runners-flink-job-server.jar'
- into "build/target"
- // Entry script
- from file("./flink-job-server.sh")
- into "build/target"
-}
-
-docker {
- name containerImageName(name: "flink-job-server")
- files "./build/"
+project.ext {
+ docker_file = file('Dockerfile')
+ startup_script = file('flink-job-server.sh')
}
-// Ensure that we build the required resources and copy and file dependencies
from related projects
-dockerPrepare.dependsOn copyDockerfileDependencies
+// Load the main build script which contains all build logic.
+apply from: 'flink_job_server_container.gradle'
diff --git
a/runners/flink/job-server-container/flink_job_server_container.gradle
b/runners/flink/job-server-container/flink_job_server_container.gradle
new file mode 100644
index 000000000000..363a99c40021
--- /dev/null
+++ b/runners/flink/job-server-container/flink_job_server_container.gradle
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/**
+ * Build a Docker image to bootstrap FlinkJobServerDriver which requires a
Java environment.
+ * Alternatively, it can also be bootstrapped through the runShadow goal
+ * or by directly running the generated JAR file.
+ */
+
+apply plugin: org.apache.beam.gradle.BeamModulePlugin
+applyDockerNature()
+
+// Resolve the Flink project name (and version) the job-server-container is
based on
+def flinkJobServerProject = ":${project.name.replace('-container', '')}"
+
+description = project(flinkJobServerProject).description + " :: Container"
+
+configurations {
+ dockerDependency
+}
+
+dependencies {
+ dockerDependency project(path: flinkJobServerProject, configuration:
"shadow")
+}
+
+task copyDockerfileDependencies(type: Copy) {
+ // Required Jars
+ from configurations.dockerDependency
+ rename 'beam-runners-flink.*-job-server.*.jar',
'beam-runners-flink-job-server.jar'
+ into "build/target"
+ // Entry script
+ from startup_script
+ into "build/target"
+ // Dockerfile
+ from docker_file
+ into "build/target"
+}
+
+docker {
+ name containerImageName(name: project.name)
+ files "./build/"
+}
+
+// Ensure that we build the required resources and copy and file dependencies
from related projects
+dockerPrepare.dependsOn copyDockerfileDependencies
diff --git a/runners/flink/job-server/build.gradle
b/runners/flink/job-server/build.gradle
index 08a6e2eead9c..52dd41aeba03 100644
--- a/runners/flink/job-server/build.gradle
+++ b/runners/flink/job-server/build.gradle
@@ -15,101 +15,16 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-import groovy.json.JsonOutput
-apply plugin: org.apache.beam.gradle.BeamModulePlugin
-applyJavaNature(
- exportJavadoc: false,
- validateShadowJar: false,
- shadowClosure: {
- append "reference.conf"
- },
-)
+def basePath = '.'
-description = "Apache Beam :: Runners :: Flink :: Job Server"
-
-apply plugin: "application"
-
-mainClassName = "org.apache.beam.runners.flink.FlinkJobServerDriver"
-
-configurations {
- validatesPortableRunner
-}
-
-configurations.all {
- // replace commons logging with the jcl-over-slf4j bridge
- exclude group: "commons-logging", module: "commons-logging"
-}
-
-dependencies {
- compile project(path: ":beam-runners-flink_2.11", configuration: "shadow")
- compile group: "org.slf4j", name: "jcl-over-slf4j", version:
dependencies.create(project.library.java.slf4j_api).getVersion()
- validatesPortableRunner project(path: ":beam-runners-flink_2.11",
configuration: "shadowTest")
- validatesPortableRunner project(path: ":beam-sdks-java-core", configuration:
"shadowTest")
- validatesPortableRunner project(path: ":beam-runners-core-java",
configuration: "shadowTest")
- validatesPortableRunner project(path: ":beam-runners-reference-java",
configuration: "shadowTest")
- compile project(path:
":beam-sdks-java-extensions-google-cloud-platform-core", configuration:
"shadow")
- compile library.java.slf4j_simple
-// TODO: Enable AWS and HDFS file system.
+project.ext {
+ // Look for the source code in the parent module
+ main_source_dirs = ["$basePath/src/main/java"]
+ test_source_dirs = ["$basePath/src/test/java"]
+ main_resources_dirs = ["$basePath/src/main/resources"]
+ test_resources_dirs = ["$basePath/src/test/resources"]
}
-// NOTE: runShadow must be used in order to run the job server. The standard
run
-// task will not work because the flink runner classes only exist in the shadow
-// jar.
-runShadow {
- args = []
- if (project.hasProperty('jobHost'))
- args += ["--job-host=${project.property('jobHost')}"]
- if (project.hasProperty('artifactsDir'))
- args += ["--artifacts-dir=${project.property('artifactsDir')}"]
- if (project.hasProperty('cleanArtifactsPerJob'))
- args += ["--clean-artifacts-per-job"]
- if (project.hasProperty('flinkMasterUrl'))
- args += ["--flink-master-url=${project.property('flinkMasterUrl')}"]
- if (project.hasProperty('flinkConfDir'))
- args += ["--flink-conf-dir=${project.property('flinkConfDir')}"]
-
- // Enable remote debugging.
- jvmArgs = ["-Xdebug",
"-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5005"]
- if (project.hasProperty("logLevel"))
- jvmArgs +=
["-Dorg.slf4j.simpleLogger.defaultLogLevel=${project.property('logLevel')}"]
-}
-
-def portableValidatesRunnerTask(String name, Boolean streaming) {
- createPortableValidatesRunnerTask(
- name: "validatesPortableRunner${name}",
- jobServerDriver:
"org.apache.beam.runners.flink.FlinkJobServerDriver",
- jobServerConfig:
"--clean-artifacts-per-job,--job-host=localhost,--job-port=0,--artifact-port=0",
- testClasspathConfiguration: configurations.validatesPortableRunner,
- parallelism: 2,
- pipelineOpts: streaming ? ["--streaming"] : [],
- testCategories: {
- includeCategories 'org.apache.beam.sdk.testing.ValidatesRunner'
- excludeCategories
'org.apache.beam.sdk.testing.FlattenWithHeterogeneousCoders'
- excludeCategories
'org.apache.beam.sdk.testing.LargeKeys$Above100MB'
- excludeCategories
'org.apache.beam.sdk.testing.UsesAttemptedMetrics'
- excludeCategories
'org.apache.beam.sdk.testing.UsesCommittedMetrics'
- excludeCategories 'org.apache.beam.sdk.testing.UsesCounterMetrics'
- excludeCategories
'org.apache.beam.sdk.testing.UsesCustomWindowMerging'
- excludeCategories
'org.apache.beam.sdk.testing.UsesDistributionMetrics'
- excludeCategories 'org.apache.beam.sdk.testing.UsesFailureMessage'
- excludeCategories 'org.apache.beam.sdk.testing.UsesGaugeMetrics'
- excludeCategories 'org.apache.beam.sdk.testing.UsesParDoLifecycle'
- excludeCategories 'org.apache.beam.sdk.testing.UsesMapState'
- excludeCategories 'org.apache.beam.sdk.testing.UsesSetState'
- excludeCategories 'org.apache.beam.sdk.testing.UsesTestStream'
- //SplitableDoFnTests
- excludeCategories
'org.apache.beam.sdk.testing.UsesBoundedSplittableParDo'
- excludeCategories
'org.apache.beam.sdk.testing.UsesSplittableParDoWithWindowedSideInputs'
- excludeCategories
'org.apache.beam.sdk.testing.UsesUnboundedSplittableParDo'
- },
- )
-}
-
-project.ext.validatesPortableRunnerBatch =
portableValidatesRunnerTask("Batch", false)
-project.ext.validatesPortableRunnerStreaming =
portableValidatesRunnerTask("Streaming", true)
-
-task validatesPortableRunner() {
- dependsOn validatesPortableRunnerBatch
- dependsOn validatesPortableRunnerStreaming
-}
+// Load the main build script which contains all build logic.
+apply from: "$basePath/flink_job_server.gradle"
diff --git a/runners/flink/job-server/flink_job_server.gradle
b/runners/flink/job-server/flink_job_server.gradle
new file mode 100644
index 000000000000..8fe84f8f5588
--- /dev/null
+++ b/runners/flink/job-server/flink_job_server.gradle
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/**
+ * Flink Runner JobServer build file shared by all of its build targets.
+ *
+ * See build.gradle files for an example of how to use this script.
+ */
+
+apply plugin: org.apache.beam.gradle.BeamModulePlugin
+applyJavaNature(
+ validateShadowJar: false,
+ exportJavadoc: false,
+ shadowClosure: {
+ append "reference.conf"
+ },
+)
+apply plugin: "application"
+
+// Resolve the Flink project name (and version) the job-server is based on
+def flinkRunnerProject = ":${project.name.replace("-job-server", "")}"
+
+description = project(flinkRunnerProject).description + " :: Job Server"
+
+/*
+ * We have to explicitly set all directories here to make sure each
+ * version of Flink has the correct overrides set.
+ */
+sourceSets {
+ main {
+ java {
+ srcDirs = main_source_dirs
+ }
+ resources {
+ srcDirs = main_resources_dirs
+ }
+ }
+ test {
+ java {
+ srcDirs = test_source_dirs
+ }
+ resources {
+ srcDirs = test_resources_dirs
+ }
+ }
+}
+
+mainClassName = "org.apache.beam.runners.flink.FlinkJobServerDriver"
+
+configurations {
+ validatesPortableRunner
+}
+
+configurations.all {
+ // replace commons logging with the jcl-over-slf4j bridge
+ exclude group: "commons-logging", module: "commons-logging"
+}
+
+dependencies {
+ compile project(path: flinkRunnerProject, configuration: "shadow")
+ compile group: "org.slf4j", name: "jcl-over-slf4j", version:
dependencies.create(project.library.java.slf4j_api).getVersion()
+ validatesPortableRunner project(path: flinkRunnerProject, configuration:
"shadowTest")
+ validatesPortableRunner project(path: ":beam-sdks-java-core", configuration:
"shadowTest")
+ validatesPortableRunner project(path: ":beam-runners-core-java",
configuration: "shadowTest")
+ validatesPortableRunner project(path: ":beam-runners-reference-java",
configuration: "shadowTest")
+ compile project(path:
":beam-sdks-java-extensions-google-cloud-platform-core", configuration:
"shadow")
+ compile library.java.slf4j_simple
+// TODO: Enable AWS and HDFS file system.
+}
+
+// NOTE: runShadow must be used in order to run the job server. The standard
run
+// task will not work because the flink runner classes only exist in the shadow
+// jar.
+runShadow {
+ args = []
+ if (project.hasProperty('jobHost'))
+ args += ["--job-host=${project.property('jobHost')}"]
+ if (project.hasProperty('artifactsDir'))
+ args += ["--artifacts-dir=${project.property('artifactsDir')}"]
+ if (project.hasProperty('cleanArtifactsPerJob'))
+ args += ["--clean-artifacts-per-job"]
+ if (project.hasProperty('flinkMasterUrl'))
+ args += ["--flink-master-url=${project.property('flinkMasterUrl')}"]
+ if (project.hasProperty('flinkConfDir'))
+ args += ["--flink-conf-dir=${project.property('flinkConfDir')}"]
+
+ // Enable remote debugging.
+ jvmArgs = ["-Xdebug",
"-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5005"]
+ if (project.hasProperty("logLevel"))
+ jvmArgs +=
["-Dorg.slf4j.simpleLogger.defaultLogLevel=${project.property('logLevel')}"]
+}
+
+def portableValidatesRunnerTask(String name, Boolean streaming) {
+ createPortableValidatesRunnerTask(
+ name: "validatesPortableRunner${name}",
+ jobServerDriver: "org.apache.beam.runners.flink.FlinkJobServerDriver",
+ jobServerConfig:
"--clean-artifacts-per-job,--job-host=localhost,--job-port=0,--artifact-port=0",
+ testClasspathConfiguration: configurations.validatesPortableRunner,
+ parallelism: 2,
+ pipelineOpts: streaming ? ["--streaming"] : [],
+ testCategories: {
+ includeCategories 'org.apache.beam.sdk.testing.ValidatesRunner'
+ excludeCategories
'org.apache.beam.sdk.testing.FlattenWithHeterogeneousCoders'
+ excludeCategories 'org.apache.beam.sdk.testing.LargeKeys$Above100MB'
+ excludeCategories 'org.apache.beam.sdk.testing.UsesAttemptedMetrics'
+ excludeCategories 'org.apache.beam.sdk.testing.UsesCommittedMetrics'
+ excludeCategories 'org.apache.beam.sdk.testing.UsesCounterMetrics'
+ excludeCategories 'org.apache.beam.sdk.testing.UsesCustomWindowMerging'
+ excludeCategories 'org.apache.beam.sdk.testing.UsesDistributionMetrics'
+ excludeCategories 'org.apache.beam.sdk.testing.UsesFailureMessage'
+ excludeCategories 'org.apache.beam.sdk.testing.UsesGaugeMetrics'
+ excludeCategories 'org.apache.beam.sdk.testing.UsesParDoLifecycle'
+ excludeCategories 'org.apache.beam.sdk.testing.UsesMapState'
+ excludeCategories 'org.apache.beam.sdk.testing.UsesSetState'
+ excludeCategories 'org.apache.beam.sdk.testing.UsesTestStream'
+ //SplitableDoFnTests
+ excludeCategories
'org.apache.beam.sdk.testing.UsesBoundedSplittableParDo'
+ excludeCategories
'org.apache.beam.sdk.testing.UsesSplittableParDoWithWindowedSideInputs'
+ excludeCategories
'org.apache.beam.sdk.testing.UsesUnboundedSplittableParDo'
+ },
+ )
+}
+
+project.ext.validatesPortableRunnerBatch =
portableValidatesRunnerTask("Batch", false)
+project.ext.validatesPortableRunnerStreaming =
portableValidatesRunnerTask("Streaming", true)
+
+task validatesPortableRunner() {
+ dependsOn validatesPortableRunnerBatch
+ dependsOn validatesPortableRunnerStreaming
+}
diff --git
a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/state/FlinkKeyGroupStateInternals.java
b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/state/FlinkKeyGroupStateInternals.java
index 224d6b628714..0ca767fba0da 100644
---
a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/state/FlinkKeyGroupStateInternals.java
+++
b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/state/FlinkKeyGroupStateInternals.java
@@ -50,7 +50,6 @@
import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.runtime.state.KeyGroupsList;
import org.apache.flink.runtime.state.KeyedStateBackend;
-import org.apache.flink.streaming.api.operators.HeapInternalTimerService;
import org.apache.flink.util.InstantiationUtil;
import org.apache.flink.util.Preconditions;
@@ -59,7 +58,7 @@
*
* <p>Note: Ignore index of key. Just implement BagState.
*
- * <p>Reference from {@link HeapInternalTimerService} to the local key-group
range.
+ * <p>Reference from Flink's HeapInternalTimerService to the local key-group
range.
*/
public class FlinkKeyGroupStateInternals<K> implements StateInternals {
diff --git a/settings.gradle b/settings.gradle
index 9dc234811cea..aac5bf9b7151 100644
--- a/settings.gradle
+++ b/settings.gradle
@@ -38,12 +38,22 @@ include "beam-runners-direct-java"
project(":beam-runners-direct-java").dir = file("runners/direct-java")
include "beam-runners-extensions-java-metrics"
project(":beam-runners-extensions-java-metrics").dir =
file("runners/extensions-java/metrics")
+/* Begin Flink Runner related settings */
+// 1.5 version with Scala 2.11 suffix
include "beam-runners-flink_2.11"
project(":beam-runners-flink_2.11").dir = file("runners/flink")
include "beam-runners-flink_2.11-job-server"
project(":beam-runners-flink_2.11-job-server").dir =
file("runners/flink/job-server")
include "beam-runners-flink_2.11-job-server-container"
project(":beam-runners-flink_2.11-job-server-container").dir =
file("runners/flink/job-server-container")
+// 1.6 version
+include "beam-runners-flink-1.6"
+project(":beam-runners-flink-1.6").dir = file("runners/flink/1.6")
+include "beam-runners-flink-1.6-job-server"
+project(":beam-runners-flink-1.6-job-server").dir =
file("runners/flink/1.6/job-server")
+include "beam-runners-flink-1.6-job-server-container"
+project(":beam-runners-flink-1.6-job-server-container").dir =
file("runners/flink/1.6/job-server-container")
+/* End Flink Runner related settings */
include "beam-runners-gcp-gcemd"
project(":beam-runners-gcp-gcemd").dir = file("runners/gcp/gcemd")
include "beam-runners-gcp-gcsproxy"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 174671)
Time Spent: 3.5h (was: 3h 20m)
> Build multiple versions of the Flink Runner against different Flink versions
> ----------------------------------------------------------------------------
>
> Key: BEAM-5419
> URL: https://issues.apache.org/jira/browse/BEAM-5419
> Project: Beam
> Issue Type: New Feature
> Components: build-system, runner-flink
> Reporter: Maximilian Michels
> Assignee: Maximilian Michels
> Priority: Major
> Fix For: 2.10.0
>
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> Following up on a discussion on the mailing list.
> We want to keep the Flink version stable across different versions to avoid
> upgrade pain for long-term users. At the same time, there are users out there
> with newer Flink clusters and developers also want to utilize new Flink
> features.
> It would be great to build multiple versions of the Flink Runner against
> different Flink versions.
> When the upgrade is as simple as changing the version property in the build
> script, this should be pretty straight-forward. If not, having a "base
> version" and applying a patch during the build could be an option. We should
> avoid duplicating any Runner code.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
