[beam] branch dependabot/pip/sdks/python/container/pyyaml-5.1 updated (354179d -> 5ce20cb)
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a change to branch
dependabot/pip/sdks/python/container/pyyaml-5.1
in repository https://gitbox.apache.org/repos/asf/beam.git.
discard 354179d Bump pyyaml from 3.13 to 5.1 in /sdks/python/container
add a80ae25 [BEAM-8540] Fix CSVSink example in FileIO docs
add d7b2c0e Merge pull request #9964 from jklukas/csvsink-fix
add 90abf5d [BEAM-8513] Allow reads from exchange-bound queue without
declaring the exchange (#9937)
add 9a41ce0 [BEAM-8521] only build Python 2.7 for xlang test
add 980e655 Merge pull request #9939 from ibzib/xlang-version
add c870c63 [BEAM-7917] Fix datastore writes failing on retry (#9294)
add dfc230c Merge pull request #9741 from [BEAM-7926] Visualize
PCollection
add bebbba8 [BEAM-7389] Add code snippets for CombinePerKey (#9921)
add 9e38cbe [BEAM-7389] Add code snippets for Distinct
add 9d2d9df Merge pull request #9924 from davidcavazos/distinct-code
add 9b30d7c [BEAM-8146,BEAM-8204,BEAM-8205] Add equals and hashCode to
SchemaCoder and RowCoder (#9493)
add 56d17fb Add separate testSQLRead and testSQLWrite test in BigQueryIT
add d093625 Add PubsubJsonIT tests that pull and verify pubsub messages
add 107ac28 Add TestPubsub#{waitForNMessages,
assertThatTopicEventuallyReceives}
add ed8401d Merge pull request #9880: [BEAM-8503] Improve TestBigQuery
and TestPubsub
add 1cb675a Add Jira priority explanations to Beam site
add 2564cb9 Add release blocking section to Beam site
add 902 Merge pull request #9862: Describe Jira priorities and
release blockers on Beam site
add d651434 [BEAM-8422] Notify builds@ on ABORTED job final state
add 3d865bc Merge pull request #9816: [BEAM-8422] Notify builds@ on
ABORTED job final state
add 3626751 [BEAM-8491] Add ability for replacing transforms with
multiple outputs (#9912)
add 1664ce2 [BEAM-8467] Enabling reading compressed files (#9861)
add a6936c8 [BEAM-3713] Add pytest for unit tests (#9756)
add 5ce20cb Bump pyyaml from 3.13 to 5.1 in /sdks/python/container
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (354179d)
\
N -- N -- N
refs/heads/dependabot/pip/sdks/python/container/pyyaml-5.1 (5ce20cb)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.gitignore | 1 +
.test-infra/jenkins/CommonJobProperties.groovy | 10 +-
.test-infra/jenkins/PrecommitJobBuilder.groovy | 7 +-
.test-infra/jenkins/job_PreCommit_Python.groovy| 15 +
build.gradle | 8 +
.../org/apache/beam/gradle/BeamModulePlugin.groovy | 2 +-
runners/apex/build.gradle | 6 +-
runners/flink/flink_runner.gradle | 3 -
.../util/SchemaCoderCloudObjectTranslator.java | 15 +-
.../runners/dataflow/util/CloudObjectsTest.java| 12 +-
.../java/org/apache/beam/sdk/coders/RowCoder.java | 7 +-
.../main/java/org/apache/beam/sdk/io/AvroIO.java | 2 +
.../main/java/org/apache/beam/sdk/io/FileIO.java | 10 +-
.../apache/beam/sdk/schemas/AutoValueSchema.java | 59 ++--
.../apache/beam/sdk/schemas/AvroRecordSchema.java | 13 +-
.../apache/beam/sdk/schemas/CachingFactory.java| 18 ++
.../beam/sdk/schemas/FromRowUsingCreator.java | 30 +-
.../sdk/schemas/GetterBasedSchemaProvider.java | 73 -
.../apache/beam/sdk/schemas/JavaBeanSchema.java| 80 --
.../apache/beam/sdk/schemas/JavaFieldSchema.java | 40 ++-
.../org/apache/beam/sdk/schemas/SchemaCoder.java | 46 +++-
.../beam/sdk/schemas/UserTypeCreatorFactory.java | 5 +-
.../beam/sdk/schemas/transforms/Convert.java | 1 +
.../sdk/schemas/transforms/SchemaAggregateFn.java | 4 +-
.../apache/beam/sdk/schemas/utils/AvroUtils.java | 21 +-
.../beam/sdk/schemas/utils/ConvertHelpers.java | 10 +-
.../org/apache/beam/sdk/testing/TestStream.java| 4 +-
.../org/apache/beam/sdk/transforms/Create.java | 24 +-
.../java/org/apache/beam/sdk/transforms/ParDo.java | 10 +-
.../apache/beam/sdk/util/SerializableUtils.java| 13 +
.../org/apache/beam/sdk/values/PCollection.java| 10 +-
.../beam/sdk/schemas/AutoValueSchemaTest.java | 31 +++
.../apache/beam/sdk/schemas/AvroSchemaTest.java
[beam] branch master updated (1664ce2 -> a6936c8)
This is an automated email from the ASF dual-hosted git repository.
udim pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 1664ce2 [BEAM-8467] Enabling reading compressed files (#9861)
add a6936c8 [BEAM-3713] Add pytest for unit tests (#9756)
No new revisions were added by this update.
Summary of changes:
.gitignore | 1 +
.test-infra/jenkins/PrecommitJobBuilder.groovy | 7 +-
.test-infra/jenkins/job_PreCommit_Python.groovy| 15
build.gradle | 8 ++
.../apache_beam/coders/coders_test_common.py | 5 ++
.../io/gcp/tests/bigquery_matcher_test.py | 2 +
.../runners/dataflow/dataflow_runner_test.py | 3 +
.../apache_beam/runners/portability/stager_test.py | 3 +
sdks/python/apache_beam/transforms/ptransform.py | 4 +-
sdks/python/apache_beam/transforms/window_test.py | 4 +-
.../coders/avro_record.py => conftest.py} | 28 +++---
.../metrics/execution.pxd => pytest.ini} | 19 +++--
sdks/python/scripts/run_pytest.sh | 49 +++
sdks/python/setup.py | 3 +
sdks/python/test-suites/tox/py2/build.gradle | 16
sdks/python/test-suites/tox/py35/build.gradle | 15
sdks/python/test-suites/tox/py36/build.gradle | 14 +++
sdks/python/test-suites/tox/py37/build.gradle | 14 +++
sdks/python/test_config.py | 1 +
sdks/python/tox.ini| 99 ++
20 files changed, 281 insertions(+), 29 deletions(-)
copy sdks/python/{apache_beam/coders/avro_record.py => conftest.py} (67%)
copy sdks/python/{apache_beam/metrics/execution.pxd => pytest.ini} (63%)
create mode 100755 sdks/python/scripts/run_pytest.sh
[beam] branch master updated (664de39 -> ad7ae0d)
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 664de39 Merge pull request #9930: [BEAM-8516] only consider current version in sdist build add 10a570f Align base image requirements with Beam's setup.py. add ad7ae0d Merge pull request #9967 Align base image requirements with Beam's setup.py. No new revisions were added by this update. Summary of changes: sdks/python/container/base_image_requirements.txt | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
[beam] branch master updated (d024a4e -> 9ed9dc7)
This is an automated email from the ASF dual-hosted git repository. altay pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from d024a4e Merge pull request #9958 [BEAM-8532] Output timestamp inclusive, not exclusive, end of window. add 9ed9dc7 [BEAM-8368] [BEAM-8392] Update pyarrow to the latest version 0.15.1 (#9970) No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/io/parquetio_test.py | 3 ++- sdks/python/setup.py | 5 + 2 files changed, 3 insertions(+), 5 deletions(-)
[beam] branch master updated (a6936c8 -> e9087a0)
This is an automated email from the ASF dual-hosted git repository. lgajowy pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from a6936c8 [BEAM-3713] Add pytest for unit tests (#9756) add 55b6d05 [BEAM-8526] Turn on metrics publishing in load tests on Flink add e9087a0 Merge pull request #9948: [BEAM-8526] Turn on metrics publishing in load tests on Flink No new revisions were added by this update. Summary of changes: .../jenkins/job_LoadTests_Combine_Flink_Python.groovy | 6 +++--- .test-infra/jenkins/job_LoadTests_GBK_Flink_Python.groovy | 14 +++--- .../jenkins/job_LoadTests_ParDo_Flink_Python.groovy| 8 .../jenkins/job_LoadTests_coGBK_Flink_Python.groovy| 8 4 files changed, 18 insertions(+), 18 deletions(-)
[beam] branch master updated (e9087a0 -> 711b7aa)
This is an automated email from the ASF dual-hosted git repository. markliu pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from e9087a0 Merge pull request #9948: [BEAM-8526] Turn on metrics publishing in load tests on Flink add 101165a update trigger phrase for beam_PreCommit_Website_Stage_GCS in README add 711b7aa Merge pull request #9983: [BEAM-8500] update trigger phrase for beam_PreCommit_Website_Stage_GCS in README No new revisions were added by this update. Summary of changes: .test-infra/jenkins/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] branch master updated (ad7ae0d -> c421484)
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from ad7ae0d Merge pull request #9967 Align base image requirements with Beam's setup.py. add 81af13c [BEAM-8544] Use ccache for compiling the Beam Python SDK. add 660c618 Additional logging. add c421484 Merge pull request #9966 [BEAM-8544] Use ccache for compiling the Beam Python SDK. No new revisions were added by this update. Summary of changes: sdks/python/container/Dockerfile | 11 ++- 1 file changed, 10 insertions(+), 1 deletion(-)
[beam] branch master updated (c421484 -> d024a4e)
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from c421484 Merge pull request #9966 [BEAM-8544] Use ccache for compiling the Beam Python SDK. add 4b107d2 [BEAM-8532] Output timestamp should be the inclusive, not exclusive, end of window. add d024a4e Merge pull request #9958 [BEAM-8532] Output timestamp inclusive, not exclusive, end of window. No new revisions were added by this update. Summary of changes: .../testing/data/trigger_transcripts.yaml | 28 +++--- sdks/python/apache_beam/transforms/timeutil.py | 2 +- sdks/python/apache_beam/transforms/trigger.py | 2 +- sdks/python/apache_beam/transforms/trigger_test.py | 43 -- sdks/python/apache_beam/transforms/util_test.py| 13 --- sdks/python/apache_beam/transforms/window_test.py | 31 6 files changed, 78 insertions(+), 41 deletions(-)
[beam] branch master updated (711b7aa -> 664de39)
This is an automated email from the ASF dual-hosted git repository. markliu pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 711b7aa Merge pull request #9983: [BEAM-8500] update trigger phrase for beam_PreCommit_Website_Stage_GCS in README add 0ad6611 [BEAM-8516] only consider current version in sdist build add 664de39 Merge pull request #9930: [BEAM-8516] only consider current version in sdist build No new revisions were added by this update. Summary of changes: sdks/python/build.gradle | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] 01/01: [BEAM-8555] Update Python streaming limitations.
This is an automated email from the ASF dual-hosted git repository.
tvalentyn pushed a commit to branch tvalentyn-patch-1
in repository https://gitbox.apache.org/repos/asf/beam.git
commit 2bfac6f108c207748caf58039cad24c645337b2b
Author: tvalentyn
AuthorDate: Mon Nov 4 17:30:27 2019 -0800
[BEAM-8555] Update Python streaming limitations.
---
website/src/documentation/sdks/python-streaming.md | 14 +++---
1 file changed, 3 insertions(+), 11 deletions(-)
diff --git a/website/src/documentation/sdks/python-streaming.md
b/website/src/documentation/sdks/python-streaming.md
index ea08da9..f15d6a1 100644
--- a/website/src/documentation/sdks/python-streaming.md
+++ b/website/src/documentation/sdks/python-streaming.md
@@ -136,17 +136,17 @@ This runner is not yet available for the Python SDK.
{:.runner-flink-local}
```
-This runner is not yet available for the Python SDK.
+See https://beam.apache.org/roadmap/portability/#python-on-flink for more
information.
```
{:.runner-flink-cluster}
```
-This runner is not yet available for the Python SDK.
+See https://beam.apache.org/documentation/runners/flink/ for more information.
```
{:.runner-spark}
```
-This runner is not yet available for the Python SDK.
+See https://beam.apache.org/roadmap/portability/#python-on-spark for more
information.
```
{:.runner-dataflow}
@@ -185,11 +185,3 @@ These unsupported Beam features apply to all runners.
- Handling of late data
- User-defined custom merging `WindowFn` (with fnapi)
-### DataflowRunner specific features
-
-Additionally, `DataflowRunner` does not currently support the following Cloud
-Dataflow specific features with Python streaming execution.
-
-- Cloud Dataflow Templates
-
-
[beam] branch tvalentyn-patch-1 created (now 2bfac6f)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch tvalentyn-patch-1 in repository https://gitbox.apache.org/repos/asf/beam.git. at 2bfac6f [BEAM-8555] Update Python streaming limitations. This branch includes the following new commits: new 2bfac6f [BEAM-8555] Update Python streaming limitations. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[beam] branch tvalentyn-patch-1-1 created (now e29a0e9)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch tvalentyn-patch-1-1 in repository https://gitbox.apache.org/repos/asf/beam.git. at e29a0e9 [BEAM-8555] Update Python streaming limitations. No new revisions were added by this update.
[beam] branch master updated: Remove base tests from python precommits
This is an automated email from the ASF dual-hosted git repository.
altay pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 640ced8 Remove base tests from python precommits
new 187d71e Merge pull request #9985 from aaltay/prec
640ced8 is described below
commit 640ced80c15fa899b6343e645bf8cb99f9df11c5
Author: Ahmet Altay
AuthorDate: Mon Nov 4 13:46:35 2019 -0800
Remove base tests from python precommits
GCP and Cython test variants provide the same coverage and also ensure
that test could run with or without GCP or Cython components intalled.
---
sdks/python/test-suites/tox/py2/build.gradle | 3 +--
sdks/python/test-suites/tox/py35/build.gradle | 3 +--
sdks/python/test-suites/tox/py36/build.gradle | 2 --
sdks/python/test-suites/tox/py37/build.gradle | 2 --
4 files changed, 2 insertions(+), 8 deletions(-)
diff --git a/sdks/python/test-suites/tox/py2/build.gradle
b/sdks/python/test-suites/tox/py2/build.gradle
index 7d4cd6a..04748d0 100644
--- a/sdks/python/test-suites/tox/py2/build.gradle
+++ b/sdks/python/test-suites/tox/py2/build.gradle
@@ -40,6 +40,7 @@ test.dependsOn testPython2
toxTask "testPy2Cython", "py27-cython"
test.dependsOn testPy2Cython
+
// Ensure that testPy2Cython runs exclusively to other tests. This line is not
// actually required, since gradle doesn't do parallel execution within a
// project.
@@ -61,14 +62,12 @@ toxTask "cover", "cover"
task preCommitPy2() {
dependsOn "docs"
dependsOn "testPy2Cython"
- dependsOn "testPython2"
dependsOn "testPy2Gcp"
}
task preCommitPy2Pytest {
dependsOn "docs"
dependsOn "testPy2CythonPytest"
- dependsOn "testPython2Pytest"
dependsOn "testPy2GcpPytest"
dependsOn "lint"
}
diff --git a/sdks/python/test-suites/tox/py35/build.gradle
b/sdks/python/test-suites/tox/py35/build.gradle
index 0736792..37e13f6 100644
--- a/sdks/python/test-suites/tox/py35/build.gradle
+++ b/sdks/python/test-suites/tox/py35/build.gradle
@@ -34,6 +34,7 @@ test.dependsOn testPy35Gcp
toxTask "testPy35Cython", "py35-cython"
test.dependsOn testPy35Cython
+
// Ensure that testPy35Cython runs exclusively to other tests. This line is not
// actually required, since gradle doesn't do parallel execution within a
// project.
@@ -48,13 +49,11 @@ toxTask "testPy35CythonPytest", "py35-cython-pytest"
testPy35CythonPytest.mustRunAfter testPython35Pytest, testPy35GcpPytest
task preCommitPy35() {
-dependsOn "testPython35"
dependsOn "testPy35Gcp"
dependsOn "testPy35Cython"
}
task preCommitPy35Pytest {
- dependsOn "testPython35Pytest"
dependsOn "testPy35GcpPytest"
dependsOn "testPy35CythonPytest"
dependsOn "lint"
diff --git a/sdks/python/test-suites/tox/py36/build.gradle
b/sdks/python/test-suites/tox/py36/build.gradle
index 2422a15..8171366 100644
--- a/sdks/python/test-suites/tox/py36/build.gradle
+++ b/sdks/python/test-suites/tox/py36/build.gradle
@@ -49,13 +49,11 @@ toxTask "testPy36CythonPytest", "py36-cython-pytest"
testPy36CythonPytest.mustRunAfter testPython36Pytest, testPy36GcpPytest
task preCommitPy36() {
-dependsOn "testPython36"
dependsOn "testPy36Gcp"
dependsOn "testPy36Cython"
}
task preCommitPy36Pytest {
- dependsOn "testPython36Pytest"
dependsOn "testPy36GcpPytest"
dependsOn "testPy36CythonPytest"
}
diff --git a/sdks/python/test-suites/tox/py37/build.gradle
b/sdks/python/test-suites/tox/py37/build.gradle
index d30c393..2a57ca9 100644
--- a/sdks/python/test-suites/tox/py37/build.gradle
+++ b/sdks/python/test-suites/tox/py37/build.gradle
@@ -58,14 +58,12 @@ toxTask "testPy37CythonPytest", "py37-cython-pytest"
testPy37CythonPytest.mustRunAfter testPython37Pytest, testPy37GcpPytest
task preCommitPy37() {
-dependsOn "testPython37"
dependsOn "testPy37Gcp"
dependsOn "testPy37Cython"
dependsOn "testPy37Interactive"
}
task preCommitPy37Pytest {
- dependsOn "testPython37Pytest"
dependsOn "testPy37GcpPytest"
dependsOn "testPy37CythonPytest"
}
[beam] branch master updated (e7f1539 -> a4141ac)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from e7f1539 Merge pull request #9961 from ibzib/java-worker add e29a0e9 [BEAM-8555] Update Python streaming limitations. add a4141ac Merge pull request #9989 [BEAM-8555] Update Python streaming limitations. No new revisions were added by this update. Summary of changes: website/src/documentation/sdks/python-streaming.md | 14 +++--- 1 file changed, 3 insertions(+), 11 deletions(-)
[beam] branch master updated: [BEAM-8254] add workerRegion and workerZone options to Java SDK
This is an automated email from the ASF dual-hosted git repository.
ibzib pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 8f3818e [BEAM-8254] add workerRegion and workerZone options to Java
SDK
new e7f1539 Merge pull request #9961 from ibzib/java-worker
8f3818e is described below
commit 8f3818e9a256e3a24129ba7762e7f7c3ddd3f783
Author: Kyle Weaver
AuthorDate: Thu Oct 31 16:50:35 2019 -0700
[BEAM-8254] add workerRegion and workerZone options to Java SDK
---
.../beam/runners/dataflow/DataflowRunner.java | 34 +++
.../beam/runners/dataflow/DataflowRunnerTest.java | 50 ++
.../sdk/extensions/gcp/options/GcpOptions.java | 31 ++
3 files changed, 115 insertions(+)
diff --git
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
index 9f7394f..51ee59c 100644
---
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
+++
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
@@ -90,6 +90,7 @@ import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.coders.Coder.NonDeterministicException;
import org.apache.beam.sdk.coders.KvCoder;
import org.apache.beam.sdk.coders.VoidCoder;
+import org.apache.beam.sdk.extensions.gcp.options.GcpOptions;
import org.apache.beam.sdk.extensions.gcp.storage.PathValidator;
import org.apache.beam.sdk.io.BoundedSource;
import org.apache.beam.sdk.io.FileBasedSink;
@@ -158,6 +159,7 @@ import org.apache.beam.sdk.values.ValueWithRecordId;
import org.apache.beam.sdk.values.WindowingStrategy;
import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Joiner;
+import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Utf8;
import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
@@ -253,6 +255,8 @@ public class DataflowRunner extends
PipelineRunner {
+
"https://cloud.google.com/compute/docs/regions-zones/regions-zones;);
}
+validateWorkerSettings(PipelineOptionsValidator.validate(GcpOptions.class,
options));
+
PathValidator validator = dataflowOptions.getPathValidator();
String gcpTempLocation;
try {
@@ -358,6 +362,36 @@ public class DataflowRunner extends
PipelineRunner {
}
@VisibleForTesting
+ static void validateWorkerSettings(GcpOptions gcpOptions) {
+Preconditions.checkArgument(
+gcpOptions.getZone() == null || gcpOptions.getWorkerRegion() == null,
+"Cannot use option zone with workerRegion. Prefer either workerZone or
workerRegion.");
+Preconditions.checkArgument(
+gcpOptions.getZone() == null || gcpOptions.getWorkerZone() == null,
+"Cannot use option zone with workerZone. Prefer workerZone.");
+Preconditions.checkArgument(
+gcpOptions.getWorkerRegion() == null || gcpOptions.getWorkerZone() ==
null,
+"workerRegion and workerZone options are mutually exclusive.");
+
+DataflowPipelineOptions dataflowOptions =
gcpOptions.as(DataflowPipelineOptions.class);
+boolean hasExperimentWorkerRegion = false;
+if (dataflowOptions.getExperiments() != null) {
+ for (String experiment : dataflowOptions.getExperiments()) {
+if (experiment.startsWith("worker_region")) {
+ hasExperimentWorkerRegion = true;
+ break;
+}
+ }
+}
+Preconditions.checkArgument(
+!hasExperimentWorkerRegion || gcpOptions.getWorkerRegion() == null,
+"Experiment worker_region and option workerRegion are mutually
exclusive.");
+Preconditions.checkArgument(
+!hasExperimentWorkerRegion || gcpOptions.getWorkerZone() == null,
+"Experiment worker_region and option workerZone are mutually
exclusive.");
+ }
+
+ @VisibleForTesting
protected DataflowRunner(DataflowPipelineOptions options) {
this.options = options;
this.dataflowClient = DataflowClient.create(options);
diff --git
a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowRunnerTest.java
b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowRunnerTest.java
index f46d034..4438e14 100644
---
a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowRunnerTest.java
+++
[beam] branch asf-site updated: Publishing website 2019/11/05 01:51:12 at commit a4141ac
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new 170b6b6 Publishing website 2019/11/05 01:51:12 at commit a4141ac 170b6b6 is described below commit 170b6b627440d2109683599599d83fa9296f4eaf Author: jenkins AuthorDate: Tue Nov 5 01:51:12 2019 + Publishing website 2019/11/05 01:51:12 at commit a4141ac --- .../documentation/sdks/python-streaming/index.html | 16 +++- 1 file changed, 3 insertions(+), 13 deletions(-) diff --git a/website/generated-content/documentation/sdks/python-streaming/index.html b/website/generated-content/documentation/sdks/python-streaming/index.html index eed103a..fefdd52 100644 --- a/website/generated-content/documentation/sdks/python-streaming/index.html +++ b/website/generated-content/documentation/sdks/python-streaming/index.html @@ -293,7 +293,6 @@ Unsupported features General Beam features - DataflowRunner specific features @@ -430,15 +429,15 @@ python -m apache_beam.examples.streaming_wordcount \ -This runner is not yet available for the Python SDK. +See https://beam.apache.org/roadmap/portability/#python-on-flink for more information. -This runner is not yet available for the Python SDK. +See https://beam.apache.org/documentation/runners/flink/ for more information. -This runner is not yet available for the Python SDK. +See https://beam.apache.org/roadmap/portability/#python-on-spark for more information. @@ -480,15 +479,6 @@ about executing streaming pipelines: User-defined custom merging WindowFn (with fnapi) -DataflowRunner specific features - -Additionally, DataflowRunner does not currently support the following Cloud -Dataflow specific features with Python streaming execution. - - - Cloud Dataflow Templates - -
[beam] branch master updated (a4141ac -> 9233aa4)
This is an automated email from the ASF dual-hosted git repository. chamikara pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from a4141ac Merge pull request #9989 [BEAM-8555] Update Python streaming limitations. add 1e70f1c Fix a bug related to zero-row responses add 9233aa4 Merge pull request #9987: [BEAM-8504] Fix a bug related to zero-row responses No new revisions were added by this update. Summary of changes: .../sdk/io/gcp/bigquery/BigQueryStorageStreamSource.java | 14 +++--- .../sdk/io/gcp/bigquery/BigQueryIOStorageReadTest.java | 3 +++ 2 files changed, 10 insertions(+), 7 deletions(-)
[beam] branch spark-runner_structured-streaming updated (d224f1c -> 59ba063)
This is an automated email from the ASF dual-hosted git repository. echauchot pushed a change to branch spark-runner_structured-streaming in repository https://gitbox.apache.org/repos/asf/beam.git. from d224f1c [BEAM-8470] simplify coders in combinePerKey translation add 59ba063 [BEAM-8470] Fix combiner. Do not reuse instance of accumulator No new revisions were added by this update. Summary of changes: .../structuredstreaming/translation/batch/AggregatorCombiner.java| 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)
