[beam] branch master updated (b02bc8a -> 1deab4c)
This is an automated email from the ASF dual-hosted git repository. mxm pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from b02bc8a Merge pull request #9927 from lukecwik/beam7230 add 5c3cf5f Fail when pipeline options conversion fails due to incompatible jackson-core version add 1deab4c Merge pull request #9931: [BEAM-8518] Fail when pipeline options conversion fails due to incompatible jackson-core version No new revisions were added by this update. Summary of changes: .../construction/PipelineOptionsTranslation.java | 25 +++--- 1 file changed, 17 insertions(+), 8 deletions(-)
[beam] branch release-2.17.0 updated (0826236 -> 76b8009)
This is an automated email from the ASF dual-hosted git repository. mxm pushed a change to branch release-2.17.0 in repository https://gitbox.apache.org/repos/asf/beam.git. from 0826236 [Go SDK] Fix post commits: runtime error in logger (#9909) add 87f4339 [BEAM-7730] Add Flink 1.9 to the list of supported Flink versions new 76b8009 Merge pull request #9919: [BEAM-7730] Add Flink 1.9 to the list of supported Flink versions The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/python/apache_beam/runners/portability/flink_runner.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] 01/01: Merge pull request #9919: [BEAM-7730] Add Flink 1.9 to the list of supported Flink versions
This is an automated email from the ASF dual-hosted git repository. mxm pushed a commit to branch release-2.17.0 in repository https://gitbox.apache.org/repos/asf/beam.git commit 76b8009c72802ce00c91ef3fc64b7ac7f0df8f47 Merge: 0826236 87f4339 Author: Maximilian Michels AuthorDate: Wed Oct 30 10:13:03 2019 +0100 Merge pull request #9919: [BEAM-7730] Add Flink 1.9 to the list of supported Flink versions Backport of #9917. sdks/python/apache_beam/runners/portability/flink_runner.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] branch master updated (1deab4c -> d664592)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 1deab4c Merge pull request #9931: [BEAM-8518] Fail when pipeline options conversion fails due to incompatible jackson-core version add 174b218 [BEAM-8398] Upgrade Google Cloud Java Client APIs add ceb9a99 fix PubsubJsonClientTest NPE add d664592 [BEAM-8398] Upgrade Dataflow Java Client API No new revisions were added by this update. Summary of changes: .../groovy/org/apache/beam/gradle/BeamModulePlugin.groovy| 12 ++-- .../org/apache/beam/sdk/io/gcp/pubsub/PubsubJsonClient.java | 2 +- 2 files changed, 7 insertions(+), 7 deletions(-)
[beam] branch master updated (3226fd2 -> 1ccecd3)
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 3226fd2 Merge pull request #9928 Cleanup metrics tests. add 1ccecd3 Remove --flink-master-url alias. No new revisions were added by this update. Summary of changes: .../main/java/org/apache/beam/runners/flink/FlinkJobServerDriver.java| 1 - 1 file changed, 1 deletion(-)
[beam] branch master updated (d664592 -> 7b3a3fa)
This is an automated email from the ASF dual-hosted git repository. thw pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from d664592 [BEAM-8398] Upgrade Dataflow Java Client API add 7b3a3fa [BEAM-8507] Augment 'flink_master' pipeline option for Python FlinkRunner (#9918) No new revisions were added by this update. Summary of changes: .../runners/flink/FlinkExecutionEnvironments.java | 24 ++ .../flink/FlinkExecutionEnvironmentsTest.java | 38 +- .../runners/portability/flink_runner.py| 37 +++-- .../runners/portability/portable_runner.py | 5 +++ 4 files changed, 87 insertions(+), 17 deletions(-)
[beam] branch master updated (bb4f85a -> a23d704)
This is an automated email from the ASF dual-hosted git repository. apilloud pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from bb4f85a Merge pull request #9933 Update Dataflow container images to beam-master-20191029 add 1f63a9c BigQuery should return row fields in the selected order add a23d704 Merge pull request #9864: [BEAM-8428] [SQL] BigQuery should return row fields in the selected order No new revisions were added by this update. Summary of changes: .../extensions/sql/meta/provider/bigquery/BigQueryTable.java | 10 -- .../sql/meta/provider/bigquery/BigQueryReadWriteIT.java| 6 +++--- 2 files changed, 11 insertions(+), 5 deletions(-)
[beam] branch master updated (a23d704 -> 3226fd2)
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from a23d704 Merge pull request #9864: [BEAM-8428] [SQL] BigQuery should return row fields in the selected order add c67e7de Cleanup metrics tests. add 3226fd2 Merge pull request #9928 Cleanup metrics tests. No new revisions were added by this update. Summary of changes: .../runners/portability/fn_api_runner_test.py | 60 -- .../runners/portability/portable_runner_test.py| 5 +- 2 files changed, 4 insertions(+), 61 deletions(-)
[beam] branch master updated (7b3a3fa -> bb4f85a)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 7b3a3fa [BEAM-8507] Augment 'flink_master' pipeline option for Python FlinkRunner (#9918) add 8698862 Bump Python container versions add bb4f85a Merge pull request #9933 Update Dataflow container images to beam-master-20191029 No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/runners/dataflow/internal/names.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
[beam] branch master updated (f196d8e -> 474dcd9)
This is an automated email from the ASF dual-hosted git repository. pabloem pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from f196d8e Merge pull request #9821 from nahuellofeudo/logs-cleanup add 474dcd9 [BEAM-8474] A microbenchmark for Python FnApiRunner: (#9891) No new revisions were added by this update. Summary of changes: .../tools/fn_api_runner_microbenchmark.py | 148 + sdks/python/apache_beam/tools/utils.py | 77 +-- 2 files changed, 216 insertions(+), 9 deletions(-) create mode 100644 sdks/python/apache_beam/tools/fn_api_runner_microbenchmark.py
[beam] branch master updated (1ccecd3 -> 7a02887)
This is an automated email from the ASF dual-hosted git repository. mikhail pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 1ccecd3 Remove --flink-master-url alias. new 054fe9c [BEAM-4775] Extract code responsible for parsing MonitoringInfo new 34407f6 [BEAM-4775] Expose metrics gathered by FnApiRunner in LocalJobServicer new f1944c0 [BEAM-4775] Publish only user metrics in load tests new f53e47a [BEAM-4775] Use result property of MetricResult in load tests new 7a02887 Merge pull request #9843 from kamilwu/portable-runners-metrics The 23700 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../python/apache_beam/metrics/monitoring_infos.py | 7 +++ .../runners/portability/fn_api_runner.py | 45 -- .../runners/portability/local_job_service.py | 25 +++- .../runners/portability/portable_metrics.py| 69 ++ .../runners/portability/portable_runner.py | 32 +++--- .../apache_beam/testing/load_tests/load_test.py| 32 -- .../testing/load_tests/load_test_metrics_utils.py | 9 +-- .../apache_beam/testing/load_tests/pardo_test.py | 2 - 8 files changed, 152 insertions(+), 69 deletions(-) create mode 100644 sdks/python/apache_beam/runners/portability/portable_metrics.py
[beam] branch master updated (7a02887 -> 5fa2c7f)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 7a02887 Merge pull request #9843 from kamilwu/portable-runners-metrics add cb5a467 [Go SDK] Delete spammy log in datamgr.go add 5fa2c7f [Go SDK] Delete spammy log in datamgr.go No new revisions were added by this update. Summary of changes: sdks/go/pkg/beam/core/runtime/harness/datamgr.go | 1 - 1 file changed, 1 deletion(-)
[beam] branch master updated (5fa2c7f -> f196d8e)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 5fa2c7f [Go SDK] Delete spammy log in datamgr.go new dc6b7a3 Avoid polluting Stackdriver logs with noise during autoscaling events. Add message directing users to ignore them when appropriate. new db87ebc Spotless Java applied new f196d8e Merge pull request #9821 from nahuellofeudo/logs-cleanup The 23705 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../beam/runners/dataflow/worker/windmill/GrpcWindmillServer.java| 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
[beam] branch master updated (474dcd9 -> 18b63e4)
This is an automated email from the ASF dual-hosted git repository. thw pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 474dcd9 [BEAM-8474] A microbenchmark for Python FnApiRunner: (#9891) add d85f64b Revert "Remove --flink-master-url alias." add 18b63e4 Merge pull request #9947: Revert "Remove --flink-master-url alias." No new revisions were added by this update. Summary of changes: .../main/java/org/apache/beam/runners/flink/FlinkJobServerDriver.java| 1 + 1 file changed, 1 insertion(+)
[beam] branch master updated: Not inject pubsub into Impulse when in fnapi streaming mode.
This is an automated email from the ASF dual-hosted git repository.
boyuanz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new eb05f56 Not inject pubsub into Impulse when in fnapi streaming mode.
new 0573e10 Merge pull request #9932 from boyuanzz/windmill_create
eb05f56 is described below
commit eb05f5684e4a27ad363b507bbeef83773b8fc06c
Author: Boyuan Zhang
AuthorDate: Tue Oct 29 18:18:51 2019 -0700
Not inject pubsub into Impulse when in fnapi streaming mode.
---
.../java/org/apache/beam/runners/dataflow/DataflowRunner.java | 2 +-
sdks/python/apache_beam/runners/dataflow/dataflow_runner.py | 8 +---
2 files changed, 6 insertions(+), 4 deletions(-)
diff --git
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
index 9f7394f..f6f78a8 100644
---
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
+++
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
@@ -1427,7 +1427,7 @@ public class DataflowRunner extends
PipelineRunner {
private static class ImpulseTranslator implements
TransformTranslator {
@Override
public void translate(Impulse transform, TranslationContext context) {
- if (context.getPipelineOptions().isStreaming()) {
+ if (context.getPipelineOptions().isStreaming() && !context.isFnApi()) {
StepTranslationContext stepContext = context.addStep(transform,
"ParallelRead");
stepContext.addInput(PropertyNames.FORMAT, "pubsub");
stepContext.addInput(PropertyNames.PUBSUB_SUBSCRIPTION,
"_starting_signal/");
diff --git a/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py
b/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py
index 4928550..039eaf0 100644
--- a/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py
+++ b/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py
@@ -622,9 +622,12 @@ class DataflowRunner(PipelineRunner):
def run_Impulse(self, transform_node, options):
standard_options = options.view_as(StandardOptions)
+debug_options = options.view_as(DebugOptions)
+use_fn_api = (debug_options.experiments and
+ 'beam_fn_api' in debug_options.experiments)
step = self._add_step(
TransformNames.READ, transform_node.full_label, transform_node)
-if standard_options.streaming:
+if standard_options.streaming and not use_fn_api:
step.add_property(PropertyNames.FORMAT, 'pubsub')
step.add_property(PropertyNames.PUBSUB_SUBSCRIPTION, '_starting_signal/')
else:
@@ -634,8 +637,7 @@ class DataflowRunner(PipelineRunner):
coders.coders.GlobalWindowCoder()).get_impl().encode_nested(
window.GlobalWindows.windowed_value(b''))
- from apache_beam.runners.dataflow.internal import apiclient
- if apiclient._use_fnapi(options):
+ if use_fn_api:
encoded_impulse_as_str = self.byte_array_to_json_string(
encoded_impulse_element)
else:
