[beam] branch master updated: Fix a typo in SelectHelpers.java
This is an automated email from the ASF dual-hosted git repository.
gleb pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 9983d25 Fix a typo in SelectHelpers.java
new 5b3dae3 Merge pull request #8353: Fix a typo in SelectHelpers.java
9983d25 is described below
commit 9983d25e49002f835af3c713947b77740343ba8d
Author: tvalentyn
AuthorDate: Thu Apr 18 19:48:15 2019 -0700
Fix a typo in SelectHelpers.java
---
.../src/main/java/org/apache/beam/sdk/schemas/utils/SelectHelpers.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/SelectHelpers.java
b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/SelectHelpers.java
index 37742a5..a8f1274 100644
---
a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/SelectHelpers.java
+++
b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/SelectHelpers.java
@@ -64,7 +64,7 @@ public class SelectHelpers {
* {@code
* class Location {
* double latitude;
- * double longtitude;
+ * double longitude;
* }
* }
*
[beam] branch new-io-Dashboards-link created (now f69e960)
This is an automated email from the ASF dual-hosted git repository. lgajowy pushed a change to branch new-io-Dashboards-link in repository https://gitbox.apache.org/repos/asf/beam.git. at f69e960 Update IOIT Dashbards url No new revisions were added by this update.
[beam] branch master updated: [BEAM-7095] Upgrade to RabbitMQ amqp-client 4.9.3 in RabbitMqIO
This is an automated email from the ASF dual-hosted git repository.
aromanenko pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new acdaa78 [BEAM-7095] Upgrade to RabbitMQ amqp-client 4.9.3 in
RabbitMqIO
new c2c371a Merge pull request #8329: [BEAM-7095] Upgrade to RabbitMQ
amqp-client 4.9.3 in RabbitMqIO
acdaa78 is described below
commit acdaa78208e998212a80d412a64c318e9374fb37
Author: Jean-Baptiste Onofré
AuthorDate: Wed Apr 17 15:16:01 2019 +0200
[BEAM-7095] Upgrade to RabbitMQ amqp-client 4.9.3 in RabbitMqIO
---
sdks/java/io/rabbitmq/OWNERS | 4
sdks/java/io/rabbitmq/build.gradle | 3 ++-
2 files changed, 6 insertions(+), 1 deletion(-)
diff --git a/sdks/java/io/rabbitmq/OWNERS b/sdks/java/io/rabbitmq/OWNERS
new file mode 100644
index 000..8b0b055
--- /dev/null
+++ b/sdks/java/io/rabbitmq/OWNERS
@@ -0,0 +1,4 @@
+# See the OWNERS docs at https://s.apache.org/beam-owners
+
+reviewers:
+ - jbonofre
diff --git a/sdks/java/io/rabbitmq/build.gradle
b/sdks/java/io/rabbitmq/build.gradle
index 520c37f..62c106b 100644
--- a/sdks/java/io/rabbitmq/build.gradle
+++ b/sdks/java/io/rabbitmq/build.gradle
@@ -26,7 +26,8 @@ dependencies {
shadow library.java.vendored_guava_20_0
shadow project(path: ":beam-sdks-java-core", configuration: "shadow")
shadow library.java.joda_time
- shadow "com.rabbitmq:amqp-client:4.6.0"
+ shadow "com.rabbitmq:amqp-client:4.9.3"
+ testCompile project(path: ":beam-runners-direct-java", configuration:
"shadow")
testCompile project(path: ":beam-sdks-java-io-common", configuration:
"shadowTest")
testCompile "org.apache.qpid:qpid-broker:0.28"
testCompile "org.apache.qpid:qpid-broker-core:0.28"
[beam] branch master updated: Mahatma Gandhi is spelt wrong.
This is an automated email from the ASF dual-hosted git repository.
lcwik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new b29cc7c Mahatma Gandhi is spelt wrong.
new d11cb27 Mahatma Gandhi is spelt wrong.
b29cc7c is described below
commit b29cc7cbff755dfaa4b80d095dff3b1f58345b9a
Author: Pranay Nanda
AuthorDate: Fri Apr 19 02:19:00 2019 +0530
Mahatma Gandhi is spelt wrong.
https://en.wikipedia.org/wiki/Mahatma_Gandhi
---
sdks/python/apache_beam/examples/snippets/snippets.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/sdks/python/apache_beam/examples/snippets/snippets.py
b/sdks/python/apache_beam/examples/snippets/snippets.py
index f5cc4c9..7a04c4c 100644
--- a/sdks/python/apache_beam/examples/snippets/snippets.py
+++ b/sdks/python/apache_beam/examples/snippets/snippets.py
@@ -1127,7 +1127,7 @@ def model_bigqueryio(p, write_project='',
write_dataset='', write_table=''):
# [START model_bigqueryio_write_input]
quotes = p | beam.Create([
- {'source': 'Mahatma Ghandi', 'quote': 'My life is my message.'},
+ {'source': 'Mahatma Gandhi', 'quote': 'My life is my message.'},
{'source': 'Yoda', 'quote': "Do, or do not. There is no 'try'."},
])
# [END model_bigqueryio_write_input]
[beam] branch asf-site updated: Publishing website 2019/04/19 16:05:04 at commit d11cb27
This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new c179131 Publishing website 2019/04/19 16:05:04 at commit d11cb27
c179131 is described below
commit c17913195de9beca458b6f47ea7a1ec760741e6d
Author: jenkins
AuthorDate: Fri Apr 19 16:05:04 2019 +
Publishing website 2019/04/19 16:05:04 at commit d11cb27
---
.../documentation/io/built-in/google-bigquery/index.html| 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/website/generated-content/documentation/io/built-in/google-bigquery/index.html
b/website/generated-content/documentation/io/built-in/google-bigquery/index.html
index 7dfaf67..69e4cdf 100644
---
a/website/generated-content/documentation/io/built-in/google-bigquery/index.html
+++
b/website/generated-content/documentation/io/built-in/google-bigquery/index.html
@@ -1047,7 +1047,7 @@ static class Quote {
quotes = p | beam.Create([
-{'source': 'Mahatma Ghandi', 'quote': 'My life is my message.'},
+{'source': 'Mahatma Gandhi', 'quote': 'My life is my message.'},
{'source': 'Yoda', 'quote': "Do, or do
not. There is no 'try'."},
])
[beam] branch master updated: [BEAM-7100] BeamValuesRel should accept empty tuples
This is an automated email from the ASF dual-hosted git repository.
anton pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new e99dd29 [BEAM-7100] BeamValuesRel should accept empty tuples
new 3128cf5 Merge pull request #8339 from
amaliujia/rw_empty_join_on_one_side
e99dd29 is described below
commit e99dd2970e1dc075c0bb3e553c5afb8a1f317b40
Author: amaliujia
AuthorDate: Wed Apr 17 14:21:46 2019 -0700
[BEAM-7100] BeamValuesRel should accept empty tuples
---
.../sdk/extensions/sql/impl/rel/BeamValuesRel.java | 6 ---
.../impl/rel/BeamJoinRelBoundedVsBoundedTest.java | 54 ++
2 files changed, 54 insertions(+), 6 deletions(-)
diff --git
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamValuesRel.java
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamValuesRel.java
index dbb5bd8..b681738 100644
---
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamValuesRel.java
+++
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamValuesRel.java
@@ -80,14 +80,8 @@ public class BeamValuesRel extends Values implements
BeamRelNode {
BeamValuesRel.class.getSimpleName(),
pinput);
- if (tuples.isEmpty()) {
-throw new IllegalStateException("Values with empty tuples!");
- }
-
Schema schema = CalciteUtils.toSchema(getRowType());
-
List rows = tuples.stream().map(tuple -> tupleToRow(schema,
tuple)).collect(toList());
-
return
pinput.getPipeline().begin().apply(Create.of(rows)).setRowSchema(schema);
}
}
diff --git
a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamJoinRelBoundedVsBoundedTest.java
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamJoinRelBoundedVsBoundedTest.java
index 3286042..162b0ef 100644
---
a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamJoinRelBoundedVsBoundedTest.java
+++
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamJoinRelBoundedVsBoundedTest.java
@@ -106,6 +106,60 @@ public class BeamJoinRelBoundedVsBoundedTest extends
BaseRelTest {
}
@Test
+ public void testLeftOuterJoinWithEmptyTuplesOnRightSide() throws Exception {
+String sql =
+"SELECT * "
++ "FROM ORDER_DETAILS1 o1"
++ " LEFT OUTER JOIN (SELECT * FROM ORDER_DETAILS2 WHERE FALSE) o2"
++ " on "
++ " o1.order_id=o2.site_id AND o2.price=o1.site_id";
+
+PCollection rows = compilePipeline(sql, pipeline);
+pipeline.enableAbandonedNodeEnforcement(false);
+PAssert.that(rows)
+.containsInAnyOrder(
+TestUtils.RowsBuilder.of(
+Schema.builder()
+.addField("order_id", Schema.FieldType.INT32)
+.addField("site_id", Schema.FieldType.INT32)
+.addField("price", Schema.FieldType.INT32)
+.addNullableField("order_id0", Schema.FieldType.INT32)
+.addNullableField("site_id0", Schema.FieldType.INT32)
+.addNullableField("price0", Schema.FieldType.INT32)
+.build())
+.addRows(
+1, 2, 3, null, null, null, 2, 3, 3, null, null, null, 3,
4, 5, null, null, null)
+.getRows());
+pipeline.run();
+ }
+
+ @Test
+ public void testInnerJoinWithEmptyTuplesOnRightSide() throws Exception {
+String sql =
+"SELECT * "
++ "FROM ORDER_DETAILS1 o1"
++ " INNER JOIN (SELECT * FROM ORDER_DETAILS2 WHERE FALSE) o2"
++ " on "
++ " o1.order_id=o2.site_id AND o2.price=o1.site_id";
+
+PCollection rows = compilePipeline(sql, pipeline);
+pipeline.enableAbandonedNodeEnforcement(false);
+PAssert.that(rows)
+.containsInAnyOrder(
+TestUtils.RowsBuilder.of(
+Schema.builder()
+.addField("order_id", Schema.FieldType.INT32)
+.addField("site_id", Schema.FieldType.INT32)
+.addField("price", Schema.FieldType.INT32)
+.addNullableField("order_id0", Schema.FieldType.INT32)
+.addNullableField("site_id0", Schema.FieldType.INT32)
+.addNullableField("price0", Schema.FieldType.INT32)
+.build())
+.getRows());
+pipeline.run();
+ }
+
+ @Test
public void testRightOuterJoin() throws Exception {
String sql =
"SELECT * "
[beam] branch master updated (3128cf5 -> d368c9d)
This is an automated email from the ASF dual-hosted git repository. melap pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 3128cf5 Merge pull request #8339 from amaliujia/rw_empty_join_on_one_side add f69e960 Update IOIT Dashbards url new d368c9d Merge pull request #8356: [BEAM-6627] Update IOIT Dashbards url in docs The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: website/src/documentation/io/testing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] 01/01: Merge pull request #8356: [BEAM-6627] Update IOIT Dashbards url in docs
This is an automated email from the ASF dual-hosted git repository. melap pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit d368c9d219345d3d73b209a46cd2f669c8916d12 Merge: 3128cf5 f69e960 Author: Melissa Pashniak AuthorDate: Fri Apr 19 11:09:51 2019 -0700 Merge pull request #8356: [BEAM-6627] Update IOIT Dashbards url in docs website/src/documentation/io/testing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] branch asf-site updated: Publishing website 2019/04/19 18:10:43 at commit d368c9d
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new 0d13830 Publishing website 2019/04/19 18:10:43 at commit d368c9d 0d13830 is described below commit 0d13830f5f214ae4cdc8d2e7c057a1568825241c Author: jenkins AuthorDate: Fri Apr 19 18:10:43 2019 + Publishing website 2019/04/19 18:10:43 at commit d368c9d --- website/generated-content/documentation/io/testing/index.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/generated-content/documentation/io/testing/index.html b/website/generated-content/documentation/io/testing/index.html index 05833ae..78c59b9 100644 --- a/website/generated-content/documentation/io/testing/index.html +++ b/website/generated-content/documentation/io/testing/index.html @@ -780,7 +780,7 @@ If you modified/added new Jenkins job definitions in your Pull Request, run the We measure the performance of IOITs by gathering test execution times from Jenkins jobs that run periodically. The consequent results are stored in a database (BigQuery), therefore we can display them in a form of plots. -The dashboard gathering all the results is available here: https://apache-beam-testing.appspot.com/explore?dashboard=5755685136498688";>Performance Testing Dashboard +The dashboard gathering all the results is available here: https://s.apache.org/io-test-dashboards";>Performance Testing Dashboard Implementing Integration Tests
[beam] branch master updated (d368c9d -> ccbad61)
This is an automated email from the ASF dual-hosted git repository. pabloem pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from d368c9d Merge pull request #8356: [BEAM-6627] Update IOIT Dashbards url in docs new 332d5c1 Stage Dataflow worker jar in Dataflow PostCommit tests on Python 3. new d76a9fe Fix bug with use_fn_api_runner in Python. new ccbad61 Merge pull request #8352 from tvalentyn/stage_df_worker_jar The 21073 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/python/apache_beam/runners/direct/direct_runner.py | 3 +-- sdks/python/test-suites/dataflow/py3/build.gradle | 17 ++--- 2 files changed, 15 insertions(+), 5 deletions(-)
[beam] branch pabloem-patch-1 deleted (was 6466089)
This is an automated email from the ASF dual-hosted git repository. pabloem pushed a change to branch pabloem-patch-1 in repository https://gitbox.apache.org/repos/asf/beam.git. was 6466089 testing release The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
[beam] branch master updated: [SQL] Add Data Catalog Table Provider
This is an automated email from the ASF dual-hosted git repository.
anton pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 9a99664 [SQL] Add Data Catalog Table Provider
new 9c8a8dc Merge pull request #8349 from
akedin/datacatalog-table-provider
9a99664 is described below
commit 9a99664822a8fd79bb5491e206159fce064c9882
Author: akedin
AuthorDate: Wed Apr 17 22:55:01 2019 -0700
[SQL] Add Data Catalog Table Provider
---
.../org/apache/beam/gradle/BeamModulePlugin.groovy | 3 +
sdks/java/extensions/sql/datacatalog/build.gradle | 63 +++
.../sql/example/BeamSqlDataCatalogExample.java | 103 +
.../sdk/extensions/sql/example/package-info.java | 20
.../meta/provider/datacatalog/BigQueryUtils.java | 58 ++
.../datacatalog/DataCatalogClientAdapter.java | 94
.../datacatalog/DataCatalogPipelineOptions.java| 35 ++
.../datacatalog/DataCatalogTableProvider.java | 122 +
.../sql/meta/provider/datacatalog/PubsubUtils.java | 53 +
.../sql/meta/provider/datacatalog/SchemaUtils.java | 98 +
.../sql/meta/provider/datacatalog/TableUtils.java | 59 ++
.../meta/provider/datacatalog/package-info.java| 20
.../beam/sdk/io/gcp/bigquery/BigQueryUtils.java| 4 +-
settings.gradle| 2 +
14 files changed, 733 insertions(+), 1 deletion(-)
diff --git
a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
index 74bc02f..6614954 100644
--- a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
+++ b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
@@ -350,6 +350,7 @@ class BeamModulePlugin implements Plugin {
def cassandra_driver_version = "3.6.0"
def generated_grpc_beta_version = "0.44.0"
def generated_grpc_ga_version = "1.43.0"
+def generated_grpc_dc_beta_version = "0.1.0-alpha"
def google_auth_version = "0.12.0"
def google_clients_version = "1.27.0"
def google_cloud_bigdataoss_version = "1.9.16"
@@ -442,6 +443,7 @@ class BeamModulePlugin implements Plugin {
grpc_all:
"io.grpc:grpc-all:$grpc_version",
grpc_auth :
"io.grpc:grpc-auth:$grpc_version",
grpc_core :
"io.grpc:grpc-core:$grpc_version",
+grpc_google_cloud_datacatalog_v1beta1 :
"com.google.api.grpc:grpc-google-cloud-datacatalog-v1beta1:$generated_grpc_dc_beta_version",
grpc_google_cloud_pubsub_v1 :
"com.google.api.grpc:grpc-google-cloud-pubsub-v1:$generated_grpc_ga_version",
grpc_protobuf :
"io.grpc:grpc-protobuf:$grpc_version",
grpc_protobuf_lite :
"io.grpc:grpc-protobuf-lite:$grpc_version",
@@ -480,6 +482,7 @@ class BeamModulePlugin implements Plugin {
powermock :
"org.powermock:powermock-mockito-release-full:1.6.4",
protobuf_java :
"com.google.protobuf:protobuf-java:$protobuf_version",
protobuf_java_util :
"com.google.protobuf:protobuf-java-util:$protobuf_version",
+proto_google_cloud_datacatalog_v1beta1 :
"com.google.api.grpc:proto-google-cloud-datacatalog-v1beta1:$generated_grpc_dc_beta_version",
proto_google_cloud_pubsub_v1:
"com.google.api.grpc:proto-google-cloud-pubsub-v1:$generated_grpc_ga_version",
proto_google_cloud_spanner_admin_database_v1:
"com.google.api.grpc:proto-google-cloud-spanner-admin-database-v1:$google_cloud_spanner_version",
proto_google_common_protos :
"com.google.api.grpc:proto-google-common-protos:$proto_google_common_protos_version",
diff --git a/sdks/java/extensions/sql/datacatalog/build.gradle
b/sdks/java/extensions/sql/datacatalog/build.gradle
new file mode 100644
index 000..9bf974a
--- /dev/null
+++ b/sdks/java/extensions/sql/datacatalog/build.gradle
@@ -0,0 +1,63 @@
+import groovy.json.JsonOutput
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an A
[beam] branch master updated: [BEAM-7088] Implement switch to using MonitoringInfo labels for Name and Namespace. (#8316)
This is an automated email from the ASF dual-hosted git repository.
pabloem pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 5ff8475 [BEAM-7088] Implement switch to using MonitoringInfo labels
for Name and Namespace. (#8316)
5ff8475 is described below
commit 5ff8475e370c5a746ddc6e6cfd7b63004d8c40e3
Author: Mikhail Gryzykhin <12602502+arda...@users.noreply.github.com>
AuthorDate: Fri Apr 19 16:08:28 2019 -0700
[BEAM-7088] Implement switch to using MonitoringInfo labels for Name and
Namespace. (#8316)
* Implement switch to using MonitoringInfo labels for Name and Namespace
in Java.
* Add backwards compatibility to MetricName and patch flink tests
---
model/pipeline/src/main/proto/metrics.proto| 16 +++-
.../beam/runners/core/metrics/MetricUrns.java | 41
.../runners/core/metrics/MetricsContainerImpl.java | 87 +
.../core/metrics/MonitoringInfoConstants.java | 7 +-
.../core/metrics/MonitoringInfoMetricName.java | 43 -
.../core/metrics/SimpleMonitoringInfoBuilder.java | 50 +-
.../core/metrics/MetricsContainerImplTest.java | 27 +++---
.../core/metrics/MetricsContainerStepMapTest.java | 9 +-
.../core/metrics/MonitoringInfoMetricNameTest.java | 29 +++---
.../core/metrics/MonitoringInfoTestUtil.java | 2 +-
.../metrics/SimpleMonitoringInfoBuilderTest.java | 40 ++--
.../core/metrics/SimpleStateRegistryTest.java | 6 +-
.../metrics/SpecMonitoringInfoValidatorTest.java | 6 +-
.../flink/metrics/FlinkMetricContainerTest.java| 68 +++--
...onMonitoringInfoToCounterUpdateTransformer.java | 29 ++
...erMonitoringInfoToCounterUpdateTransformer.java | 25 ++---
.../fn/control/BeamFnMapTaskExecutorTest.java | 6 +-
...nitoringInfoToCounterUpdateTransformerTest.java | 8 +-
...nitoringInfoToCounterUpdateTransformerTest.java | 8 +-
.../fnexecution/control/RemoteExecutionTest.java | 47 ++---
.../beam/fn/harness/FnApiDoFnRunnerTest.java | 20 ++--
.../data/ElementCountFnDataReceiverTest.java | 4 +-
sdks/python/apache_beam/metrics/execution.py | 14 ++-
.../python/apache_beam/metrics/monitoring_infos.py | 106 +++--
.../apache_beam/metrics/monitoring_infos_test.py | 18 ++--
sdks/python/apache_beam/portability/common_urns.py | 4 +
26 files changed, 335 insertions(+), 385 deletions(-)
diff --git a/model/pipeline/src/main/proto/metrics.proto
b/model/pipeline/src/main/proto/metrics.proto
index 43ec994..3b45efb 100644
--- a/model/pipeline/src/main/proto/metrics.proto
+++ b/model/pipeline/src/main/proto/metrics.proto
@@ -62,8 +62,13 @@ message MonitoringInfoSpecs {
// TODO(BEAM-6926): Add the PTRANSFORM name as a required label after
// upgrading the python SDK.
USER_COUNTER = 0 [(monitoring_info_spec) = {
- urn: "beam:metric:user:",
+ urn: "beam:metric:user",
type_urn: "beam:metrics:sum_int_64",
+ required_labels: ["PTRANSFORM", "NAMESPACE", "NAME"],
+ annotations: [{
+key: "description",
+value: "URN utilized to report user numeric counters."
+ }]
}];
ELEMENT_COUNT = 1 [(monitoring_info_spec) = {
@@ -122,8 +127,13 @@ message MonitoringInfoSpecs {
// TODO(BEAM-6926): Add the PTRANSFORM name as a required label after
// upgrading the python SDK.
USER_DISTRIBUTION_COUNTER = 6 [(monitoring_info_spec) = {
- urn: "beam:metric:user_distribution:",
+ urn: "beam:metric:user_distribution",
type_urn: "beam:metrics:distribution_int_64",
+ required_labels: ["PTRANSFORM", "NAMESPACE", "NAME"],
+ annotations: [{
+key: "description",
+value: "URN utilized to report user distribution counters."
+ }]
}];
}
}
@@ -171,6 +181,8 @@ message MonitoringInfo {
WINDOWING_STRATEGY = 2 [(label_props) = { name: "WINDOWING_STRATEGY" }];
CODER = 3 [(label_props) = { name: "CODER" }];
ENVIRONMENT = 4 [(label_props) = { name: "ENVIRONMENT" }];
+NAMESPACE = 5 [(label_props) = { name: "NAMESPACE" }];
+NAME = 6 [(label_props) = { name: "NAME" }];
}
// A set of key+value labels which define the scope of the metric.
// Either a well defined entity id for matching the enum names in
diff --git
a/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MetricUrns.java
b/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MetricUrns.java
deleted file mode 100644
index 4ed5fc4..000
---
a/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MetricUrns.java
+++ /dev/null
@@ -1,41 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ow
[beam] branch master updated: [BEAM-7118] Link Spark portable runner design doc on website
This is an automated email from the ASF dual-hosted git repository. goenka pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new fbe357f [BEAM-7118] Link Spark portable runner design doc on website new 40fe671 Merge pull request #8362 from ibzib/spark-doc fbe357f is described below commit fbe357f39403a5c9ff2739290634b0b22c055a92 Author: Kyle Weaver AuthorDate: Fri Apr 19 10:36:42 2019 -0700 [BEAM-7118] Link Spark portable runner design doc on website --- website/src/contribute/design-documents.md | 1 + 1 file changed, 1 insertion(+) diff --git a/website/src/contribute/design-documents.md b/website/src/contribute/design-documents.md index c483f16..eccf5e6 100644 --- a/website/src/contribute/design-documents.md +++ b/website/src/contribute/design-documents.md @@ -118,6 +118,7 @@ This is a collection of documents that may or may not be up to date. - Portability API: How to Checkpoint and Split Bundles [[doc](https://s.apache.org/beam-checkpoint-and-split-bundles)] - Portability API: How to Finalize Bundles [[doc](https://s.apache.org/beam-finalizing-bundles)] - Side Input in Universal Reference Runner [[doc](https://docs.google.com/document/d/13N0OJ7QJm81wcgu13pi9GuN29UUxN2iIFn_H8lKpDks)] +- Spark Portable Runner Overview [[doc](https://docs.google.com/document/d/1j8GERTiHUuc6CzzCXZHc38rBn41uWfATBh2-5JN8hro)] ### Build / Testing - More Expressive PAsserts [[doc](https://docs.google.com/document/d/1fZUUbG2LxBtqCVabQshldXIhkMcXepsbv2vuuny8Ix4)]
[beam] branch asf-site updated: Publishing website 2019/04/20 00:51:32 at commit 40fe671
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new d097cd2 Publishing website 2019/04/20 00:51:32 at commit 40fe671 d097cd2 is described below commit d097cd2004a85113530b682384ca281498fbd28b Author: jenkins AuthorDate: Sat Apr 20 00:51:32 2019 + Publishing website 2019/04/20 00:51:32 at commit 40fe671 --- website/generated-content/contribute/design-documents/index.html | 1 + 1 file changed, 1 insertion(+) diff --git a/website/generated-content/contribute/design-documents/index.html b/website/generated-content/contribute/design-documents/index.html index ff7b114..52a2873 100644 --- a/website/generated-content/contribute/design-documents/index.html +++ b/website/generated-content/contribute/design-documents/index.html @@ -357,6 +357,7 @@ limitations under the License. Portability API: How to Checkpoint and Split Bundles [https://s.apache.org/beam-checkpoint-and-split-bundles";>doc] Portability API: How to Finalize Bundles [https://s.apache.org/beam-finalizing-bundles";>doc] Side Input in Universal Reference Runner [https://docs.google.com/document/d/13N0OJ7QJm81wcgu13pi9GuN29UUxN2iIFn_H8lKpDks";>doc] + Spark Portable Runner Overview [https://docs.google.com/document/d/1j8GERTiHUuc6CzzCXZHc38rBn41uWfATBh2-5JN8hro";>doc] Build / Testing
