[GitHub] [beam] lostluck commented on pull request #11870: [BEAM-9951] Adding integration tests for synthetic pipelines in Go
lostluck commented on pull request #11870: URL: https://github.com/apache/beam/pull/11870#issuecomment-642414004 Run Go PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] lostluck commented on a change in pull request #11976: [BEAM-10169] - ParDo functions with correct output N in their error messages.
lostluck commented on a change in pull request #11976:
URL: https://github.com/apache/beam/pull/11976#discussion_r438475207
##
File path: sdks/go/pkg/beam/core/util/reflectx/functions_test.go
##
@@ -41,3 +45,19 @@ func TestLoadFunction(t *testing.T) {
t.Errorf("got %d, wanted %d", out[0].Int(), testFunction())
}
}
+
+func TestFunctionOutputSize(t *testing.T) {
+ expected := 1
+ received := FunctionOutputSize(testFunction)
+ if received != expected {
+ t.Errorf("got %d, wanted %d", received, expected)
+ }
+}
+
+func TestFunction2OutputSize(t *testing.T) {
Review comment:
Conventionally, if there are multiple cases for a given test, they
should be combined into the same function, and run as a loop.
See https://gobyexample.com/testing for an excellent example.
Otherwise, this function should be named TestFunctionOutputSize_2
##
File path: sdks/go/pkg/beam/pardo.go
##
@@ -414,7 +414,45 @@ func ParDo6(s Scope, dofn interface{}, col PCollection,
opts ...Option) (PCollec
func ParDo7(s Scope, dofn interface{}, col PCollection, opts ...Option)
(PCollection, PCollection, PCollection, PCollection, PCollection, PCollection,
PCollection) {
ret := MustN(TryParDo(s, dofn, col, opts...))
if len(ret) != 7 {
- panic(fmt.Sprintf("expected 7 output. Found: %v", ret))
+ panic(ParDoErrorFormatter(dofn, ParDo7))
}
return ret[0], ret[1], ret[2], ret[3], ret[4], ret[5], ret[6]
}
+
+//ParDoErrorFormatter is a helper function to provide a more concise error
+// message to the users when a DoFn and its ParDo pairing is incorrect.
+func ParDoErrorFormatter(doFn interface{}, parDo interface{}) string {
+ doFnName := reflectx.FunctionName(doFn)
+ doFnOutSize := reflectx.FunctionOutputSize(doFn)
+
+ parDoName := reflectx.FunctionName(parDo)
+ parDoOutSize := reflectx.FunctionOutputSize(parDo)
+
+ useParDo := reflectx.FunctionName(RecommendParDo(doFnOutSize))
+ return fmt.Sprintf("DoFn %v has %v outptus, but %v requires %v outputs,
Use %v instead.", doFnName, doFnOutSize, parDoName, parDoOutSize, useParDo)
+
+}
+
+// recommendParDo takes a in a DoFns emit dimension and recommends the correct
+// ParDo to use.
+func RecommendParDo(emitDim int) interface{} {
+ switch {
+ case emitDim == 0:
+ return ParDo0
+ case emitDim == 1:
+ return ParDo
+ case emitDim == 2:
+ return ParDo2
+ case emitDim == 3:
+ return ParDo3
+ case emitDim == 4:
+ return ParDo4
+ case emitDim == 5:
+ return ParDo5
+ case emitDim == 6:
+ return ParDo6
+ case emitDim == 7:
+ return ParDo7
+ }
+ return ParDoN
+}
Review comment:
What I'd recommend here instead though is to return a string instead of
the function.
```
switch emitDim {
case 0,2,3,4,5,6,7:
return fmt.Sprintf("ParDo%d", emitDim)
case 1:
return "ParDo"
default:
return "ParDoN"
}
```
Easier to read, and see if it's correct. Read [Effective Go for more about
Switches](https://golang.org/doc/effective_go.html#switch)
##
File path: sdks/go/pkg/beam/pardo.go
##
@@ -414,7 +414,45 @@ func ParDo6(s Scope, dofn interface{}, col PCollection,
opts ...Option) (PCollec
func ParDo7(s Scope, dofn interface{}, col PCollection, opts ...Option)
(PCollection, PCollection, PCollection, PCollection, PCollection, PCollection,
PCollection) {
ret := MustN(TryParDo(s, dofn, col, opts...))
if len(ret) != 7 {
- panic(fmt.Sprintf("expected 7 output. Found: %v", ret))
+ panic(ParDoErrorFormatter(dofn, ParDo7))
}
return ret[0], ret[1], ret[2], ret[3], ret[4], ret[5], ret[6]
}
+
+//ParDoErrorFormatter is a helper function to provide a more concise error
+// message to the users when a DoFn and its ParDo pairing is incorrect.
+func ParDoErrorFormatter(doFn interface{}, parDo interface{}) string {
+ doFnName := reflectx.FunctionName(doFn)
+ doFnOutSize := reflectx.FunctionOutputSize(doFn)
+
+ parDoName := reflectx.FunctionName(parDo)
+ parDoOutSize := reflectx.FunctionOutputSize(parDo)
+
+ useParDo := reflectx.FunctionName(RecommendParDo(doFnOutSize))
+ return fmt.Sprintf("DoFn %v has %v outptus, but %v requires %v outputs,
Use %v instead.", doFnName, doFnOutSize, parDoName, parDoOutSize, useParDo)
+
+}
+
+// recommendParDo takes a in a DoFns emit dimension and recommends the correct
+// ParDo to use.
+func RecommendParDo(emitDim int) interface{} {
Review comment:
Same comment here, WRT exported functions being part of the external
API. Please rename this to recommendParDo.
##
File path: sdks/go/test/regression/pardo_test.go
##
@@ -16,6 +16,8 @@
[GitHub] [beam] rezarokni commented on pull request #11929: [BEAM-10201] Add deadletter support to JsonToRow
rezarokni commented on pull request #11929: URL: https://github.com/apache/beam/pull/11929#issuecomment-642368157 @reuvenlax @pabloem The latest commit covers most of the points now, with the exception of the three open questions I raised. - Return of the Schema that failed. @pabloem - Discussion about withExtendedErrMsg @reuvenlax - use of transient with ObjectMapper @reuvenlax This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] rezarokni commented on a change in pull request #11929: [BEAM-10201] Add deadletter support to JsonToRow
rezarokni commented on a change in pull request #11929:
URL: https://github.com/apache/beam/pull/11929#discussion_r438508022
##
File path:
sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/JsonToRow.java
##
@@ -116,4 +131,267 @@ private ObjectMapper objectMapper() {
return this.objectMapper;
}
}
+
+ /**
+ * Enable Dead letter support. If this value is set errors in the parsing
layer are returned as
+ * Row objects within a {@link ParseResult}
+ *
+ * You can access the results by using:
+ *
+ * ParseResult results =
jsonPersons.apply(JsonToRow.withDeadLetter(PERSON_SCHEMA));
+ *
+ * {@link ParseResult#getResults()}
+ *
+ * {@Code PCollection personRows = results.getResults()}
+ *
+ * {@link ParseResult#getFailedToParseLines()}
+ *
+ * {@Code PCollection errorsLines = results.getFailedToParseLines()}
+ *
+ * To access the reason for the failure you will need to first enable
extended error reporting.
+ * {@Code ParseResult results =
+ *
jsonPersons.apply(JsonToRow.withDeadLetter(PERSON_SCHEMA).withExtendedErrorInfo());
}
+ *
+ * {@link ParseResult#getFailedToParseLinesWithErr()}
+ *
+ * {@Code PCollection errorsLinesWithErrMsg =
results.getFailedToParseLines()}
+ *
+ * @return {@link JsonToRowWithErrFn}
+ */
+ @Experimental(Kind.SCHEMAS)
+ public static JsonToRowWithErrFn withDeadLetter(Schema rowSchema) {
+return JsonToRowWithErrFn.forSchema(rowSchema);
+ }
+
+ @AutoValue
+ abstract static class JsonToRowWithErrFn extends
PTransform, ParseResult> {
+
+private Pipeline pipeline;
+
+private PCollection parsedLine;
+private PCollection failedParse;
+private PCollection failedParseWithErr;
+
+private static final String LINE_FIELD_NAME = "line";
+private static final String ERROR_FIELD_NAME = "err";
+
+public static final Schema ERROR_ROW_SCHEMA =
+Schema.of(Field.of(LINE_FIELD_NAME, FieldType.STRING));
+
+public static final Schema ERROR_ROW_WITH_ERR_MSG_SCHEMA =
+Schema.of(
+Field.of(LINE_FIELD_NAME, FieldType.STRING),
+Field.of(ERROR_FIELD_NAME, FieldType.STRING));
+
+static final TupleTag PARSED_LINE = new TupleTag() {};
+static final TupleTag PARSE_ERROR_LINE = new TupleTag() {};
+static final TupleTag PARSE_ERROR_LINE_WITH_MSG = new TupleTag()
{};
+
+public abstract Schema getSchema();
+
+public abstract String getLineFieldName();
+
+public abstract String getErrorFieldName();
+
+public abstract boolean getExtendedErrorInfo();
+
+PCollection deadLetterCollection;
+
+public abstract Builder toBuilder();
+
+@AutoValue.Builder
+public abstract static class Builder {
+ public abstract Builder setSchema(Schema value);
+
+ public abstract Builder setLineFieldName(String value);
+
+ public abstract Builder setErrorFieldName(String value);
+
+ public abstract Builder setExtendedErrorInfo(boolean value);
+
+ public abstract JsonToRowWithErrFn build();
+}
+
+public static JsonToRowWithErrFn forSchema(Schema rowSchema) {
+ // Throw exception if this schema is not supported by RowJson
+ RowJson.verifySchemaSupported(rowSchema);
+ return new AutoValue_JsonToRow_JsonToRowWithErrFn.Builder()
+ .setSchema(rowSchema)
+ .setExtendedErrorInfo(false)
+ .setLineFieldName(LINE_FIELD_NAME)
+ .setErrorFieldName(ERROR_FIELD_NAME)
+ .build();
+}
+
+/**
+ * Adds the error message to the returned error Row.
+ *
+ * @return {@link JsonToRow}
+ */
+public JsonToRowWithErrFn withExtendedErrorInfo() {
+ return this.toBuilder().setExtendedErrorInfo(true).build();
+}
+
+/**
+ * Sets the field name for the line field in the returned Row.
+ *
+ * @return {@link JsonToRow}
+ */
+public JsonToRowWithErrFn setLineField(String lineField) {
+ return this.toBuilder().setLineFieldName(lineField).build();
+}
+
+/**
+ * Adds the error message to the returned error Row.
+ *
+ * @return {@link JsonToRow}
+ */
+public JsonToRowWithErrFn setErrorField(String errorField) {
+ if (!this.getExtendedErrorInfo()) {
+throw new IllegalArgumentException(
+"This option is only available with Extended Error Info.");
+ }
+ return this.toBuilder().setErrorFieldName(errorField).build();
+}
+
+@Override
+public ParseResult expand(PCollection jsonStrings) {
+
+ PCollectionTuple result =
+ jsonStrings.apply(
+ ParDo.of(new ParseWithError(this.getSchema(),
getExtendedErrorInfo()))
+ .withOutputTags(
+ PARSED_LINE,
+
TupleTagList.of(PARSE_ERROR_LINE).and(PARSE_ERROR_LINE_WITH_MSG)));
+
+ this.parsedLine = result.get(PARSED_LINE).setRowSchema(this.getSchema());
Review comment:
Changed to be passed in via
[GitHub] [beam] rezarokni commented on pull request #11929: [BEAM-10201] Add deadletter support to JsonToRow
rezarokni commented on pull request #11929: URL: https://github.com/apache/beam/pull/11929#issuecomment-642367042 @TheNeuralBit If you have time, would be great if you could scan the ParseResult and see if the API would suite all the things PubSubIO needs. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] ibzib commented on pull request #11987: [BEAM-10232] Restrict rsa package version.
ibzib commented on pull request #11987: URL: https://github.com/apache/beam/pull/11987#issuecomment-642360956 R: @aaltay @udim This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] ibzib commented on pull request #11977: Remove "whitelist" and "blacklist" terminology from repository where possible
ibzib commented on pull request #11977: URL: https://github.com/apache/beam/pull/11977#issuecomment-642359468 Thanks for doing this Kenn. Another one: https://github.com/apache/beam/blob/d5dd47b47cbdf0739ac1a28cb8fbd06becbdbae7/sdks/java/io/hadoop-format/src/test/resources/cassandra.yaml#L66 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] ibzib opened a new pull request #11987: [BEAM-10232] Restrict rsa package version.
ibzib opened a new pull request #11987: URL: https://github.com/apache/beam/pull/11987 **Please** add a meaningful description for your change here Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Samza | Spark --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[![Build
[GitHub] [beam] youngoli commented on a change in pull request #11870: [BEAM-9951] Adding integration tests for synthetic pipelines in Go
youngoli commented on a change in pull request #11870:
URL: https://github.com/apache/beam/pull/11870#discussion_r438490190
##
File path: sdks/go/pkg/beam/testing/passert/count.go
##
@@ -0,0 +1,52 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+//http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package passert
+
+import (
+ "fmt"
+
+ "github.com/apache/beam/sdks/go/pkg/beam"
+ "github.com/apache/beam/sdks/go/pkg/beam/core/typex"
+)
+
+func Count(s beam.Scope, col beam.PCollection, name string, count int) {
+ s = s.Scope(fmt.Sprintf("passert.Count(%v)", name))
+
+ if typex.IsKV(col.Type()) {
+ col = beam.DropKey(s, col)
+ }
+ counted := beam.Combine(s, {}, col)
+ Equals(s, counted, count)
Review comment:
I thought so, that's what I was referring to. The error message in
passert.Sum (while better than passert.Equals) is still a bit unclear, just
because it specifically mentions passert.Sum instead of passert.Count.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #11985: Bump default Pubsub gRPC timeout to 60 seconds
TheNeuralBit commented on pull request #11985: URL: https://github.com/apache/beam/pull/11985#issuecomment-642346883 R: @chamikaramj This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] youngoli opened a new pull request #11986: [BEAM-10235] Adding a CountElms transform to the Go SDK.
youngoli opened a new pull request #11986: URL: https://github.com/apache/beam/pull/11986 This transform counts the number of elements in a PTransform. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [x] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [x] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Samza | Spark --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[![Build
[GitHub] [beam] youngoli commented on pull request #11986: [BEAM-10235] Adding a CountElms transform to the Go SDK.
youngoli commented on pull request #11986: URL: https://github.com/apache/beam/pull/11986#issuecomment-642346309 R: @lostluck This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] TheNeuralBit opened a new pull request #11985: Bump default Pubsub gRPC timeout to 60 seconds
TheNeuralBit opened a new pull request #11985: URL: https://github.com/apache/beam/pull/11985 I'm seeing occasional timeouts when TestPubsub creates a subscription. It looks like the [official pubsub client](https://github.com/googleapis/java-pubsub/blob/71f15a4a90475c8e82c6cc6393d4232228273656/google-cloud-pubsub/src/main/java/com/google/cloud/pubsub/v1/stub/SubscriberStubSettings.java#L513) actually uses a 60 second timeout (with retries) for most calls. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Samza | Spark --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)[![Build
[GitHub] [beam] ibzib edited a comment on pull request #11972: [BEAM-9852] Do not create data channel for empty timer descriptor.
ibzib edited a comment on pull request #11972:
URL: https://github.com/apache/beam/pull/11972#issuecomment-642336725
PythonDocker is failing with a license issue:
```
File "/tmp/license_scripts/pull_licenses_py.py", line 149, in
py_ver=py_ver, license_list=sorted(','.join(no_licenses)),
how_to=how_to))
RuntimeError: Could not retrieve licences for packages [['3', 'a', 'h', 'p',
's', 'y']] in Python3.5 environment.
```
Package names are scrambled due to BEAM-10234, but I'm assuming it's pysha3.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [beam] ibzib commented on pull request #11972: [BEAM-9852] Do not create data channel for empty timer descriptor.
ibzib commented on pull request #11972: URL: https://github.com/apache/beam/pull/11972#issuecomment-642340383 Run PythonDocker PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] codeBehindMe commented on pull request #11976: [BEAM-10169] - ParDo functions with correct output N in their error messages.
codeBehindMe commented on pull request #11976: URL: https://github.com/apache/beam/pull/11976#issuecomment-642339945 R: @damondouglas This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] ibzib commented on pull request #11972: [BEAM-9852] Do not create data channel for empty timer descriptor.
ibzib commented on pull request #11972:
URL: https://github.com/apache/beam/pull/11972#issuecomment-642336725
I'm getting a weird license issue too:
```
File "/tmp/license_scripts/pull_licenses_py.py", line 149, in
py_ver=py_ver, license_list=sorted(','.join(no_licenses)),
how_to=how_to))
RuntimeError: Could not retrieve licences for packages [['3', 'a', 'h', 'p',
's', 'y']] in Python3.5 environment.
```
Package names are scrambled due to BEAM-10234, but I'm assuming it's pysha3.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [beam] TheNeuralBit opened a new pull request #11984: [BEAM-9217] Update DoFn javadoc for schema type translation
TheNeuralBit opened a new pull request #11984: URL: https://github.com/apache/beam/pull/11984 Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Samza | Spark --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [![Build
[GitHub] [beam] TheNeuralBit commented on pull request #11984: [BEAM-9217] Update DoFn javadoc for schema type translation
TheNeuralBit commented on pull request #11984: URL: https://github.com/apache/beam/pull/11984#issuecomment-642335762 R: @reuvenlax This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] ibzib opened a new pull request #11983: [BEAM-10234] Fix error message for missing licenses.
ibzib opened a new pull request #11983: URL: https://github.com/apache/beam/pull/11983 I also yapf'd it while I was at it. R: @udim Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Samza | Spark --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[![Build
[GitHub] [beam] pabloem commented on pull request #11982: [BEAM-6892] Supporting bucket auto-creation for Dataflow.
pabloem commented on pull request #11982: URL: https://github.com/apache/beam/pull/11982#issuecomment-642331957 Run Python2_PVR_Flink PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #11982: [BEAM-6892] Supporting bucket auto-creation for Dataflow.
pabloem commented on pull request #11982: URL: https://github.com/apache/beam/pull/11982#issuecomment-642331818 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #11982: [BEAM-6892] Supporting bucket auto-creation for Dataflow.
pabloem commented on pull request #11982: URL: https://github.com/apache/beam/pull/11982#issuecomment-642331889 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #11929: [BEAM-10201] Add deadletter support to JsonToRow
TheNeuralBit commented on pull request #11929: URL: https://github.com/apache/beam/pull/11929#issuecomment-642330786 FYI `PubsubJsonTableProvider` has support for writing to a dead letter pubsub topic: https://github.com/apache/beam/blob/d5dd47b47cbdf0739ac1a28cb8fbd06becbdbae7/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubIOJsonTable.java#L160. Maybe there's some duplicate logic we can remove after this PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #11982: [BEAM-6892] Supporting bucket auto-creation for Dataflow.
pabloem commented on pull request #11982: URL: https://github.com/apache/beam/pull/11982#issuecomment-642327786 I've tested this on my machine This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] rezarokni commented on pull request #11955: [BEAM-10220] add support for implicit nulls for converting between beam rows and json
rezarokni commented on pull request #11955: URL: https://github.com/apache/beam/pull/11955#issuecomment-642327425 Thanx for FYI @pabloem . This is awesome! we are also looking to add deadletter pattern to JsonToRow: https://github.com/apache/beam/pull/11929 The null support your adding is great! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] rezarokni commented on a change in pull request #11929: [BEAM-10201] Add deadletter support to JsonToRow
rezarokni commented on a change in pull request #11929:
URL: https://github.com/apache/beam/pull/11929#discussion_r438470939
##
File path:
sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/JsonToRow.java
##
@@ -72,10 +79,44 @@
@Experimental(Kind.SCHEMAS)
public class JsonToRow {
+ private static final String LINE_FIELD_NAME = "line";
+ private static final String ERROR_FIELD_NAME = "err";
+
+ public static final Schema ERROR_ROW_SCHEMA =
+ Schema.of(
+ Field.of(LINE_FIELD_NAME, FieldType.STRING),
+ Field.of(ERROR_FIELD_NAME, FieldType.STRING));
+
+ public static final TupleTag MAIN_TUPLE_TAG = new TupleTag() {};
+ public static final TupleTag DEAD_LETTER_TUPLE_TAG = new
TupleTag() {};
+
Review comment:
Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #11982: [BEAM-6892] Supporting bucket auto-creation for Dataflow.
pabloem commented on pull request #11982: URL: https://github.com/apache/beam/pull/11982#issuecomment-642327079 r: @udim This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] pabloem opened a new pull request #11982: [BEAM-6892] Supporting bucket auto-creation for Dataflow.
pabloem opened a new pull request #11982: URL: https://github.com/apache/beam/pull/11982 Following the logic from Java (https://github.com/apache/beam/blob/b56740f0e8cd80c2873412847d0b336837429fb9/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/options/GcpOptions.java#L332). Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Samza | Spark --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[![Build
[GitHub] [beam] pabloem commented on pull request #11955: [BEAM-10220] add support for implicit nulls for converting between beam rows and json
pabloem commented on pull request #11955: URL: https://github.com/apache/beam/pull/11955#issuecomment-642325514 and assigned BEAM-7624 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #11955: [BEAM-10220] add support for implicit nulls for converting between beam rows and json
pabloem commented on pull request #11955: URL: https://github.com/apache/beam/pull/11955#issuecomment-642325342 Added @reubenvanammers as contributor! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #11955: [BEAM-10220] add support for implicit nulls for converting between beam rows and json
pabloem commented on pull request #11955: URL: https://github.com/apache/beam/pull/11955#issuecomment-642325145 fyi @rezarokni @reuvenlax This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #11955: [BEAM-10220] add support for implicit nulls for converting between beam rows and json
TheNeuralBit commented on pull request #11955: URL: https://github.com/apache/beam/pull/11955#issuecomment-642323660 I went to assign BEAM-7624 to you, but it looks like you need to be added as a Beam contributor first. @pabloem could you add Reuben on jira? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] TheNeuralBit opened a new pull request #11981: [BEAM-8364] SchemaCoder is not consistent with equals
TheNeuralBit opened a new pull request #11981: URL: https://github.com/apache/beam/pull/11981 R: @reuvenlax CC: @nevillelyh Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Samza | Spark --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [![Build
[GitHub] [beam] TheNeuralBit commented on pull request #11980: WIP: [BEAM-9564] Dataframe schema input
TheNeuralBit commented on pull request #11980: URL: https://github.com/apache/beam/pull/11980#issuecomment-642320833 CC: @robertwb This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] TheNeuralBit opened a new pull request #11980: WIP: [BEAM-9564] Dataframe schema input
TheNeuralBit opened a new pull request #11980: URL: https://github.com/apache/beam/pull/11980 Adds batching of schema'd PCollections into dataframes based on BatchElements transform. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Samza | Spark --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [![Build
[GitHub] [beam] ibzib commented on pull request #11972: [BEAM-9852] Do not create data channel for empty timer descriptor.
ibzib commented on pull request #11972: URL: https://github.com/apache/beam/pull/11972#issuecomment-642320097 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] damondouglas opened a new pull request #11979: [BEAM-9679] Add Partition task to Core Transform katas
damondouglas opened a new pull request #11979: URL: https://github.com/apache/beam/pull/11979 This pull requests adds a Partition lesson to the Go SDK katas. I would like to request the following reviewers: (R: @lostluck ) (R: @henryken ) If accepted by both reviewers, please wait until the [Stepik course](https://stepik.org/course/70387) is updated before finally merging this PR. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [x] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [x] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Samza | Spark --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [![Build
[GitHub] [beam] amaliujia commented on pull request #11975: [BEAM-9198] BeamSQL aggregation analytics functionality
amaliujia commented on pull request #11975: URL: https://github.com/apache/beam/pull/11975#issuecomment-642302788 also cc @Mark-Zeng. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] ibzib commented on pull request #11972: [BEAM-9852] Do not create data channel for empty timer descriptor.
ibzib commented on pull request #11972: URL: https://github.com/apache/beam/pull/11972#issuecomment-642301092 Looks like a bug from new rsa pypi release, filed BEAM-10232 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] amaliujia commented on pull request #11975: [BEAM-9198] BeamSQL aggregation analytics functionality
amaliujia commented on pull request #11975: URL: https://github.com/apache/beam/pull/11975#issuecomment-642300874 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] kennknowles opened a new pull request #11978: [BEAM-6458] Link to metrics page by domain, not ip address
kennknowles opened a new pull request #11978: URL: https://github.com/apache/beam/pull/11978 Replacing a direct link by i.p. with our new domain name Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [x] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [x] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [x] Update `CHANGES.md` with noteworthy changes. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Samza | Spark --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[![Build
[GitHub] [beam] ibzib commented on pull request #11972: [BEAM-9852] Do not create data channel for empty timer descriptor.
ibzib commented on pull request #11972: URL: https://github.com/apache/beam/pull/11972#issuecomment-642294606 > Thanks for the explanation! Please run tox -e py3-yapf to fix pylint. I ran `tox -e py3-yapf` already, but pylint is still failing. `Execution failed for task ':sdks:python:test-suites:tox:py37:mypyPy37'.` ``` google-auth 1.16.1 has requirement rsa<4.1,>=3.1.4, but you have rsa 4.1. ERROR: InvocationError for command /home/jenkins/jenkins-slave/workspace/beam_PreCommit_PythonLint_Commit/src/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-mypy/py37-mypy/bin/pip check (exited with code 1) ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] kennknowles opened a new pull request #11977: Remove "whitelist" and "blacklist" terminology from repository where possible
kennknowles opened a new pull request #11977: URL: https://github.com/apache/beam/pull/11977 For context, see: - https://tools.ietf.org/html/draft-knodel-terminology-01 - https://developers.google.com/style/inclusive-documentation#features-and-users Blocked by configuration of other projects and vendored code: - https://github.com/tox-dev/tox/issues/1491 - https://github.com/jenkinsci/ghprb-plugin/issues/784 - Generated storage_v1_messages.py (undocumented how we generated this!?) - Many golang and vendored modules Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Samza | Spark --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [![Build
[GitHub] [beam] udim edited a comment on pull request #7017: [BEAM-5788] Update storage_v1 client and messages
udim edited a comment on pull request #7017: URL: https://github.com/apache/beam/pull/7017#issuecomment-642292563 My notes from around that time: ``` pip install google-apitools[cli] .../VIRTUALENV_PATH/bin/gen_client --discovery_url storage.v1 \ --overwrite --outdir=out client ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] udim commented on pull request #7017: [BEAM-5788] Update storage_v1 client and messages
udim commented on pull request #7017: URL: https://github.com/apache/beam/pull/7017#issuecomment-642292563 My notes from around that times: ``` pip install google-apitools[cli] .../VIRTUALENV_PATH/bin/gen_client --discovery_url storage.v1 \ --overwrite --outdir=out client ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] udim edited a comment on pull request #7017: [BEAM-5788] Update storage_v1 client and messages
udim edited a comment on pull request #7017: URL: https://github.com/apache/beam/pull/7017#issuecomment-642292563 My notes from around that time: ``` pip install google-apitools[cli] .../VIRTUALENV_PATH/bin/gen_client --discovery_url storage.v1 \ --overwrite --outdir=out client ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] lukecwik commented on pull request #11938: [BEAM-9577] Remove uses of legacy artifact service in Java.
lukecwik commented on pull request #11938: URL: https://github.com/apache/beam/pull/11938#issuecomment-642291285 Won't be able to get to this for a little while as I'm on the customer support rotation currently. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] codeBehindMe commented on pull request #11976: [BEAM-10169] - ParDo functions with correct output N in their error messages.
codeBehindMe commented on pull request #11976: URL: https://github.com/apache/beam/pull/11976#issuecomment-642290717 @damondouglas If you could give me some feedback that'll be nice. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] codeBehindMe removed a comment on pull request #11976: [BEAM-10169] - ParDo functions with correct output N in their error messages.
codeBehindMe removed a comment on pull request #11976: URL: https://github.com/apache/beam/pull/11976#issuecomment-642290142 @damondouglas Hey mate, could you give this a quick once over. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] codeBehindMe commented on pull request #11976: [BEAM-10169] - ParDo functions with correct output N in their error messages.
codeBehindMe commented on pull request #11976: URL: https://github.com/apache/beam/pull/11976#issuecomment-642290142 @damondouglas Hey mate, could you give this a quick once over. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] codeBehindMe opened a new pull request #11976: [BEAM-10169] - ParDo functions with correct output N in their error messages.
codeBehindMe opened a new pull request #11976: URL: https://github.com/apache/beam/pull/11976 Previously, when a mismatch of the dimensions emitted by the DoFn and the dimensions emitted by the ParDo was encountered, a less than helpful message of `expected 1 output. Found: []` was returned. This caused some confusion especially when paired with ParDo and ParDo0 which may be counterintuitive arrangement to most. In this patch, we provide a more helpful error message which correctly reference the DoFn which has the misalignment and the ParDo which the misalignment of dimensions were captured by. It also recommends a ParDo function which may be used by the DoFn in question. We introduce two new functions to the pardo package as well as a new function to the reflectx package to help us construct this error message. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [![Build
[GitHub] [beam] kennknowles commented on pull request #7017: [BEAM-5788] Update storage_v1 client and messages
kennknowles commented on pull request #7017: URL: https://github.com/apache/beam/pull/7017#issuecomment-642285619 Ping? Can you comment here with a link to the documentation on regenerating these? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] jhnmora000 commented on pull request #11975: [BEAM-9198] BeamSQL aggregation analytics functionality
jhnmora000 commented on pull request #11975: URL: https://github.com/apache/beam/pull/11975#issuecomment-642285036 R:@amaliujia This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] jhnmora000 opened a new pull request #11975: [BEAM-9198] BeamSQL aggregation analytics functionality
jhnmora000 opened a new pull request #11975: URL: https://github.com/apache/beam/pull/11975 An concept-proof implementation for a cumulative sum using Analytic functions. Implemented in: BeamAnalyticFunctionsExperimentTest.testOverCumulativeSum() Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Samza | Spark --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[![Build
[GitHub] [beam] lostluck merged pull request #11936: [BEAM-9679] Add CombinePerKey to Core Transforms Go Katas
lostluck merged pull request #11936: URL: https://github.com/apache/beam/pull/11936 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] damondouglas commented on pull request #11936: [BEAM-9679] Add CombinePerKey to Core Transforms Go Katas
damondouglas commented on pull request #11936: URL: https://github.com/apache/beam/pull/11936#issuecomment-642274790 This is ready to merge now. Thank you @lostluck and @henryken. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] lostluck commented on a change in pull request #11870: [BEAM-9951] Adding integration tests for synthetic pipelines in Go
lostluck commented on a change in pull request #11870:
URL: https://github.com/apache/beam/pull/11870#discussion_r438411053
##
File path: sdks/go/pkg/beam/testing/passert/count.go
##
@@ -0,0 +1,52 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+//http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package passert
+
+import (
+ "fmt"
+
+ "github.com/apache/beam/sdks/go/pkg/beam"
+ "github.com/apache/beam/sdks/go/pkg/beam/core/typex"
+)
+
+func Count(s beam.Scope, col beam.PCollection, name string, count int) {
+ s = s.Scope(fmt.Sprintf("passert.Count(%v)", name))
+
+ if typex.IsKV(col.Type()) {
+ col = beam.DropKey(s, col)
+ }
+ counted := beam.Combine(s, {}, col)
+ Equals(s, counted, count)
Review comment:
I meant passert.Sum. Sorry that should have been clearer.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [beam] lostluck commented on a change in pull request #11870: [BEAM-9951] Adding integration tests for synthetic pipelines in Go
lostluck commented on a change in pull request #11870:
URL: https://github.com/apache/beam/pull/11870#discussion_r438411053
##
File path: sdks/go/pkg/beam/testing/passert/count.go
##
@@ -0,0 +1,52 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+//http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package passert
+
+import (
+ "fmt"
+
+ "github.com/apache/beam/sdks/go/pkg/beam"
+ "github.com/apache/beam/sdks/go/pkg/beam/core/typex"
+)
+
+func Count(s beam.Scope, col beam.PCollection, name string, count int) {
+ s = s.Scope(fmt.Sprintf("passert.Count(%v)", name))
+
+ if typex.IsKV(col.Type()) {
+ col = beam.DropKey(s, col)
+ }
+ counted := beam.Combine(s, {}, col)
+ Equals(s, counted, count)
Review comment:
I meant passert.Sum
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [beam] pabloem commented on a change in pull request #11702: [BEAM-9990] Add Conditional Update and Conditional Create to FhirIO
pabloem commented on a change in pull request #11702:
URL: https://github.com/apache/beam/pull/11702#discussion_r438404800
##
File path:
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/healthcare/FhirIO.java
##
@@ -1173,4 +1276,339 @@ public void executeBundles(ProcessContext context) {
}
}
}
+
+ /**
+ * Create resources fhir io . create resources.
+ *
+ * @param the type parameter
+ * @param fhirStore the fhir store
+ * @return the fhir io . create resources
+ */
+ public static FhirIO.CreateResources
createResources(ValueProvider fhirStore) {
+return new CreateResources(fhirStore);
+ }
+
+ /**
+ * Create resources fhir io . create resources.
+ *
+ * @param the type parameter
+ * @param fhirStore the fhir store
+ * @return the fhir io . create resources
+ */
+ public static FhirIO.CreateResources createResources(String
fhirStore) {
+return new CreateResources(fhirStore);
+ }
+ /**
+ * {@link PTransform} for Creating FHIR resources.
+ *
+ *
https://cloud.google.com/healthcare/docs/reference/rest/v1beta1/projects.locations.datasets.fhirStores.fhir/create
+ */
+ public static class CreateResources extends PTransform,
Write.Result> {
+private final String fhirStore;
+private SerializableFunction ifNoneExistFunction;
+private SerializableFunction formatBodyFunction;
+private SerializableFunction typeFunction;
+private static final Logger LOG =
LoggerFactory.getLogger(CreateResources.class);
+
+/**
+ * Instantiates a new Create resources transform.
+ *
+ * @param fhirStore the fhir store
+ */
+CreateResources(ValueProvider fhirStore) {
+ this.fhirStore = fhirStore.get();
+}
+
+/**
+ * Instantiates a new Create resources.
+ *
+ * @param fhirStore the fhir store
+ */
+CreateResources(String fhirStore) {
+ this.fhirStore = fhirStore;
+}
+
+/**
+ * This adds a {@link SerializableFunction} that reads an resource string
and extracts an
+ * If-None-Exists query for conditional create. Typically this will just
be extracting an ID to
+ * look for.
+ *
+ *
https://cloud.google.com/healthcare/docs/reference/rest/v1beta1/projects.locations.datasets.fhirStores.fhir/create
+ *
+ * @param ifNoneExistFunction the if none exist function
+ * @return the create resources
+ */
+public CreateResources withIfNotExistFunction(
+SerializableFunction ifNoneExistFunction) {
+ this.ifNoneExistFunction = ifNoneExistFunction;
+ return this;
+}
+
+/**
+ * This adds a {@link SerializableFunction} that reads an resource string
and extracts an
+ * resource type.
+ *
+ *
https://cloud.google.com/healthcare/docs/reference/rest/v1beta1/projects.locations.datasets.fhirStores.fhir/create
+ *
+ * @param typeFunction for extracting type from a resource.
+ * @return the create resources
+ */
+public CreateResources withTypeFunction(SerializableFunction
typeFunction) {
+ this.typeFunction = typeFunction;
+ return this;
+}
+/**
+ * With format body function create resources.
+ *
+ * @param formatBodyFunction the format body function
+ * @return the create resources
+ */
+public CreateResources withFormatBodyFunction(
Review comment:
Hm - I am not sure. I feel that `Resource` may be better? Since we take
in any type, and format a resource that gets inserted? But I trust your
judgement here.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [beam] pabloem commented on a change in pull request #11702: [BEAM-9990] Add Conditional Update and Conditional Create to FhirIO
pabloem commented on a change in pull request #11702:
URL: https://github.com/apache/beam/pull/11702#discussion_r438403098
##
File path:
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/healthcare/FhirIO.java
##
@@ -155,17 +168,53 @@
* FhirIO.Write.Result writeResult =
* output.apply("Execute FHIR Bundles",
FhirIO.executeBundles(options.getExistingFhirStore()));
*
+ * // Alternatively you could use import for high throughput to a new store.
+ * FhirIO.Write.Result writeResult =
+ * output.apply("Import FHIR Resources",
FhirIO.executeBundles(options.getNewFhirStore()));
+ * // [End Writing ]
+ *
* PCollection> failedBundles =
writeResult.getFailedInsertsWithErr();
*
+ * // [Begin Writing to Dead Letter Queue]
* failedBundles.apply("Write failed bundles to BigQuery",
* BigQueryIO
* .write()
* .to(option.getBQFhirExecuteBundlesDeadLetterTable())
* .withFormatFunction(new HealthcareIOErrorToTableRow()));
+ * // [End Writing to Dead Letter Queue]
+ *
+ * // Alternatively you may want to handle DeadLetter with conditional update
+ * // [Begin Reconciliation with Conditional Update]
+ * failedBundles
+ * .apply("Reconcile with Conditional Update",
+ * FhirIO.ConditionalUpdate(fhirStore)
+ *
.withFormatBodyFunction(HealthcareIOError::getDataResource)
+ * .withTypeFunction((HealthcareIOError err) -> {
+ * String body = err.getDataResource();
+ * // TODO(user) insert logic to exctract type.
+ * return params;
Review comment:
Sounds good. No need to add!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [beam] youngoli commented on pull request #11870: [BEAM-9951] Adding integration tests for synthetic pipelines in Go
youngoli commented on pull request #11870: URL: https://github.com/apache/beam/pull/11870#issuecomment-642253483 I added unit tests and changed the error message. I spent a bunch of time moving the Count code to Beam (specifically the stats package), but ended up having to scrap it because having it there caused a circular dependency. Testing relied on stats which relied on testing for passerts. I'll add an independent version to the stats package because I already have the code written and it seems valuable, but it's not going to be used for the passerts. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] youngoli commented on a change in pull request #11870: [BEAM-9951] Adding integration tests for synthetic pipelines in Go
youngoli commented on a change in pull request #11870:
URL: https://github.com/apache/beam/pull/11870#discussion_r438401382
##
File path: sdks/go/pkg/beam/testing/passert/count.go
##
@@ -0,0 +1,52 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+//http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package passert
+
+import (
+ "fmt"
+
+ "github.com/apache/beam/sdks/go/pkg/beam"
+ "github.com/apache/beam/sdks/go/pkg/beam/core/typex"
+)
+
+func Count(s beam.Scope, col beam.PCollection, name string, count int) {
+ s = s.Scope(fmt.Sprintf("passert.Count(%v)", name))
+
+ if typex.IsKV(col.Type()) {
+ col = beam.DropKey(s, col)
+ }
+ counted := beam.Combine(s, {}, col)
+ Equals(s, counted, count)
Review comment:
I think it could, and I considered it, but the error message isn't too
helpful either (better than equals though). I added a ParDo0 with a custom
error message which I think ends up being the most user-friendly.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [beam] robertwb commented on pull request #11963: Add relational GroupBy transform to Python.
robertwb commented on pull request #11963: URL: https://github.com/apache/beam/pull/11963#issuecomment-642251071 CC: @TheNeuralBit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] robertwb commented on pull request #11974: [BEAM-10035] Add more methods to deferred dataframes.
robertwb commented on pull request #11974: URL: https://github.com/apache/beam/pull/11974#issuecomment-642250609 CC: @TheNeuralBit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] boyuanzz commented on pull request #11972: [BEAM-9852] Do not create data channel for empty timer descriptor.
boyuanzz commented on pull request #11972: URL: https://github.com/apache/beam/pull/11972#issuecomment-642246885 Thanks for the explanation! Please run `tox -e py3-yapf` to fix pylint. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] robertwb merged pull request #11935: [BEAM-9577] Remove use of legacy artifact service in Python.
robertwb merged pull request #11935: URL: https://github.com/apache/beam/pull/11935 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] robertwb commented on a change in pull request #11935: [BEAM-9577] Remove use of legacy artifact service in Python.
robertwb commented on a change in pull request #11935:
URL: https://github.com/apache/beam/pull/11935#discussion_r438395866
##
File path: sdks/python/apache_beam/runners/portability/artifact_service.py
##
@@ -55,272 +52,6 @@
from typing import Iterable
from typing import MutableMapping
-# The legacy artifact staging and retrieval services.
-
-
-class AbstractArtifactService(
-beam_artifact_api_pb2_grpc.LegacyArtifactStagingServiceServicer,
-beam_artifact_api_pb2_grpc.LegacyArtifactRetrievalServiceServicer):
-
- _DEFAULT_CHUNK_SIZE = 2 << 20 # 2mb
-
- def __init__(self, root, chunk_size=None):
-self._root = root
-self._chunk_size = chunk_size or self._DEFAULT_CHUNK_SIZE
-
- def _sha256(self, string):
-return hashlib.sha256(string.encode('utf-8')).hexdigest()
-
- def _join(self, *args):
-# type: (*str) -> str
-raise NotImplementedError(type(self))
-
- def _dirname(self, path):
-# type: (str) -> str
-raise NotImplementedError(type(self))
-
- def _temp_path(self, path):
-# type: (str) -> str
-return path + '.tmp'
-
- def _open(self, path, mode):
-raise NotImplementedError(type(self))
-
- def _rename(self, src, dest):
-# type: (str, str) -> None
-raise NotImplementedError(type(self))
-
- def _delete(self, path):
-# type: (str) -> None
-raise NotImplementedError(type(self))
-
- def _artifact_path(self, retrieval_token, name):
-# type: (str, str) -> str
-return self._join(self._dirname(retrieval_token), self._sha256(name))
-
- def _manifest_path(self, retrieval_token):
-# type: (str) -> str
-return retrieval_token
-
- def _get_manifest_proxy(self, retrieval_token):
-# type: (str) -> beam_artifact_api_pb2.ProxyManifest
-with self._open(self._manifest_path(retrieval_token), 'r') as fin:
- return json_format.Parse(
- fin.read().decode('utf-8'), beam_artifact_api_pb2.ProxyManifest())
-
- def retrieval_token(self, staging_session_token):
-# type: (str) -> str
-return self._join(
-self._root, self._sha256(staging_session_token), 'MANIFEST')
-
- def PutArtifact(self, request_iterator, context=None):
-# type: (...) -> beam_artifact_api_pb2.PutArtifactResponse
-first = True
-for request in request_iterator:
- if first:
-first = False
-metadata = request.metadata.metadata
-retrieval_token = self.retrieval_token(
-request.metadata.staging_session_token)
-artifact_path = self._artifact_path(retrieval_token, metadata.name)
-temp_path = self._temp_path(artifact_path)
-fout = self._open(temp_path, 'w')
-hasher = hashlib.sha256()
- else:
-hasher.update(request.data.data)
-fout.write(request.data.data)
-fout.close()
-data_hash = hasher.hexdigest()
-if metadata.sha256 and metadata.sha256 != data_hash:
- self._delete(temp_path)
- raise ValueError(
- 'Bad metadata hash: %s vs %s' % (metadata.sha256, data_hash))
-self._rename(temp_path, artifact_path)
-return beam_artifact_api_pb2.PutArtifactResponse()
-
- def CommitManifest(self,
- request, # type:
beam_artifact_api_pb2.CommitManifestRequest
- context=None):
-# type: (...) -> beam_artifact_api_pb2.CommitManifestResponse
-retrieval_token = self.retrieval_token(request.staging_session_token)
-proxy_manifest = beam_artifact_api_pb2.ProxyManifest(
-manifest=request.manifest,
-location=[
-beam_artifact_api_pb2.ProxyManifest.Location(
-name=metadata.name,
-uri=self._artifact_path(retrieval_token, metadata.name))
-for metadata in request.manifest.artifact
-])
-with self._open(self._manifest_path(retrieval_token), 'w') as fout:
- fout.write(json_format.MessageToJson(proxy_manifest).encode('utf-8'))
-return beam_artifact_api_pb2.CommitManifestResponse(
-retrieval_token=retrieval_token)
-
- def GetManifest(self,
- request, # type: beam_artifact_api_pb2.GetManifestRequest
- context=None):
-# type: (...) -> beam_artifact_api_pb2.GetManifestResponse
-return beam_artifact_api_pb2.GetManifestResponse(
-manifest=self._get_manifest_proxy(request.retrieval_token).manifest)
-
- def GetArtifact(self,
- request, # type:
beam_artifact_api_pb2.LegacyGetArtifactRequest
- context=None):
-# type: (...) -> Iterator[beam_artifact_api_pb2.ArtifactChunk]
-for artifact in self._get_manifest_proxy(request.retrieval_token).location:
- if artifact.name == request.name:
-with self._open(artifact.uri, 'r') as fin:
- # This value is not emitted, but lets us yield a single empty
- # chunk on an empty file.
- chunk = b'1'
- while chunk:
-chunk = fin.read(self._chunk_size)
-yield
[GitHub] [beam] kennknowles merged pull request #11960: [BEAM-9999] Remove Gearpump runner.
kennknowles merged pull request #11960: URL: https://github.com/apache/beam/pull/11960 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] kennknowles commented on pull request #11960: [BEAM-9999] Remove Gearpump runner.
kennknowles commented on pull request #11960: URL: https://github.com/apache/beam/pull/11960#issuecomment-642236775 We've got enough LGTM and test greenness. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] ibzib commented on pull request #11972: [BEAM-9852] Do not create data channel for empty timer descriptor.
ibzib commented on pull request #11972: URL: https://github.com/apache/beam/pull/11972#issuecomment-642237210 > Can you also fix the state one as well (beam/sdks/python/apache_beam/runners/worker/sdk_worker.py Line 691)? Good catch. Fixed This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] TheNeuralBit merged pull request #11973: Finalize CHANGES.md for 2.22.0
TheNeuralBit merged pull request #11973: URL: https://github.com/apache/beam/pull/11973 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #11973: Finalize CHANGES.md for 2.22.0
TheNeuralBit commented on pull request #11973: URL: https://github.com/apache/beam/pull/11973#issuecomment-642236509 Thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] amaliujia commented on pull request #11960: [BEAM-9999] Remove Gearpump runner.
amaliujia commented on pull request #11960: URL: https://github.com/apache/beam/pull/11960#issuecomment-642236503 Re-trigger the Java_Examples_Dataflow_Java11. Now it seems get scheduled. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] amaliujia commented on pull request #11960: [BEAM-9999] Remove Gearpump runner.
amaliujia commented on pull request #11960: URL: https://github.com/apache/beam/pull/11960#issuecomment-642236279 Run Java_Examples_Dataflow_Java11 Precommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] amaliujia commented on pull request #11973: Finalize CHANGES.md for 2.22.0
amaliujia commented on pull request #11973: URL: https://github.com/apache/beam/pull/11973#issuecomment-642234960 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] lukecwik commented on pull request #11972: [BEAM-9852] Do not create data channel for empty timer descriptor.
lukecwik commented on pull request #11972: URL: https://github.com/apache/beam/pull/11972#issuecomment-642233639 > > I'm curious with which runner we will run into this issue. > > All portable runners as far as I know (confirmed for Flink, Spark, Dataflow Python streaming). > > > The problem is that protos default isn't null but the default empty instance so the check should really be whether it is not the default instance. > > Thanks Luke, looks like `message.HasField` is what we want here. Can you also fix the state one as well (beam/sdks/python/apache_beam/runners/worker/sdk_worker.py Line 691)? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] ibzib commented on pull request #11972: [BEAM-9852] Do not create data channel for empty timer descriptor.
ibzib commented on pull request #11972: URL: https://github.com/apache/beam/pull/11972#issuecomment-642232521 > I'm curious with which runner we will run into this issue. All portable runners as far as I know (confirmed for Flink, Spark, Dataflow Python streaming). > The problem is that protos default isn't null but the default empty instance so the check should really be whether it is not the default instance. Thanks Luke, looks like `message.HasField` is what we want here. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] InigoSJ commented on a change in pull request #11943: [BEAM-10209] Add without_defaults to Mean combiner in Python
InigoSJ commented on a change in pull request #11943: URL: https://github.com/apache/beam/pull/11943#discussion_r438368619 ## File path: sdks/python/apache_beam/transforms/combiners.py ## @@ -66,8 +66,21 @@ class Mean(object): """Combiners for computing arithmetic means of elements.""" class Globally(ptransform.PTransform): """combiners.Mean.Globally computes the arithmetic mean of the elements.""" +def __init__(self, has_defaults=True, *args, **kwargs): + super(Mean.Globally, self).__init__() + self.has_defaults = has_defaults + self.args = args + self.kwargs = kwargs + def expand(self, pcoll): - return pcoll | core.CombineGlobally(MeanCombineFn()) + if self.has_defaults: +return pcoll | core.CombineGlobally(MeanCombineFn()) + else: +return pcoll | core.CombineGlobally(MeanCombineFn()).without_defaults() + +def without_defaults(self): + self.has_defaults = False Review comment: I'm not sure if I understand it correctly, but I made a modification to return ``` return self | core.CombineGlobally(MeanCombineFn()).without_defaults() ``` I personally think it's less intuitive, but you know more :D. I also tried directly returning without modifying `self.has_defaults`, but it executes the first sequence of the Reduce fine, but the second returns an error saying that I need the `without_defaults()`, since it directly goes to the `expand()`. If you have suggestions, I'm all ears. Thanks again This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] ihji commented on a change in pull request #11935: [BEAM-9577] Remove use of legacy artifact service in Python.
ihji commented on a change in pull request #11935:
URL: https://github.com/apache/beam/pull/11935#discussion_r438370617
##
File path: sdks/python/apache_beam/runners/portability/artifact_service.py
##
@@ -55,272 +52,6 @@
from typing import Iterable
from typing import MutableMapping
-# The legacy artifact staging and retrieval services.
-
-
-class AbstractArtifactService(
-beam_artifact_api_pb2_grpc.LegacyArtifactStagingServiceServicer,
-beam_artifact_api_pb2_grpc.LegacyArtifactRetrievalServiceServicer):
-
- _DEFAULT_CHUNK_SIZE = 2 << 20 # 2mb
-
- def __init__(self, root, chunk_size=None):
-self._root = root
-self._chunk_size = chunk_size or self._DEFAULT_CHUNK_SIZE
-
- def _sha256(self, string):
-return hashlib.sha256(string.encode('utf-8')).hexdigest()
-
- def _join(self, *args):
-# type: (*str) -> str
-raise NotImplementedError(type(self))
-
- def _dirname(self, path):
-# type: (str) -> str
-raise NotImplementedError(type(self))
-
- def _temp_path(self, path):
-# type: (str) -> str
-return path + '.tmp'
-
- def _open(self, path, mode):
-raise NotImplementedError(type(self))
-
- def _rename(self, src, dest):
-# type: (str, str) -> None
-raise NotImplementedError(type(self))
-
- def _delete(self, path):
-# type: (str) -> None
-raise NotImplementedError(type(self))
-
- def _artifact_path(self, retrieval_token, name):
-# type: (str, str) -> str
-return self._join(self._dirname(retrieval_token), self._sha256(name))
-
- def _manifest_path(self, retrieval_token):
-# type: (str) -> str
-return retrieval_token
-
- def _get_manifest_proxy(self, retrieval_token):
-# type: (str) -> beam_artifact_api_pb2.ProxyManifest
-with self._open(self._manifest_path(retrieval_token), 'r') as fin:
- return json_format.Parse(
- fin.read().decode('utf-8'), beam_artifact_api_pb2.ProxyManifest())
-
- def retrieval_token(self, staging_session_token):
-# type: (str) -> str
-return self._join(
-self._root, self._sha256(staging_session_token), 'MANIFEST')
-
- def PutArtifact(self, request_iterator, context=None):
-# type: (...) -> beam_artifact_api_pb2.PutArtifactResponse
-first = True
-for request in request_iterator:
- if first:
-first = False
-metadata = request.metadata.metadata
-retrieval_token = self.retrieval_token(
-request.metadata.staging_session_token)
-artifact_path = self._artifact_path(retrieval_token, metadata.name)
-temp_path = self._temp_path(artifact_path)
-fout = self._open(temp_path, 'w')
-hasher = hashlib.sha256()
- else:
-hasher.update(request.data.data)
-fout.write(request.data.data)
-fout.close()
-data_hash = hasher.hexdigest()
-if metadata.sha256 and metadata.sha256 != data_hash:
- self._delete(temp_path)
- raise ValueError(
- 'Bad metadata hash: %s vs %s' % (metadata.sha256, data_hash))
-self._rename(temp_path, artifact_path)
-return beam_artifact_api_pb2.PutArtifactResponse()
-
- def CommitManifest(self,
- request, # type:
beam_artifact_api_pb2.CommitManifestRequest
- context=None):
-# type: (...) -> beam_artifact_api_pb2.CommitManifestResponse
-retrieval_token = self.retrieval_token(request.staging_session_token)
-proxy_manifest = beam_artifact_api_pb2.ProxyManifest(
-manifest=request.manifest,
-location=[
-beam_artifact_api_pb2.ProxyManifest.Location(
-name=metadata.name,
-uri=self._artifact_path(retrieval_token, metadata.name))
-for metadata in request.manifest.artifact
-])
-with self._open(self._manifest_path(retrieval_token), 'w') as fout:
- fout.write(json_format.MessageToJson(proxy_manifest).encode('utf-8'))
-return beam_artifact_api_pb2.CommitManifestResponse(
-retrieval_token=retrieval_token)
-
- def GetManifest(self,
- request, # type: beam_artifact_api_pb2.GetManifestRequest
- context=None):
-# type: (...) -> beam_artifact_api_pb2.GetManifestResponse
-return beam_artifact_api_pb2.GetManifestResponse(
-manifest=self._get_manifest_proxy(request.retrieval_token).manifest)
-
- def GetArtifact(self,
- request, # type:
beam_artifact_api_pb2.LegacyGetArtifactRequest
- context=None):
-# type: (...) -> Iterator[beam_artifact_api_pb2.ArtifactChunk]
-for artifact in self._get_manifest_proxy(request.retrieval_token).location:
- if artifact.name == request.name:
-with self._open(artifact.uri, 'r') as fin:
- # This value is not emitted, but lets us yield a single empty
- # chunk on an empty file.
- chunk = b'1'
- while chunk:
-chunk = fin.read(self._chunk_size)
-yield
[GitHub] [beam] amaliujia merged pull request #11971: [BEAM-10230] @Ignore: BYTES works with LIKE.
amaliujia merged pull request #11971: URL: https://github.com/apache/beam/pull/11971 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] amaliujia commented on pull request #11971: [BEAM-10230] @Ignore: BYTES works with LIKE.
amaliujia commented on pull request #11971: URL: https://github.com/apache/beam/pull/11971#issuecomment-642227049 Thank you Robin! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] lukecwik commented on a change in pull request #11972: [BEAM-9852] Do not create data channel for empty timer descriptor.
lukecwik commented on a change in pull request #11972: URL: https://github.com/apache/beam/pull/11972#discussion_r438371296 ## File path: sdks/python/apache_beam/runners/worker/bundle_processor.py ## @@ -819,7 +819,8 @@ def __init__(self, # There is no guarantee that the runner only set # timer_api_service_descriptor when having timers. So this field cannot be # used as an indicator of timers. -if self.process_bundle_descriptor.timer_api_service_descriptor: +if (self.process_bundle_descriptor.timer_api_service_descriptor and Review comment: I think the same problem exists here: https://github.com/apache/beam/blob/89fc35b87d5dc074d25d60a97bb96d71e04be283/sdks/python/apache_beam/runners/worker/sdk_worker.py#L691 Is there a way to see if the api service descriptor isn't the default instance instead of checking for the URL parameter? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] lukecwik commented on pull request #11972: [BEAM-9852] Do not create data channel for empty timer descriptor.
lukecwik commented on pull request #11972: URL: https://github.com/apache/beam/pull/11972#issuecomment-642224159 > Thanks, Kyle! > > I'm curious with which runner we will run into this issue. Based on the proto, the `url` should be required for `ApiServiceDescriptor`: > > https://github.com/apache/beam/blob/c496ae56d1b066c895fdf273d3975bd8baf1a56f/model/pipeline/src/main/proto/endpoints.proto#L32-L40 The problem is that protos default isn't `null` but the default empty instance so the check should really be whether it is not the default instance. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] pabloem commented on a change in pull request #11950: [BEAM-8596]: Add SplunkIO transform to write messages to Splunk
pabloem commented on a change in pull request #11950:
URL: https://github.com/apache/beam/pull/11950#discussion_r438284843
##
File path:
sdks/java/io/splunk/src/main/java/org/apache/beam/sdk/io/splunk/SplunkEventWriter.java
##
@@ -0,0 +1,395 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.splunk;
+
+import static
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
+
+import com.google.api.client.http.HttpResponse;
+import com.google.api.client.http.HttpResponseException;
+import com.google.auto.value.AutoValue;
+import com.google.gson.Gson;
+import com.google.gson.GsonBuilder;
+import java.io.IOException;
+import java.io.UnsupportedEncodingException;
+import java.security.KeyManagementException;
+import java.security.KeyStoreException;
+import java.security.NoSuchAlgorithmException;
+import java.util.List;
+import javax.annotation.Nullable;
+import org.apache.beam.sdk.metrics.Counter;
+import org.apache.beam.sdk.metrics.Metrics;
+import org.apache.beam.sdk.options.ValueProvider;
+import org.apache.beam.sdk.state.BagState;
+import org.apache.beam.sdk.state.StateSpec;
+import org.apache.beam.sdk.state.StateSpecs;
+import org.apache.beam.sdk.state.TimeDomain;
+import org.apache.beam.sdk.state.Timer;
+import org.apache.beam.sdk.state.TimerSpec;
+import org.apache.beam.sdk.state.TimerSpecs;
+import org.apache.beam.sdk.state.ValueState;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.values.KV;
+import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.joda.time.Duration;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** A {@link DoFn} to write {@link SplunkEvent}s to Splunk's HEC endpoint. */
+@AutoValue
+abstract class SplunkEventWriter extends DoFn,
SplunkWriteError> {
+
+ private static final Integer DEFAULT_BATCH_COUNT = 1;
+ private static final Boolean DEFAULT_DISABLE_CERTIFICATE_VALIDATION = false;
+ private static final Logger LOG =
LoggerFactory.getLogger(SplunkEventWriter.class);
+ private static final long DEFAULT_FLUSH_DELAY = 2;
+ private static final Counter INPUT_COUNTER =
+ Metrics.counter(SplunkEventWriter.class, "inbound-events");
+ private static final Counter SUCCESS_WRITES =
+ Metrics.counter(SplunkEventWriter.class, "outbound-successful-events");
+ private static final Counter FAILED_WRITES =
+ Metrics.counter(SplunkEventWriter.class, "outbound-failed-events");
+ private static final String BUFFER_STATE_NAME = "buffer";
+ private static final String COUNT_STATE_NAME = "count";
+ private static final String TIME_ID_NAME = "expiry";
+
+ @StateId(BUFFER_STATE_NAME)
+ private final StateSpec> buffer = StateSpecs.bag();
+
+ @StateId(COUNT_STATE_NAME)
+ private final StateSpec> count = StateSpecs.value();
+
+ @TimerId(TIME_ID_NAME)
+ private final TimerSpec expirySpec = TimerSpecs.timer(TimeDomain.EVENT_TIME);
Review comment:
I wonder if this should be processing time, to avoid getting stuck when
the watermark slows down?
##
File path:
sdks/java/io/splunk/src/main/java/org/apache/beam/sdk/io/splunk/SplunkEventWriter.java
##
@@ -0,0 +1,395 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License
[GitHub] [beam] InigoSJ commented on a change in pull request #11943: [BEAM-10209] Add without_defaults to Mean combiner in Python
InigoSJ commented on a change in pull request #11943: URL: https://github.com/apache/beam/pull/11943#discussion_r438368619 ## File path: sdks/python/apache_beam/transforms/combiners.py ## @@ -66,8 +66,21 @@ class Mean(object): """Combiners for computing arithmetic means of elements.""" class Globally(ptransform.PTransform): """combiners.Mean.Globally computes the arithmetic mean of the elements.""" +def __init__(self, has_defaults=True, *args, **kwargs): + super(Mean.Globally, self).__init__() + self.has_defaults = has_defaults + self.args = args + self.kwargs = kwargs + def expand(self, pcoll): - return pcoll | core.CombineGlobally(MeanCombineFn()) + if self.has_defaults: +return pcoll | core.CombineGlobally(MeanCombineFn()) + else: +return pcoll | core.CombineGlobally(MeanCombineFn()).without_defaults() + +def without_defaults(self): + self.has_defaults = False Review comment: I'm not sure if I understand it correctly, but I made a modification to return ``` return self | core.CombineGlobally(MeanCombineFn()).without_defaults() ``` I personally think it's less intuitive, but you know more :D. I also tried directly returning without modifying `self.has_defaults`, but it executes the first sequence of the Reduce fine, but the second returns an error saying that I need the `without_defaults()`. If you have suggestions, I'm all ears. Thanks again This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] InigoSJ commented on a change in pull request #11943: [BEAM-10209] Add without_defaults to Mean combiner in Python
InigoSJ commented on a change in pull request #11943: URL: https://github.com/apache/beam/pull/11943#discussion_r438366946 ## File path: sdks/python/apache_beam/transforms/combiners_test.py ## @@ -105,6 +109,16 @@ def test_builtin_combines(self): assert_that(result_mean, equal_to([mean]), label='assert:mean') assert_that(result_count, equal_to([size]), label='assert:size') + # Now for global combines without default + timestamped = pcoll | Map(lambda x: TimestampedValue(timestamp)) Review comment: Thanks for spotting this, I think I missed it when moving from my test env to the PR env! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] InigoSJ commented on a change in pull request #11943: [BEAM-10209] Add without_defaults to Mean combiner in Python
InigoSJ commented on a change in pull request #11943: URL: https://github.com/apache/beam/pull/11943#discussion_r438366434 ## File path: sdks/python/apache_beam/transforms/combiners_test.py ## @@ -97,6 +100,7 @@ def test_builtin_combines(self): vals = [6, 3, 1, 1, 9, 1, 5, 2, 0, 6] mean = sum(vals) / float(len(vals)) size = len(vals) + timestamp = 1591485720 Review comment: Modified! Thanks This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] pabloem edited a comment on pull request #11969: Remove a hack used to retrieve schematized data from HL7v2 messages, …
pabloem edited a comment on pull request #11969: URL: https://github.com/apache/beam/pull/11969#issuecomment-642210806 Note the spotless complaints: https://builds.apache.org/job/beam_PreCommit_Spotless_Commit/9383/console you can run `./gradlew :my-project...:spotlessApply` to fix those (where `my-project` is your actual gradle project) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #11969: Remove a hack used to retrieve schematized data from HL7v2 messages, …
pabloem commented on pull request #11969: URL: https://github.com/apache/beam/pull/11969#issuecomment-642210806 Note the spotless complaints: https://builds.apache.org/job/beam_PreCommit_Spotless_Commit/9383/console you can run `./gradlew :my-project...:spotlessApply` to fix those This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #11969: Remove a hack used to retrieve schematized data from HL7v2 messages, …
pabloem commented on pull request #11969: URL: https://github.com/apache/beam/pull/11969#issuecomment-642210975 Note some build errors: https://builds.apache.org/job/beam_PreCommit_JavaPortabilityApi_Commit/8189/console This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #11969: Remove a hack used to retrieve schematized data from HL7v2 messages, …
pabloem commented on pull request #11969: URL: https://github.com/apache/beam/pull/11969#issuecomment-642208223 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] robertwb merged pull request #11949: Simplify Python on Flink runner instructions.
robertwb merged pull request #11949: URL: https://github.com/apache/beam/pull/11949 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] udim commented on pull request #11966: [BEAM-10217] CALL_FUNCTION and CALL_METHOD fixes
udim commented on pull request #11966: URL: https://github.com/apache/beam/pull/11966#issuecomment-642205856 > Jenkins hooks are mad at me. Dependencies are still broken so it won't help This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] TheNeuralBit opened a new pull request #11973: Finalize CHANGES.md for 2.22.0
TheNeuralBit opened a new pull request #11973: URL: https://github.com/apache/beam/pull/11973 R: @aaltay Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)[![Build
[GitHub] [beam] boyuanzz commented on pull request #11972: [BEAM-9852] Do not create data channel for empty timer descriptor.
boyuanzz commented on pull request #11972: URL: https://github.com/apache/beam/pull/11972#issuecomment-642199648 Thanks, Kyle! I'm curious with which runner we will run into this issue. Based on the proto, the `url` should be required for `ApiServiceDescriptor`: https://github.com/apache/beam/blob/c496ae56d1b066c895fdf273d3975bd8baf1a56f/model/pipeline/src/main/proto/endpoints.proto#L32-L40 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] aromanenko-dev merged pull request #11396: [BEAM-9742] Add Configurable FluentBackoff to JdbcIO Write
aromanenko-dev merged pull request #11396: URL: https://github.com/apache/beam/pull/11396 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] robertwb commented on pull request #11949: Simplify Python on Flink runner instructions.
robertwb commented on pull request #11949: URL: https://github.com/apache/beam/pull/11949#issuecomment-642188187 Done. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] ibzib opened a new pull request #11972: [BEAM-9852] Do not create data channel for empty timer descriptor.
ibzib opened a new pull request #11972: URL: https://github.com/apache/beam/pull/11972 This is possibly a naive solution, please let me know if you think there is a greater underlying issue. R: @boyuanzz cc: @robertwb Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [![Build
[GitHub] [beam] kamilwu removed a comment on pull request #11956: [BEAM-8133] Publishing results of Nexmark tests to InfluxDB
kamilwu removed a comment on pull request #11956: URL: https://github.com/apache/beam/pull/11956#issuecomment-642144427 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
