[GitHub] [beam] piotr-szuberski commented on pull request #12758: [BEAM-9898] Move snowflake.py from apache_beam.io to apache_beam.io.external for backwards compatibility
piotr-szuberski commented on pull request #12758: URL: https://github.com/apache/beam/pull/12758#issuecomment-692507849 @chamikaramj ping This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12758: [BEAM-9898] Move snowflake.py from apache_beam.io to apache_beam.io.external for backwards compatibility
codecov[bot] edited a comment on pull request #12758: URL: https://github.com/apache/beam/pull/12758#issuecomment-685598683 # [Codecov](https://codecov.io/gh/apache/beam/pull/12758?src=pr=h1) Report > Merging [#12758](https://codecov.io/gh/apache/beam/pull/12758?src=pr=desc) into [master](https://codecov.io/gh/apache/beam/commit/a2126e571cfddc90e3819e66ef1f2da3d52833ed?el=desc) will **decrease** coverage by `41.97%`. > The diff coverage is `0.00%`. [](https://codecov.io/gh/apache/beam/pull/12758?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #12758 +/- ## === - Coverage 82.35% 40.38% -41.98% === Files 450 452+2 Lines 5370853725 +17 === - Hits4423121696-22535 - Misses 947732029+22552 ``` | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12758?src=pr=tree) | Coverage Δ | | |---|---|---| | [sdks/python/apache\_beam/io/external/snowflake.py](https://codecov.io/gh/apache/beam/pull/12758/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZXh0ZXJuYWwvc25vd2ZsYWtlLnB5) | `0.00% <0.00%> (ø)` | | | [...python/apache\_beam/examples/complete/distribopt.py](https://codecov.io/gh/apache/beam/pull/12758/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29tcGxldGUvZGlzdHJpYm9wdC5weQ==) | `0.00% <0.00%> (-98.59%)` | :arrow_down: | | [...dks/python/apache\_beam/transforms/create\_source.py](https://codecov.io/gh/apache/beam/pull/12758/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9jcmVhdGVfc291cmNlLnB5) | `0.00% <0.00%> (-98.19%)` | :arrow_down: | | [...on/apache\_beam/runners/direct/helper\_transforms.py](https://codecov.io/gh/apache/beam/pull/12758/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvaGVscGVyX3RyYW5zZm9ybXMucHk=) | `0.00% <0.00%> (-98.15%)` | :arrow_down: | | [...e\_beam/runners/interactive/testing/mock\_ipython.py](https://codecov.io/gh/apache/beam/pull/12758/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS90ZXN0aW5nL21vY2tfaXB5dGhvbi5weQ==) | `7.14% <0.00%> (-92.86%)` | :arrow_down: | | [.../examples/snippets/transforms/elementwise/pardo.py](https://codecov.io/gh/apache/beam/pull/12758/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvc25pcHBldHMvdHJhbnNmb3Jtcy9lbGVtZW50d2lzZS9wYXJkby5weQ==) | `11.36% <0.00%> (-88.64%)` | :arrow_down: | | [sdks/python/apache\_beam/typehints/opcodes.py](https://codecov.io/gh/apache/beam/pull/12758/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL29wY29kZXMucHk=) | `0.00% <0.00%> (-87.66%)` | :arrow_down: | | [...s/snippets/transforms/aggregation/combineperkey.py](https://codecov.io/gh/apache/beam/pull/12758/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvc25pcHBldHMvdHJhbnNmb3Jtcy9hZ2dyZWdhdGlvbi9jb21iaW5lcGVya2V5LnB5) | `11.95% <0.00%> (-86.96%)` | :arrow_down: | | [...xamples/snippets/transforms/elementwise/flatmap.py](https://codecov.io/gh/apache/beam/pull/12758/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvc25pcHBldHMvdHJhbnNmb3Jtcy9lbGVtZW50d2lzZS9mbGF0bWFwLnB5) | `14.28% <0.00%> (-85.72%)` | :arrow_down: | | [...mples/snippets/transforms/elementwise/partition.py](https://codecov.io/gh/apache/beam/pull/12758/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvc25pcHBldHMvdHJhbnNmb3Jtcy9lbGVtZW50d2lzZS9wYXJ0aXRpb24ucHk=) | `11.90% <0.00%> (-85.72%)` | :arrow_down: | | ... and [272 more](https://codecov.io/gh/apache/beam/pull/12758/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12758?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12758?src=pr=footer). Last update [a2126e5...ded8d41](https://codecov.io/gh/apache/beam/pull/12758?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] piotr-szuberski commented on pull request #12809: [BEAM-10870] Add raw private key param to snowflake cross-language python wrapper
piotr-szuberski commented on pull request #12809: URL: https://github.com/apache/beam/pull/12809#issuecomment-692508024 @chamikaramj ping This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] piotr-szuberski commented on pull request #12847: [BEAM-7523] Fix starting Kafka container twice in KafkaCSVTableIT
piotr-szuberski commented on pull request #12847: URL: https://github.com/apache/beam/pull/12847#issuecomment-692544119 @TheNeuralBit Could you review this? I think it's worth to have it done asap. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] piotr-szuberski opened a new pull request #12847: [BEAM-7523] Fix starting Kafka container twice in KafkaCSVTableIT
piotr-szuberski opened a new pull request #12847: URL: https://github.com/apache/beam/pull/12847 I made a mistake and Kafka Container starts twice in KafkaCSVTableIT (@Rule KafkaContainer makes it start and stop automatically). It may result in leaving redundant containers on the VMs. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2 --- | --- | --- | --- | --- | --- | --- Go | [](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | --- Java | [](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/) Python | [](https://ci-beam.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[![Build
[GitHub] [beam] runningcode commented on pull request #12846: [BEAM-10886] Fix Java Wordcount Direct Runner (windows-latest)
runningcode commented on pull request #12846: URL: https://github.com/apache/beam/pull/12846#issuecomment-692523923 R: @TheNeuralBit R: @lukecwik R: @kennknowles R: @mxm This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] piotr-szuberski commented on a change in pull request #12611: [BEAM-10139][BEAM-10140] Add cross-language support for Java SpannerIO with python wrapper
piotr-szuberski commented on a change in pull request #12611:
URL: https://github.com/apache/beam/pull/12611#discussion_r488422134
##
File path:
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/StructUtils.java
##
@@ -0,0 +1,545 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.spanner;
+
+import static java.util.stream.Collectors.toList;
+import static
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+
+import com.google.cloud.ByteArray;
+import com.google.cloud.Timestamp;
+import com.google.cloud.spanner.Struct;
+import com.google.cloud.spanner.Type;
+import java.math.BigDecimal;
+import java.util.List;
+import java.util.stream.StreamSupport;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.values.Row;
+import org.joda.time.DateTime;
+import org.joda.time.Instant;
+
+final class StructUtils {
+ public static Row translateStructToRow(Struct struct, Schema schema) {
+checkForSchemasEquality(schema.getFields(),
struct.getType().getStructFields(), false);
+
+List fields = schema.getFields();
+Row.FieldValueBuilder valueBuilder = null;
+// TODO: Remove this null-checking once nullable fields are supported in
cross-language
Review comment:
I'm not sure where my message has gone, but I wrote that nulls come up
with no problems, I've just used ImmutableMap which does not allow null values.
Replacing it with java.util.HashMap solved the issue.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [beam] runningcode opened a new pull request #12846: [BEAM-10886] Fix Java Wordcount Direct Runner (windows-latest)
runningcode opened a new pull request #12846: URL: https://github.com/apache/beam/pull/12846 This fixes the java wordcount direct runner test on windows by using the Gradle Github action. Gradle Github action will use the `.bat` script on windows instead of the gradlew bash script. The issue was that the previous test was running the test on windows using the gradlew bash script instead of the gradlew.bat which is what should be used on Windows. Github actions has msys installed which would mean that the gradlew bash script would not parse the arguments as expected. The quotes would be removed from the pipelineOptionsString which would mean it was invalid JSON. See https://issues.apache.org/jira/browse/BEAM-10886 for more details. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2 --- | --- | --- | --- | --- | --- | --- Go | [](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | --- Java | [](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) |
[GitHub] [beam] purbanow commented on pull request #12823: [BEAM-10882] Update Snowflake docs
purbanow commented on pull request #12823: URL: https://github.com/apache/beam/pull/12823#issuecomment-692653308 @TheNeuralBit will you find a moment for a CR? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #12704: [BEAM-10603] Implement the new Large Source Recording API.
pabloem commented on pull request #12704: URL: https://github.com/apache/beam/pull/12704#issuecomment-692746763 LGTM. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] mxm commented on a change in pull request #12576: [BEAM-10671] Add environment configuration fields as first-class pipeline options.
mxm commented on a change in pull request #12576:
URL: https://github.com/apache/beam/pull/12576#discussion_r488571222
##
File path: sdks/python/apache_beam/options/pipeline_options.py
##
@@ -1044,7 +1044,12 @@ def _add_argparse_args(cls, parser):
'form {"os": "", "arch": "", "command": '
'"", "env":{"": '
'""} }. All fields in the json are optional except '
-'command.'))
+'command.\n\nPlease consider using the following '
+'options instead, depending on the environment type:\n'
+' --docker_env_container_image\n'
+' --process_env_command\n'
+' --process_env_variables\n'
+' --external_env_service_address'))
Review comment:
Could we change the option name pattern to `env__`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [beam] kamilwu commented on pull request #12848: [BEAM-10616] Add missing ParDo test cases for streaming/Flink
kamilwu commented on pull request #12848: URL: https://github.com/apache/beam/pull/12848#issuecomment-692698764 Run Seed Job This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] entimaniac commented on pull request #12564: [BEAM-10612] add Flink 1.11 runner
entimaniac commented on pull request #12564: URL: https://github.com/apache/beam/pull/12564#issuecomment-692709810 thanks! that was all the information I was looking to hear This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] tanya-borisova commented on pull request #12734: Add authentication story via pipeline options and tests using Azurite.
tanya-borisova commented on pull request #12734: URL: https://github.com/apache/beam/pull/12734#issuecomment-692744673 FYI we at Wayve are going to use Beam for our data pipelines, and we store our data in Azure. Naturally, we're very interested in these features getting merged and getting to the new Beam release :) Please do let me know if I can help in any way. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] pabloem merged pull request #12808: add readme file to python nexmark
pabloem merged pull request #12808: URL: https://github.com/apache/beam/pull/12808 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] kamilwu commented on pull request #12848: [BEAM-10616] Add missing ParDo test cases for streaming/Flink
kamilwu commented on pull request #12848: URL: https://github.com/apache/beam/pull/12848#issuecomment-692645050 Run Load Tests Python ParDo Combine Streaming This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] kamilwu commented on pull request #12848: [BEAM-10616] Add missing ParDo test cases for streaming/Flink
kamilwu commented on pull request #12848: URL: https://github.com/apache/beam/pull/12848#issuecomment-692645216 Run Load Tests Python Combine Flink Streaming This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] kamilwu removed a comment on pull request #12848: [BEAM-10616] Add missing ParDo test cases for streaming/Flink
kamilwu removed a comment on pull request #12848: URL: https://github.com/apache/beam/pull/12848#issuecomment-692645050 Run Load Tests Python ParDo Combine Streaming This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] kamilwu commented on pull request #12848: [BEAM-10616] Add missing ParDo test cases for streaming/Flink
kamilwu commented on pull request #12848: URL: https://github.com/apache/beam/pull/12848#issuecomment-692644949 Run Load Tests Python ParDo Flink Streaming This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] jayendra13 commented on a change in pull request #12540: [BEAM-10620] Eliminate nullability errors from :sdks:java:extensions:sql:perf-tests
jayendra13 commented on a change in pull request #12540:
URL: https://github.com/apache/beam/pull/12540#discussion_r488690404
##
File path:
sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/IOTestPipelineOptions.java
##
@@ -45,19 +45,16 @@
void setBigQueryTable(@Nullable String tableName);
@Description("InfluxDB measurement to publish results to.")
- @Nullable
String getInfluxMeasurement();
void setInfluxMeasurement(@Nullable String measurement);
@Description("InfluxDB host.")
- @Nullable
String getInfluxHost();
void setInfluxHost(@Nullable String host);
@Description("InfluxDB database.")
- @Nullable
Review comment:
So these three getters (for measurements, host and database) are fine,
and we should also change these three setters to take `non-null` arguments.
Correct me if I am wrong.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [beam] pabloem merged pull request #12813: write to file ability for java suite
pabloem merged pull request #12813: URL: https://github.com/apache/beam/pull/12813 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] kamilwu commented on pull request #12848: [BEAM-10616] Add missing ParDo test cases for streaming/Flink
kamilwu commented on pull request #12848: URL: https://github.com/apache/beam/pull/12848#issuecomment-692641763 Run Seed Job This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] methodmissing commented on pull request #12794: [BEAM-10865] Support for Kafka deserialization API with headers (since Kafka API 2.1.0)
methodmissing commented on pull request #12794: URL: https://github.com/apache/beam/pull/12794#issuecomment-692706541 Hi Luke, Apologies for the lag on this. TLDR I have the integration test against `KafkaIOTest` going but seem to not be able to force the `kafka-clients` module override. Executed with `./gradlew clean sdks:java:io:kafka:buildDependent` Main test task `:sdks:java:io:kafka:test` ``` <===--> 86% EXECUTING [2m 7s] > IDLE > :sdks:java:io:expansion-service:shadowDistZip > :sdks:java:extensions:sql:expansion-service:shadowJar > :runners:spark:spotbugsMain > :sdks:java:io:kafka:test > Executing test org.apache.beam.sdk.io.kafka.KafkaIOTest << > :sdks:java:testing:nexmark:spotbugsMain > :runners:spark:job-server:shadowJar > :sdks:java:io:kafka:test > 8 tests completed ``` Kafka client API 2.1.0 specific task `:sdks:java:io:kafka:kafkaVersion210Test` ``` <===--> 88% EXECUTING [4m 2s] > :runners:spark:test > Executing test org.apache.beam...metrics.sink.SparkMetricsSinkTest > :sdks:java:extensions:sql:expansion-service:shadowJar > :sdks:java:testing:nexmark:test > 9 tests completed, 1 skipped > :sdks:java:testing:nexmark:test > Executing test org.apache.beam.sdk.nexmark.queries.QueryTest > :sdks:java:io:kafka:kafkaVersion210Test > Executing test org.apache.beam.sdk.io.kafka.KafkaIOTest << > :sdks:java:io:kafka:kafkaVersion210Test > 1 test completed > :runners:spark:job-server:shadowJar > :runners:spark:test > 10 tests completed > :runners:spark:test > Executing test org.apache.beam.runners.spark.metrics.SparkMetricsPusherTest > :runners:spark:test > Executing test org.apache...streaming.TrackStreamingSourcesTest ``` ``` methodmissing:beam lourens$ ls -la /Users/lourens/src/github.com/Shopify/cdc/beam/sdks/java/io/kafka/build/reports/tests total 0 drwxr-xr-x 4 lourens staff 128 Sep 15 11:00 . drwxr-xr-x 5 lourens staff 160 Sep 15 11:06 .. drwxr-xr-x 7 lourens staff 224 Sep 15 11:06 kafkaVersion210Test << drwxr-xr-x 7 lourens staff 224 Sep 15 11:04 test ``` Any failure in task `:sdks:java:io:kafka:test` means the 2.1.0 specific test doesn't run though. Given this is an integration test, units that fail should not run the dependent integration test either, that is default Gradle behavior and should be fine for this case as well. I opted to run the whole `KafkaIOTest` test with the new task as it exercises a larger net of the deserialization flows and doesn't add significantly to runtime. **!!! FAIL:** However I'm failing at being able to inject the 2.1.0 kafka-client module dependency for the integration test and I'm wondering if you have any obvious thoughts on what I've done wrong in specifying the test module dependency? --- I have some ideas (based on the python specific tasks and multi-version support) on how to support test tasks for all Kafka client API versions, but beyond the scope of this pull request. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] samatix commented on pull request #12811: [BEAM-10705] Fixes a bug when passing whl files in --sdk_location from https locations
samatix commented on pull request #12811: URL: https://github.com/apache/beam/pull/12811#issuecomment-692656216 @tvalentyn, I've corrected the problem with python 2.7 and corrected two formatting issues. Now I'm able to run tox locally on my machine. For the next time, I will avoid such issues after the PR. However, I'm unable to reproduce the last two issues (they seem to be random). Any ideas on what went wrong? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] pabloem merged pull request #12704: [BEAM-10603] Implement the new Large Source Recording API.
pabloem merged pull request #12704: URL: https://github.com/apache/beam/pull/12704 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] kamilwu opened a new pull request #12848: [BEAM-10616] Add missing ParDo test cases for streaming/Flink
kamilwu opened a new pull request #12848: URL: https://github.com/apache/beam/pull/12848 Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2 --- | --- | --- | --- | --- | --- | --- Go | [](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | --- Java | [](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/) Python | [](https://ci-beam.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[![Build
[GitHub] [beam] entimaniac commented on pull request #12564: [BEAM-10612] add Flink 1.11 runner
entimaniac commented on pull request #12564: URL: https://github.com/apache/beam/pull/12564#issuecomment-692685878 Hey, I know this is a closed PR but I'm hoping someone can help me. I'm trying to understand how I might be able to take advantage of this PR. The environment I'm trying to run my beam job on has flink 1.11 installed but the only runner package I see available is for 1.10 (https://mvnrepository.com/artifact/org.apache.beam/beam-runners-flink-1.10) Is there another way of utilizing this code? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] mxm commented on pull request #12564: [BEAM-10612] add Flink 1.11 runner
mxm commented on pull request #12564: URL: https://github.com/apache/beam/pull/12564#issuecomment-692699588 Support for Flink 1.11 will be included in Beam 2.25.0. Until then, you'll have to manually build the jar from source using the following command: ./gradlew :runners:flink:1.11:jar This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] tanya-borisova commented on pull request #12492: [BEAM-6807] Implement an Azure blobstore filesystem for Python SDK
tanya-borisova commented on pull request #12492: URL: https://github.com/apache/beam/pull/12492#issuecomment-692624622 This is some great functionality in this PR, is it expected to be in a release soon? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] kamilwu commented on pull request #12848: [BEAM-10616] Add missing ParDo test cases for streaming/Flink
kamilwu commented on pull request #12848: URL: https://github.com/apache/beam/pull/12848#issuecomment-692703201 Run Load Tests Python ParDo Flink Streaming This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #12721: [BEAM-10871] Add deidentify for FhirIO connector
pabloem commented on pull request #12721: URL: https://github.com/apache/beam/pull/12721#issuecomment-692734363 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] tanya-borisova commented on pull request #12734: Add authentication story via pipeline options and tests using Azurite.
tanya-borisova commented on pull request #12734: URL: https://github.com/apache/beam/pull/12734#issuecomment-692740987 @AldairCoronel With this PR, will it possible to connect to several different storage accounts using different connection strings? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] PoojaChandak opened a new pull request #12849: [Minor] Typo/grammatical changes
PoojaChandak opened a new pull request #12849: URL: https://github.com/apache/beam/pull/12849 **Please** add a meaningful description for your change here Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2 --- | --- | --- | --- | --- | --- | --- Go | [](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | --- Java | [](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/) Python | [](https://ci-beam.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[![Build
[GitHub] [beam] codecov[bot] edited a comment on pull request #12812: [BEAM-10873] Stronger testing of dataframe partitioning declarations.
codecov[bot] edited a comment on pull request #12812: URL: https://github.com/apache/beam/pull/12812#issuecomment-690611762 # [Codecov](https://codecov.io/gh/apache/beam/pull/12812?src=pr=h1) Report > Merging [#12812](https://codecov.io/gh/apache/beam/pull/12812?src=pr=desc) into [master](https://codecov.io/gh/apache/beam/commit/803efdd931699c24e60350e5ccf6e54482f5916f?el=desc) will **decrease** coverage by `0.01%`. > The diff coverage is `87.06%`. [](https://codecov.io/gh/apache/beam/pull/12812?src=pr=tree) ```diff @@Coverage Diff @@ ## master #12812 +/- ## == - Coverage 82.38% 82.37% -0.02% == Files 451 451 Lines 5377554459 +684 == + Hits4430344858 +555 - Misses 9472 9601 +129 ``` | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12812?src=pr=tree) | Coverage Δ | | |---|---|---| | [...n/apache\_beam/runners/dataflow/dataflow\_metrics.py](https://codecov.io/gh/apache/beam/pull/12812/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9kYXRhZmxvd19tZXRyaWNzLnB5) | `74.32% <33.33%> (-0.85%)` | :arrow_down: | | [sdks/python/apache\_beam/dataframe/frame\_base.py](https://codecov.io/gh/apache/beam/pull/12812/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lX2Jhc2UucHk=) | `84.97% <70.00%> (+0.26%)` | :arrow_up: | | [sdks/python/apache\_beam/typehints/opcodes.py](https://codecov.io/gh/apache/beam/pull/12812/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL29wY29kZXMucHk=) | `87.65% <83.33%> (-0.35%)` | :arrow_down: | | [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/12812/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `89.90% <85.18%> (-0.52%)` | :arrow_down: | | [sdks/python/apache\_beam/dataframe/doctests.py](https://codecov.io/gh/apache/beam/pull/12812/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2RvY3Rlc3RzLnB5) | `96.83% <85.71%> (-0.93%)` | :arrow_down: | | [sdks/python/apache\_beam/transforms/stats.py](https://codecov.io/gh/apache/beam/pull/12812/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9zdGF0cy5weQ==) | `90.41% <86.36%> (+3.02%)` | :arrow_up: | | [sdks/python/apache\_beam/typehints/schemas.py](https://codecov.io/gh/apache/beam/pull/12812/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL3NjaGVtYXMucHk=) | `93.20% <86.66%> (-3.41%)` | :arrow_down: | | [sdks/python/apache\_beam/coders/row\_coder.py](https://codecov.io/gh/apache/beam/pull/12812/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL3Jvd19jb2Rlci5weQ==) | `94.36% <90.90%> (-0.64%)` | :arrow_down: | | [sdks/python/apache\_beam/dataframe/expressions.py](https://codecov.io/gh/apache/beam/pull/12812/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2V4cHJlc3Npb25zLnB5) | `89.70% <96.29%> (+1.63%)` | :arrow_up: | | [sdks/python/apache\_beam/dataframe/partitionings.py](https://codecov.io/gh/apache/beam/pull/12812/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL3BhcnRpdGlvbmluZ3MucHk=) | `89.04% <100.00%> (+5.43%)` | :arrow_up: | | ... and [30 more](https://codecov.io/gh/apache/beam/pull/12812/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12812?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12812?src=pr=footer). Last update [8e7014d...7433760](https://codecov.io/gh/apache/beam/pull/12812?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12841: [BEAM-10894] Basic CSV reading and writing.
codecov[bot] edited a comment on pull request #12841: URL: https://github.com/apache/beam/pull/12841#issuecomment-692395796 # [Codecov](https://codecov.io/gh/apache/beam/pull/12841?src=pr=h1) Report > Merging [#12841](https://codecov.io/gh/apache/beam/pull/12841?src=pr=desc) into [master](https://codecov.io/gh/apache/beam/commit/803efdd931699c24e60350e5ccf6e54482f5916f?el=desc) will **decrease** coverage by `0.09%`. > The diff coverage is `87.09%`. [](https://codecov.io/gh/apache/beam/pull/12841?src=pr=tree) ```diff @@Coverage Diff @@ ## master #12841 +/- ## == - Coverage 82.38% 82.28% -0.10% == Files 451 452 +1 Lines 5377553838 +63 == - Hits4430344302 -1 - Misses 9472 9536 +64 ``` | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12841?src=pr=tree) | Coverage Δ | | |---|---|---| | [...n/apache\_beam/runners/dataflow/dataflow\_metrics.py](https://codecov.io/gh/apache/beam/pull/12841/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9kYXRhZmxvd19tZXRyaWNzLnB5) | `74.32% <33.33%> (-0.85%)` | :arrow_down: | | [sdks/python/apache\_beam/dataframe/frame\_base.py](https://codecov.io/gh/apache/beam/pull/12841/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lX2Jhc2UucHk=) | `84.70% <57.14%> (ø)` | | | [sdks/python/apache\_beam/dataframe/doctests.py](https://codecov.io/gh/apache/beam/pull/12841/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2RvY3Rlc3RzLnB5) | `96.83% <83.33%> (-0.93%)` | :arrow_down: | | [sdks/python/apache\_beam/typehints/opcodes.py](https://codecov.io/gh/apache/beam/pull/12841/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL29wY29kZXMucHk=) | `87.65% <83.33%> (-0.35%)` | :arrow_down: | | [sdks/python/apache\_beam/transforms/stats.py](https://codecov.io/gh/apache/beam/pull/12841/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9zdGF0cy5weQ==) | `90.41% <86.36%> (+3.02%)` | :arrow_up: | | [sdks/python/apache\_beam/typehints/schemas.py](https://codecov.io/gh/apache/beam/pull/12841/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL3NjaGVtYXMucHk=) | `93.20% <86.66%> (-3.41%)` | :arrow_down: | | [sdks/python/apache\_beam/dataframe/io.py](https://codecov.io/gh/apache/beam/pull/12841/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2lvLnB5) | `89.90% <89.90%> (ø)` | | | [sdks/python/apache\_beam/coders/row\_coder.py](https://codecov.io/gh/apache/beam/pull/12841/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL3Jvd19jb2Rlci5weQ==) | `94.36% <90.90%> (-0.64%)` | :arrow_down: | | [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/12841/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `90.60% <100.00%> (+0.18%)` | :arrow_up: | | [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/12841/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `76.02% <0.00%> (-10.53%)` | :arrow_down: | | ... and [29 more](https://codecov.io/gh/apache/beam/pull/12841/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12841?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12841?src=pr=footer). Last update [8e7014d...35d86ce](https://codecov.io/gh/apache/beam/pull/12841?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #12847: [BEAM-7523] Fix starting Kafka container twice in KafkaCSVTableIT
TheNeuralBit commented on pull request #12847: URL: https://github.com/apache/beam/pull/12847#issuecomment-692819294 Looks like moving the imports around broke spotless, I'll push a commit to fix so we can get this merged today. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12841: [BEAM-10894] Basic CSV reading and writing.
codecov[bot] edited a comment on pull request #12841: URL: https://github.com/apache/beam/pull/12841#issuecomment-692395796 # [Codecov](https://codecov.io/gh/apache/beam/pull/12841?src=pr=h1) Report > Merging [#12841](https://codecov.io/gh/apache/beam/pull/12841?src=pr=desc) into [master](https://codecov.io/gh/apache/beam/commit/803efdd931699c24e60350e5ccf6e54482f5916f?el=desc) will **decrease** coverage by `0.10%`. > The diff coverage is `87.09%`. [](https://codecov.io/gh/apache/beam/pull/12841?src=pr=tree) ```diff @@Coverage Diff @@ ## master #12841 +/- ## == - Coverage 82.38% 82.27% -0.11% == Files 451 452 +1 Lines 5377553838 +63 == - Hits4430344296 -7 - Misses 9472 9542 +70 ``` | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12841?src=pr=tree) | Coverage Δ | | |---|---|---| | [...n/apache\_beam/runners/dataflow/dataflow\_metrics.py](https://codecov.io/gh/apache/beam/pull/12841/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9kYXRhZmxvd19tZXRyaWNzLnB5) | `74.32% <33.33%> (-0.85%)` | :arrow_down: | | [sdks/python/apache\_beam/dataframe/frame\_base.py](https://codecov.io/gh/apache/beam/pull/12841/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lX2Jhc2UucHk=) | `84.70% <57.14%> (ø)` | | | [sdks/python/apache\_beam/dataframe/doctests.py](https://codecov.io/gh/apache/beam/pull/12841/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2RvY3Rlc3RzLnB5) | `96.83% <83.33%> (-0.93%)` | :arrow_down: | | [sdks/python/apache\_beam/typehints/opcodes.py](https://codecov.io/gh/apache/beam/pull/12841/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL29wY29kZXMucHk=) | `87.65% <83.33%> (-0.35%)` | :arrow_down: | | [sdks/python/apache\_beam/transforms/stats.py](https://codecov.io/gh/apache/beam/pull/12841/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9zdGF0cy5weQ==) | `90.41% <86.36%> (+3.02%)` | :arrow_up: | | [sdks/python/apache\_beam/typehints/schemas.py](https://codecov.io/gh/apache/beam/pull/12841/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL3NjaGVtYXMucHk=) | `93.20% <86.66%> (-3.41%)` | :arrow_down: | | [sdks/python/apache\_beam/dataframe/io.py](https://codecov.io/gh/apache/beam/pull/12841/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2lvLnB5) | `89.90% <89.90%> (ø)` | | | [sdks/python/apache\_beam/coders/row\_coder.py](https://codecov.io/gh/apache/beam/pull/12841/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL3Jvd19jb2Rlci5weQ==) | `94.36% <90.90%> (-0.64%)` | :arrow_down: | | [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/12841/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `90.60% <100.00%> (+0.18%)` | :arrow_up: | | [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/12841/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `76.02% <0.00%> (-10.53%)` | :arrow_down: | | ... and [29 more](https://codecov.io/gh/apache/beam/pull/12841/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12841?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12841?src=pr=footer). Last update [8e7014d...ab66e1c](https://codecov.io/gh/apache/beam/pull/12841?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12841: [BEAM-10894] Basic CSV reading and writing.
codecov[bot] edited a comment on pull request #12841: URL: https://github.com/apache/beam/pull/12841#issuecomment-692395796 # [Codecov](https://codecov.io/gh/apache/beam/pull/12841?src=pr=h1) Report > Merging [#12841](https://codecov.io/gh/apache/beam/pull/12841?src=pr=desc) into [master](https://codecov.io/gh/apache/beam/commit/803efdd931699c24e60350e5ccf6e54482f5916f?el=desc) will **decrease** coverage by `0.10%`. > The diff coverage is `87.09%`. [](https://codecov.io/gh/apache/beam/pull/12841?src=pr=tree) ```diff @@Coverage Diff @@ ## master #12841 +/- ## == - Coverage 82.38% 82.27% -0.11% == Files 451 452 +1 Lines 5377553838 +63 == - Hits4430344296 -7 - Misses 9472 9542 +70 ``` | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12841?src=pr=tree) | Coverage Δ | | |---|---|---| | [...n/apache\_beam/runners/dataflow/dataflow\_metrics.py](https://codecov.io/gh/apache/beam/pull/12841/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9kYXRhZmxvd19tZXRyaWNzLnB5) | `74.32% <33.33%> (-0.85%)` | :arrow_down: | | [sdks/python/apache\_beam/dataframe/frame\_base.py](https://codecov.io/gh/apache/beam/pull/12841/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lX2Jhc2UucHk=) | `84.70% <57.14%> (ø)` | | | [sdks/python/apache\_beam/dataframe/doctests.py](https://codecov.io/gh/apache/beam/pull/12841/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2RvY3Rlc3RzLnB5) | `96.83% <83.33%> (-0.93%)` | :arrow_down: | | [sdks/python/apache\_beam/typehints/opcodes.py](https://codecov.io/gh/apache/beam/pull/12841/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL29wY29kZXMucHk=) | `87.65% <83.33%> (-0.35%)` | :arrow_down: | | [sdks/python/apache\_beam/transforms/stats.py](https://codecov.io/gh/apache/beam/pull/12841/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9zdGF0cy5weQ==) | `90.41% <86.36%> (+3.02%)` | :arrow_up: | | [sdks/python/apache\_beam/typehints/schemas.py](https://codecov.io/gh/apache/beam/pull/12841/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL3NjaGVtYXMucHk=) | `93.20% <86.66%> (-3.41%)` | :arrow_down: | | [sdks/python/apache\_beam/dataframe/io.py](https://codecov.io/gh/apache/beam/pull/12841/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2lvLnB5) | `89.90% <89.90%> (ø)` | | | [sdks/python/apache\_beam/coders/row\_coder.py](https://codecov.io/gh/apache/beam/pull/12841/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL3Jvd19jb2Rlci5weQ==) | `94.36% <90.90%> (-0.64%)` | :arrow_down: | | [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/12841/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `90.60% <100.00%> (+0.18%)` | :arrow_up: | | [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/12841/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `76.02% <0.00%> (-10.53%)` | :arrow_down: | | ... and [29 more](https://codecov.io/gh/apache/beam/pull/12841/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12841?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12841?src=pr=footer). Last update [8e7014d...ab66e1c](https://codecov.io/gh/apache/beam/pull/12841?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] tvalentyn commented on pull request #12811: [BEAM-10705] Fixes a bug when passing whl files in --sdk_location from https locations
tvalentyn commented on pull request #12811: URL: https://github.com/apache/beam/pull/12811#issuecomment-692845652 Actually found the previous green PostCommit result: https://ci-beam.apache.org/job/beam_PostCommit_Python36_PR/93/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] kw2542 commented on a change in pull request #12617: [BEAM-10670] Update Samza to be opt-out for SplittableDoFn powering the Read transform.
kw2542 commented on a change in pull request #12617:
URL: https://github.com/apache/beam/pull/12617#discussion_r488820872
##
File path:
runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaRunner.java
##
@@ -106,6 +107,7 @@ public PortablePipelineResult
runPortablePipeline(RunnerApi.Pipeline pipeline) {
@Override
public SamzaPipelineResult run(Pipeline pipeline) {
+SplittableParDo.validateNoPrimitiveReads(pipeline);
Review comment:
is it expected that we only validate in non-portable mode? i.e. this
validation does not exist in runPortablePipeline()
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [beam] pabloem merged pull request #12770: [BEAM-10545] Assembled the extension with components
pabloem merged pull request #12770: URL: https://github.com/apache/beam/pull/12770 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] kamilwu commented on pull request #12848: [BEAM-10616] Add missing ParDo test cases for streaming/Flink
kamilwu commented on pull request #12848: URL: https://github.com/apache/beam/pull/12848#issuecomment-692780948 R: @mxm This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] robertwb commented on pull request #12812: [BEAM-10873] Stronger testing of dataframe partitioning declarations.
robertwb commented on pull request #12812: URL: https://github.com/apache/beam/pull/12812#issuecomment-692799520 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12841: [BEAM-10894] Basic CSV reading and writing.
codecov[bot] edited a comment on pull request #12841: URL: https://github.com/apache/beam/pull/12841#issuecomment-692395796 # [Codecov](https://codecov.io/gh/apache/beam/pull/12841?src=pr=h1) Report > Merging [#12841](https://codecov.io/gh/apache/beam/pull/12841?src=pr=desc) into [master](https://codecov.io/gh/apache/beam/commit/803efdd931699c24e60350e5ccf6e54482f5916f?el=desc) will **decrease** coverage by `0.00%`. > The diff coverage is `87.09%`. [](https://codecov.io/gh/apache/beam/pull/12841?src=pr=tree) ```diff @@Coverage Diff @@ ## master #12841 +/- ## == - Coverage 82.38% 82.37% -0.01% == Files 451 451 Lines 5377553816 +41 == + Hits4430344333 +30 - Misses 9472 9483 +11 ``` | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12841?src=pr=tree) | Coverage Δ | | |---|---|---| | [...n/apache\_beam/runners/dataflow/dataflow\_metrics.py](https://codecov.io/gh/apache/beam/pull/12841/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9kYXRhZmxvd19tZXRyaWNzLnB5) | `74.32% <33.33%> (-0.85%)` | :arrow_down: | | [sdks/python/apache\_beam/dataframe/frame\_base.py](https://codecov.io/gh/apache/beam/pull/12841/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lX2Jhc2UucHk=) | `84.70% <57.14%> (ø)` | | | [sdks/python/apache\_beam/dataframe/doctests.py](https://codecov.io/gh/apache/beam/pull/12841/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2RvY3Rlc3RzLnB5) | `96.83% <83.33%> (-0.93%)` | :arrow_down: | | [sdks/python/apache\_beam/typehints/opcodes.py](https://codecov.io/gh/apache/beam/pull/12841/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL29wY29kZXMucHk=) | `87.65% <83.33%> (-0.35%)` | :arrow_down: | | [sdks/python/apache\_beam/transforms/stats.py](https://codecov.io/gh/apache/beam/pull/12841/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9zdGF0cy5weQ==) | `90.41% <86.36%> (+3.02%)` | :arrow_up: | | [sdks/python/apache\_beam/typehints/schemas.py](https://codecov.io/gh/apache/beam/pull/12841/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL3NjaGVtYXMucHk=) | `93.20% <86.66%> (-3.41%)` | :arrow_down: | | [sdks/python/apache\_beam/dataframe/io.py](https://codecov.io/gh/apache/beam/pull/12841/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2lvLnB5) | `89.90% <89.90%> (ø)` | | | [sdks/python/apache\_beam/coders/row\_coder.py](https://codecov.io/gh/apache/beam/pull/12841/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL3Jvd19jb2Rlci5weQ==) | `94.36% <90.90%> (-0.64%)` | :arrow_down: | | [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/12841/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `90.60% <100.00%> (+0.18%)` | :arrow_up: | | [...s/snippets/transforms/aggregation/combinevalues.py](https://codecov.io/gh/apache/beam/pull/12841/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvc25pcHBldHMvdHJhbnNmb3Jtcy9hZ2dyZWdhdGlvbi9jb21iaW5ldmFsdWVzLnB5) | `87.36% <0.00%> (-7.37%)` | :arrow_down: | | ... and [16 more](https://codecov.io/gh/apache/beam/pull/12841/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12841?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12841?src=pr=footer). Last update [8e7014d...35d86ce](https://codecov.io/gh/apache/beam/pull/12841?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] tvalentyn commented on pull request #12811: [BEAM-10705] Fixes a bug when passing whl files in --sdk_location from https locations
tvalentyn commented on pull request #12811: URL: https://github.com/apache/beam/pull/12811#issuecomment-692844451 > able to reproduce the last two issues (they seem to be random). Any ideas on what went wrong? Those seem to be flaky tests :(. Found this in the logs: ``` self = def test_time_based_flush_grpc_data_channel(self): > self._grpc_data_channel_test(True) apache_beam/runners/worker/data_plane_test.py:44: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ apache_beam/runners/worker/data_plane_test.py:74: in _grpc_data_channel_test data_channel_service, data_channel_client, time_based_flush) apache_beam/runners/worker/data_plane_test.py:86: in _data_channel_test self._data_channel_test_one_direction(server, client, time_based_flush) apache_beam/runners/worker/data_plane_test.py:129: in _data_channel_test_one_direction instruction_id='2', transform_id=transform_2, data=b'ghi') E AssertionError: Lists differ: [inst[26 chars]id: "2" E data: "ghi" E , instruction_id: "2" E tran[22 chars]ef" E ] != [inst[26 chars]id: "1" E data: "def" E , instruction_id: "2" E tran[22 chars]hi" E ] E E First differing element 0: E instruction_id: "2" E transform_id: "2" E data: "ghi" E E instruction_id: "2" E transform_id: "1" E data: "def" E E E [instruction_id: "2" E + transform_id: "1" E + data: "def" E + , E + instruction_id: "2" E transform_id: "2" E data: "ghi" E - , E - instruction_id: "2" E - transform_id: "1" E - data: "def" E ] ``` There is an open issue about it: https://issues.apache.org/jira/browse/BEAM-10768. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] chamikaramj commented on a change in pull request #12505: [BEAM-8106] Add version to java container image name
chamikaramj commented on a change in pull request #12505: URL: https://github.com/apache/beam/pull/12505#discussion_r488818825 ## File path: website/www/site/content/en/documentation/runtime/environments.md ## @@ -116,8 +116,8 @@ By default, no licenses/notices are added to the docker images. To examine the containers that you built, run `docker images` from anywhere in the command line. If you successfully built all of the container images, the command prints a table like the following: ``` -REPOSITORY TAG IMAGE ID CREATED SIZE -apache/beam_java_sdk latest 16ca619d489e2 weeks ago550MB +REPOSITORY TAG IMAGE ID CREATED SIZE +apache/beam_java_8sdk latest 16ca619d489e2 weeks ago550MB Review comment: Let's make sure that the correct container name is set in the environment send to the remote SDKs that use Java transforms as cross-language transforms. (We don't have integration tests setup for this by we can just try existing Kafka/SQL x-lang examples to confirm) https://github.com/apache/beam/tree/master/sdks/python/apache_beam/examples/kafkataxi https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/sql_taxi.py This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] lukecwik commented on a change in pull request #12617: [BEAM-10670] Update Samza to be opt-out for SplittableDoFn powering the Read transform.
lukecwik commented on a change in pull request #12617:
URL: https://github.com/apache/beam/pull/12617#discussion_r488825043
##
File path:
runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaRunner.java
##
@@ -106,6 +107,7 @@ public PortablePipelineResult
runPortablePipeline(RunnerApi.Pipeline pipeline) {
@Override
public SamzaPipelineResult run(Pipeline pipeline) {
+SplittableParDo.validateNoPrimitiveReads(pipeline);
Review comment:
Yes since portable pipelines only support SDF. This is about migrating
non-portable pipeline runner implementations to use SDF.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [beam] tvalentyn commented on a change in pull request #12811: [BEAM-10705] Fixes a bug when passing whl files in --sdk_location from https locations
tvalentyn commented on a change in pull request #12811: URL: https://github.com/apache/beam/pull/12811#discussion_r488824939 ## File path: sdks/python/apache_beam/runners/portability/stager_test.py ## @@ -448,7 +448,8 @@ def test_sdk_location_remote_source_file(self, *unused_mocks): def test_sdk_location_remote_wheel_file(self, *unused_mocks): staging_dir = self.make_temp_dir() sdk_filename = 'apache_beam-1.0.0-cp27-cp27mu-manylinux1_x86_64.whl' -sdk_location = 'https://storage.googleapis.com/my-gcs-bucket/' + sdk_filename +sdk_location = 'https://storage.googleapis.com/my-gcs-bucket/' + \ Review comment: Nit: implicit line joining (with parenthesis instead of backslash) is preferable by PIP8. Since Beam does not officially use PIP8, I will merge as is, just FYI. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] pabloem commented on a change in pull request #12823: [BEAM-10882] Update Snowflake docs
pabloem commented on a change in pull request #12823:
URL: https://github.com/apache/beam/pull/12823#discussion_r488745935
##
File path: website/www/site/content/en/documentation/io/built-in/snowflake.md
##
@@ -362,3 +635,206 @@ static SnowflakeIO.CsvMapper
getCsvMapper() {
};
}
{{< /highlight >}}
+## Cross Language
Review comment:
I wonder if this should be called 'Using SnowflakeIO in Python SDK' -
what do you think? Users may not care if it's cross-lang, but rather that it
works in their language of choice. Thoughts?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [beam] robertwb commented on pull request #12834: Document GroupBy transform.
robertwb commented on pull request #12834: URL: https://github.com/apache/beam/pull/12834#issuecomment-692796395 Run Website PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12834: Document GroupBy transform.
codecov[bot] edited a comment on pull request #12834: URL: https://github.com/apache/beam/pull/12834#issuecomment-691369145 # [Codecov](https://codecov.io/gh/apache/beam/pull/12834?src=pr=h1) Report > Merging [#12834](https://codecov.io/gh/apache/beam/pull/12834?src=pr=desc) into [master](https://codecov.io/gh/apache/beam/commit/803efdd931699c24e60350e5ccf6e54482f5916f?el=desc) will **decrease** coverage by `0.01%`. > The diff coverage is `84.84%`. [](https://codecov.io/gh/apache/beam/pull/12834?src=pr=tree) ```diff @@Coverage Diff @@ ## master #12834 +/- ## == - Coverage 82.38% 82.36% -0.02% == Files 451 450 -1 Lines 5377553702 -73 == - Hits4430344233 -70 + Misses 9472 9469 -3 ``` | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12834?src=pr=tree) | Coverage Δ | | |---|---|---| | [...n/apache\_beam/runners/dataflow/dataflow\_metrics.py](https://codecov.io/gh/apache/beam/pull/12834/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9kYXRhZmxvd19tZXRyaWNzLnB5) | `74.32% <33.33%> (-0.85%)` | :arrow_down: | | [sdks/python/apache\_beam/dataframe/frame\_base.py](https://codecov.io/gh/apache/beam/pull/12834/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lX2Jhc2UucHk=) | `84.70% <57.14%> (ø)` | | | [sdks/python/apache\_beam/dataframe/doctests.py](https://codecov.io/gh/apache/beam/pull/12834/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2RvY3Rlc3RzLnB5) | `96.83% <83.33%> (-0.93%)` | :arrow_down: | | [sdks/python/apache\_beam/typehints/opcodes.py](https://codecov.io/gh/apache/beam/pull/12834/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL29wY29kZXMucHk=) | `87.65% <83.33%> (-0.35%)` | :arrow_down: | | [sdks/python/apache\_beam/transforms/stats.py](https://codecov.io/gh/apache/beam/pull/12834/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9zdGF0cy5weQ==) | `90.41% <86.36%> (+3.02%)` | :arrow_up: | | [sdks/python/apache\_beam/typehints/schemas.py](https://codecov.io/gh/apache/beam/pull/12834/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL3NjaGVtYXMucHk=) | `93.20% <86.66%> (-3.41%)` | :arrow_down: | | [sdks/python/apache\_beam/coders/row\_coder.py](https://codecov.io/gh/apache/beam/pull/12834/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL3Jvd19jb2Rlci5weQ==) | `94.36% <90.90%> (-0.64%)` | :arrow_down: | | [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/12834/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `90.48% <100.00%> (+0.07%)` | :arrow_up: | | [...s/snippets/transforms/aggregation/combinevalues.py](https://codecov.io/gh/apache/beam/pull/12834/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvc25pcHBldHMvdHJhbnNmb3Jtcy9hZ2dyZWdhdGlvbi9jb21iaW5ldmFsdWVzLnB5) | `87.36% <0.00%> (-7.37%)` | :arrow_down: | | [.../python/apache\_beam/transforms/periodicsequence.py](https://codecov.io/gh/apache/beam/pull/12834/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9wZXJpb2RpY3NlcXVlbmNlLnB5) | `96.49% <0.00%> (-1.76%)` | :arrow_down: | | ... and [18 more](https://codecov.io/gh/apache/beam/pull/12834/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12834?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12834?src=pr=footer). Last update [8e7014d...4e0e911](https://codecov.io/gh/apache/beam/pull/12834?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] terryxian78 commented on pull request #12818: [BEAM-10875] Support NUMERIC in spanner schema parser
terryxian78 commented on pull request #12818: URL: https://github.com/apache/beam/pull/12818#issuecomment-692816048 ping, @chamikaramj, could you review this PR? Thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #12846: [BEAM-10886] Fix Java Wordcount Direct Runner (windows-latest)
TheNeuralBit commented on pull request #12846: URL: https://github.com/apache/beam/pull/12846#issuecomment-692816874 Thanks @runningcode! Looks like this adds a dependency on https://github.com/eskatos/gradle-command-action but it is MIT licensed so I don't think there's any concern. It's too bad the workflow doesn't trigger for this change.. and I don't think there's a way to trigger it manually. It looks like the change was tested in your fork though (https://github.com/runningcode/beam/actions/runs/25520654), so I'll go ahead and merge. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] ajamato commented on a change in pull request #12822: [BEAM-10880] Log error counts to debug BigQuery streaming insert requ…
ajamato commented on a change in pull request #12822: URL: https://github.com/apache/beam/pull/12822#discussion_r488797757 ## File path: sdks/python/apache_beam/io/gcp/bigquery.py ## @@ -1210,20 +1236,32 @@ def process(self, element, *schema_side_inputs): return self._flush_all_batches() def finish_bundle(self): -if BigQueryWriteFn.LATENCY_LOGGING_LOCK.acquire(False): +if BigQueryWriteFn.STREAMING_API_LOGGING_LOCK.acquire(False): Review comment: Can all of this be encapsulated entirely in a new class? or at least a helper that can be called in finish_bundle Ideally we can instrument this without adding so much to the original implementation. ## File path: sdks/python/apache_beam/io/gcp/bigquery.py ## @@ -1210,20 +1236,32 @@ def process(self, element, *schema_side_inputs): return self._flush_all_batches() def finish_bundle(self): -if BigQueryWriteFn.LATENCY_LOGGING_LOCK.acquire(False): +if BigQueryWriteFn.STREAMING_API_LOGGING_LOCK.acquire(False): try: current_millis = int(time.time() * 1000) -if (BigQueryWriteFn.LATENCY_LOGGING_HISTOGRAM.total_count() > 0 and -(current_millis - - BigQueryWriteFn.LATENCY_LOGGING_LAST_REPORTED_MILLIS) > -self._latency_logging_frequency_msec): - _LOGGER.info( - BigQueryWriteFn.LATENCY_LOGGING_HISTOGRAM.get_percentile_info( - 'streaming insert requests', 'ms')) - BigQueryWriteFn.LATENCY_LOGGING_HISTOGRAM.clear() - BigQueryWriteFn.LATENCY_LOGGING_LAST_REPORTED_MILLIS = current_millis +if ((current_millis - + BigQueryWriteFn.STREAMING_API_LOGGING_LAST_REPORTED_MILLIS) > +self._streaming_api_logging_frequency_msec): + streaming_api_info = [ + '[Streaming Insert API Statistics since %s]' % + datetime.datetime.fromtimestamp( + BigQueryWriteFn.STREAMING_API_LOGGING_LAST_REPORTED_MILLIS / + 1000.0) + ] + if BigQueryWriteFn.LATENCY_LOGGING_HISTOGRAM.total_count() > 0: +streaming_api_info.append( Review comment: Please address code coverage warnings Codecov / codecov/patch sdks/python/apache_beam/io/gcp/bigquery.py#L1252 Added line #L1252 was not covered by tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] timclemons opened a new pull request #12850: [BEAM-10481] Ensure registration of the accumulator occurs.
timclemons opened a new pull request #12850: URL: https://github.com/apache/beam/pull/12850 Move registration of accumulator so as to include those recovered from checkpoints. Prior to this, if the accumulator was recovered from a checkpoint it would not be registered, resulting in a runtime exception. R: @jbonofre Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [X] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [X] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2 --- | --- | --- | --- | --- | --- | --- Go | [](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | --- Java | [](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/) Python | [](https://ci-beam.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[![Build
[GitHub] [beam] tvalentyn commented on pull request #12637: [BEAM-10768] Don't assert the order in which elements are received.
tvalentyn commented on pull request #12637: URL: https://github.com/apache/beam/pull/12637#issuecomment-692843176 seeing this flake in precommits, thanks for fixing it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] amaliujia commented on pull request #12843: [BEAM-10895] Support UNNEST an (possibly nested) array field of an struct column
amaliujia commented on pull request #12843: URL: https://github.com/apache/beam/pull/12843#issuecomment-692848083 Thanks @robinyqiu will take a look soon. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] monicadsong commented on pull request #12851: [BEAM-10900] add ability for ApproximateUniqueCombineFn to handle numpy input data
monicadsong commented on pull request #12851: URL: https://github.com/apache/beam/pull/12851#issuecomment-692856430 R: @tvalentyn This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] codecov[bot] commented on pull request #12779: [BEAM-10856] Support for NestedValueProvider for Python SDK
codecov[bot] commented on pull request #12779: URL: https://github.com/apache/beam/pull/12779#issuecomment-692856347 # [Codecov](https://codecov.io/gh/apache/beam/pull/12779?src=pr=h1) Report > Merging [#12779](https://codecov.io/gh/apache/beam/pull/12779?src=pr=desc) into [master](https://codecov.io/gh/apache/beam/commit/a2126e571cfddc90e3819e66ef1f2da3d52833ed?el=desc) will **decrease** coverage by `0.08%`. > The diff coverage is `44.11%`. [](https://codecov.io/gh/apache/beam/pull/12779?src=pr=tree) ```diff @@Coverage Diff @@ ## master #12779 +/- ## == - Coverage 82.36% 82.28% -0.09% == Files 450 451 +1 Lines 5370853738 +30 == - Hits4423844217 -21 - Misses 9470 9521 +51 ``` | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12779?src=pr=tree) | Coverage Δ | | |---|---|---| | [...beam/testing/benchmarks/nexmark/queries/query10.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvcXVlcmllcy9xdWVyeTEwLnB5) | `0.00% <0.00%> (ø)` | | | [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `76.02% <64.40%> (-10.53%)` | :arrow_down: | | [...runners/interactive/display/pcoll\_visualization.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kaXNwbGF5L3Bjb2xsX3Zpc3VhbGl6YXRpb24ucHk=) | `85.26% <87.50%> (-0.88%)` | :arrow_down: | | [sdks/python/apache\_beam/options/value\_provider.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vb3B0aW9ucy92YWx1ZV9wcm92aWRlci5weQ==) | `91.76% <93.33%> (+0.21%)` | :arrow_up: | | [...ive/messaging/interactive\_environment\_inspector.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9tZXNzYWdpbmcvaW50ZXJhY3RpdmVfZW52aXJvbm1lbnRfaW5zcGVjdG9yLnB5) | `97.43% <100.00%> (ø)` | | | [...eam/runners/interactive/options/capture\_control.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9vcHRpb25zL2NhcHR1cmVfY29udHJvbC5weQ==) | `92.00% <0.00%> (-8.00%)` | :arrow_down: | | [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `88.98% <0.00%> (-0.36%)` | :arrow_down: | | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `94.45% <0.00%> (-0.14%)` | :arrow_down: | | [sdks/python/apache\_beam/pvalue.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcHZhbHVlLnB5) | `92.01% <0.00%> (+0.38%)` | :arrow_up: | | ... and [5 more](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12779?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12779?src=pr=footer). Last update [5206131...af2c14c](https://codecov.io/gh/apache/beam/pull/12779?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] monicadsong opened a new pull request #12851: [BEAM-10900] add ability for ApproximateUniqueCombineFn to handle numpy input data
monicadsong opened a new pull request #12851: URL: https://github.com/apache/beam/pull/12851 If element for ApproximateUniqueCombineFn.add_input() is numpy type, convert to python type. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2 --- | --- | --- | --- | --- | --- | --- Go | [](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | --- Java | [](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/) Python | [](https://ci-beam.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[![Build
[GitHub] [beam] robertwb commented on pull request #12812: [BEAM-10873] Stronger testing of dataframe partitioning declarations.
robertwb commented on pull request #12812: URL: https://github.com/apache/beam/pull/12812#issuecomment-692861888 The github actions failures look completely unrelated and the tests all pass on jenkins (and locally). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] robertwb merged pull request #12812: [BEAM-10873] Stronger testing of dataframe partitioning declarations.
robertwb merged pull request #12812: URL: https://github.com/apache/beam/pull/12812 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on a change in pull request #12823: [BEAM-10882] Update Snowflake docs
TheNeuralBit commented on a change in pull request #12823:
URL: https://github.com/apache/beam/pull/12823#discussion_r488776570
##
File path: website/www/site/content/en/documentation/io/built-in/snowflake.md
##
@@ -362,3 +635,206 @@ static SnowflakeIO.CsvMapper
getCsvMapper() {
};
}
{{< /highlight >}}
+## Cross Language
Review comment:
+1
It's worth having a note somewhere like "SnowflakeIO in Python can only be
used on runners that support cross-language transforms". I'm wary of listing
specific runners here since it will get out of date. It seems we should have a
central location that lists runners that support cross-language, seems like a
good thing to have on the compatibility matrix.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [beam] robertwb merged pull request #12834: Document GroupBy transform.
robertwb merged pull request #12834: URL: https://github.com/apache/beam/pull/12834 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] TheNeuralBit merged pull request #12846: [BEAM-10886] Fix Java Wordcount Direct Runner (windows-latest)
TheNeuralBit merged pull request #12846: URL: https://github.com/apache/beam/pull/12846 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] ajamato commented on a change in pull request #12822: [BEAM-10880] Log error counts to debug BigQuery streaming insert requ…
ajamato commented on a change in pull request #12822: URL: https://github.com/apache/beam/pull/12822#discussion_r488813424 ## File path: sdks/python/apache_beam/io/gcp/bigquery_tools.py ## @@ -538,6 +539,13 @@ def _insert_all_rows( try: response = self.client.tabledata.InsertAll(request) # response.insertErrors is not [] if errors encountered. +except HttpError as exn: + if error_counter: +content = json.loads(exn.content) +error_counter.record( +'%s(%s)' % +(content['error']['errors'][0]['reason'], exn.status_code)) Review comment: Please convert this to the human readable format instead of the numeric http code. See the doc I shared with you. At least before logging https://cloud.google.com/apis/design/errors This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] ibzib commented on a change in pull request #12576: [BEAM-10671] Add environment configuration fields as first-class pipeline options.
ibzib commented on a change in pull request #12576:
URL: https://github.com/apache/beam/pull/12576#discussion_r488839465
##
File path: sdks/python/apache_beam/options/pipeline_options.py
##
@@ -1044,7 +1044,12 @@ def _add_argparse_args(cls, parser):
'form {"os": "", "arch": "", "command": '
'"", "env":{"": '
'""} }. All fields in the json are optional except '
-'command.'))
+'command.\n\nPlease consider using the following '
+'options instead, depending on the environment type:\n'
+' --docker_env_container_image\n'
+' --process_env_command\n'
+' --process_env_variables\n'
+' --external_env_service_address'))
Review comment:
Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [beam] boyuanzz commented on pull request #12806: [BEAM-10869] Use PubsubMessagePayloadOnlyCoder when writing to Pubsub.
boyuanzz commented on pull request #12806: URL: https://github.com/apache/beam/pull/12806#issuecomment-692865584 r: @chamikaramj This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #12721: [BEAM-10871] Add deidentify for FhirIO connector
pabloem commented on pull request #12721: URL: https://github.com/apache/beam/pull/12721#issuecomment-692790229 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #12505: [BEAM-8106] Add version to java container image name
TheNeuralBit commented on pull request #12505: URL: https://github.com/apache/beam/pull/12505#issuecomment-692824435 Note the release manager that will likely be impacted is @robinyqiu for 2.25.0, since the 2.24.0 branch has already been cut. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #12505: [BEAM-8106] Add version to java container image name
TheNeuralBit commented on pull request #12505: URL: https://github.com/apache/beam/pull/12505#issuecomment-692827332 Run Dataflow PortabilityApi ValidatesRunner with Java 11 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12505: [BEAM-8106] Add version to java container image name
codecov[bot] edited a comment on pull request #12505: URL: https://github.com/apache/beam/pull/12505#issuecomment-684502100 # [Codecov](https://codecov.io/gh/apache/beam/pull/12505?src=pr=h1) Report > Merging [#12505](https://codecov.io/gh/apache/beam/pull/12505?src=pr=desc) into [master](https://codecov.io/gh/apache/beam/commit/803efdd931699c24e60350e5ccf6e54482f5916f?el=desc) will **decrease** coverage by `0.02%`. > The diff coverage is `86.93%`. [](https://codecov.io/gh/apache/beam/pull/12505?src=pr=tree) ```diff @@Coverage Diff @@ ## master #12505 +/- ## == - Coverage 82.38% 82.35% -0.03% == Files 451 450 -1 Lines 5377553702 -73 == - Hits4430344228 -75 - Misses 9472 9474 +2 ``` | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12505?src=pr=tree) | Coverage Δ | | |---|---|---| | [...n/apache\_beam/runners/dataflow/dataflow\_metrics.py](https://codecov.io/gh/apache/beam/pull/12505/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9kYXRhZmxvd19tZXRyaWNzLnB5) | `74.32% <33.33%> (-0.85%)` | :arrow_down: | | [sdks/python/apache\_beam/transforms/combiners.py](https://codecov.io/gh/apache/beam/pull/12505/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9jb21iaW5lcnMucHk=) | `92.14% <50.00%> (-0.02%)` | :arrow_down: | | [sdks/python/apache\_beam/dataframe/frame\_base.py](https://codecov.io/gh/apache/beam/pull/12505/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lX2Jhc2UucHk=) | `84.70% <57.14%> (ø)` | | | [sdks/python/apache\_beam/dataframe/doctests.py](https://codecov.io/gh/apache/beam/pull/12505/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2RvY3Rlc3RzLnB5) | `96.83% <83.33%> (-0.93%)` | :arrow_down: | | [sdks/python/apache\_beam/typehints/opcodes.py](https://codecov.io/gh/apache/beam/pull/12505/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL29wY29kZXMucHk=) | `87.65% <83.33%> (-0.35%)` | :arrow_down: | | [sdks/python/apache\_beam/transforms/stats.py](https://codecov.io/gh/apache/beam/pull/12505/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9zdGF0cy5weQ==) | `90.41% <86.36%> (+3.02%)` | :arrow_up: | | [sdks/python/apache\_beam/typehints/schemas.py](https://codecov.io/gh/apache/beam/pull/12505/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL3NjaGVtYXMucHk=) | `93.20% <86.66%> (-3.41%)` | :arrow_down: | | [sdks/python/apache\_beam/coders/row\_coder.py](https://codecov.io/gh/apache/beam/pull/12505/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL3Jvd19jb2Rlci5weQ==) | `94.36% <90.90%> (-0.64%)` | :arrow_down: | | [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/12505/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `90.48% <100.00%> (+0.07%)` | :arrow_up: | | [sdks/python/apache\_beam/io/gcp/pubsub.py](https://codecov.io/gh/apache/beam/pull/12505/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3B1YnN1Yi5weQ==) | `93.58% <100.00%> (+1.27%)` | :arrow_up: | | ... and [15 more](https://codecov.io/gh/apache/beam/pull/12505/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12505?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12505?src=pr=footer). Last update [63f54fd...ba2a1fa](https://codecov.io/gh/apache/beam/pull/12505?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] tvalentyn commented on pull request #12779: [BEAM-10856] Support for NestedValueProvider for Python SDK
tvalentyn commented on pull request #12779: URL: https://github.com/apache/beam/pull/12779#issuecomment-692833522 > A user has written a feature that they would find useful, and that will not change the experience for other users (if anything, it should improve it). The feature looks correct, and similar to what we do in Java. If we reject the PR, we may push the user to run on a fork. Can we let this in? @tvalentyn I agree with this assessment, feel free to merge once tests pass. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on a change in pull request #12505: [BEAM-8106] Add version to java container image name
TheNeuralBit commented on a change in pull request #12505:
URL: https://github.com/apache/beam/pull/12505#discussion_r488809349
##
File path: sdks/java/container/Dockerfile-java11
##
@@ -16,6 +16,10 @@
# limitations under the License.
###
+##
+# NOTE: This image is now deprecated.
+# Use Dockerfile with the appropriate java_version build argument.
+##
Review comment:
I'm +1 for just removing this. Looks like it was added in
https://github.com/apache/beam/pull/8053 in order to start running a job for
testing Java 11. As long as the jobs that use Java 11 still pass after removing
this I think it's fine.
I don't think users should be referencing this Dockerfile directly, if
they're building a Java 11 container at all it should be through the gradle
command (and even then we don't support Java 11 so they shouldn't be doing it).
##
File path: website/www/site/content/en/documentation/runtime/environments.md
##
@@ -116,8 +116,8 @@ By default, no licenses/notices are added to the docker
images.
To examine the containers that you built, run `docker images` from anywhere in
the command line. If you successfully built all of the container images, the
command prints a table like the following:
```
-REPOSITORY TAG IMAGE ID
CREATED SIZE
-apache/beam_java_sdk latest 16ca619d489e2
weeks ago550MB
+REPOSITORY TAG IMAGE ID
CREATED SIZE
+apache/beam_java_8sdk latest 16ca619d489e2
weeks ago550MB
Review comment:
typo here:
```suggestion
apache/beam_java8_sdk latest 16ca619d489e2
weeks ago550MB
```
I'm assuming this change and the others like it are the result of a search
to find and update all the references to `beam_java_sdk`, so we don't need to
worry about there being other references?
##
File path:
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/Environments.java
##
@@ -357,4 +354,14 @@ private static File zipDirectory(File directory) throws
IOException {
return env;
}
}
+
+ private static String getDefaultJavaSdkHarnessContainerUrl() {
+String javaVersionId =
+Float.parseFloat(System.getProperty("java.specification.version")) >=
9 ? "java11" : "java8";
Review comment:
nit: could you make this an exact check for 8 and 11 and throw an
exception for other (unsupported) versions
##
File path: sdks/java/container/Dockerfile-java11
##
@@ -16,6 +16,10 @@
# limitations under the License.
###
+##
+# NOTE: This image is now deprecated.
+# Use Dockerfile with the appropriate java_version build argument.
+##
Review comment:
Also the fact that this Dockerfile doesn't copy LICENSE and NOTICE is
problematic.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [beam] tvalentyn edited a comment on pull request #12779: [BEAM-10856] Support for NestedValueProvider for Python SDK
tvalentyn edited a comment on pull request #12779: URL: https://github.com/apache/beam/pull/12779#issuecomment-692833522 > A user has written a feature that they would find useful, and that will not change the experience for other users (if anything, it should improve it). The feature looks correct, and similar to what we do in Java. If we reject the PR, we may push the user to run on a fork. Can we let this in? @tvalentyn I agree with this assessment, feel free to merge once tests & linter pass. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12779: [BEAM-10856] Support for NestedValueProvider for Python SDK
codecov[bot] edited a comment on pull request #12779: URL: https://github.com/apache/beam/pull/12779#issuecomment-692856347 # [Codecov](https://codecov.io/gh/apache/beam/pull/12779?src=pr=h1) Report > Merging [#12779](https://codecov.io/gh/apache/beam/pull/12779?src=pr=desc) into [master](https://codecov.io/gh/apache/beam/commit/a2126e571cfddc90e3819e66ef1f2da3d52833ed?el=desc) will **decrease** coverage by `0.08%`. > The diff coverage is `44.11%`. [](https://codecov.io/gh/apache/beam/pull/12779?src=pr=tree) ```diff @@Coverage Diff @@ ## master #12779 +/- ## == - Coverage 82.36% 82.28% -0.09% == Files 450 451 +1 Lines 5370853738 +30 == - Hits4423844217 -21 - Misses 9470 9521 +51 ``` | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12779?src=pr=tree) | Coverage Δ | | |---|---|---| | [...beam/testing/benchmarks/nexmark/queries/query10.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvcXVlcmllcy9xdWVyeTEwLnB5) | `0.00% <0.00%> (ø)` | | | [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `76.02% <64.40%> (-10.53%)` | :arrow_down: | | [...runners/interactive/display/pcoll\_visualization.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kaXNwbGF5L3Bjb2xsX3Zpc3VhbGl6YXRpb24ucHk=) | `85.26% <87.50%> (-0.88%)` | :arrow_down: | | [sdks/python/apache\_beam/options/value\_provider.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vb3B0aW9ucy92YWx1ZV9wcm92aWRlci5weQ==) | `91.76% <93.33%> (+0.21%)` | :arrow_up: | | [...ive/messaging/interactive\_environment\_inspector.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9tZXNzYWdpbmcvaW50ZXJhY3RpdmVfZW52aXJvbm1lbnRfaW5zcGVjdG9yLnB5) | `97.43% <100.00%> (ø)` | | | [...eam/runners/interactive/options/capture\_control.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9vcHRpb25zL2NhcHR1cmVfY29udHJvbC5weQ==) | `92.00% <0.00%> (-8.00%)` | :arrow_down: | | [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `88.98% <0.00%> (-0.36%)` | :arrow_down: | | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `94.45% <0.00%> (-0.14%)` | :arrow_down: | | [sdks/python/apache\_beam/pvalue.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcHZhbHVlLnB5) | `92.01% <0.00%> (+0.38%)` | :arrow_up: | | ... and [5 more](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12779?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12779?src=pr=footer). Last update [5206131...af2c14c](https://codecov.io/gh/apache/beam/pull/12779?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #12721: [BEAM-10871] Add deidentify for FhirIO connector
pabloem commented on pull request #12721: URL: https://github.com/apache/beam/pull/12721#issuecomment-692763416 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] yifanmai commented on pull request #12656: [BEAM-10781] Add PTransformOverride.get_replacement_transform_for_applied_ptransform()
yifanmai commented on pull request #12656: URL: https://github.com/apache/beam/pull/12656#issuecomment-692808999 Awesome, thanks for your help! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #12505: [BEAM-8106] Add version to java container image name
TheNeuralBit commented on pull request #12505: URL: https://github.com/apache/beam/pull/12505#issuecomment-692822795 It looks like the spotless (Java code format) check is failing. FYI you can auto-format locally with `./gradlew spotlessApply`. It's not critical until we're ready to merge though. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] tvalentyn commented on pull request #12811: [BEAM-10705] Fixes a bug when passing whl files in --sdk_location from https locations
tvalentyn commented on pull request #12811: URL: https://github.com/apache/beam/pull/12811#issuecomment-692844750 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] pabloem merged pull request #12807: [BEAM-2855] implement query 10
pabloem merged pull request #12807: URL: https://github.com/apache/beam/pull/12807 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12841: [BEAM-10894] Basic CSV reading and writing.
codecov[bot] edited a comment on pull request #12841: URL: https://github.com/apache/beam/pull/12841#issuecomment-692395796 # [Codecov](https://codecov.io/gh/apache/beam/pull/12841?src=pr=h1) Report > Merging [#12841](https://codecov.io/gh/apache/beam/pull/12841?src=pr=desc) into [master](https://codecov.io/gh/apache/beam/commit/803efdd931699c24e60350e5ccf6e54482f5916f?el=desc) will **decrease** coverage by `0.09%`. > The diff coverage is `87.09%`. [](https://codecov.io/gh/apache/beam/pull/12841?src=pr=tree) ```diff @@Coverage Diff @@ ## master #12841 +/- ## == - Coverage 82.38% 82.28% -0.10% == Files 451 452 +1 Lines 5377553838 +63 == - Hits4430344302 -1 - Misses 9472 9536 +64 ``` | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12841?src=pr=tree) | Coverage Δ | | |---|---|---| | [...n/apache\_beam/runners/dataflow/dataflow\_metrics.py](https://codecov.io/gh/apache/beam/pull/12841/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9kYXRhZmxvd19tZXRyaWNzLnB5) | `74.32% <33.33%> (-0.85%)` | :arrow_down: | | [sdks/python/apache\_beam/dataframe/frame\_base.py](https://codecov.io/gh/apache/beam/pull/12841/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lX2Jhc2UucHk=) | `84.70% <57.14%> (ø)` | | | [sdks/python/apache\_beam/dataframe/doctests.py](https://codecov.io/gh/apache/beam/pull/12841/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2RvY3Rlc3RzLnB5) | `96.83% <83.33%> (-0.93%)` | :arrow_down: | | [sdks/python/apache\_beam/typehints/opcodes.py](https://codecov.io/gh/apache/beam/pull/12841/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL29wY29kZXMucHk=) | `87.65% <83.33%> (-0.35%)` | :arrow_down: | | [sdks/python/apache\_beam/transforms/stats.py](https://codecov.io/gh/apache/beam/pull/12841/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9zdGF0cy5weQ==) | `90.41% <86.36%> (+3.02%)` | :arrow_up: | | [sdks/python/apache\_beam/typehints/schemas.py](https://codecov.io/gh/apache/beam/pull/12841/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL3NjaGVtYXMucHk=) | `93.20% <86.66%> (-3.41%)` | :arrow_down: | | [sdks/python/apache\_beam/dataframe/io.py](https://codecov.io/gh/apache/beam/pull/12841/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2lvLnB5) | `89.90% <89.90%> (ø)` | | | [sdks/python/apache\_beam/coders/row\_coder.py](https://codecov.io/gh/apache/beam/pull/12841/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL3Jvd19jb2Rlci5weQ==) | `94.36% <90.90%> (-0.64%)` | :arrow_down: | | [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/12841/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `90.60% <100.00%> (+0.18%)` | :arrow_up: | | [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/12841/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `76.02% <0.00%> (-10.53%)` | :arrow_down: | | ... and [29 more](https://codecov.io/gh/apache/beam/pull/12841/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12841?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12841?src=pr=footer). Last update [8e7014d...ab66e1c](https://codecov.io/gh/apache/beam/pull/12841?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12819: [BEAM-9561] Initial framework for testing pandas website docs.
codecov[bot] edited a comment on pull request #12819: URL: https://github.com/apache/beam/pull/12819#issuecomment-691336298 # [Codecov](https://codecov.io/gh/apache/beam/pull/12819?src=pr=h1) Report > Merging [#12819](https://codecov.io/gh/apache/beam/pull/12819?src=pr=desc) into [master](https://codecov.io/gh/apache/beam/commit/803efdd931699c24e60350e5ccf6e54482f5916f?el=desc) will **increase** coverage by `0.03%`. > The diff coverage is `88.93%`. [](https://codecov.io/gh/apache/beam/pull/12819?src=pr=tree) ```diff @@Coverage Diff @@ ## master #12819 +/- ## == + Coverage 82.38% 82.42% +0.03% == Files 451 451 Lines 5377553983 +208 == + Hits4430344493 +190 - Misses 9472 9490 +18 ``` | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12819?src=pr=tree) | Coverage Δ | | |---|---|---| | [...n/apache\_beam/runners/dataflow/dataflow\_metrics.py](https://codecov.io/gh/apache/beam/pull/12819/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9kYXRhZmxvd19tZXRyaWNzLnB5) | `74.32% <33.33%> (-0.85%)` | :arrow_down: | | [sdks/python/apache\_beam/dataframe/frame\_base.py](https://codecov.io/gh/apache/beam/pull/12819/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lX2Jhc2UucHk=) | `84.70% <57.14%> (ø)` | | | [sdks/python/apache\_beam/typehints/opcodes.py](https://codecov.io/gh/apache/beam/pull/12819/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL29wY29kZXMucHk=) | `87.65% <83.33%> (-0.35%)` | :arrow_down: | | [sdks/python/apache\_beam/transforms/stats.py](https://codecov.io/gh/apache/beam/pull/12819/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9zdGF0cy5weQ==) | `87.39% <86.36%> (ø)` | | | [sdks/python/apache\_beam/typehints/schemas.py](https://codecov.io/gh/apache/beam/pull/12819/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL3NjaGVtYXMucHk=) | `93.20% <86.66%> (-3.41%)` | :arrow_down: | | [sdks/python/apache\_beam/coders/row\_coder.py](https://codecov.io/gh/apache/beam/pull/12819/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL3Jvd19jb2Rlci5weQ==) | `94.36% <90.90%> (-0.64%)` | :arrow_down: | | [sdks/python/apache\_beam/dataframe/doctests.py](https://codecov.io/gh/apache/beam/pull/12819/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2RvY3Rlc3RzLnB5) | `96.39% <95.09%> (-1.37%)` | :arrow_down: | | [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/12819/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `90.48% <100.00%> (+0.07%)` | :arrow_up: | | [...n/apache\_beam/runners/direct/evaluation\_context.py](https://codecov.io/gh/apache/beam/pull/12819/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvZXZhbHVhdGlvbl9jb250ZXh0LnB5) | `91.91% <100.00%> (ø)` | | | [.../python/apache\_beam/transforms/periodicsequence.py](https://codecov.io/gh/apache/beam/pull/12819/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9wZXJpb2RpY3NlcXVlbmNlLnB5) | `96.49% <0.00%> (-1.76%)` | :arrow_down: | | ... and [9 more](https://codecov.io/gh/apache/beam/pull/12819/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12819?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12819?src=pr=footer). Last update [8e7014d...0e3c400](https://codecov.io/gh/apache/beam/pull/12819?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12812: [BEAM-10873] Stronger testing of dataframe partitioning declarations.
codecov[bot] edited a comment on pull request #12812: URL: https://github.com/apache/beam/pull/12812#issuecomment-690611762 # [Codecov](https://codecov.io/gh/apache/beam/pull/12812?src=pr=h1) Report > Merging [#12812](https://codecov.io/gh/apache/beam/pull/12812?src=pr=desc) into [master](https://codecov.io/gh/apache/beam/commit/803efdd931699c24e60350e5ccf6e54482f5916f?el=desc) will **decrease** coverage by `0.01%`. > The diff coverage is `87.06%`. [](https://codecov.io/gh/apache/beam/pull/12812?src=pr=tree) ```diff @@Coverage Diff @@ ## master #12812 +/- ## == - Coverage 82.38% 82.37% -0.02% == Files 451 451 Lines 5377554459 +684 == + Hits4430344858 +555 - Misses 9472 9601 +129 ``` | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12812?src=pr=tree) | Coverage Δ | | |---|---|---| | [...n/apache\_beam/runners/dataflow/dataflow\_metrics.py](https://codecov.io/gh/apache/beam/pull/12812/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9kYXRhZmxvd19tZXRyaWNzLnB5) | `74.32% <33.33%> (-0.85%)` | :arrow_down: | | [sdks/python/apache\_beam/dataframe/frame\_base.py](https://codecov.io/gh/apache/beam/pull/12812/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lX2Jhc2UucHk=) | `84.97% <70.00%> (+0.26%)` | :arrow_up: | | [sdks/python/apache\_beam/typehints/opcodes.py](https://codecov.io/gh/apache/beam/pull/12812/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL29wY29kZXMucHk=) | `87.65% <83.33%> (-0.35%)` | :arrow_down: | | [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/12812/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `89.90% <85.18%> (-0.52%)` | :arrow_down: | | [sdks/python/apache\_beam/dataframe/doctests.py](https://codecov.io/gh/apache/beam/pull/12812/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2RvY3Rlc3RzLnB5) | `96.83% <85.71%> (-0.93%)` | :arrow_down: | | [sdks/python/apache\_beam/transforms/stats.py](https://codecov.io/gh/apache/beam/pull/12812/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9zdGF0cy5weQ==) | `90.41% <86.36%> (+3.02%)` | :arrow_up: | | [sdks/python/apache\_beam/typehints/schemas.py](https://codecov.io/gh/apache/beam/pull/12812/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL3NjaGVtYXMucHk=) | `93.20% <86.66%> (-3.41%)` | :arrow_down: | | [sdks/python/apache\_beam/coders/row\_coder.py](https://codecov.io/gh/apache/beam/pull/12812/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL3Jvd19jb2Rlci5weQ==) | `94.36% <90.90%> (-0.64%)` | :arrow_down: | | [sdks/python/apache\_beam/dataframe/expressions.py](https://codecov.io/gh/apache/beam/pull/12812/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2V4cHJlc3Npb25zLnB5) | `89.70% <96.29%> (+1.63%)` | :arrow_up: | | [sdks/python/apache\_beam/dataframe/partitionings.py](https://codecov.io/gh/apache/beam/pull/12812/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL3BhcnRpdGlvbmluZ3MucHk=) | `89.04% <100.00%> (+5.43%)` | :arrow_up: | | ... and [30 more](https://codecov.io/gh/apache/beam/pull/12812/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12812?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12812?src=pr=footer). Last update [8e7014d...7433760](https://codecov.io/gh/apache/beam/pull/12812?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] tvalentyn commented on pull request #12811: [BEAM-10705] Fixes a bug when passing whl files in --sdk_location from https locations
tvalentyn commented on pull request #12811: URL: https://github.com/apache/beam/pull/12811#issuecomment-692850902 Thanks, @samatix ! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] tvalentyn merged pull request #12811: [BEAM-10705] Fixes a bug when passing whl files in --sdk_location from https locations
tvalentyn merged pull request #12811: URL: https://github.com/apache/beam/pull/12811 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12806: [BEAM-10869] Make WriteToPubsub output serialized PubsubMessage proto bytes when using runner v2
codecov[bot] edited a comment on pull request #12806: URL: https://github.com/apache/beam/pull/12806#issuecomment-692876818 # [Codecov](https://codecov.io/gh/apache/beam/pull/12806?src=pr=h1) Report > Merging [#12806](https://codecov.io/gh/apache/beam/pull/12806?src=pr=desc) into [master](https://codecov.io/gh/apache/beam/commit/2bb60c323095340240ec4982c1e9dabc397107e5?el=desc) will **increase** coverage by `41.83%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/beam/pull/12806?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #12806 +/- ## === + Coverage 40.43% 82.27% +41.83% === Files 449 451+2 Lines 5353053785 +255 === + Hits2164344249+22606 + Misses 31887 9536-22351 ``` | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12806?src=pr=tree) | Coverage Δ | | |---|---|---| | [sdks/python/apache\_beam/io/gcp/pubsub.py](https://codecov.io/gh/apache/beam/pull/12806/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3B1YnN1Yi5weQ==) | `93.54% <ø> (+51.53%)` | :arrow_up: | | [...apache\_beam/portability/api/beam\_runner\_api\_pb2.py](https://codecov.io/gh/apache/beam/pull/12806/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjIucHk=) | `100.00% <0.00%> (ø)` | | | [...he\_beam/testing/benchmarks/nexmark/nexmark\_util.py](https://codecov.io/gh/apache/beam/pull/12806/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvbmV4bWFya191dGlsLnB5) | `0.00% <0.00%> (ø)` | | | [...e\_beam/portability/api/beam\_runner\_api\_pb2\_urns.py](https://codecov.io/gh/apache/beam/pull/12806/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjJfdXJucy5weQ==) | `100.00% <0.00%> (ø)` | | | [...eam/testing/benchmarks/nexmark/nexmark\_launcher.py](https://codecov.io/gh/apache/beam/pull/12806/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvbmV4bWFya19sYXVuY2hlci5weQ==) | `0.00% <0.00%> (ø)` | | | [...ache\_beam/runners/interactive/pipeline\_analyzer.py](https://codecov.io/gh/apache/beam/pull/12806/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9waXBlbGluZV9hbmFseXplci5weQ==) | | | | [...beam/testing/benchmarks/nexmark/queries/query10.py](https://codecov.io/gh/apache/beam/pull/12806/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvcXVlcmllcy9xdWVyeTEwLnB5) | `0.00% <0.00%> (ø)` | | | [sdks/python/apache\_beam/utils/histogram.py](https://codecov.io/gh/apache/beam/pull/12806/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvaGlzdG9ncmFtLnB5) | `94.28% <0.00%> (ø)` | | | [...he\_beam/testing/benchmarks/nexmark/nexmark\_perf.py](https://codecov.io/gh/apache/beam/pull/12806/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvbmV4bWFya19wZXJmLnB5) | `0.00% <0.00%> (ø)` | | | [sdks/python/apache\_beam/coders/slow\_stream.py](https://codecov.io/gh/apache/beam/pull/12806/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL3Nsb3dfc3RyZWFtLnB5) | `92.43% <0.00%> (+1.68%)` | :arrow_up: | | ... and [275 more](https://codecov.io/gh/apache/beam/pull/12806/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12806?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12806?src=pr=footer). Last update [b6d0abb...60b0229](https://codecov.io/gh/apache/beam/pull/12806?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] ibzib commented on pull request #12576: [BEAM-10671] Add environment configuration fields as first-class pipeline options.
ibzib commented on pull request #12576: URL: https://github.com/apache/beam/pull/12576#issuecomment-692885360 (Filed BEAM-10901 for test flakes.) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12799: [BEAM-10603] Add record_pipeline, clear to RM and fix duration limiter
codecov[bot] edited a comment on pull request #12799: URL: https://github.com/apache/beam/pull/12799#issuecomment-692960218 # [Codecov](https://codecov.io/gh/apache/beam/pull/12799?src=pr=h1) Report > Merging [#12799](https://codecov.io/gh/apache/beam/pull/12799?src=pr=desc) into [master](https://codecov.io/gh/apache/beam/commit/a2126e571cfddc90e3819e66ef1f2da3d52833ed?el=desc) will **decrease** coverage by `0.06%`. > The diff coverage is `58.56%`. [](https://codecov.io/gh/apache/beam/pull/12799?src=pr=tree) ```diff @@Coverage Diff @@ ## master #12799 +/- ## == - Coverage 82.36% 82.30% -0.07% == Files 450 451 +1 Lines 5370853849 +141 == + Hits4423844319 +81 - Misses 9470 9530 +60 ``` | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12799?src=pr=tree) | Coverage Δ | | |---|---|---| | [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `96.52% <ø> (+1.73%)` | :arrow_up: | | [...beam/testing/benchmarks/nexmark/queries/query10.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvcXVlcmllcy9xdWVyeTEwLnB5) | `0.00% <0.00%> (ø)` | | | [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `76.02% <64.40%> (-10.53%)` | :arrow_down: | | [...runners/interactive/display/pcoll\_visualization.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kaXNwbGF5L3Bjb2xsX3Zpc3VhbGl6YXRpb24ucHk=) | `85.26% <87.50%> (-0.88%)` | :arrow_down: | | [...ive/messaging/interactive\_environment\_inspector.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9tZXNzYWdpbmcvaW50ZXJhY3RpdmVfZW52aXJvbm1lbnRfaW5zcGVjdG9yLnB5) | `97.43% <100.00%> (ø)` | | | [...am/runners/interactive/options/capture\_limiters.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9vcHRpb25zL2NhcHR1cmVfbGltaXRlcnMucHk=) | `90.47% <100.00%> (-3.08%)` | :arrow_down: | | [...ache\_beam/runners/interactive/recording\_manager.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9yZWNvcmRpbmdfbWFuYWdlci5weQ==) | `98.33% <100.00%> (+5.47%)` | :arrow_up: | | [...ks/python/apache\_beam/runners/interactive/utils.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS91dGlscy5weQ==) | `92.38% <100.00%> (+0.30%)` | :arrow_up: | | [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `88.68% <0.00%> (-1.23%)` | :arrow_down: | | ... and [19 more](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12799?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12799?src=pr=footer). Last update [5206131...9ba76c5](https://codecov.io/gh/apache/beam/pull/12799?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] monicadsong commented on a change in pull request #12851: [BEAM-10900] add ability for ApproximateUniqueCombineFn to handle numpy input data
monicadsong commented on a change in pull request #12851: URL: https://github.com/apache/beam/pull/12851#discussion_r488863704 ## File path: sdks/python/apache_beam/transforms/stats_test.py ## @@ -89,6 +90,12 @@ def setUp(self): None, 0.1, 'assert:global_by_error_with_large_population'), + ( + 'numpy_input_data', + np.array(range(10)), Review comment: No. For numpy input I am assuming that for ApproxUniqueCombineFn.add_input(self, accumulator, element), element is a numpy scalar (not a numpy array). However, if the element is a multidimensional numpy array, then calling element.item(0) returns the value at the first index of the array as a python type. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] ibzib commented on a change in pull request #12827: [BEAM-10885] Add Avro support to Kafka table provider
ibzib commented on a change in pull request #12827:
URL: https://github.com/apache/beam/pull/12827#discussion_r488862816
##
File path:
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/BeamKafkaTableTest.java
##
@@ -41,27 +40,49 @@
import
org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataTypeSystem;
import
org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.type.SqlTypeName;
import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
-import org.apache.commons.csv.CSVFormat;
import org.junit.Assert;
import org.junit.Rule;
import org.junit.Test;
/** Test for BeamKafkaCSVTable. */
-public class BeamKafkaCSVTableTest {
+public abstract class BeamKafkaTableTest {
@Rule public TestPipeline pipeline = TestPipeline.create();
- private static final Row ROW1 = Row.withSchema(genSchema()).addValues(1L, 1,
1.0).build();
+ protected static final Schema BEAM_SQL_SCHEMA =
+ TestTableUtils.buildBeamSqlSchema(
+ Schema.FieldType.INT32,
+ "order_id",
+ Schema.FieldType.INT32,
+ "site_id",
+ Schema.FieldType.INT32,
+ "price");
- private static final Row ROW2 = Row.withSchema(genSchema()).addValues(2L, 2,
2.0).build();
+ protected static final List TOPICS = ImmutableList.of("topic1",
"topic2");
+
+ protected static final Schema SCHEMA = genSchema();
+
+ protected static final Row ROW1 = Row.withSchema(SCHEMA).addValues(1L, 1,
1.0).build();
+
+ protected static final Row ROW2 = Row.withSchema(SCHEMA).addValues(2L, 2,
2.0).build();
+
+ private static final Map tables = new HashMap<>();
- private static Map tables = new HashMap<>();
protected static BeamSqlEnv env = BeamSqlEnv.readOnly("test", tables);
+ protected abstract KafkaTestRecord createKafkaTestRecord(
Review comment:
Nit: Can we change this to `createKafkaTestRecord(String key, List
values, int timestamp)`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [beam] robertwb merged pull request #12787: [BEAM-10641] Add eliminate_common_key_with_none graph optimizer
robertwb merged pull request #12787: URL: https://github.com/apache/beam/pull/12787 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12576: [BEAM-10671] Add environment configuration fields as first-class pipeline options.
codecov[bot] edited a comment on pull request #12576: URL: https://github.com/apache/beam/pull/12576#issuecomment-692353567 # [Codecov](https://codecov.io/gh/apache/beam/pull/12576?src=pr=h1) Report > Merging [#12576](https://codecov.io/gh/apache/beam/pull/12576?src=pr=desc) into [master](https://codecov.io/gh/apache/beam/commit/803efdd931699c24e60350e5ccf6e54482f5916f?el=desc) will **decrease** coverage by `0.08%`. > The diff coverage is `89.89%`. [](https://codecov.io/gh/apache/beam/pull/12576?src=pr=tree) ```diff @@Coverage Diff @@ ## master #12576 +/- ## == - Coverage 82.38% 82.30% -0.09% == Files 451 451 Lines 5377553807 +32 == - Hits4430344284 -19 - Misses 9472 9523 +51 ``` | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12576?src=pr=tree) | Coverage Δ | | |---|---|---| | [...n/apache\_beam/runners/dataflow/dataflow\_metrics.py](https://codecov.io/gh/apache/beam/pull/12576/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9kYXRhZmxvd19tZXRyaWNzLnB5) | `74.32% <33.33%> (-0.85%)` | :arrow_down: | | [sdks/python/apache\_beam/transforms/combiners.py](https://codecov.io/gh/apache/beam/pull/12576/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9jb21iaW5lcnMucHk=) | `92.14% <50.00%> (-0.02%)` | :arrow_down: | | [sdks/python/apache\_beam/dataframe/frame\_base.py](https://codecov.io/gh/apache/beam/pull/12576/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lX2Jhc2UucHk=) | `84.70% <57.14%> (ø)` | | | [sdks/python/apache\_beam/dataframe/doctests.py](https://codecov.io/gh/apache/beam/pull/12576/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2RvY3Rlc3RzLnB5) | `96.83% <83.33%> (-0.93%)` | :arrow_down: | | [sdks/python/apache\_beam/typehints/opcodes.py](https://codecov.io/gh/apache/beam/pull/12576/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL29wY29kZXMucHk=) | `87.65% <83.33%> (-0.35%)` | :arrow_down: | | [sdks/python/apache\_beam/transforms/stats.py](https://codecov.io/gh/apache/beam/pull/12576/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9zdGF0cy5weQ==) | `90.41% <86.36%> (+3.02%)` | :arrow_up: | | [sdks/python/apache\_beam/typehints/schemas.py](https://codecov.io/gh/apache/beam/pull/12576/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL3NjaGVtYXMucHk=) | `93.20% <86.66%> (-3.41%)` | :arrow_down: | | [sdks/python/apache\_beam/transforms/environments.py](https://codecov.io/gh/apache/beam/pull/12576/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9lbnZpcm9ubWVudHMucHk=) | `84.19% <90.47%> (+0.37%)` | :arrow_up: | | [sdks/python/apache\_beam/coders/row\_coder.py](https://codecov.io/gh/apache/beam/pull/12576/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL3Jvd19jb2Rlci5weQ==) | `94.36% <90.90%> (-0.64%)` | :arrow_down: | | [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/12576/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `90.48% <100.00%> (+0.07%)` | :arrow_up: | | ... and [33 more](https://codecov.io/gh/apache/beam/pull/12576/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12576?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12576?src=pr=footer). Last update [91a18f1...2710a0e](https://codecov.io/gh/apache/beam/pull/12576?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] monicadsong closed pull request #12851: [BEAM-10900] add ability for ApproximateUniqueCombineFn to handle numpy input data
monicadsong closed pull request #12851: URL: https://github.com/apache/beam/pull/12851 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] tvalentyn commented on a change in pull request #12851: [BEAM-10900] add ability for ApproximateUniqueCombineFn to handle numpy input data
tvalentyn commented on a change in pull request #12851: URL: https://github.com/apache/beam/pull/12851#discussion_r488853550 ## File path: sdks/python/apache_beam/transforms/stats_test.py ## @@ -89,6 +90,12 @@ def setUp(self): None, 0.1, 'assert:global_by_error_with_large_population'), + ( + 'numpy_input_data', + np.array(range(10)), Review comment: Do multidimensional arrays also work? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] mxm commented on pull request #12576: [BEAM-10671] Add environment configuration fields as first-class pipeline options.
mxm commented on pull request #12576: URL: https://github.com/apache/beam/pull/12576#issuecomment-692877960 Thanks! I'll have another look tomorrow but generally looks good to me. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12806: [BEAM-10869] Make WriteToPubsub output serialized PubsubMessage proto bytes when using runner v2
codecov[bot] edited a comment on pull request #12806: URL: https://github.com/apache/beam/pull/12806#issuecomment-692876818 # [Codecov](https://codecov.io/gh/apache/beam/pull/12806?src=pr=h1) Report > Merging [#12806](https://codecov.io/gh/apache/beam/pull/12806?src=pr=desc) into [master](https://codecov.io/gh/apache/beam/commit/2bb60c323095340240ec4982c1e9dabc397107e5?el=desc) will **increase** coverage by `41.83%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/beam/pull/12806?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #12806 +/- ## === + Coverage 40.43% 82.27% +41.83% === Files 449 451+2 Lines 5353053785 +255 === + Hits2164344249+22606 + Misses 31887 9536-22351 ``` | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12806?src=pr=tree) | Coverage Δ | | |---|---|---| | [sdks/python/apache\_beam/io/gcp/pubsub.py](https://codecov.io/gh/apache/beam/pull/12806/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3B1YnN1Yi5weQ==) | `93.54% <ø> (+51.53%)` | :arrow_up: | | [...apache\_beam/portability/api/beam\_runner\_api\_pb2.py](https://codecov.io/gh/apache/beam/pull/12806/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjIucHk=) | `100.00% <0.00%> (ø)` | | | [...he\_beam/testing/benchmarks/nexmark/nexmark\_util.py](https://codecov.io/gh/apache/beam/pull/12806/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvbmV4bWFya191dGlsLnB5) | `0.00% <0.00%> (ø)` | | | [...e\_beam/portability/api/beam\_runner\_api\_pb2\_urns.py](https://codecov.io/gh/apache/beam/pull/12806/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjJfdXJucy5weQ==) | `100.00% <0.00%> (ø)` | | | [...eam/testing/benchmarks/nexmark/nexmark\_launcher.py](https://codecov.io/gh/apache/beam/pull/12806/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvbmV4bWFya19sYXVuY2hlci5weQ==) | `0.00% <0.00%> (ø)` | | | [...ache\_beam/runners/interactive/pipeline\_analyzer.py](https://codecov.io/gh/apache/beam/pull/12806/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9waXBlbGluZV9hbmFseXplci5weQ==) | | | | [sdks/python/apache\_beam/utils/histogram.py](https://codecov.io/gh/apache/beam/pull/12806/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvaGlzdG9ncmFtLnB5) | `94.28% <0.00%> (ø)` | | | [...beam/testing/benchmarks/nexmark/queries/query10.py](https://codecov.io/gh/apache/beam/pull/12806/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvcXVlcmllcy9xdWVyeTEwLnB5) | `0.00% <0.00%> (ø)` | | | [...he\_beam/testing/benchmarks/nexmark/nexmark\_perf.py](https://codecov.io/gh/apache/beam/pull/12806/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvbmV4bWFya19wZXJmLnB5) | `0.00% <0.00%> (ø)` | | | [sdks/python/apache\_beam/coders/slow\_stream.py](https://codecov.io/gh/apache/beam/pull/12806/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL3Nsb3dfc3RyZWFtLnB5) | `92.43% <0.00%> (+1.68%)` | :arrow_up: | | ... and [275 more](https://codecov.io/gh/apache/beam/pull/12806/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12806?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12806?src=pr=footer). Last update [b6d0abb...60b0229](https://codecov.io/gh/apache/beam/pull/12806?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12806: [BEAM-10869] Make WriteToPubsub output serialized PubsubMessage proto bytes when using runner v2
codecov[bot] edited a comment on pull request #12806: URL: https://github.com/apache/beam/pull/12806#issuecomment-692876818 # [Codecov](https://codecov.io/gh/apache/beam/pull/12806?src=pr=h1) Report > Merging [#12806](https://codecov.io/gh/apache/beam/pull/12806?src=pr=desc) into [master](https://codecov.io/gh/apache/beam/commit/2bb60c323095340240ec4982c1e9dabc397107e5?el=desc) will **increase** coverage by `41.83%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/beam/pull/12806?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #12806 +/- ## === + Coverage 40.43% 82.27% +41.83% === Files 449 451+2 Lines 5353053785 +255 === + Hits2164344249+22606 + Misses 31887 9536-22351 ``` | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12806?src=pr=tree) | Coverage Δ | | |---|---|---| | [sdks/python/apache\_beam/io/gcp/pubsub.py](https://codecov.io/gh/apache/beam/pull/12806/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3B1YnN1Yi5weQ==) | `93.54% <ø> (+51.53%)` | :arrow_up: | | [...apache\_beam/portability/api/beam\_runner\_api\_pb2.py](https://codecov.io/gh/apache/beam/pull/12806/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjIucHk=) | `100.00% <0.00%> (ø)` | | | [...he\_beam/testing/benchmarks/nexmark/nexmark\_util.py](https://codecov.io/gh/apache/beam/pull/12806/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvbmV4bWFya191dGlsLnB5) | `0.00% <0.00%> (ø)` | | | [...e\_beam/portability/api/beam\_runner\_api\_pb2\_urns.py](https://codecov.io/gh/apache/beam/pull/12806/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjJfdXJucy5weQ==) | `100.00% <0.00%> (ø)` | | | [...eam/testing/benchmarks/nexmark/nexmark\_launcher.py](https://codecov.io/gh/apache/beam/pull/12806/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvbmV4bWFya19sYXVuY2hlci5weQ==) | `0.00% <0.00%> (ø)` | | | [...ache\_beam/runners/interactive/pipeline\_analyzer.py](https://codecov.io/gh/apache/beam/pull/12806/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9waXBlbGluZV9hbmFseXplci5weQ==) | | | | [sdks/python/apache\_beam/utils/histogram.py](https://codecov.io/gh/apache/beam/pull/12806/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvaGlzdG9ncmFtLnB5) | `94.28% <0.00%> (ø)` | | | [...he\_beam/testing/benchmarks/nexmark/nexmark\_perf.py](https://codecov.io/gh/apache/beam/pull/12806/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvbmV4bWFya19wZXJmLnB5) | `0.00% <0.00%> (ø)` | | | [...beam/testing/benchmarks/nexmark/queries/query10.py](https://codecov.io/gh/apache/beam/pull/12806/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvcXVlcmllcy9xdWVyeTEwLnB5) | `0.00% <0.00%> (ø)` | | | [sdks/python/apache\_beam/coders/slow\_stream.py](https://codecov.io/gh/apache/beam/pull/12806/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL3Nsb3dfc3RyZWFtLnB5) | `92.43% <0.00%> (+1.68%)` | :arrow_up: | | ... and [275 more](https://codecov.io/gh/apache/beam/pull/12806/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12806?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12806?src=pr=footer). Last update [b6d0abb...60b0229](https://codecov.io/gh/apache/beam/pull/12806?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12576: [BEAM-10671] Add environment configuration fields as first-class pipeline options.
codecov[bot] edited a comment on pull request #12576: URL: https://github.com/apache/beam/pull/12576#issuecomment-692353567 # [Codecov](https://codecov.io/gh/apache/beam/pull/12576?src=pr=h1) Report > Merging [#12576](https://codecov.io/gh/apache/beam/pull/12576?src=pr=desc) into [master](https://codecov.io/gh/apache/beam/commit/803efdd931699c24e60350e5ccf6e54482f5916f?el=desc) will **decrease** coverage by `0.08%`. > The diff coverage is `89.89%`. [](https://codecov.io/gh/apache/beam/pull/12576?src=pr=tree) ```diff @@Coverage Diff @@ ## master #12576 +/- ## == - Coverage 82.38% 82.30% -0.09% == Files 451 451 Lines 5377553807 +32 == - Hits4430344284 -19 - Misses 9472 9523 +51 ``` | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12576?src=pr=tree) | Coverage Δ | | |---|---|---| | [...n/apache\_beam/runners/dataflow/dataflow\_metrics.py](https://codecov.io/gh/apache/beam/pull/12576/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9kYXRhZmxvd19tZXRyaWNzLnB5) | `74.32% <33.33%> (-0.85%)` | :arrow_down: | | [sdks/python/apache\_beam/transforms/combiners.py](https://codecov.io/gh/apache/beam/pull/12576/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9jb21iaW5lcnMucHk=) | `92.14% <50.00%> (-0.02%)` | :arrow_down: | | [sdks/python/apache\_beam/dataframe/frame\_base.py](https://codecov.io/gh/apache/beam/pull/12576/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lX2Jhc2UucHk=) | `84.70% <57.14%> (ø)` | | | [sdks/python/apache\_beam/dataframe/doctests.py](https://codecov.io/gh/apache/beam/pull/12576/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2RvY3Rlc3RzLnB5) | `96.83% <83.33%> (-0.93%)` | :arrow_down: | | [sdks/python/apache\_beam/typehints/opcodes.py](https://codecov.io/gh/apache/beam/pull/12576/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL29wY29kZXMucHk=) | `87.65% <83.33%> (-0.35%)` | :arrow_down: | | [sdks/python/apache\_beam/transforms/stats.py](https://codecov.io/gh/apache/beam/pull/12576/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9zdGF0cy5weQ==) | `90.41% <86.36%> (+3.02%)` | :arrow_up: | | [sdks/python/apache\_beam/typehints/schemas.py](https://codecov.io/gh/apache/beam/pull/12576/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL3NjaGVtYXMucHk=) | `93.20% <86.66%> (-3.41%)` | :arrow_down: | | [sdks/python/apache\_beam/transforms/environments.py](https://codecov.io/gh/apache/beam/pull/12576/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9lbnZpcm9ubWVudHMucHk=) | `84.19% <90.47%> (+0.37%)` | :arrow_up: | | [sdks/python/apache\_beam/coders/row\_coder.py](https://codecov.io/gh/apache/beam/pull/12576/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL3Jvd19jb2Rlci5weQ==) | `94.36% <90.90%> (-0.64%)` | :arrow_down: | | [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/12576/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `90.48% <100.00%> (+0.07%)` | :arrow_up: | | ... and [33 more](https://codecov.io/gh/apache/beam/pull/12576/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12576?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12576?src=pr=footer). Last update [91a18f1...2710a0e](https://codecov.io/gh/apache/beam/pull/12576?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12806: [BEAM-10869] Make WriteToPubsub output serialized PubsubMessage proto bytes when using runner v2
codecov[bot] edited a comment on pull request #12806: URL: https://github.com/apache/beam/pull/12806#issuecomment-692876818 # [Codecov](https://codecov.io/gh/apache/beam/pull/12806?src=pr=h1) Report > Merging [#12806](https://codecov.io/gh/apache/beam/pull/12806?src=pr=desc) into [master](https://codecov.io/gh/apache/beam/commit/2bb60c323095340240ec4982c1e9dabc397107e5?el=desc) will **increase** coverage by `41.83%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/beam/pull/12806?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #12806 +/- ## === + Coverage 40.43% 82.27% +41.83% === Files 449 451+2 Lines 5353053785 +255 === + Hits2164344249+22606 + Misses 31887 9536-22351 ``` | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12806?src=pr=tree) | Coverage Δ | | |---|---|---| | [sdks/python/apache\_beam/io/gcp/pubsub.py](https://codecov.io/gh/apache/beam/pull/12806/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3B1YnN1Yi5weQ==) | `93.54% <ø> (+51.53%)` | :arrow_up: | | [...apache\_beam/portability/api/beam\_runner\_api\_pb2.py](https://codecov.io/gh/apache/beam/pull/12806/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjIucHk=) | `100.00% <0.00%> (ø)` | | | [...he\_beam/testing/benchmarks/nexmark/nexmark\_util.py](https://codecov.io/gh/apache/beam/pull/12806/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvbmV4bWFya191dGlsLnB5) | `0.00% <0.00%> (ø)` | | | [...e\_beam/portability/api/beam\_runner\_api\_pb2\_urns.py](https://codecov.io/gh/apache/beam/pull/12806/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjJfdXJucy5weQ==) | `100.00% <0.00%> (ø)` | | | [...eam/testing/benchmarks/nexmark/nexmark\_launcher.py](https://codecov.io/gh/apache/beam/pull/12806/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvbmV4bWFya19sYXVuY2hlci5weQ==) | `0.00% <0.00%> (ø)` | | | [...ache\_beam/runners/interactive/pipeline\_analyzer.py](https://codecov.io/gh/apache/beam/pull/12806/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9waXBlbGluZV9hbmFseXplci5weQ==) | | | | [sdks/python/apache\_beam/utils/histogram.py](https://codecov.io/gh/apache/beam/pull/12806/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvaGlzdG9ncmFtLnB5) | `94.28% <0.00%> (ø)` | | | [...he\_beam/testing/benchmarks/nexmark/nexmark\_perf.py](https://codecov.io/gh/apache/beam/pull/12806/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvbmV4bWFya19wZXJmLnB5) | `0.00% <0.00%> (ø)` | | | [...beam/testing/benchmarks/nexmark/queries/query10.py](https://codecov.io/gh/apache/beam/pull/12806/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvcXVlcmllcy9xdWVyeTEwLnB5) | `0.00% <0.00%> (ø)` | | | [sdks/python/apache\_beam/coders/slow\_stream.py](https://codecov.io/gh/apache/beam/pull/12806/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL3Nsb3dfc3RyZWFtLnB5) | `92.43% <0.00%> (+1.68%)` | :arrow_up: | | ... and [275 more](https://codecov.io/gh/apache/beam/pull/12806/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12806?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12806?src=pr=footer). Last update [b6d0abb...60b0229](https://codecov.io/gh/apache/beam/pull/12806?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
