[GitHub] [beam] chamikaramj merged pull request #12656: [BEAM-10781] Add PTransformOverride.get_replacement_transform_for_applied_ptransform()
chamikaramj merged pull request #12656: URL: https://github.com/apache/beam/pull/12656 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] youngoli commented on pull request #12845: [BEAM-6928] Update changelog with WriteToBigQuery changed requirements.
youngoli commented on pull request #12845: URL: https://github.com/apache/beam/pull/12845#issuecomment-692466752 R: @pabloem @chamikaramj I tested this fix and confirmed that it works in #12792 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] youngoli opened a new pull request #12845: [BEAM-6928] Update changelog with WriteToBigQuery changed requirements.
youngoli opened a new pull request #12845: URL: https://github.com/apache/beam/pull/12845 Also update a test in the release validation script. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [x] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [x] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2 --- | --- | --- | --- | --- | --- | --- Go | [](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | --- Java | [](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/) Python | [](https://ci-beam.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[![Build
[GitHub] [beam] chamikaramj commented on pull request #12656: [BEAM-10781] Add PTransformOverride.get_replacement_transform_for_applied_ptransform()
chamikaramj commented on pull request #12656: URL: https://github.com/apache/beam/pull/12656#issuecomment-692443824 PreCommit test suite passed and macos failure seems to be unrelated - https://issues.apache.org/jira/browse/BEAM-10866. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] codecov[bot] commented on pull request #12845: [BEAM-6928] Update changelog with WriteToBigQuery changed requirements.
codecov[bot] commented on pull request #12845: URL: https://github.com/apache/beam/pull/12845#issuecomment-692472368 # [Codecov](https://codecov.io/gh/apache/beam/pull/12845?src=pr=h1) Report > Merging [#12845](https://codecov.io/gh/apache/beam/pull/12845?src=pr=desc) into [master](https://codecov.io/gh/apache/beam/commit/636bf8bf944b7e0c085480456a7a3df586255ab6?el=desc) will **increase** coverage by `0.00%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/beam/pull/12845?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #12845 +/- ## === Coverage 82.35% 82.35% === Files 450 450 Lines 5370253708+6 === + Hits4422544230+5 - Misses 9477 9478+1 ``` | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12845?src=pr=tree) | Coverage Δ | | |---|---|---| | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/12845/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `94.05% <0.00%> (-0.40%)` | :arrow_down: | | [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/12845/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `88.98% <0.00%> (-0.36%)` | :arrow_down: | | [...he\_beam/runners/interactive/pipeline\_instrument.py](https://codecov.io/gh/apache/beam/pull/12845/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9waXBlbGluZV9pbnN0cnVtZW50LnB5) | `98.56% <0.00%> (-0.24%)` | :arrow_down: | | [...on/apache\_beam/runners/direct/sdf\_direct\_runner.py](https://codecov.io/gh/apache/beam/pull/12845/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3Qvc2RmX2RpcmVjdF9ydW5uZXIucHk=) | `36.06% <0.00%> (-0.15%)` | :arrow_down: | | [...python/apache\_beam/runners/direct/direct\_runner.py](https://codecov.io/gh/apache/beam/pull/12845/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvZGlyZWN0X3J1bm5lci5weQ==) | `93.70% <0.00%> (+0.02%)` | :arrow_up: | | [sdks/python/apache\_beam/pipeline.py](https://codecov.io/gh/apache/beam/pull/12845/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcGlwZWxpbmUucHk=) | `88.83% <0.00%> (+0.03%)` | :arrow_up: | | [...ache\_beam/runners/dataflow/ptransform\_overrides.py](https://codecov.io/gh/apache/beam/pull/12845/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9wdHJhbnNmb3JtX292ZXJyaWRlcy5weQ==) | `90.00% <0.00%> (+0.14%)` | :arrow_up: | | [...eam/runners/interactive/caching/streaming\_cache.py](https://codecov.io/gh/apache/beam/pull/12845/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9jYWNoaW5nL3N0cmVhbWluZ19jYWNoZS5weQ==) | `95.33% <0.00%> (+1.27%)` | :arrow_up: | | [...ive/messaging/interactive\_environment\_inspector.py](https://codecov.io/gh/apache/beam/pull/12845/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9tZXNzYWdpbmcvaW50ZXJhY3RpdmVfZW52aXJvbm1lbnRfaW5zcGVjdG9yLnB5) | `97.43% <0.00%> (+1.28%)` | :arrow_up: | | [...am/runners/interactive/options/capture\_limiters.py](https://codecov.io/gh/apache/beam/pull/12845/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9vcHRpb25zL2NhcHR1cmVfbGltaXRlcnMucHk=) | `93.54% <0.00%> (+1.61%)` | :arrow_up: | | ... and [1 more](https://codecov.io/gh/apache/beam/pull/12845/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12845?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12845?src=pr=footer). Last update [636bf8b...e0e5fb5](https://codecov.io/gh/apache/beam/pull/12845?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12845: [BEAM-6928] Update changelog with WriteToBigQuery changed requirements.
codecov[bot] edited a comment on pull request #12845: URL: https://github.com/apache/beam/pull/12845#issuecomment-692472368 # [Codecov](https://codecov.io/gh/apache/beam/pull/12845?src=pr=h1) Report > Merging [#12845](https://codecov.io/gh/apache/beam/pull/12845?src=pr=desc) into [master](https://codecov.io/gh/apache/beam/commit/636bf8bf944b7e0c085480456a7a3df586255ab6?el=desc) will **increase** coverage by `0.00%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/beam/pull/12845?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #12845 +/- ## === Coverage 82.35% 82.35% === Files 450 450 Lines 5370253708+6 === + Hits4422544230+5 - Misses 9477 9478+1 ``` | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12845?src=pr=tree) | Coverage Δ | | |---|---|---| | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/12845/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `94.05% <0.00%> (-0.40%)` | :arrow_down: | | [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/12845/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `88.98% <0.00%> (-0.36%)` | :arrow_down: | | [...he\_beam/runners/interactive/pipeline\_instrument.py](https://codecov.io/gh/apache/beam/pull/12845/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9waXBlbGluZV9pbnN0cnVtZW50LnB5) | `98.56% <0.00%> (-0.24%)` | :arrow_down: | | [...on/apache\_beam/runners/direct/sdf\_direct\_runner.py](https://codecov.io/gh/apache/beam/pull/12845/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3Qvc2RmX2RpcmVjdF9ydW5uZXIucHk=) | `36.06% <0.00%> (-0.15%)` | :arrow_down: | | [...python/apache\_beam/runners/direct/direct\_runner.py](https://codecov.io/gh/apache/beam/pull/12845/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvZGlyZWN0X3J1bm5lci5weQ==) | `93.70% <0.00%> (+0.02%)` | :arrow_up: | | [sdks/python/apache\_beam/pipeline.py](https://codecov.io/gh/apache/beam/pull/12845/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcGlwZWxpbmUucHk=) | `88.83% <0.00%> (+0.03%)` | :arrow_up: | | [...ache\_beam/runners/dataflow/ptransform\_overrides.py](https://codecov.io/gh/apache/beam/pull/12845/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9wdHJhbnNmb3JtX292ZXJyaWRlcy5weQ==) | `90.00% <0.00%> (+0.14%)` | :arrow_up: | | [...eam/runners/interactive/caching/streaming\_cache.py](https://codecov.io/gh/apache/beam/pull/12845/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9jYWNoaW5nL3N0cmVhbWluZ19jYWNoZS5weQ==) | `95.33% <0.00%> (+1.27%)` | :arrow_up: | | [...ive/messaging/interactive\_environment\_inspector.py](https://codecov.io/gh/apache/beam/pull/12845/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9tZXNzYWdpbmcvaW50ZXJhY3RpdmVfZW52aXJvbm1lbnRfaW5zcGVjdG9yLnB5) | `97.43% <0.00%> (+1.28%)` | :arrow_up: | | [...am/runners/interactive/options/capture\_limiters.py](https://codecov.io/gh/apache/beam/pull/12845/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9vcHRpb25zL2NhcHR1cmVfbGltaXRlcnMucHk=) | `93.54% <0.00%> (+1.61%)` | :arrow_up: | | ... and [1 more](https://codecov.io/gh/apache/beam/pull/12845/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12845?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12845?src=pr=footer). Last update [636bf8b...e0e5fb5](https://codecov.io/gh/apache/beam/pull/12845?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12845: [BEAM-6928] Update changelog with WriteToBigQuery changed requirements.
codecov[bot] edited a comment on pull request #12845: URL: https://github.com/apache/beam/pull/12845#issuecomment-692472368 # [Codecov](https://codecov.io/gh/apache/beam/pull/12845?src=pr=h1) Report > Merging [#12845](https://codecov.io/gh/apache/beam/pull/12845?src=pr=desc) into [master](https://codecov.io/gh/apache/beam/commit/636bf8bf944b7e0c085480456a7a3df586255ab6?el=desc) will **increase** coverage by `0.00%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/beam/pull/12845?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #12845 +/- ## === Coverage 82.35% 82.35% === Files 450 450 Lines 5370253708+6 === + Hits4422544230+5 - Misses 9477 9478+1 ``` | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12845?src=pr=tree) | Coverage Δ | | |---|---|---| | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/12845/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `94.05% <0.00%> (-0.40%)` | :arrow_down: | | [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/12845/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `88.98% <0.00%> (-0.36%)` | :arrow_down: | | [...he\_beam/runners/interactive/pipeline\_instrument.py](https://codecov.io/gh/apache/beam/pull/12845/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9waXBlbGluZV9pbnN0cnVtZW50LnB5) | `98.56% <0.00%> (-0.24%)` | :arrow_down: | | [...on/apache\_beam/runners/direct/sdf\_direct\_runner.py](https://codecov.io/gh/apache/beam/pull/12845/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3Qvc2RmX2RpcmVjdF9ydW5uZXIucHk=) | `36.06% <0.00%> (-0.15%)` | :arrow_down: | | [...python/apache\_beam/runners/direct/direct\_runner.py](https://codecov.io/gh/apache/beam/pull/12845/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvZGlyZWN0X3J1bm5lci5weQ==) | `93.70% <0.00%> (+0.02%)` | :arrow_up: | | [sdks/python/apache\_beam/pipeline.py](https://codecov.io/gh/apache/beam/pull/12845/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcGlwZWxpbmUucHk=) | `88.83% <0.00%> (+0.03%)` | :arrow_up: | | [...ache\_beam/runners/dataflow/ptransform\_overrides.py](https://codecov.io/gh/apache/beam/pull/12845/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9wdHJhbnNmb3JtX292ZXJyaWRlcy5weQ==) | `90.00% <0.00%> (+0.14%)` | :arrow_up: | | [...eam/runners/interactive/caching/streaming\_cache.py](https://codecov.io/gh/apache/beam/pull/12845/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9jYWNoaW5nL3N0cmVhbWluZ19jYWNoZS5weQ==) | `95.33% <0.00%> (+1.27%)` | :arrow_up: | | [...ive/messaging/interactive\_environment\_inspector.py](https://codecov.io/gh/apache/beam/pull/12845/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9tZXNzYWdpbmcvaW50ZXJhY3RpdmVfZW52aXJvbm1lbnRfaW5zcGVjdG9yLnB5) | `97.43% <0.00%> (+1.28%)` | :arrow_up: | | [...am/runners/interactive/options/capture\_limiters.py](https://codecov.io/gh/apache/beam/pull/12845/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9vcHRpb25zL2NhcHR1cmVfbGltaXRlcnMucHk=) | `93.54% <0.00%> (+1.61%)` | :arrow_up: | | ... and [1 more](https://codecov.io/gh/apache/beam/pull/12845/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12845?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12845?src=pr=footer). Last update [636bf8b...e0e5fb5](https://codecov.io/gh/apache/beam/pull/12845?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] scwhittle commented on pull request #12825: [BEAM-9399] Change DataflowWorkerLoggingHandler to report errors to t…
scwhittle commented on pull request #12825: URL: https://github.com/apache/beam/pull/12825#issuecomment-691867771 I am only changing how errors from within the DataflowWorkerLoggingHandler itself are reported, for example an error publishing to stackdriver. I agree that logging to stderr is difficult to view in the UI, but that seems separate from the deadlock we originally fixed (the Jira has more details) and the erroneous precondition introduced when that was added. Kenn, does the limited scope of this change address your concerns? I am not changing it to use the original System.err generally This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] rworley-monster commented on a change in pull request #12367: [BEAM-10564] Support more Avro field name formats when mapping to Jav…
rworley-monster commented on a change in pull request #12367:
URL: https://github.com/apache/beam/pull/12367#discussion_r487819806
##
File path:
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/AvroUtils.java
##
@@ -601,12 +601,26 @@ private void readObject(ObjectInputStream in) throws
IOException, ClassNotFoundE
private Map getMapping(Schema schema) {
Map mapping = Maps.newHashMap();
for (Field field : schema.getFields()) {
-String underscore =
CaseFormat.LOWER_UNDERSCORE.to(CaseFormat.LOWER_CAMEL, field.getName());
-mapping.put(underscore, field.getName());
+String fieldName = field.getName();
+String getter;
+if (fieldName.contains("_")) {
+ if (Character.isLowerCase(fieldName.charAt(0))) {
+// field_name -> fieldName
+getter = CaseFormat.LOWER_UNDERSCORE.to(CaseFormat.LOWER_CAMEL,
fieldName);
+ } else {
+// FIELD_NAME -> fIELDNAME
+getter = CaseFormat.UPPER_CAMEL.to(CaseFormat.LOWER_CAMEL,
fieldName.replace("_", ""));
Review comment:
I have added an extended comment for this conversion and updated the
existing test Avro schema to cover the new mappable field name styles. Can you
please confirm that this is acceptable?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [beam] piotr-szuberski opened a new pull request #12839: [BEAM-10893] Add Json support to Kafka Table Provider
piotr-szuberski opened a new pull request #12839: URL: https://github.com/apache/beam/pull/12839 Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2 --- | --- | --- | --- | --- | --- | --- Go | [](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | --- Java | [](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/) Python | [](https://ci-beam.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[![Build
[GitHub] [beam] piotr-szuberski commented on pull request #12839: [BEAM-10893] Add Json support to Kafka Table Provider
piotr-szuberski commented on pull request #12839: URL: https://github.com/apache/beam/pull/12839#issuecomment-692057876 Run SQL PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] piotr-szuberski commented on pull request #12838: [BEAM-10892] Add Proto support to Kafka Table Provider
piotr-szuberski commented on pull request #12838: URL: https://github.com/apache/beam/pull/12838#issuecomment-692058196 @TheNeuralBit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] piotr-szuberski commented on pull request #12838: [BEAM-10892] Add Proto support to Kafka Table Provider
piotr-szuberski commented on pull request #12838: URL: https://github.com/apache/beam/pull/12838#issuecomment-692057621 Run SQL PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] runningcode opened a new pull request #12840: [BEAM-10886] Also publish build scans from Github actions.
runningcode opened a new pull request #12840: URL: https://github.com/apache/beam/pull/12840 This will help in investigating the issue in the failing build on windows and other future Github Actions failures. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2 --- | --- | --- | --- | --- | --- | --- Go | [](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | --- Java | [](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/) Python | [](https://ci-beam.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[![Build
[GitHub] [beam] pabloem commented on pull request #12779: [BEAM-10856] Support for NestedValueProvider for Python SDK
pabloem commented on pull request #12779: URL: https://github.com/apache/beam/pull/12779#issuecomment-692107896 A user has written a feature that they would find useful, and that will not change the experience for other users (if anything, it should improve it). The feature looks correct, and similar to what we do in Java. If we reject the PR, we may push the user to run on a fork. Can we let this in? @tvalentyn Flex templates have numerous benefits that are not being undone or discounted by adding this improvement to traditional templates, so I don't see a great disadvantage to merge this. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12731: [BEAM-10830] Twister2 quickstarts and the runner maven archetypes
codecov[bot] edited a comment on pull request #12731: URL: https://github.com/apache/beam/pull/12731#issuecomment-683376404 # [Codecov](https://codecov.io/gh/apache/beam/pull/12731?src=pr=h1) Report > Merging [#12731](https://codecov.io/gh/apache/beam/pull/12731?src=pr=desc) into [master](https://codecov.io/gh/apache/beam/commit/1d25e2ebeb4a0f74278dbd0cfbaa00f36abd73dc?el=desc) will **not change** coverage. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/beam/pull/12731?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #12731 +/- ## === Coverage 40.22% 40.22% === Files 454 454 Lines 5366953669 === Hits2158721587 Misses 3208232082 ``` -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12731?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12731?src=pr=footer). Last update [1d25e2e...8339055](https://codecov.io/gh/apache/beam/pull/12731?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] piotr-szuberski opened a new pull request #12838: [BEAM-10892] Add Proto support to Kafka Table Provider
piotr-szuberski opened a new pull request #12838: URL: https://github.com/apache/beam/pull/12838 Added support for protocol buffers to Kafka Table Providers Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2 --- | --- | --- | --- | --- | --- | --- Go | [](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | --- Java | [](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/) Python | [](https://ci-beam.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[![Build
[GitHub] [beam] piotr-szuberski commented on pull request #12827: [BEAM-10885] Add Avro support to Kafka table provider
piotr-szuberski commented on pull request #12827: URL: https://github.com/apache/beam/pull/12827#issuecomment-692003556 Run SQL PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] rezarokni commented on a change in pull request #12645: [BEAM-10124] Add ContextualTextIO
rezarokni commented on a change in pull request #12645:
URL: https://github.com/apache/beam/pull/12645#discussion_r487899783
##
File path:
sdks/java/io/contextual-text-io/src/main/java/org/apache/beam/sdk/io/contextualtextio/RecordWithMetadata.java
##
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.contextualtextio;
+
+import com.google.auto.value.AutoValue;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.schemas.AutoValueSchema;
+import org.apache.beam.sdk.schemas.annotations.DefaultSchema;
+
+/**
+ * Helper Class based on {@link AutoValueSchema}, it provides Metadata
associated with each Record
+ * when reading from file(s) using {@link ContextualTextIO}.
+ *
+ * Fields:
+ *
+ *
+ * recordOffset: The offset of a record (the byte at which the record
begins) in a file. This
+ * information can be useful if you wish to reconstruct the file. {@link
+ * RecordWithMetadata#getRecordOffset()}
+ * recordNum: The ordinal number of the record in its file. {@link
+ * RecordWithMetadata#getRecordNum()}
+ * recordValue: The value / contents of the record {@link
RecordWithMetadata#getRecordValue()}
+ * rangeOffset: The starting offset of the range (split), which
contained the record, when the
+ * record was read. {@link RecordWithMetadata#getRangeOffset()}
+ * recordNumInOffset: The record number relative to the Range. (line
number within the range)
+ * {@link RecordWithMetadata#getRecordNumInOffset()}
+ * fileName: Name of the file to which the record belongs (this is the
full filename,
+ * eg:path/to/file.txt) {@link RecordWithMetadata#getFileName()}
+ *
+ */
+@Experimental(Experimental.Kind.SCHEMAS)
+@DefaultSchema(AutoValueSchema.class)
+@AutoValue
+public abstract class RecordWithMetadata {
+ public abstract Long getRecordOffset();
+
+ public abstract Long getRecordNum();
+
+ public abstract String getRecordValue();
+
+ public abstract Long getRangeOffset();
+
+ public abstract Long getRecordNumInOffset();
+
+ public abstract Builder toBuilder();
+
+ public abstract String getFileName();
Review comment:
See note above as to why we cant use ResourceID
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [beam] piotr-szuberski commented on pull request #12827: [BEAM-10885] Add Avro support to Kafka table provider
piotr-szuberski commented on pull request #12827: URL: https://github.com/apache/beam/pull/12827#issuecomment-692058121 @TheNeuralBit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] piotr-szuberski commented on pull request #12839: [BEAM-10893] Add Json support to Kafka Table Provider
piotr-szuberski commented on pull request #12839: URL: https://github.com/apache/beam/pull/12839#issuecomment-692058319 @TheNeuralBit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] pabloem merged pull request #12686: Removing dead code from Interactive Beam.
pabloem merged pull request #12686: URL: https://github.com/apache/beam/pull/12686 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #12686: Removing dead code from Interactive Beam.
pabloem commented on pull request #12686: URL: https://github.com/apache/beam/pull/12686#issuecomment-692088106 Thanks @KevinGG @qinyeli ! We can merge as is. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #12721: [BEAM-10871] Add deidentify for FhirIO connector
pabloem commented on pull request #12721: URL: https://github.com/apache/beam/pull/12721#issuecomment-692081450 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] piotr-szuberski commented on pull request #12838: [BEAM-10892] Add Proto support to Kafka Table Provider
piotr-szuberski commented on pull request #12838: URL: https://github.com/apache/beam/pull/12838#issuecomment-692003648 Run SQL PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] rezarokni commented on a change in pull request #12645: [BEAM-10124] Add ContextualTextIO
rezarokni commented on a change in pull request #12645:
URL: https://github.com/apache/beam/pull/12645#discussion_r487897086
##
File path:
sdks/java/io/contextual-text-io/src/main/java/org/apache/beam/sdk/io/contextualtextio/ContextualTextIOSource.java
##
@@ -0,0 +1,364 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.contextualtextio;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.channels.ReadableByteChannel;
+import java.nio.channels.SeekableByteChannel;
+import java.util.NoSuchElementException;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.io.FileBasedSource;
+import org.apache.beam.sdk.io.fs.EmptyMatchTreatment;
+import org.apache.beam.sdk.io.fs.MatchResult;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.ValueProvider;
+import org.apache.beam.sdk.schemas.NoSuchSchemaException;
+import org.apache.beam.sdk.schemas.SchemaCoder;
+import org.apache.beam.sdk.schemas.SchemaRegistry;
+import org.apache.beam.vendor.grpc.v1p26p0.com.google.protobuf.ByteString;
+import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
+import org.checkerframework.checker.nullness.qual.Nullable;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation detail of {@link ContextualTextIO.Read}.
+ *
+ * A {@link FileBasedSource} which can decode records delimited by newline
characters.
+ *
+ * This source splits the data into records using {@code UTF-8} {@code \n},
{@code \r}, or {@code
+ * \r\n} as the delimiter. This source is not strict and supports decoding the
last record even if
+ * it is not delimited. Finally, no records are decoded if the stream is empty.
+ *
+ * This source supports reading from any arbitrary byte position within the
stream. If the
+ * starting position is not {@code 0}, then bytes are skipped until the first
delimiter is found
+ * representing the beginning of the first record to be decoded.
+ */
+@VisibleForTesting
+class ContextualTextIOSource extends FileBasedSource {
+ byte[] delimiter;
+
+ private static final Logger LOG =
LoggerFactory.getLogger(ContextualTextIOSource.class);
+
+ // Used to Override isSplittable
+ private boolean hasMultilineCSVRecords;
+
+ @Override
+ protected boolean isSplittable() throws Exception {
+if (hasMultilineCSVRecords) {
+ // When Having Multiline CSV Records,
+ // Splitting the file may cause a split to be within a record,
+ // Disabling split prevents this from happening
+ return false;
+}
+return super.isSplittable();
+ }
+
+ ContextualTextIOSource(
+ ValueProvider fileSpec,
+ EmptyMatchTreatment emptyMatchTreatment,
+ byte[] delimiter,
+ boolean hasMultilineCSVRecords) {
+super(fileSpec, emptyMatchTreatment, 1L);
+this.delimiter = delimiter;
+this.hasMultilineCSVRecords = hasMultilineCSVRecords;
+ }
+
+ private ContextualTextIOSource(
+ MatchResult.Metadata metadata,
+ long start,
+ long end,
+ byte[] delimiter,
+ boolean hasMultilineCSVRecords) {
+super(metadata, 1L, start, end);
+this.delimiter = delimiter;
+this.hasMultilineCSVRecords = hasMultilineCSVRecords;
+ }
+
+ @Override
+ protected FileBasedSource createForSubrangeOfFile(
+ MatchResult.Metadata metadata, long start, long end) {
+return new ContextualTextIOSource(metadata, start, end, delimiter,
hasMultilineCSVRecords);
+ }
+
+ @Override
+ protected FileBasedReader
createSingleFileReader(PipelineOptions options) {
+return new MultiLineTextBasedReader(this, delimiter,
hasMultilineCSVRecords);
+ }
+
+ @Override
+ public Coder getOutputCoder() {
+SchemaCoder coder = null;
+try {
+ coder =
SchemaRegistry.createDefault().getSchemaCoder(RecordWithMetadata.class);
+} catch (NoSuchSchemaException e) {
+ LOG.error("No Coder Found for RecordWithMetadata");
+}
+return coder;
+ }
+
+ /**
+ * A {@link FileBasedReader FileBasedReader} which can decode records
delimited by delimiter
+
[GitHub] [beam] iemejia merged pull request #12744: [BEAM-10463] Twister2 Beam Runner Documentation
iemejia merged pull request #12744: URL: https://github.com/apache/beam/pull/12744 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] rezarokni commented on a change in pull request #12645: [BEAM-10124] Add ContextualTextIO
rezarokni commented on a change in pull request #12645:
URL: https://github.com/apache/beam/pull/12645#discussion_r487927710
##
File path:
sdks/java/io/contextual-text-io/src/main/java/org/apache/beam/sdk/io/contextualtextio/RecordWithMetadata.java
##
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.contextualtextio;
+
+import com.google.auto.value.AutoValue;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.schemas.AutoValueSchema;
+import org.apache.beam.sdk.schemas.annotations.DefaultSchema;
+
+/**
+ * Helper Class based on {@link AutoValueSchema}, it provides Metadata
associated with each Record
+ * when reading from file(s) using {@link ContextualTextIO}.
+ *
+ * Fields:
+ *
+ *
+ * recordOffset: The offset of a record (the byte at which the record
begins) in a file. This
+ * information can be useful if you wish to reconstruct the file. {@link
+ * RecordWithMetadata#getRecordOffset()}
+ * recordNum: The ordinal number of the record in its file. {@link
+ * RecordWithMetadata#getRecordNum()}
+ * recordValue: The value / contents of the record {@link
RecordWithMetadata#getRecordValue()}
+ * rangeOffset: The starting offset of the range (split), which
contained the record, when the
+ * record was read. {@link RecordWithMetadata#getRangeOffset()}
+ * recordNumInOffset: The record number relative to the Range. (line
number within the range)
+ * {@link RecordWithMetadata#getRecordNumInOffset()}
+ * fileName: Name of the file to which the record belongs (this is the
full filename,
+ * eg:path/to/file.txt) {@link RecordWithMetadata#getFileName()}
+ *
+ */
+@Experimental(Experimental.Kind.SCHEMAS)
+@DefaultSchema(AutoValueSchema.class)
+@AutoValue
+public abstract class RecordWithMetadata {
Review comment:
@lukecwik This class is a Schema class, looking into this we cant make
it generic, until schemas supports generics. But for ContextualTextIO we would
be setting it to ContextualTextIO anyway ... So while it will be useful
in the future, I am not sure it adds much for now? WDYT?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [beam] rezarokni commented on a change in pull request #12645: [BEAM-10124] Add ContextualTextIO
rezarokni commented on a change in pull request #12645:
URL: https://github.com/apache/beam/pull/12645#discussion_r487934071
##
File path:
sdks/java/io/contextual-text-io/src/main/java/org/apache/beam/sdk/io/contextualtextio/ContextualTextIO.java
##
@@ -0,0 +1,631 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.contextualtextio;
+
+import static org.apache.beam.sdk.io.FileIO.ReadMatches.DirectoryTreatment;
+import static
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
+
+import com.google.auto.value.AutoValue;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.SortedMap;
+import java.util.TreeMap;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
+import org.apache.beam.sdk.io.CompressedSource;
+import org.apache.beam.sdk.io.Compression;
+import org.apache.beam.sdk.io.FileBasedSource;
+import org.apache.beam.sdk.io.FileIO;
+import org.apache.beam.sdk.io.FileIO.MatchConfiguration;
+import org.apache.beam.sdk.io.ReadAllViaFileBasedSource;
+import org.apache.beam.sdk.io.TextIO;
+import org.apache.beam.sdk.io.fs.EmptyMatchTreatment;
+import org.apache.beam.sdk.options.ValueProvider;
+import org.apache.beam.sdk.options.ValueProvider.StaticValueProvider;
+import org.apache.beam.sdk.schemas.NoSuchSchemaException;
+import org.apache.beam.sdk.schemas.SchemaCoder;
+import org.apache.beam.sdk.transforms.Count;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.SerializableFunction;
+import org.apache.beam.sdk.transforms.View;
+import org.apache.beam.sdk.transforms.Watch.Growth.TerminationCondition;
+import org.apache.beam.sdk.transforms.display.DisplayData;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PBegin;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionView;
+import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.checkerframework.checker.nullness.qual.Nullable;
+import org.joda.time.Duration;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * {@link PTransform}s that read text files and collect contextual information
of the elements in
+ * the input.
+ *
+ * Use {@link TextIO} when not reading file with Multiline Records or
additional metadata is not
+ * required.
+ *
+ * Reading from text files
+ *
+ * To read a {@link PCollection} from one or more text files, use {@code
+ * ContextualTextIO.read()}. To instantiate a transform use {@link
+ * ContextualTextIO.Read#from(String)} and specify the path of the file(s) to
be read.
+ * Alternatively, if the filenames to be read are themselves in a {@link
PCollection} you can use
+ * {@link FileIO} to match them and {@link ContextualTextIO#readFiles()} to
read them.
+ *
+ * {@link #read} returns a {@link PCollection} of {@link RecordWithMetadata
RecordWithMetadata},
+ * each corresponding to one line of an input UTF-8 text file (split into
lines delimited by '\n',
+ * '\r', '\r\n', or specified delimiter see {@link
ContextualTextIO.Read#withDelimiter})
+ *
+ * Filepattern expansion and watching
+ *
+ * By default, the filepatterns are expanded only once. The combination of
{@link
+ * FileIO.Match#continuously(Duration, TerminationCondition)} and {@link
#readFiles()} allow
+ * streaming of new files matching the filepattern(s).
+ *
+ * By default, {@link #read} prohibits filepatterns that match no files,
and {@link #readFiles()}
+ * allows them in case the filepattern contains a glob wildcard character. Use
{@link
+ * ContextualTextIO.Read#withEmptyMatchTreatment} or {@link
+ * FileIO.Match#withEmptyMatchTreatment(EmptyMatchTreatment)} plus {@link
#readFiles()} to configure
+ * this behavior.
+ *
+ * Example 1: reading a file or filepattern.
+ *
+ * {@code
+ * Pipeline p =
[GitHub] [beam] rezarokni commented on a change in pull request #12645: [BEAM-10124] Add ContextualTextIO
rezarokni commented on a change in pull request #12645:
URL: https://github.com/apache/beam/pull/12645#discussion_r487933083
##
File path:
sdks/java/io/contextual-text-io/src/main/java/org/apache/beam/sdk/io/contextualtextio/ContextualTextIO.java
##
@@ -0,0 +1,631 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.contextualtextio;
+
+import static org.apache.beam.sdk.io.FileIO.ReadMatches.DirectoryTreatment;
+import static
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
+
+import com.google.auto.value.AutoValue;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.SortedMap;
+import java.util.TreeMap;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
+import org.apache.beam.sdk.io.CompressedSource;
+import org.apache.beam.sdk.io.Compression;
+import org.apache.beam.sdk.io.FileBasedSource;
+import org.apache.beam.sdk.io.FileIO;
+import org.apache.beam.sdk.io.FileIO.MatchConfiguration;
+import org.apache.beam.sdk.io.ReadAllViaFileBasedSource;
+import org.apache.beam.sdk.io.TextIO;
+import org.apache.beam.sdk.io.fs.EmptyMatchTreatment;
+import org.apache.beam.sdk.options.ValueProvider;
+import org.apache.beam.sdk.options.ValueProvider.StaticValueProvider;
+import org.apache.beam.sdk.schemas.NoSuchSchemaException;
+import org.apache.beam.sdk.schemas.SchemaCoder;
+import org.apache.beam.sdk.transforms.Count;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.SerializableFunction;
+import org.apache.beam.sdk.transforms.View;
+import org.apache.beam.sdk.transforms.Watch.Growth.TerminationCondition;
+import org.apache.beam.sdk.transforms.display.DisplayData;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PBegin;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionView;
+import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.checkerframework.checker.nullness.qual.Nullable;
+import org.joda.time.Duration;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * {@link PTransform}s that read text files and collect contextual information
of the elements in
+ * the input.
+ *
+ * Use {@link TextIO} when not reading file with Multiline Records or
additional metadata is not
+ * required.
+ *
+ * Reading from text files
+ *
+ * To read a {@link PCollection} from one or more text files, use {@code
+ * ContextualTextIO.read()}. To instantiate a transform use {@link
+ * ContextualTextIO.Read#from(String)} and specify the path of the file(s) to
be read.
+ * Alternatively, if the filenames to be read are themselves in a {@link
PCollection} you can use
+ * {@link FileIO} to match them and {@link ContextualTextIO#readFiles()} to
read them.
+ *
+ * {@link #read} returns a {@link PCollection} of {@link RecordWithMetadata
RecordWithMetadata},
+ * each corresponding to one line of an input UTF-8 text file (split into
lines delimited by '\n',
+ * '\r', '\r\n', or specified delimiter see {@link
ContextualTextIO.Read#withDelimiter})
+ *
+ * Filepattern expansion and watching
+ *
+ * By default, the filepatterns are expanded only once. The combination of
{@link
+ * FileIO.Match#continuously(Duration, TerminationCondition)} and {@link
#readFiles()} allow
+ * streaming of new files matching the filepattern(s).
+ *
+ * By default, {@link #read} prohibits filepatterns that match no files,
and {@link #readFiles()}
+ * allows them in case the filepattern contains a glob wildcard character. Use
{@link
+ * ContextualTextIO.Read#withEmptyMatchTreatment} or {@link
+ * FileIO.Match#withEmptyMatchTreatment(EmptyMatchTreatment)} plus {@link
#readFiles()} to configure
+ * this behavior.
+ *
+ * Example 1: reading a file or filepattern.
+ *
+ * {@code
+ * Pipeline p =
[GitHub] [beam] iemejia commented on a change in pull request #12731: [BEAM-10830] Twister2 quickstarts and the runner maven archetypes
iemejia commented on a change in pull request #12731:
URL: https://github.com/apache/beam/pull/12731#discussion_r487696110
##
File path:
runners/twister2/src/main/java/org/apache/beam/runners/twister2/Twister2PipelineResult.java
##
@@ -17,15 +17,21 @@
*/
package org.apache.beam.runners.twister2;
+import edu.iu.dsc.tws.api.scheduler.Twister2JobState;
import java.io.IOException;
import org.apache.beam.sdk.PipelineResult;
import org.apache.beam.sdk.metrics.MetricResults;
import org.joda.time.Duration;
+import org.mortbay.log.Log;
Review comment:
Is there any particular reason to use a different logger kind here?
We use slf4j in most of the Beam code base for logging so maybe good to
follow this on the Twister2 runner. Notice that this can be done in a separate
PR.
##
File path:
sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml
##
@@ -281,6 +281,17 @@
+
+ twister2-runner
Review comment:
:+1:
##
File path:
runners/twister2/src/main/java/org/apache/beam/runners/twister2/Twister2Runner.java
##
@@ -119,19 +129,27 @@ public PipelineResult run(Pipeline pipeline) {
.addComputeResource(options.getWorkerCPUs(),
options.getRamMegaBytes(), workers)
.setConfig(jobConfig)
.build();
-Twister2JobState jobState = Twister2Submitter.submitJob(twister2Job,
config);
-Twister2PipelineResult result = new Twister2PipelineResult();
-// TODO: Need to fix the check for "RUNNING" once fix for this is done on
Twister2 end.
-if (jobState.getJobstate() == DriverJobState.FAILED
-|| jobState.getJobstate() == DriverJobState.RUNNING) {
- throw new RuntimeException("Pipeline execution failed",
jobState.getCause());
+Twister2JobState jobState;
+if (isLocalMode(options)) {
+ jobState = LocalSubmitter.submitJob(twister2Job, config);
} else {
- result.setState(PipelineResult.State.DONE);
+ jobState = Twister2Submitter.submitJob(twister2Job, config);
}
+
+Twister2PipelineResult result = new Twister2PipelineResult(jobState);
return result;
}
+ /** Check if the Runner is set to use Twister local mode or pointing to a
deployment. */
+ private boolean isLocalMode(Twister2PipelineOptions options) {
+if (options.getTwister2Home() == null ||
"".equals(options.getTwister2Home())) {
Review comment:
with guava (use vendor if so) this can be simpler, but that's not
blocking at all for this PR
`return Strings.isNullOrEmpty(options.getTwister2Home());`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [beam] iemejia merged pull request #12731: [BEAM-10830] Twister2 quickstarts and the runner maven archetypes
iemejia merged pull request #12731: URL: https://github.com/apache/beam/pull/12731 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] iemejia commented on pull request #12731: [BEAM-10830] Twister2 quickstarts and the runner maven archetypes
iemejia commented on pull request #12731: URL: https://github.com/apache/beam/pull/12731#issuecomment-692041712 Run Twister2 ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] rezarokni commented on a change in pull request #12645: [BEAM-10124] Add ContextualTextIO
rezarokni commented on a change in pull request #12645:
URL: https://github.com/apache/beam/pull/12645#discussion_r487927710
##
File path:
sdks/java/io/contextual-text-io/src/main/java/org/apache/beam/sdk/io/contextualtextio/RecordWithMetadata.java
##
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.contextualtextio;
+
+import com.google.auto.value.AutoValue;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.schemas.AutoValueSchema;
+import org.apache.beam.sdk.schemas.annotations.DefaultSchema;
+
+/**
+ * Helper Class based on {@link AutoValueSchema}, it provides Metadata
associated with each Record
+ * when reading from file(s) using {@link ContextualTextIO}.
+ *
+ * Fields:
+ *
+ *
+ * recordOffset: The offset of a record (the byte at which the record
begins) in a file. This
+ * information can be useful if you wish to reconstruct the file. {@link
+ * RecordWithMetadata#getRecordOffset()}
+ * recordNum: The ordinal number of the record in its file. {@link
+ * RecordWithMetadata#getRecordNum()}
+ * recordValue: The value / contents of the record {@link
RecordWithMetadata#getRecordValue()}
+ * rangeOffset: The starting offset of the range (split), which
contained the record, when the
+ * record was read. {@link RecordWithMetadata#getRangeOffset()}
+ * recordNumInOffset: The record number relative to the Range. (line
number within the range)
+ * {@link RecordWithMetadata#getRecordNumInOffset()}
+ * fileName: Name of the file to which the record belongs (this is the
full filename,
+ * eg:path/to/file.txt) {@link RecordWithMetadata#getFileName()}
+ *
+ */
+@Experimental(Experimental.Kind.SCHEMAS)
+@DefaultSchema(AutoValueSchema.class)
+@AutoValue
+public abstract class RecordWithMetadata {
Review comment:
@lukecwik This class is a Schema class, looking into this we cant make
it generic, until schemas supports generics. But for ContextualTextIO we would
be setting it to ContextualTextIO`` anyway ... So while it will be
useful in the future, I am not sure it adds much for now? WDYT?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [beam] piotr-szuberski commented on pull request #12827: [BEAM-10885] Add Avro support to Kafka table provider
piotr-szuberski commented on pull request #12827: URL: https://github.com/apache/beam/pull/12827#issuecomment-692062465 Run SQL PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #12813: write to file ability for java suite
pabloem commented on pull request #12813: URL: https://github.com/apache/beam/pull/12813#issuecomment-692116395 this LGTM. Waiting for @y1chi if he has any comments : ) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] kennknowles merged pull request #12840: [BEAM-10886] Also publish build scans from Github actions.
kennknowles merged pull request #12840: URL: https://github.com/apache/beam/pull/12840 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] lukecwik commented on pull request #12616: [BEAM-10670] Update Jet to be opt-out for SplittableDoFn powering the Read transform.
lukecwik commented on pull request #12616: URL: https://github.com/apache/beam/pull/12616#issuecomment-692196137 > What kind of test coverage is there for these transform replacements on the Jet runner? The `@ValidatesRunner` junit category gives us coverage. The `@ValidatesRunner` tests within https://github.com/apache/beam/blob/master/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/SplittableDoFnTest.java are comprehensive for this feature but Jet can't support them all due to lack of features and or/bugs (e.g. support for bundle finalization, assuming byte[] is equals comparable, ...). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] kennknowles commented on a change in pull request #12540: [BEAM-10620] Eliminate nullability errors from :sdks:java:extensions:sql:perf-tests
kennknowles commented on a change in pull request #12540:
URL: https://github.com/apache/beam/pull/12540#discussion_r488104689
##
File path:
sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/IOTestPipelineOptions.java
##
@@ -45,19 +45,16 @@
void setBigQueryTable(@Nullable String tableName);
@Description("InfluxDB measurement to publish results to.")
- @Nullable
String getInfluxMeasurement();
void setInfluxMeasurement(@Nullable String measurement);
@Description("InfluxDB host.")
- @Nullable
String getInfluxHost();
void setInfluxHost(@Nullable String host);
@Description("InfluxDB database.")
- @Nullable
Review comment:
Ah that makes sense. What you should do, I think, is alter
`setInfluxDatabase` so the parameter is non-nullable. Otherwise somehow the
pipeline options would be magically turning `null` into non-`null`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [beam] rworley-monster commented on pull request #12367: [BEAM-10564] Support more Avro field name formats when mapping to Jav…
rworley-monster commented on pull request #12367: URL: https://github.com/apache/beam/pull/12367#issuecomment-692166530 I have looked into both of the failing checks and the first appears to be a syntax error in `beam\examples\java\build.gradle` and the second I am not able to reproduce locally with the same schema and method call (`AvroUtils.toBeamRowStrict`). Though I haven't found the values that were used for the `GenericRecord`, so that could make the difference. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] lukecwik commented on pull request #12825: [BEAM-9399] Change DataflowWorkerLoggingHandler to report errors to t…
lukecwik commented on pull request #12825: URL: https://github.com/apache/beam/pull/12825#issuecomment-692173705 @scwhittle I see how the deadlock occurs but why introduce the synchronized(buffer) if we are swapping out the error manager? Also, wouldn't we solve the locking problem if we always held the same "flush" lock when interacting with the error manager? We could do this by sharing a single lock object across the error manager, print stream, ... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12811: [BEAM-10705] Fixes a bug when passing whl files in --sdk_location from https locations
codecov[bot] edited a comment on pull request #12811: URL: https://github.com/apache/beam/pull/12811#issuecomment-690521374 # [Codecov](https://codecov.io/gh/apache/beam/pull/12811?src=pr=h1) Report > Merging [#12811](https://codecov.io/gh/apache/beam/pull/12811?src=pr=desc) into [master](https://codecov.io/gh/apache/beam/commit/d4e3729460dc710f954e22c92c36e8f00a24bd4c?el=desc) will **increase** coverage by `41.78%`. > The diff coverage is `100.00%`. [](https://codecov.io/gh/apache/beam/pull/12811?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #12811 +/- ## === + Coverage 40.26% 82.05% +41.78% === Files 455 457+2 Lines 5382254222 +400 === + Hits2167444492+22818 + Misses 32148 9730-22418 ``` | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12811?src=pr=tree) | Coverage Δ | | |---|---|---| | [...s/python/apache\_beam/runners/portability/stager.py](https://codecov.io/gh/apache/beam/pull/12811/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9zdGFnZXIucHk=) | `82.55% <100.00%> (+45.55%)` | :arrow_up: | | [...apache\_beam/portability/api/beam\_runner\_api\_pb2.py](https://codecov.io/gh/apache/beam/pull/12811/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjIucHk=) | `100.00% <0.00%> (ø)` | | | [...he\_beam/testing/benchmarks/nexmark/nexmark\_util.py](https://codecov.io/gh/apache/beam/pull/12811/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvbmV4bWFya191dGlsLnB5) | `0.00% <0.00%> (ø)` | | | [...e\_beam/portability/api/beam\_runner\_api\_pb2\_urns.py](https://codecov.io/gh/apache/beam/pull/12811/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjJfdXJucy5weQ==) | `100.00% <0.00%> (ø)` | | | [...eam/testing/benchmarks/nexmark/nexmark\_launcher.py](https://codecov.io/gh/apache/beam/pull/12811/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvbmV4bWFya19sYXVuY2hlci5weQ==) | `0.00% <0.00%> (ø)` | | | [sdks/python/apache\_beam/utils/histogram.py](https://codecov.io/gh/apache/beam/pull/12811/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvaGlzdG9ncmFtLnB5) | `94.28% <0.00%> (ø)` | | | [...he\_beam/testing/benchmarks/nexmark/nexmark\_perf.py](https://codecov.io/gh/apache/beam/pull/12811/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvbmV4bWFya19wZXJmLnB5) | `0.00% <0.00%> (ø)` | | | [sdks/python/apache\_beam/coders/slow\_stream.py](https://codecov.io/gh/apache/beam/pull/12811/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL3Nsb3dfc3RyZWFtLnB5) | `92.43% <0.00%> (+1.68%)` | :arrow_up: | | [sdks/python/apache\_beam/utils/profiler.py](https://codecov.io/gh/apache/beam/pull/12811/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvcHJvZmlsZXIucHk=) | `32.11% <0.00%> (+1.83%)` | :arrow_up: | | [...on/apache\_beam/runners/direct/sdf\_direct\_runner.py](https://codecov.io/gh/apache/beam/pull/12811/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3Qvc2RmX2RpcmVjdF9ydW5uZXIucHk=) | `36.21% <0.00%> (+2.46%)` | :arrow_up: | | ... and [275 more](https://codecov.io/gh/apache/beam/pull/12811/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12811?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12811?src=pr=footer). Last update [d4e3729...a99c520](https://codecov.io/gh/apache/beam/pull/12811?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #12721: [BEAM-10871] Add deidentify for FhirIO connector
pabloem commented on pull request #12721: URL: https://github.com/apache/beam/pull/12721#issuecomment-692114831 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] yifanmai commented on pull request #12656: [BEAM-10781] Add PTransformOverride.get_replacement_transform_for_applied_ptransform()
yifanmai commented on pull request #12656: URL: https://github.com/apache/beam/pull/12656#issuecomment-692172601 The failures are on `DataChannelTest.test_time_based_flush_grpc_data_channel` which does not seem related to the PR. @chamikaramj is this ready for merge or is there more testing needed? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] y1chi commented on pull request #12807: [BEAM-2855] implement query 10
y1chi commented on pull request #12807: URL: https://github.com/apache/beam/pull/12807#issuecomment-692203332 @pabloem Pablo could you help merge this? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] lukecwik commented on pull request #12367: [BEAM-10564] Support more Avro field name formats when mapping to Jav…
lukecwik commented on pull request #12367: URL: https://github.com/apache/beam/pull/12367#issuecomment-692185073 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] nehsyc commented on pull request #12726: [BEAM-10475] Add max buffering duration to GroupIntoBatches (Java)
nehsyc commented on pull request #12726: URL: https://github.com/apache/beam/pull/12726#issuecomment-692200394 > Please fix spotlessCheck and squash all commits into me. Then I can merge it. Thanks! Done. Thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] boyuanzz commented on pull request #12773: [BEAM-10505][BEAM-10530] Add truncate capability.
boyuanzz commented on pull request #12773: URL: https://github.com/apache/beam/pull/12773#issuecomment-692206185 Kindly pinging : ) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] boyuanzz commented on pull request #12806: [BEAM-10869] Use PubsubMessagePayloadOnlyCoder when writing to Pubsub.
boyuanzz commented on pull request #12806: URL: https://github.com/apache/beam/pull/12806#issuecomment-692206454 Kindly pinging : ) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] lukecwik commented on pull request #12367: [BEAM-10564] Support more Avro field name formats when mapping to Jav…
lukecwik commented on pull request #12367: URL: https://github.com/apache/beam/pull/12367#issuecomment-692210480 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #12808: add readme file to python nexmark
pabloem commented on pull request #12808: URL: https://github.com/apache/beam/pull/12808#issuecomment-692113058 This looks fine to me. I'd rather it be more hand-holdy, but it has the commands we need, and Leiyi is done with internship, so I'm happy to merge. LMK what you think @y1chi This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] y1chi commented on pull request #12808: add readme file to python nexmark
y1chi commented on pull request #12808: URL: https://github.com/apache/beam/pull/12808#issuecomment-692169272 > This looks fine to me. I'd rather it be more hand-holdy, but it has the commands we need, and Leiyi is done with internship, so I'm happy to merge. LMK what you think @y1chi it looks fine to me as examples This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] robertwb commented on pull request #12841: [BEAM-10894] Basic CSV reading and writing.
robertwb commented on pull request #12841: URL: https://github.com/apache/beam/pull/12841#issuecomment-692174531 R: @chamikaramj This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] robertwb opened a new pull request #12841: [BEAM-10894] Basic CSV reading and writing.
robertwb opened a new pull request #12841: URL: https://github.com/apache/beam/pull/12841 Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2 --- | --- | --- | --- | --- | --- | --- Go | [](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | --- Java | [](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/) Python | [](https://ci-beam.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[![Build
[GitHub] [beam] lukecwik commented on pull request #12825: [BEAM-9399] Change DataflowWorkerLoggingHandler to report errors to t…
lukecwik commented on pull request #12825: URL: https://github.com/apache/beam/pull/12825#issuecomment-692175069 @kennknowles `system.err` is always intended to log at ERROR level. Are you saying that logging is being misconfigured somehow and logs for other severities are being sent to `system.err`? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] lukecwik commented on a change in pull request #12616: [BEAM-10670] Update Jet to be opt-out for SplittableDoFn powering the Read transform.
lukecwik commented on a change in pull request #12616:
URL: https://github.com/apache/beam/pull/12616#discussion_r488095723
##
File path: runners/jet/src/main/java/org/apache/beam/runners/jet/JetRunner.java
##
@@ -186,7 +189,15 @@ private JetInstance getJetInstance(JetPipelineOptions
options) {
}
private static List getDefaultOverrides() {
-return Collections.emptyList();
+return Arrays.asList(
Review comment:
There is no exhaustive list as each runner can choose to implement each
transform however they wish. We have to rely on the `@ValidatesRunner` suite of
tests for validation.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [beam] boyuanzz commented on pull request #12726: [BEAM-10475] Add max buffering duration to GroupIntoBatches (Java)
boyuanzz commented on pull request #12726: URL: https://github.com/apache/beam/pull/12726#issuecomment-692205395 Run Dataflow ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12811: [BEAM-10705] Fixes a bug when passing whl files in --sdk_location from https locations
codecov[bot] edited a comment on pull request #12811: URL: https://github.com/apache/beam/pull/12811#issuecomment-690521374 # [Codecov](https://codecov.io/gh/apache/beam/pull/12811?src=pr=h1) Report > Merging [#12811](https://codecov.io/gh/apache/beam/pull/12811?src=pr=desc) into [master](https://codecov.io/gh/apache/beam/commit/d4e3729460dc710f954e22c92c36e8f00a24bd4c?el=desc) will **increase** coverage by `41.73%`. > The diff coverage is `100.00%`. [](https://codecov.io/gh/apache/beam/pull/12811?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #12811 +/- ## === + Coverage 40.26% 82.00% +41.73% === Files 455 456+1 Lines 5382254025 +203 === + Hits2167444302+22628 + Misses 32148 9723-22425 ``` | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12811?src=pr=tree) | Coverage Δ | | |---|---|---| | [...s/python/apache\_beam/runners/portability/stager.py](https://codecov.io/gh/apache/beam/pull/12811/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9zdGFnZXIucHk=) | `82.55% <100.00%> (+45.55%)` | :arrow_up: | | [...apache\_beam/portability/api/beam\_runner\_api\_pb2.py](https://codecov.io/gh/apache/beam/pull/12811/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjIucHk=) | `100.00% <0.00%> (ø)` | | | [...he\_beam/testing/benchmarks/nexmark/nexmark\_util.py](https://codecov.io/gh/apache/beam/pull/12811/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvbmV4bWFya191dGlsLnB5) | `0.00% <0.00%> (ø)` | | | [...e\_beam/portability/api/beam\_runner\_api\_pb2\_urns.py](https://codecov.io/gh/apache/beam/pull/12811/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjJfdXJucy5weQ==) | `100.00% <0.00%> (ø)` | | | [...eam/testing/benchmarks/nexmark/nexmark\_launcher.py](https://codecov.io/gh/apache/beam/pull/12811/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvbmV4bWFya19sYXVuY2hlci5weQ==) | `0.00% <0.00%> (ø)` | | | [...ache\_beam/runners/interactive/pipeline\_analyzer.py](https://codecov.io/gh/apache/beam/pull/12811/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9waXBlbGluZV9hbmFseXplci5weQ==) | | | | [...he\_beam/testing/benchmarks/nexmark/nexmark\_perf.py](https://codecov.io/gh/apache/beam/pull/12811/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvbmV4bWFya19wZXJmLnB5) | `0.00% <0.00%> (ø)` | | | [sdks/python/apache\_beam/utils/histogram.py](https://codecov.io/gh/apache/beam/pull/12811/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvaGlzdG9ncmFtLnB5) | `94.28% <0.00%> (ø)` | | | [sdks/python/apache\_beam/coders/slow\_stream.py](https://codecov.io/gh/apache/beam/pull/12811/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL3Nsb3dfc3RyZWFtLnB5) | `92.43% <0.00%> (+1.68%)` | :arrow_up: | | [sdks/python/apache\_beam/utils/profiler.py](https://codecov.io/gh/apache/beam/pull/12811/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvcHJvZmlsZXIucHk=) | `32.11% <0.00%> (+1.83%)` | :arrow_up: | | ... and [275 more](https://codecov.io/gh/apache/beam/pull/12811/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12811?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12811?src=pr=footer). Last update [d4e3729...a99c520](https://codecov.io/gh/apache/beam/pull/12811?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12811: [BEAM-10705] Fixes a bug when passing whl files in --sdk_location from https locations
codecov[bot] edited a comment on pull request #12811: URL: https://github.com/apache/beam/pull/12811#issuecomment-690521374 # [Codecov](https://codecov.io/gh/apache/beam/pull/12811?src=pr=h1) Report > Merging [#12811](https://codecov.io/gh/apache/beam/pull/12811?src=pr=desc) into [master](https://codecov.io/gh/apache/beam/commit/d4e3729460dc710f954e22c92c36e8f00a24bd4c?el=desc) will **increase** coverage by `41.73%`. > The diff coverage is `100.00%`. [](https://codecov.io/gh/apache/beam/pull/12811?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #12811 +/- ## === + Coverage 40.26% 82.00% +41.73% === Files 455 456+1 Lines 5382254025 +203 === + Hits2167444302+22628 + Misses 32148 9723-22425 ``` | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12811?src=pr=tree) | Coverage Δ | | |---|---|---| | [...s/python/apache\_beam/runners/portability/stager.py](https://codecov.io/gh/apache/beam/pull/12811/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9zdGFnZXIucHk=) | `82.55% <100.00%> (+45.55%)` | :arrow_up: | | [...apache\_beam/portability/api/beam\_runner\_api\_pb2.py](https://codecov.io/gh/apache/beam/pull/12811/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjIucHk=) | `100.00% <0.00%> (ø)` | | | [...he\_beam/testing/benchmarks/nexmark/nexmark\_util.py](https://codecov.io/gh/apache/beam/pull/12811/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvbmV4bWFya191dGlsLnB5) | `0.00% <0.00%> (ø)` | | | [...e\_beam/portability/api/beam\_runner\_api\_pb2\_urns.py](https://codecov.io/gh/apache/beam/pull/12811/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjJfdXJucy5weQ==) | `100.00% <0.00%> (ø)` | | | [...eam/testing/benchmarks/nexmark/nexmark\_launcher.py](https://codecov.io/gh/apache/beam/pull/12811/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvbmV4bWFya19sYXVuY2hlci5weQ==) | `0.00% <0.00%> (ø)` | | | [...ache\_beam/runners/interactive/pipeline\_analyzer.py](https://codecov.io/gh/apache/beam/pull/12811/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9waXBlbGluZV9hbmFseXplci5weQ==) | | | | [...he\_beam/testing/benchmarks/nexmark/nexmark\_perf.py](https://codecov.io/gh/apache/beam/pull/12811/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvbmV4bWFya19wZXJmLnB5) | `0.00% <0.00%> (ø)` | | | [sdks/python/apache\_beam/utils/histogram.py](https://codecov.io/gh/apache/beam/pull/12811/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvaGlzdG9ncmFtLnB5) | `94.28% <0.00%> (ø)` | | | [sdks/python/apache\_beam/coders/slow\_stream.py](https://codecov.io/gh/apache/beam/pull/12811/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL3Nsb3dfc3RyZWFtLnB5) | `92.43% <0.00%> (+1.68%)` | :arrow_up: | | [sdks/python/apache\_beam/utils/profiler.py](https://codecov.io/gh/apache/beam/pull/12811/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvcHJvZmlsZXIucHk=) | `32.11% <0.00%> (+1.83%)` | :arrow_up: | | ... and [275 more](https://codecov.io/gh/apache/beam/pull/12811/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12811?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12811?src=pr=footer). Last update [d4e3729...a99c520](https://codecov.io/gh/apache/beam/pull/12811?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] kennknowles commented on pull request #12825: [BEAM-9399] Change DataflowWorkerLoggingHandler to report errors to t…
kennknowles commented on pull request #12825: URL: https://github.com/apache/beam/pull/12825#issuecomment-692236055 @lukecwik that is how I feel about System.err as well. Java disagrees and writes all logs to stderr unless you disconnect that. IIRC I discovered through some other work that it is not disconnected here. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] tvalentyn commented on pull request #12833: Updates Dataflow containers used by unreleased SDKs.
tvalentyn commented on pull request #12833: URL: https://github.com/apache/beam/pull/12833#issuecomment-692261224 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] kennknowles commented on pull request #12825: [BEAM-9399] Change DataflowWorkerLoggingHandler to report errors to t…
kennknowles commented on pull request #12825: URL: https://github.com/apache/beam/pull/12825#issuecomment-692268316 Had a chat and we confirmed that the worker logging setup already handles my concern appropriately. Please disregard all comments. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] lukecwik commented on pull request #12794: [BEAM-10865] Support for Kafka deserialization API with headers (since Kafka API 2.1.0)
lukecwik commented on pull request #12794: URL: https://github.com/apache/beam/pull/12794#issuecomment-692278596 Waiting on tests that use Kafka 2.1.0 explicitly as described in my first review. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] boyuanzz commented on pull request #12726: [BEAM-10475] Add max buffering duration to GroupIntoBatches (Java)
boyuanzz commented on pull request #12726: URL: https://github.com/apache/beam/pull/12726#issuecomment-692291612 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] qinyeli commented on pull request #12686: Removing dead code from Interactive Beam.
qinyeli commented on pull request #12686: URL: https://github.com/apache/beam/pull/12686#issuecomment-692296995 Thank you! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] lukecwik merged pull request #12773: [BEAM-10505][BEAM-10530] Add truncate capability.
lukecwik merged pull request #12773: URL: https://github.com/apache/beam/pull/12773 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12811: [BEAM-10705] Fixes a bug when passing whl files in --sdk_location from https locations
codecov[bot] edited a comment on pull request #12811: URL: https://github.com/apache/beam/pull/12811#issuecomment-690521374 # [Codecov](https://codecov.io/gh/apache/beam/pull/12811?src=pr=h1) Report > Merging [#12811](https://codecov.io/gh/apache/beam/pull/12811?src=pr=desc) into [master](https://codecov.io/gh/apache/beam/commit/d4e3729460dc710f954e22c92c36e8f00a24bd4c?el=desc) will **increase** coverage by `41.73%`. > The diff coverage is `100.00%`. [](https://codecov.io/gh/apache/beam/pull/12811?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #12811 +/- ## === + Coverage 40.26% 82.00% +41.73% === Files 455 456+1 Lines 5382254025 +203 === + Hits2167444302+22628 + Misses 32148 9723-22425 ``` | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12811?src=pr=tree) | Coverage Δ | | |---|---|---| | [...s/python/apache\_beam/runners/portability/stager.py](https://codecov.io/gh/apache/beam/pull/12811/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9zdGFnZXIucHk=) | `82.55% <100.00%> (+45.55%)` | :arrow_up: | | [...apache\_beam/portability/api/beam\_runner\_api\_pb2.py](https://codecov.io/gh/apache/beam/pull/12811/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjIucHk=) | `100.00% <0.00%> (ø)` | | | [...he\_beam/testing/benchmarks/nexmark/nexmark\_util.py](https://codecov.io/gh/apache/beam/pull/12811/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvbmV4bWFya191dGlsLnB5) | `0.00% <0.00%> (ø)` | | | [...e\_beam/portability/api/beam\_runner\_api\_pb2\_urns.py](https://codecov.io/gh/apache/beam/pull/12811/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjJfdXJucy5weQ==) | `100.00% <0.00%> (ø)` | | | [...eam/testing/benchmarks/nexmark/nexmark\_launcher.py](https://codecov.io/gh/apache/beam/pull/12811/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvbmV4bWFya19sYXVuY2hlci5weQ==) | `0.00% <0.00%> (ø)` | | | [...ache\_beam/runners/interactive/pipeline\_analyzer.py](https://codecov.io/gh/apache/beam/pull/12811/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9waXBlbGluZV9hbmFseXplci5weQ==) | | | | [sdks/python/apache\_beam/utils/histogram.py](https://codecov.io/gh/apache/beam/pull/12811/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvaGlzdG9ncmFtLnB5) | `94.28% <0.00%> (ø)` | | | [...he\_beam/testing/benchmarks/nexmark/nexmark\_perf.py](https://codecov.io/gh/apache/beam/pull/12811/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvbmV4bWFya19wZXJmLnB5) | `0.00% <0.00%> (ø)` | | | [sdks/python/apache\_beam/coders/slow\_stream.py](https://codecov.io/gh/apache/beam/pull/12811/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL3Nsb3dfc3RyZWFtLnB5) | `92.43% <0.00%> (+1.68%)` | :arrow_up: | | [sdks/python/apache\_beam/utils/profiler.py](https://codecov.io/gh/apache/beam/pull/12811/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvcHJvZmlsZXIucHk=) | `32.11% <0.00%> (+1.83%)` | :arrow_up: | | ... and [275 more](https://codecov.io/gh/apache/beam/pull/12811/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12811?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12811?src=pr=footer). Last update [d4e3729...a99c520](https://codecov.io/gh/apache/beam/pull/12811?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] NikeNano commented on pull request #12842: Update quickstart-java.md to used -P correctly in the documentations.
NikeNano commented on pull request #12842: URL: https://github.com/apache/beam/pull/12842#issuecomment-692302200 R: @iemejia This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] emilymye commented on pull request #12505: [WIP][BEAM-8106] Add version to java container image name
emilymye commented on pull request #12505: URL: https://github.com/apache/beam/pull/12505#issuecomment-692228990 run java postcommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12811: [BEAM-10705] Fixes a bug when passing whl files in --sdk_location from https locations
codecov[bot] edited a comment on pull request #12811: URL: https://github.com/apache/beam/pull/12811#issuecomment-690521374 # [Codecov](https://codecov.io/gh/apache/beam/pull/12811?src=pr=h1) Report > Merging [#12811](https://codecov.io/gh/apache/beam/pull/12811?src=pr=desc) into [master](https://codecov.io/gh/apache/beam/commit/d4e3729460dc710f954e22c92c36e8f00a24bd4c?el=desc) will **increase** coverage by `41.73%`. > The diff coverage is `100.00%`. [](https://codecov.io/gh/apache/beam/pull/12811?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #12811 +/- ## === + Coverage 40.26% 82.00% +41.73% === Files 455 456+1 Lines 5382254025 +203 === + Hits2167444302+22628 + Misses 32148 9723-22425 ``` | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12811?src=pr=tree) | Coverage Δ | | |---|---|---| | [...s/python/apache\_beam/runners/portability/stager.py](https://codecov.io/gh/apache/beam/pull/12811/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9zdGFnZXIucHk=) | `82.55% <100.00%> (+45.55%)` | :arrow_up: | | [...apache\_beam/portability/api/beam\_runner\_api\_pb2.py](https://codecov.io/gh/apache/beam/pull/12811/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjIucHk=) | `100.00% <0.00%> (ø)` | | | [...he\_beam/testing/benchmarks/nexmark/nexmark\_util.py](https://codecov.io/gh/apache/beam/pull/12811/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvbmV4bWFya191dGlsLnB5) | `0.00% <0.00%> (ø)` | | | [...e\_beam/portability/api/beam\_runner\_api\_pb2\_urns.py](https://codecov.io/gh/apache/beam/pull/12811/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjJfdXJucy5weQ==) | `100.00% <0.00%> (ø)` | | | [...eam/testing/benchmarks/nexmark/nexmark\_launcher.py](https://codecov.io/gh/apache/beam/pull/12811/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvbmV4bWFya19sYXVuY2hlci5weQ==) | `0.00% <0.00%> (ø)` | | | [...ache\_beam/runners/interactive/pipeline\_analyzer.py](https://codecov.io/gh/apache/beam/pull/12811/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9waXBlbGluZV9hbmFseXplci5weQ==) | | | | [sdks/python/apache\_beam/utils/histogram.py](https://codecov.io/gh/apache/beam/pull/12811/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvaGlzdG9ncmFtLnB5) | `94.28% <0.00%> (ø)` | | | [...he\_beam/testing/benchmarks/nexmark/nexmark\_perf.py](https://codecov.io/gh/apache/beam/pull/12811/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvbmV4bWFya19wZXJmLnB5) | `0.00% <0.00%> (ø)` | | | [sdks/python/apache\_beam/coders/slow\_stream.py](https://codecov.io/gh/apache/beam/pull/12811/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL3Nsb3dfc3RyZWFtLnB5) | `92.43% <0.00%> (+1.68%)` | :arrow_up: | | [sdks/python/apache\_beam/utils/profiler.py](https://codecov.io/gh/apache/beam/pull/12811/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvcHJvZmlsZXIucHk=) | `32.11% <0.00%> (+1.83%)` | :arrow_up: | | ... and [275 more](https://codecov.io/gh/apache/beam/pull/12811/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12811?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12811?src=pr=footer). Last update [d4e3729...a99c520](https://codecov.io/gh/apache/beam/pull/12811?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12505: [WIP][BEAM-8106] Add version to java container image name
codecov[bot] edited a comment on pull request #12505: URL: https://github.com/apache/beam/pull/12505#issuecomment-684502100 # [Codecov](https://codecov.io/gh/apache/beam/pull/12505?src=pr=h1) Report > Merging [#12505](https://codecov.io/gh/apache/beam/pull/12505?src=pr=desc) into [master](https://codecov.io/gh/apache/beam/commit/a19d06eaf5d9639df330915ec175c8c8082c4980?el=desc) will **decrease** coverage by `0.01%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/beam/pull/12505?src=pr=tree) ```diff @@Coverage Diff @@ ## master #12505 +/- ## == - Coverage 82.41% 82.40% -0.02% == Files 451 451 Lines 5389853898 == - Hits4442244413 -9 - Misses 9476 9485 +9 ``` | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12505?src=pr=tree) | Coverage Δ | | |---|---|---| | [...am/runners/interactive/options/capture\_limiters.py](https://codecov.io/gh/apache/beam/pull/12505/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9vcHRpb25zL2NhcHR1cmVfbGltaXRlcnMucHk=) | `91.93% <0.00%> (-1.62%)` | :arrow_down: | | [...ive/messaging/interactive\_environment\_inspector.py](https://codecov.io/gh/apache/beam/pull/12505/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9tZXNzYWdpbmcvaW50ZXJhY3RpdmVfZW52aXJvbm1lbnRfaW5zcGVjdG9yLnB5) | `96.15% <0.00%> (-1.29%)` | :arrow_down: | | [...eam/runners/interactive/caching/streaming\_cache.py](https://codecov.io/gh/apache/beam/pull/12505/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9jYWNoaW5nL3N0cmVhbWluZ19jYWNoZS5weQ==) | `94.06% <0.00%> (-1.28%)` | :arrow_down: | | [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/12505/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `88.98% <0.00%> (-0.54%)` | :arrow_down: | | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/12505/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `94.45% <0.00%> (-0.14%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12505?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12505?src=pr=footer). Last update [a19d06e...2d221be](https://codecov.io/gh/apache/beam/pull/12505?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] scwhittle commented on pull request #12825: [BEAM-9399] Change DataflowWorkerLoggingHandler to report errors to t…
scwhittle commented on pull request #12825: URL: https://github.com/apache/beam/pull/12825#issuecomment-692275145 @lukecwik I think to have a single lock, we would need to use the PrintStream lock itself, as that is what is synchronized on by Throwable.printStackTrace and which can be synchronize outside and inside other synchronization blocks if we choose another lock to guard the buffer, ErrorManager, handler. However using the PrintStream as a lock within the DataflowWorkerLoggingHandler seemed perhaps error prone as that extends a Handler which might have it's own synchronized methods (on itself not the PrintStream) or callers that synchronize on the Handler (and not on the PrintStream). Additionally it seemed a bit gross since there is not always a PrintStream for a handler, though we could just allow injecting a lock Object into the handler to hide that. If you think that approach is still preferrable, I can put that together. Regarding on why to synchronize on buffer, that allows us to keep the preconditioncheck to sanity check our implementation. But that seems overkill, so I will just remove it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] NikeNano opened a new pull request #12842: Update quickstart-java.md to used -P correctly in the documentations in "Getting started Java"
NikeNano opened a new pull request #12842: URL: https://github.com/apache/beam/pull/12842 Update to clarify that the `-P` is used for maven. **Please** add a meaningful description for your change here Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2 --- | --- | --- | --- | --- | --- | --- Go | [](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | --- Java | [](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/) Python | [](https://ci-beam.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[![Build
[GitHub] [beam] lukecwik commented on a change in pull request #12836: [BEAM-10887] Expose clear() to FnApiTimer.
lukecwik commented on a change in pull request #12836: URL: https://github.com/apache/beam/pull/12836#discussion_r488206104 ## File path: sdks/java/core/src/main/java/org/apache/beam/sdk/state/Timer.java ## @@ -81,6 +81,9 @@ */ void setRelative(); + /** Clears the timer. */ Review comment: This contract is vague. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12834: Document GroupBy transform.
codecov[bot] edited a comment on pull request #12834: URL: https://github.com/apache/beam/pull/12834#issuecomment-691369145 # [Codecov](https://codecov.io/gh/apache/beam/pull/12834?src=pr=h1) Report > Merging [#12834](https://codecov.io/gh/apache/beam/pull/12834?src=pr=desc) into [master](https://codecov.io/gh/apache/beam/commit/803efdd931699c24e60350e5ccf6e54482f5916f?el=desc) will **decrease** coverage by `0.02%`. > The diff coverage is `84.84%`. [](https://codecov.io/gh/apache/beam/pull/12834?src=pr=tree) ```diff @@Coverage Diff @@ ## master #12834 +/- ## == - Coverage 82.38% 82.35% -0.03% == Files 451 450 -1 Lines 5377553702 -73 == - Hits4430344228 -75 - Misses 9472 9474 +2 ``` | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12834?src=pr=tree) | Coverage Δ | | |---|---|---| | [...n/apache\_beam/runners/dataflow/dataflow\_metrics.py](https://codecov.io/gh/apache/beam/pull/12834/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9kYXRhZmxvd19tZXRyaWNzLnB5) | `74.32% <33.33%> (-0.85%)` | :arrow_down: | | [sdks/python/apache\_beam/dataframe/frame\_base.py](https://codecov.io/gh/apache/beam/pull/12834/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lX2Jhc2UucHk=) | `84.70% <57.14%> (ø)` | | | [sdks/python/apache\_beam/dataframe/doctests.py](https://codecov.io/gh/apache/beam/pull/12834/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2RvY3Rlc3RzLnB5) | `96.83% <83.33%> (-0.93%)` | :arrow_down: | | [sdks/python/apache\_beam/typehints/opcodes.py](https://codecov.io/gh/apache/beam/pull/12834/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL29wY29kZXMucHk=) | `87.65% <83.33%> (-0.35%)` | :arrow_down: | | [sdks/python/apache\_beam/transforms/stats.py](https://codecov.io/gh/apache/beam/pull/12834/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9zdGF0cy5weQ==) | `90.41% <86.36%> (+3.02%)` | :arrow_up: | | [sdks/python/apache\_beam/typehints/schemas.py](https://codecov.io/gh/apache/beam/pull/12834/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL3NjaGVtYXMucHk=) | `93.20% <86.66%> (-3.41%)` | :arrow_down: | | [sdks/python/apache\_beam/coders/row\_coder.py](https://codecov.io/gh/apache/beam/pull/12834/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL3Jvd19jb2Rlci5weQ==) | `94.36% <90.90%> (-0.64%)` | :arrow_down: | | [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/12834/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `90.48% <100.00%> (+0.07%)` | :arrow_up: | | [...s/snippets/transforms/aggregation/combinevalues.py](https://codecov.io/gh/apache/beam/pull/12834/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvc25pcHBldHMvdHJhbnNmb3Jtcy9hZ2dyZWdhdGlvbi9jb21iaW5ldmFsdWVzLnB5) | `87.36% <0.00%> (-7.37%)` | :arrow_down: | | [...n/apache\_beam/runners/interactive/cache\_manager.py](https://codecov.io/gh/apache/beam/pull/12834/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9jYWNoZV9tYW5hZ2VyLnB5) | `89.58% <0.00%> (-0.70%)` | :arrow_down: | | ... and [18 more](https://codecov.io/gh/apache/beam/pull/12834/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12834?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12834?src=pr=footer). Last update [8e7014d...cd5ada2](https://codecov.io/gh/apache/beam/pull/12834?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12834: Document GroupBy transform.
codecov[bot] edited a comment on pull request #12834: URL: https://github.com/apache/beam/pull/12834#issuecomment-691369145 # [Codecov](https://codecov.io/gh/apache/beam/pull/12834?src=pr=h1) Report > Merging [#12834](https://codecov.io/gh/apache/beam/pull/12834?src=pr=desc) into [master](https://codecov.io/gh/apache/beam/commit/803efdd931699c24e60350e5ccf6e54482f5916f?el=desc) will **decrease** coverage by `0.02%`. > The diff coverage is `84.84%`. [](https://codecov.io/gh/apache/beam/pull/12834?src=pr=tree) ```diff @@Coverage Diff @@ ## master #12834 +/- ## == - Coverage 82.38% 82.35% -0.03% == Files 451 450 -1 Lines 5377553702 -73 == - Hits4430344228 -75 - Misses 9472 9474 +2 ``` | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12834?src=pr=tree) | Coverage Δ | | |---|---|---| | [...n/apache\_beam/runners/dataflow/dataflow\_metrics.py](https://codecov.io/gh/apache/beam/pull/12834/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9kYXRhZmxvd19tZXRyaWNzLnB5) | `74.32% <33.33%> (-0.85%)` | :arrow_down: | | [sdks/python/apache\_beam/dataframe/frame\_base.py](https://codecov.io/gh/apache/beam/pull/12834/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lX2Jhc2UucHk=) | `84.70% <57.14%> (ø)` | | | [sdks/python/apache\_beam/dataframe/doctests.py](https://codecov.io/gh/apache/beam/pull/12834/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2RvY3Rlc3RzLnB5) | `96.83% <83.33%> (-0.93%)` | :arrow_down: | | [sdks/python/apache\_beam/typehints/opcodes.py](https://codecov.io/gh/apache/beam/pull/12834/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL29wY29kZXMucHk=) | `87.65% <83.33%> (-0.35%)` | :arrow_down: | | [sdks/python/apache\_beam/transforms/stats.py](https://codecov.io/gh/apache/beam/pull/12834/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9zdGF0cy5weQ==) | `90.41% <86.36%> (+3.02%)` | :arrow_up: | | [sdks/python/apache\_beam/typehints/schemas.py](https://codecov.io/gh/apache/beam/pull/12834/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL3NjaGVtYXMucHk=) | `93.20% <86.66%> (-3.41%)` | :arrow_down: | | [sdks/python/apache\_beam/coders/row\_coder.py](https://codecov.io/gh/apache/beam/pull/12834/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL3Jvd19jb2Rlci5weQ==) | `94.36% <90.90%> (-0.64%)` | :arrow_down: | | [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/12834/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `90.48% <100.00%> (+0.07%)` | :arrow_up: | | [...s/snippets/transforms/aggregation/combinevalues.py](https://codecov.io/gh/apache/beam/pull/12834/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvc25pcHBldHMvdHJhbnNmb3Jtcy9hZ2dyZWdhdGlvbi9jb21iaW5ldmFsdWVzLnB5) | `87.36% <0.00%> (-7.37%)` | :arrow_down: | | [...n/apache\_beam/runners/interactive/cache\_manager.py](https://codecov.io/gh/apache/beam/pull/12834/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9jYWNoZV9tYW5hZ2VyLnB5) | `89.58% <0.00%> (-0.70%)` | :arrow_down: | | ... and [18 more](https://codecov.io/gh/apache/beam/pull/12834/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12834?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12834?src=pr=footer). Last update [8e7014d...cd5ada2](https://codecov.io/gh/apache/beam/pull/12834?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] TheNeuralBit merged pull request #12831: [BEAM-10252] Add null check on logical type override
TheNeuralBit merged pull request #12831: URL: https://github.com/apache/beam/pull/12831 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12832: Add basic ToRows transform.
codecov[bot] edited a comment on pull request #12832: URL: https://github.com/apache/beam/pull/12832#issuecomment-691348838 # [Codecov](https://codecov.io/gh/apache/beam/pull/12832?src=pr=h1) Report > Merging [#12832](https://codecov.io/gh/apache/beam/pull/12832?src=pr=desc) into [master](https://codecov.io/gh/apache/beam/commit/803efdd931699c24e60350e5ccf6e54482f5916f?el=desc) will **increase** coverage by `0.00%`. > The diff coverage is `85.85%`. [](https://codecov.io/gh/apache/beam/pull/12832?src=pr=tree) ```diff @@ Coverage Diff@@ ## master #12832+/- ## Coverage 82.38% 82.39% Files 451 451 Lines 5377553929 +154 + Hits4430344433 +130 - Misses 9472 9496+24 ``` | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12832?src=pr=tree) | Coverage Δ | | |---|---|---| | [...n/apache\_beam/runners/dataflow/dataflow\_metrics.py](https://codecov.io/gh/apache/beam/pull/12832/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9kYXRhZmxvd19tZXRyaWNzLnB5) | `74.32% <33.33%> (-0.85%)` | :arrow_down: | | [sdks/python/apache\_beam/dataframe/frame\_base.py](https://codecov.io/gh/apache/beam/pull/12832/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lX2Jhc2UucHk=) | `84.70% <57.14%> (ø)` | | | [sdks/python/apache\_beam/transforms/core.py](https://codecov.io/gh/apache/beam/pull/12832/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9jb3JlLnB5) | `88.42% <80.00%> (-0.11%)` | :arrow_down: | | [sdks/python/apache\_beam/dataframe/doctests.py](https://codecov.io/gh/apache/beam/pull/12832/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2RvY3Rlc3RzLnB5) | `96.83% <83.33%> (-0.93%)` | :arrow_down: | | [sdks/python/apache\_beam/transforms/stats.py](https://codecov.io/gh/apache/beam/pull/12832/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9zdGF0cy5weQ==) | `87.39% <86.36%> (ø)` | | | [sdks/python/apache\_beam/typehints/schemas.py](https://codecov.io/gh/apache/beam/pull/12832/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL3NjaGVtYXMucHk=) | `93.20% <86.66%> (-3.41%)` | :arrow_down: | | [sdks/python/apache\_beam/typehints/opcodes.py](https://codecov.io/gh/apache/beam/pull/12832/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL29wY29kZXMucHk=) | `87.91% <90.00%> (-0.09%)` | :arrow_down: | | [sdks/python/apache\_beam/coders/row\_coder.py](https://codecov.io/gh/apache/beam/pull/12832/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL3Jvd19jb2Rlci5weQ==) | `94.36% <90.90%> (-0.64%)` | :arrow_down: | | [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/12832/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `90.48% <100.00%> (+0.07%)` | :arrow_up: | | [sdks/python/apache\_beam/pvalue.py](https://codecov.io/gh/apache/beam/pull/12832/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcHZhbHVlLnB5) | `91.69% <100.00%> (+0.06%)` | :arrow_up: | | ... and [8 more](https://codecov.io/gh/apache/beam/pull/12832/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12832?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12832?src=pr=footer). Last update [8e7014d...0e92ffc](https://codecov.io/gh/apache/beam/pull/12832?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] codecov[bot] commented on pull request #12576: [BEAM-10671] Add environment configuration fields as first-class pipeline options.
codecov[bot] commented on pull request #12576: URL: https://github.com/apache/beam/pull/12576#issuecomment-692353567 # [Codecov](https://codecov.io/gh/apache/beam/pull/12576?src=pr=h1) Report > :exclamation: No coverage uploaded for pull request head (`BEAM-10671@e96b70d`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-head-commit). > The diff coverage is `n/a`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12505: [WIP][BEAM-8106] Add version to java container image name
codecov[bot] edited a comment on pull request #12505: URL: https://github.com/apache/beam/pull/12505#issuecomment-684502100 # [Codecov](https://codecov.io/gh/apache/beam/pull/12505?src=pr=h1) Report > Merging [#12505](https://codecov.io/gh/apache/beam/pull/12505?src=pr=desc) into [master](https://codecov.io/gh/apache/beam/commit/dbdc019b77f5c5df17b08debaf25b5446a1ba013?el=desc) will **increase** coverage by `0.05%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/beam/pull/12505?src=pr=tree) ```diff @@Coverage Diff @@ ## master #12505 +/- ## == + Coverage 82.35% 82.40% +0.05% == Files 450 451 +1 Lines 5370153898 +197 == + Hits4422344413 +190 - Misses 9478 9485 +7 ``` | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12505?src=pr=tree) | Coverage Δ | | |---|---|---| | [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/12505/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `88.98% <0.00%> (-0.18%)` | :arrow_down: | | [...ache\_beam/runners/interactive/pipeline\_analyzer.py](https://codecov.io/gh/apache/beam/pull/12505/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9waXBlbGluZV9hbmFseXplci5weQ==) | `95.43% <0.00%> (ø)` | | | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/12505/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `94.45% <0.00%> (+0.13%)` | :arrow_up: | | [...eam/runners/portability/fn\_api\_runner/fn\_runner.py](https://codecov.io/gh/apache/beam/pull/12505/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL2ZuX3J1bm5lci5weQ==) | `89.74% <0.00%> (+0.20%)` | :arrow_up: | | [...n/apache\_beam/runners/interactive/cache\_manager.py](https://codecov.io/gh/apache/beam/pull/12505/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9jYWNoZV9tYW5hZ2VyLnB5) | `90.27% <0.00%> (+0.69%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12505?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12505?src=pr=footer). Last update [63f54fd...ba2a1fa](https://codecov.io/gh/apache/beam/pull/12505?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] emilymye commented on pull request #12505: [WIP][BEAM-8106] Add version to java container image name
emilymye commented on pull request #12505: URL: https://github.com/apache/beam/pull/12505#issuecomment-692362081 R: @chamikaramj @TheNeuralBit cc: @kennknowles I think Kenneth mentioned y'all might be good reviewers for this PR. I previously ran the Java postcommit and it passed, but I'm uncertain if I correctly ran all the tests I need to actually verify this change. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] lukecwik commented on a change in pull request #12806: [BEAM-10869] Use PubsubMessagePayloadOnlyCoder when writing to Pubsub.
lukecwik commented on a change in pull request #12806:
URL: https://github.com/apache/beam/pull/12806#discussion_r488208550
##
File path:
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.java
##
@@ -1012,6 +1012,8 @@ public PDone expand(PCollection input) {
case UNBOUNDED:
return input
.apply(MapElements.into(new TypeDescriptor()
{}).via(getFormatFn()))
+ // TODO(BEAM-10869): Also plump through
PubsubMessageWithAttributesCoder if needed.
+ .setCoder(PubsubMessagePayloadOnlyCoder.of())
Review comment:
PubsubUnboundedSink only consumes the data blob and sets the other
attributes using properties from the element (e.g. timestamp).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [beam] lukecwik commented on a change in pull request #12806: [BEAM-10869] Use PubsubMessagePayloadOnlyCoder when writing to Pubsub.
lukecwik commented on a change in pull request #12806:
URL: https://github.com/apache/beam/pull/12806#discussion_r488216535
##
File path:
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.java
##
@@ -1010,20 +1011,27 @@ public PDone expand(PCollection input) {
getMaxBatchBytesSize(),
MAX_PUBLISH_BATCH_BYTE_SIZE_DEFAULT;
return PDone.in(input.getPipeline());
case UNBOUNDED:
- return input
- .apply(MapElements.into(new TypeDescriptor()
{}).via(getFormatFn()))
- .apply(
- new PubsubUnboundedSink(
- getPubsubClientFactory(),
- NestedValueProvider.of(getTopicProvider(), new
TopicPathTranslator()),
- getTimestampAttribute(),
- getIdAttribute(),
- 100 /* numShards */,
- MoreObjects.firstNonNull(
- getMaxBatchSize(),
PubsubUnboundedSink.DEFAULT_PUBLISH_BATCH_SIZE),
- MoreObjects.firstNonNull(
- getMaxBatchBytesSize(),
- PubsubUnboundedSink.DEFAULT_PUBLISH_BATCH_BYTES)));
+ PCollection output =
+ input.apply(
+ MapElements.into(new TypeDescriptor()
{}).via(getFormatFn()));
+ if
(ExperimentalOptions.hasExperiment(input.getPipeline().getOptions(),
"beam_fn_api")
Review comment:
I think this should be "beam_fn_api" || "use_runner_v2" ||
"use_unified_worker"
Also, can we use the full pubsub with message attributes coder, this will
allow for future changes to the PubsubUnboundedSink without needing to break
update compatibility
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [beam] lostluck commented on a change in pull request #12588: [BEAM-7009] Add Go SDK Standard Coders yaml tests.
lostluck commented on a change in pull request #12588:
URL: https://github.com/apache/beam/pull/12588#discussion_r488222388
##
File path: sdks/go/pkg/beam/core/runtime/exec/coder.go
##
@@ -81,24 +82,82 @@ func MakeElementEncoder(c *coder.Coder) ElementEncoder {
return {}
case coder.Custom:
- return {
+ enc := {
t: c.Custom.Type,
enc: makeEncoder(c.Custom.Enc.Fn),
}
+ if c.Custom.Name != "schema" {
+ return enc
+ }
+ // Custom schema coding is shorthand for using beam
infrastructure
+ // wrapped in a custom coder.
Review comment:
"Custom schema coding uses the beam row coder, but wrapping it in a
length prefix and treats it as an opaque coder."
Does that make more sense?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [beam] lostluck commented on a change in pull request #12588: [BEAM-7009] Add Go SDK Standard Coders yaml tests.
lostluck commented on a change in pull request #12588:
URL: https://github.com/apache/beam/pull/12588#discussion_r488226218
##
File path: sdks/go/pkg/beam/core/runtime/exec/coder.go
##
@@ -434,6 +600,267 @@ func convertIfNeeded(v interface{}, allocated *FullValue)
*FullValue {
return allocated
}
+type iterableEncoder struct {
+ t reflect.Type
+ enc ElementEncoder
+}
+
+func (c *iterableEncoder) Encode(val *FullValue, w io.Writer) error {
+ // Do a reflect, get the length.
+ rv := reflect.ValueOf(val.Elm)
+ size := rv.Len()
+ if err := coder.EncodeInt32((int32)(size), w); err != nil {
+ return err
+ }
+ var e FullValue
+ for i := 0; i < size; i++ {
+ e.Elm = rv.Index(i).Interface()
+ err := c.enc.Encode(, w)
+ if err != nil {
+ return err
+ }
+ }
+ return nil
+}
+
+type iterableDecoder struct {
+ t reflect.Type
+ dec ElementDecoder
+}
+
+func (c *iterableDecoder) DecodeTo(r io.Reader, fv *FullValue) error {
+ // (1) Read count prefixed encoded data
+
+ size, err := coder.DecodeInt32(r)
+ if err != nil {
+ return err
+ }
+ n := int(size)
+ switch {
+ case n >= 0:
+ rv, err := c.decodeToSlice(int(n), r)
+ if err != nil {
+ return err
+ }
+ *fv = FullValue{Elm: rv.Interface()}
+ return nil
+ case n == -1:
+ rv := reflect.MakeSlice(c.t, 0, 0)
+ chunk, err := coder.DecodeVarInt(r)
+ if err != nil {
+ return err
+ }
+ for chunk != 0 {
+ rvi, err := c.decodeToSlice(int(chunk), r)
+ if err != nil {
+ return err
+ }
+ rv = reflect.AppendSlice(rv, rvi)
+ chunk, err = coder.DecodeVarInt(r)
+ if err != nil {
+ return err
+ }
+ }
+ *fv = FullValue{Elm: rv.Interface()}
+ }
+
+ return nil
+}
+
+func (c *iterableDecoder) decodeToSlice(n int, r io.Reader) (reflect.Value,
error) {
+ var e FullValue
+ rv := reflect.MakeSlice(c.t, n, n)
+ for i := 0; i < int(n); i++ {
+ err := c.dec.DecodeTo(r, )
+ if err != nil {
+ return reflect.Value{}, err
+ }
+ if e.Elm != nil {
+ rv.Index(i).Set(reflect.ValueOf(e.Elm))
+ } else {
+ rv.Index(i).Set(reflect.ValueOf(e.Windows[0]))
Review comment:
In this case it's to set it to the 0th window element. In this case it's
a small hack to support "window's as element values" which is only necessary to
dramatically simplify the yaml tests which actually test the coder values here.
Deserves a comment to explain it, as it is a hack.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [beam] lostluck commented on a change in pull request #12588: [BEAM-7009] Add Go SDK Standard Coders yaml tests.
lostluck commented on a change in pull request #12588:
URL: https://github.com/apache/beam/pull/12588#discussion_r488227779
##
File path: sdks/go/test/regression/coders/fromyaml/fromyaml.go
##
@@ -0,0 +1,415 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+//http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// fromyaml generates a resource file from the standard_coders.yaml
+// file for use in these coder regression tests.
+//
+// It expects to be run in it's test directory, or via it's go test.
+package main
+
+import (
+ "bytes"
+ "fmt"
+ "io/ioutil"
+ "log"
+ "math"
+ "reflect"
+ "runtime/debug"
+ "strconv"
+ "strings"
+
+ "github.com/apache/beam/sdks/go/pkg/beam/core/graph/mtime"
+ "github.com/apache/beam/sdks/go/pkg/beam/core/graph/window"
+ "github.com/apache/beam/sdks/go/pkg/beam/core/runtime/exec"
+ "github.com/apache/beam/sdks/go/pkg/beam/core/runtime/graphx"
+ "github.com/apache/beam/sdks/go/pkg/beam/core/util/reflectx"
+ pipepb "github.com/apache/beam/sdks/go/pkg/beam/model/pipeline_v1"
+ "github.com/google/go-cmp/cmp"
+ "golang.org/x/text/encoding/charmap"
+ yaml "gopkg.in/yaml.v2"
+)
+
+var unimplementedCoders = map[string]bool{
+ "beam:coder:param_windowed_value:v1": true,
+ "beam:coder:timer:v1":true,
+}
+
+// Coder is a representation a serialized beam coder.
+type Coder struct {
+ Urn string `yaml:"urn,omitempty"`
+ Payload string `yaml:"payload,omitempty"`
+ Components []Coder `yaml:"components,omitempty"`
+ NonDeterministic bool`yaml:"non_deterministic,omitempty"`
+}
+
+type logger interface {
+ Errorf(string, ...interface{})
+ Logf(string, ...interface{})
+}
+
+// Spec is a set of conditions that a coder must pass.
+type Spec struct {
+ CoderCoder `yaml:"coder,omitempty"`
+ Nested *bool `yaml:"nested,omitempty"`
+ Examples yaml.MapSlice `yaml:"examples,omitempty"`
+ Log logger
+
+ id int // for generating coder ids.
+ coderPBs map[string]*pipepb.Coder
+}
+
+func (s *Spec) nextID() string {
+ ret := fmt.Sprintf("%d", s.id)
+ s.id++
+ return ret
+}
+
+func (s *Spec) testStandardCoder() (err error) {
+ if unimplementedCoders[s.Coder.Urn] {
+ log.Printf("skipping unimplemented coder urn: %v", s.Coder.Urn)
+ return nil
+ }
+ // Construct the coder proto equivalents.
+
+ // Only nested tests need to be run, since nestedness is a
pre-portability
+ // concept.
+ // For legacy Java reasons, the row coder examples are all marked
nested: false
+ // so we need to check that before skipping unnested tests.
+ if s.Coder.Urn != "beam:coder:row:v1" && s.Nested != nil && !*s.Nested {
+ log.Printf("skipping unnested coder spec: %v\n", s.Coder)
+ return nil
+ }
+
+ s.coderPBs = make(map[string]*pipepb.Coder)
+ id := s.parseCoder(s.Coder)
+ b := graphx.NewCoderUnmarshaller(s.coderPBs)
+ underTest, err := b.Coder(id)
+ if err != nil {
+ return fmt.Errorf("unable to create coder: %v", err)
+ }
+
+ defer func() {
+ if e := recover(); e != nil {
+ err = fmt.Errorf("panicked on coder %v || %v:\n\t%v
:\n%s", underTest, s.Coder, e, debug.Stack())
+ }
+ }()
+
+ var decFails, encFails int
+ for _, eg := range s.Examples {
+
+ // Test Decoding
+ // Ideally we'd use the beam package coders, but KVs make that
complicated.
+ // This can be cleaned up once a type parametered beam.KV type
exists.
+ dec := exec.MakeElementDecoder(underTest)
+ encoded := eg.Key.(string)
+ var elem exec.FullValue
+
+ // What I would have expected.
+ // r :=
charmap.ISO8859_1.NewDecoder().Reader(strings.NewReader(encoded))
+ recoded, err := charmap.ISO8859_1.NewEncoder().String(encoded)
+ if err != nil {
+ return err
+
[GitHub] [beam] boyuanzz merged pull request #12726: [BEAM-10475] Add max buffering duration to GroupIntoBatches (Java)
boyuanzz merged pull request #12726: URL: https://github.com/apache/beam/pull/12726 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] boyuanzz commented on pull request #12726: [BEAM-10475] Add max buffering duration to GroupIntoBatches (Java)
boyuanzz commented on pull request #12726: URL: https://github.com/apache/beam/pull/12726#issuecomment-692328746 The test failure is not related. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12553: [BEAM-9615] Add initial schema proto documentation.
codecov[bot] edited a comment on pull request #12553: URL: https://github.com/apache/beam/pull/12553#issuecomment-692340545 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12553: [BEAM-9615] Add initial schema proto documentation.
codecov[bot] edited a comment on pull request #12553: URL: https://github.com/apache/beam/pull/12553#issuecomment-692340545 # [Codecov](https://codecov.io/gh/apache/beam/pull/12553?src=pr=h1) Report > Merging [#12553](https://codecov.io/gh/apache/beam/pull/12553?src=pr=desc) into [master](https://codecov.io/gh/apache/beam/commit/e164d170eb6b5ec199f09e79dfb0147b84ae?el=desc) will **increase** coverage by `47.88%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/beam/pull/12553?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #12553 +/- ## === + Coverage 34.47% 82.36% +47.88% === Files 684 450 -234 Lines 8148353702-27781 Branches 91800 -9180 === + Hits2809044230+16140 + Misses 52972 9472-43500 + Partials 4210 -421 ``` | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12553?src=pr=tree) | Coverage Δ | | |---|---|---| | [transforms/validate\_runner\_xlang\_test.py](https://codecov.io/gh/apache/beam/pull/12553/diff?src=pr=tree#diff-dHJhbnNmb3Jtcy92YWxpZGF0ZV9ydW5uZXJfeGxhbmdfdGVzdC5weQ==) | | | | [runners/direct/direct\_runner.py](https://codecov.io/gh/apache/beam/pull/12553/diff?src=pr=tree#diff-cnVubmVycy9kaXJlY3QvZGlyZWN0X3J1bm5lci5weQ==) | | | | [metrics/execution\_test.py](https://codecov.io/gh/apache/beam/pull/12553/diff?src=pr=tree#diff-bWV0cmljcy9leGVjdXRpb25fdGVzdC5weQ==) | | | | [coders/coders\_test\_common.py](https://codecov.io/gh/apache/beam/pull/12553/diff?src=pr=tree#diff-Y29kZXJzL2NvZGVyc190ZXN0X2NvbW1vbi5weQ==) | | | | [io/aws/clients/s3/boto3\_client.py](https://codecov.io/gh/apache/beam/pull/12553/diff?src=pr=tree#diff-aW8vYXdzL2NsaWVudHMvczMvYm90bzNfY2xpZW50LnB5) | | | | [...s/snippets/transforms/aggregation/combineperkey.py](https://codecov.io/gh/apache/beam/pull/12553/diff?src=pr=tree#diff-ZXhhbXBsZXMvc25pcHBldHMvdHJhbnNmb3Jtcy9hZ2dyZWdhdGlvbi9jb21iaW5lcGVya2V5LnB5) | | | | [io/utils\_test.py](https://codecov.io/gh/apache/beam/pull/12553/diff?src=pr=tree#diff-aW8vdXRpbHNfdGVzdC5weQ==) | | | | [runners/interactive/pipeline\_fragment.py](https://codecov.io/gh/apache/beam/pull/12553/diff?src=pr=tree#diff-cnVubmVycy9pbnRlcmFjdGl2ZS9waXBlbGluZV9mcmFnbWVudC5weQ==) | | | | [testing/util.py](https://codecov.io/gh/apache/beam/pull/12553/diff?src=pr=tree#diff-dGVzdGluZy91dGlsLnB5) | | | | [io/fileio\_test.py](https://codecov.io/gh/apache/beam/pull/12553/diff?src=pr=tree#diff-aW8vZmlsZWlvX3Rlc3QucHk=) | | | | ... and [1124 more](https://codecov.io/gh/apache/beam/pull/12553/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12553?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12553?src=pr=footer). Last update [f2192c6...b1a2916](https://codecov.io/gh/apache/beam/pull/12553?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] tvalentyn commented on pull request #12833: Updates Dataflow containers used by unreleased SDKs.
tvalentyn commented on pull request #12833: URL: https://github.com/apache/beam/pull/12833#issuecomment-692351130 Python PreCommits passed, flake in MacOS unit test suite does not exercise this codepath. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] tvalentyn merged pull request #12833: Updates Dataflow containers used by unreleased SDKs.
tvalentyn merged pull request #12833: URL: https://github.com/apache/beam/pull/12833 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] robinyqiu opened a new pull request #12843: [WIP] Support UNNEST an (possibly nested) array field of an struct column
robinyqiu opened a new pull request #12843: URL: https://github.com/apache/beam/pull/12843 See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2 --- | --- | --- | --- | --- | --- | --- Go | [](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | --- Java | [](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/) Python | [](https://ci-beam.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)[![Build
[GitHub] [beam] lostluck commented on a change in pull request #12553: [BEAM-9615] Add initial schema proto documentation.
lostluck commented on a change in pull request #12553:
URL: https://github.com/apache/beam/pull/12553#discussion_r488240030
##
File path: model/pipeline/src/main/proto/schema.proto
##
@@ -32,16 +32,27 @@ option java_package = "org.apache.beam.model.pipeline.v1";
option java_outer_classname = "SchemaApi";
message Schema {
+ // List of fields for this schema. Two fields may not share a name.
repeated Field fields = 1;
+ // REQUIRED. An RFC 4122 UUID.
string id = 2;
repeated Option options = 3;
}
message Field {
+ // REQUIRED. Name of this field within the schema.
string name = 1;
+ // OPTIONAL. Human readable description of this field, such as the query
that generated it.
string description = 2;
FieldType type = 3;
+
int32 id = 4;
+ // OPTIONAL. The position of this field's data in the encoding.
+ // Either no fields in a given row are have encoding position populated,
+ // or all of them are. Used to support backwards compatibility with schema
+ // changes.
+ // Two schemas, one with encoding positions and one without, are equal
+ // if the fields when sorted by encoding position would otherwise be equal.
Review comment:
Ack. There's nothing wrong with only using strict equivalence, rather
than assignable equivalence. We do need to document which we're using at some
point or all we'll be using is strict equivalence forever. Without a definition
of assignable, we can't really use assignable equivalence at all.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12834: Document GroupBy transform.
codecov[bot] edited a comment on pull request #12834: URL: https://github.com/apache/beam/pull/12834#issuecomment-691369145 # [Codecov](https://codecov.io/gh/apache/beam/pull/12834?src=pr=h1) Report > Merging [#12834](https://codecov.io/gh/apache/beam/pull/12834?src=pr=desc) into [master](https://codecov.io/gh/apache/beam/commit/803efdd931699c24e60350e5ccf6e54482f5916f?el=desc) will **decrease** coverage by `0.02%`. > The diff coverage is `84.84%`. [](https://codecov.io/gh/apache/beam/pull/12834?src=pr=tree) ```diff @@Coverage Diff @@ ## master #12834 +/- ## == - Coverage 82.38% 82.35% -0.03% == Files 451 450 -1 Lines 5377553702 -73 == - Hits4430344228 -75 - Misses 9472 9474 +2 ``` | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12834?src=pr=tree) | Coverage Δ | | |---|---|---| | [...n/apache\_beam/runners/dataflow/dataflow\_metrics.py](https://codecov.io/gh/apache/beam/pull/12834/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9kYXRhZmxvd19tZXRyaWNzLnB5) | `74.32% <33.33%> (-0.85%)` | :arrow_down: | | [sdks/python/apache\_beam/dataframe/frame\_base.py](https://codecov.io/gh/apache/beam/pull/12834/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lX2Jhc2UucHk=) | `84.70% <57.14%> (ø)` | | | [sdks/python/apache\_beam/dataframe/doctests.py](https://codecov.io/gh/apache/beam/pull/12834/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2RvY3Rlc3RzLnB5) | `96.83% <83.33%> (-0.93%)` | :arrow_down: | | [sdks/python/apache\_beam/typehints/opcodes.py](https://codecov.io/gh/apache/beam/pull/12834/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL29wY29kZXMucHk=) | `87.65% <83.33%> (-0.35%)` | :arrow_down: | | [sdks/python/apache\_beam/transforms/stats.py](https://codecov.io/gh/apache/beam/pull/12834/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9zdGF0cy5weQ==) | `90.41% <86.36%> (+3.02%)` | :arrow_up: | | [sdks/python/apache\_beam/typehints/schemas.py](https://codecov.io/gh/apache/beam/pull/12834/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL3NjaGVtYXMucHk=) | `93.20% <86.66%> (-3.41%)` | :arrow_down: | | [sdks/python/apache\_beam/coders/row\_coder.py](https://codecov.io/gh/apache/beam/pull/12834/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL3Jvd19jb2Rlci5weQ==) | `94.36% <90.90%> (-0.64%)` | :arrow_down: | | [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/12834/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `90.48% <100.00%> (+0.07%)` | :arrow_up: | | [...s/snippets/transforms/aggregation/combinevalues.py](https://codecov.io/gh/apache/beam/pull/12834/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvc25pcHBldHMvdHJhbnNmb3Jtcy9hZ2dyZWdhdGlvbi9jb21iaW5ldmFsdWVzLnB5) | `87.36% <0.00%> (-7.37%)` | :arrow_down: | | [...n/apache\_beam/runners/interactive/cache\_manager.py](https://codecov.io/gh/apache/beam/pull/12834/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9jYWNoZV9tYW5hZ2VyLnB5) | `89.58% <0.00%> (-0.70%)` | :arrow_down: | | ... and [18 more](https://codecov.io/gh/apache/beam/pull/12834/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12834?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12834?src=pr=footer). Last update [8e7014d...cd5ada2](https://codecov.io/gh/apache/beam/pull/12834?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] codecov[bot] commented on pull request #12553: [BEAM-9615] Add initial schema proto documentation.
codecov[bot] commented on pull request #12553: URL: https://github.com/apache/beam/pull/12553#issuecomment-692340545 # [Codecov](https://codecov.io/gh/apache/beam/pull/12553?src=pr=h1) Report > Merging [#12553](https://codecov.io/gh/apache/beam/pull/12553?src=pr=desc) into [master](https://codecov.io/gh/apache/beam/commit/e164d170eb6b5ec199f09e79dfb0147b84ae?el=desc) will **increase** coverage by `47.88%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/beam/pull/12553?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #12553 +/- ## === + Coverage 34.47% 82.36% +47.88% === Files 684 450 -234 Lines 8148353702-27781 Branches 91800 -9180 === + Hits2809044230+16140 + Misses 52972 9472-43500 + Partials 4210 -421 ``` | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12553?src=pr=tree) | Coverage Δ | | |---|---|---| | [runners/interactive/testing/test\_cache\_manager.py](https://codecov.io/gh/apache/beam/pull/12553/diff?src=pr=tree#diff-cnVubmVycy9pbnRlcmFjdGl2ZS90ZXN0aW5nL3Rlc3RfY2FjaGVfbWFuYWdlci5weQ==) | | | | [coders/slow\_coders\_test.py](https://codecov.io/gh/apache/beam/pull/12553/diff?src=pr=tree#diff-Y29kZXJzL3Nsb3dfY29kZXJzX3Rlc3QucHk=) | | | | [examples/cookbook/filters\_test.py](https://codecov.io/gh/apache/beam/pull/12553/diff?src=pr=tree#diff-ZXhhbXBsZXMvY29va2Jvb2svZmlsdGVyc190ZXN0LnB5) | | | | [runners/portability/portable\_runner.py](https://codecov.io/gh/apache/beam/pull/12553/diff?src=pr=tree#diff-cnVubmVycy9wb3J0YWJpbGl0eS9wb3J0YWJsZV9ydW5uZXIucHk=) | | | | [examples/snippets/transforms/elementwise/filter.py](https://codecov.io/gh/apache/beam/pull/12553/diff?src=pr=tree#diff-ZXhhbXBsZXMvc25pcHBldHMvdHJhbnNmb3Jtcy9lbGVtZW50d2lzZS9maWx0ZXIucHk=) | | | | [testing/synthetic\_pipeline.py](https://codecov.io/gh/apache/beam/pull/12553/diff?src=pr=tree#diff-dGVzdGluZy9zeW50aGV0aWNfcGlwZWxpbmUucHk=) | | | | [...tests/streaming/group\_by\_key\_streaming\_pipeline.py](https://codecov.io/gh/apache/beam/pull/12553/diff?src=pr=tree#diff-dGVzdGluZy9sb2FkX3Rlc3RzL3N0cmVhbWluZy9ncm91cF9ieV9rZXlfc3RyZWFtaW5nX3BpcGVsaW5lLnB5) | | | | [runners/portability/portable\_metrics.py](https://codecov.io/gh/apache/beam/pull/12553/diff?src=pr=tree#diff-cnVubmVycy9wb3J0YWJpbGl0eS9wb3J0YWJsZV9tZXRyaWNzLnB5) | | | | [testing/benchmarks/chicago\_taxi/process\_tfma.py](https://codecov.io/gh/apache/beam/pull/12553/diff?src=pr=tree#diff-dGVzdGluZy9iZW5jaG1hcmtzL2NoaWNhZ29fdGF4aS9wcm9jZXNzX3RmbWEucHk=) | | | | [utils/urns.py](https://codecov.io/gh/apache/beam/pull/12553/diff?src=pr=tree#diff-dXRpbHMvdXJucy5weQ==) | | | | ... and [1124 more](https://codecov.io/gh/apache/beam/pull/12553/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12553?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12553?src=pr=footer). Last update [f2192c6...b1a2916](https://codecov.io/gh/apache/beam/pull/12553?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12553: [BEAM-9615] Add initial schema proto documentation.
codecov[bot] edited a comment on pull request #12553: URL: https://github.com/apache/beam/pull/12553#issuecomment-692340545 # [Codecov](https://codecov.io/gh/apache/beam/pull/12553?src=pr=h1) Report > Merging [#12553](https://codecov.io/gh/apache/beam/pull/12553?src=pr=desc) into [master](https://codecov.io/gh/apache/beam/commit/e164d170eb6b5ec199f09e79dfb0147b84ae?el=desc) will **increase** coverage by `47.88%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/beam/pull/12553?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #12553 +/- ## === + Coverage 34.47% 82.36% +47.88% === Files 684 450 -234 Lines 8148353702-27781 Branches 91800 -9180 === + Hits2809044230+16140 + Misses 52972 9472-43500 + Partials 4210 -421 ``` | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12553?src=pr=tree) | Coverage Δ | | |---|---|---| | [io/gcp/experimental/spannerio.py](https://codecov.io/gh/apache/beam/pull/12553/diff?src=pr=tree#diff-aW8vZ2NwL2V4cGVyaW1lbnRhbC9zcGFubmVyaW8ucHk=) | | | | [io/mongodbio\_it\_test.py](https://codecov.io/gh/apache/beam/pull/12553/diff?src=pr=tree#diff-aW8vbW9uZ29kYmlvX2l0X3Rlc3QucHk=) | | | | [transforms/ptransform.py](https://codecov.io/gh/apache/beam/pull/12553/diff?src=pr=tree#diff-dHJhbnNmb3Jtcy9wdHJhbnNmb3JtLnB5) | | | | [io/utils\_test.py](https://codecov.io/gh/apache/beam/pull/12553/diff?src=pr=tree#diff-aW8vdXRpbHNfdGVzdC5weQ==) | | | | [examples/cookbook/mergecontacts.py](https://codecov.io/gh/apache/beam/pull/12553/diff?src=pr=tree#diff-ZXhhbXBsZXMvY29va2Jvb2svbWVyZ2Vjb250YWN0cy5weQ==) | | | | [examples/snippets/util\_test.py](https://codecov.io/gh/apache/beam/pull/12553/diff?src=pr=tree#diff-ZXhhbXBsZXMvc25pcHBldHMvdXRpbF90ZXN0LnB5) | | | | [examples/cookbook/coders.py](https://codecov.io/gh/apache/beam/pull/12553/diff?src=pr=tree#diff-ZXhhbXBsZXMvY29va2Jvb2svY29kZXJzLnB5) | | | | [io/gcp/dicomio\_integration\_test.py](https://codecov.io/gh/apache/beam/pull/12553/diff?src=pr=tree#diff-aW8vZ2NwL2RpY29taW9faW50ZWdyYXRpb25fdGVzdC5weQ==) | | | | [testing/benchmarks/nexmark/queries/query0.py](https://codecov.io/gh/apache/beam/pull/12553/diff?src=pr=tree#diff-dGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvcXVlcmllcy9xdWVyeTAucHk=) | | | | [examples/snippets/transforms/aggregation/mean.py](https://codecov.io/gh/apache/beam/pull/12553/diff?src=pr=tree#diff-ZXhhbXBsZXMvc25pcHBldHMvdHJhbnNmb3Jtcy9hZ2dyZWdhdGlvbi9tZWFuLnB5) | | | | ... and [1124 more](https://codecov.io/gh/apache/beam/pull/12553/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12553?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12553?src=pr=footer). Last update [f2192c6...b1a2916](https://codecov.io/gh/apache/beam/pull/12553?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] emilymye commented on pull request #12833: Updates Dataflow containers used by unreleased SDKs.
emilymye commented on pull request #12833: URL: https://github.com/apache/beam/pull/12833#issuecomment-692348565 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12505: [WIP][BEAM-8106] Add version to java container image name
codecov[bot] edited a comment on pull request #12505: URL: https://github.com/apache/beam/pull/12505#issuecomment-684502100 # [Codecov](https://codecov.io/gh/apache/beam/pull/12505?src=pr=h1) Report > Merging [#12505](https://codecov.io/gh/apache/beam/pull/12505?src=pr=desc) into [master](https://codecov.io/gh/apache/beam/commit/dbdc019b77f5c5df17b08debaf25b5446a1ba013?el=desc) will **increase** coverage by `0.05%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/beam/pull/12505?src=pr=tree) ```diff @@Coverage Diff @@ ## master #12505 +/- ## == + Coverage 82.35% 82.40% +0.05% == Files 450 451 +1 Lines 5370153898 +197 == + Hits4422344413 +190 - Misses 9478 9485 +7 ``` | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12505?src=pr=tree) | Coverage Δ | | |---|---|---| | [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/12505/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `88.98% <0.00%> (-0.18%)` | :arrow_down: | | [...ache\_beam/runners/interactive/pipeline\_analyzer.py](https://codecov.io/gh/apache/beam/pull/12505/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9waXBlbGluZV9hbmFseXplci5weQ==) | `95.43% <0.00%> (ø)` | | | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/12505/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `94.45% <0.00%> (+0.13%)` | :arrow_up: | | [...eam/runners/portability/fn\_api\_runner/fn\_runner.py](https://codecov.io/gh/apache/beam/pull/12505/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL2ZuX3J1bm5lci5weQ==) | `89.74% <0.00%> (+0.20%)` | :arrow_up: | | [...n/apache\_beam/runners/interactive/cache\_manager.py](https://codecov.io/gh/apache/beam/pull/12505/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9jYWNoZV9tYW5hZ2VyLnB5) | `90.27% <0.00%> (+0.69%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12505?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12505?src=pr=footer). Last update [63f54fd...48c1242](https://codecov.io/gh/apache/beam/pull/12505?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12576: [BEAM-10671] Add environment configuration fields as first-class pipeline options.
codecov[bot] edited a comment on pull request #12576: URL: https://github.com/apache/beam/pull/12576#issuecomment-692353567 # [Codecov](https://codecov.io/gh/apache/beam/pull/12576?src=pr=h1) Report > :exclamation: No coverage uploaded for pull request head (`BEAM-10671@edb10f3`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-head-commit). > The diff coverage is `n/a`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
