[GitHub] [beam] codecov[bot] edited a comment on pull request #12576: [BEAM-10671] Add environment configuration fields as first-class pipeline options.

[GitBox]

codecov[bot] edited a comment on pull request #12576:
URL: https://github.com/apache/beam/pull/12576#issuecomment-692353567


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12576?src=pr=h1) Report
   > Merging 
[#12576](https://codecov.io/gh/apache/beam/pull/12576?src=pr=desc) into 
[master](https://codecov.io/gh/apache/beam/commit/8bedfd402ccd1336f2b40fad58cd5833cecb9638?el=desc)
 will **increase** coverage by `0.06%`.
   > The diff coverage is `86.66%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/beam/pull/12576/graphs/tree.svg?width=650=150=pr=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12576?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master   #12576  +/-   ##
   ==
   + Coverage   82.35%   82.42%   +0.06% 
   ==
 Files 455  455  
 Lines   5465054978 +328 
   ==
   + Hits4500745315 +308 
   - Misses   9643 9663  +20 
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/beam/pull/12576?src=pr=tree) | Coverage 
Δ | |
   |---|---|---|
   | 
[...dks/python/apache\_beam/options/pipeline\_options.py](https://codecov.io/gh/apache/beam/pull/12576/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vb3B0aW9ucy9waXBlbGluZV9vcHRpb25zLnB5)
 | `93.76% <70.58%> (-1.03%)` | :arrow_down: |
   | 
[sdks/python/apache\_beam/transforms/environments.py](https://codecov.io/gh/apache/beam/pull/12576/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9lbnZpcm9ubWVudHMucHk=)
 | `83.73% <83.33%> (+0.19%)` | :arrow_up: |
   | 
[.../apache\_beam/options/pipeline\_options\_validator.py](https://codecov.io/gh/apache/beam/pull/12576/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vb3B0aW9ucy9waXBlbGluZV9vcHRpb25zX3ZhbGlkYXRvci5weQ==)
 | `98.69% <100.00%> (+0.25%)` | :arrow_up: |
   | 
[...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/12576/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==)
 | `89.90% <0.00%> (-1.84%)` | :arrow_down: |
   | 
[...hon/apache\_beam/runners/direct/test\_stream\_impl.py](https://codecov.io/gh/apache/beam/pull/12576/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvdGVzdF9zdHJlYW1faW1wbC5weQ==)
 | `93.38% <0.00%> (-0.74%)` | :arrow_down: |
   | 
[...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/12576/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==)
 | `89.98% <0.00%> (-0.17%)` | :arrow_down: |
   | 
[...ache\_beam/runners/interactive/recording\_manager.py](https://codecov.io/gh/apache/beam/pull/12576/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9yZWNvcmRpbmdfbWFuYWdlci5weQ==)
 | `98.89% <0.00%> (+0.01%)` | :arrow_up: |
   | 
[sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/12576/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=)
 | `89.20% <0.00%> (+0.44%)` | :arrow_up: |
   | 
[...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/12576/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5)
 | `79.72% <0.00%> (+0.86%)` | :arrow_up: |
   | 
[sdks/python/apache\_beam/io/fileio.py](https://codecov.io/gh/apache/beam/pull/12576/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZmlsZWlvLnB5)
 | `97.16% <0.00%> (+1.32%)` | :arrow_up: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/beam/pull/12576?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/beam/pull/12576?src=pr=footer). Last 
update 
[8bedfd4...c032963](https://codecov.io/gh/apache/beam/pull/12576?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] rezarokni commented on pull request #12973: [BEAM-10234] Create ApproximateDistinct using HLL Impl

[GitBox]

rezarokni commented on pull request #12973:
URL: https://github.com/apache/beam/pull/12973#issuecomment-701219619


   @robinyqiu 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] TobKed commented on a change in pull request #12318: [BEAM-8665] Add test suites to run Beam tests on Linux/Windows/Mac platforms.

[GitBox]

TobKed commented on a change in pull request #12318:
URL: https://github.com/apache/beam/pull/12318#discussion_r497319272



##
File path: sdks/python/apache_beam/runners/portability/stager_test.py
##
@@ -167,6 +167,9 @@ def test_no_main_session(self):
 
   # xdist adds unpicklable modules to the main session.
   @pytest.mark.no_xdist
+  @pytest.mark.skip(
+  sys.platform == 'win32',
+  "Windows raise TypeError: can't pickle zipimport.zipimporter objects")

Review comment:
   I see it this skip was not included in 
https://github.com/apache/beam/pull/12452.
   As far i remember after some rebasings and further work on the tests it 
started to work and for many test runs it didn't fail so I removed this skip.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] mxm commented on pull request #12576: [BEAM-10671] Add environment configuration fields as first-class pipeline options.

[GitBox]

mxm commented on pull request #12576:
URL: https://github.com/apache/beam/pull/12576#issuecomment-701244754


   Run Portable_Python PreCommit



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] rezarokni opened a new pull request #12973: [BEAM-10234] Create ApproximateDistinct using HLL Impl

[GitBox]

rezarokni opened a new pull request #12973:
URL: https://github.com/apache/beam/pull/12973


   [BEAM-10234] Create ApproximateDistinct using HLL Impl
   Create ApproximateDistinct which uses HLL Impl
   Marked ApproximateUnique as deprecated in preparation for removal in favour 
of ApproximateDistinct. 
   
   
   
   Thank you for your contribution! Follow this checklist to help us 
incorporate your contribution quickly and easily:
   
- [ ] [**Choose 
reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and 
mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more 
tips on [how to make review process 
smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
   --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/)
 | --- | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/)
 | --- | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
 | ---
   Java | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/)
 | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[![Build
 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/)
 | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[![Build
 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/badge/i
 
con)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)[![Build
 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[![Build
 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/)
 | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/)
 | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[![Build
 Status](htt
 
ps://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[![Build
 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
 | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
   Python | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[![Build
 

[GitHub] [beam] mxm commented on pull request #12576: [BEAM-10671] Add environment configuration fields as first-class pipeline options.

[GitBox]

mxm commented on pull request #12576:
URL: https://github.com/apache/beam/pull/12576#issuecomment-701245561


   Run Python PreCommit



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] nielsbasjes commented on a change in pull request #12962: [WIP][BEAM-9379] Update calcite to 1.25

[GitBox]

nielsbasjes commented on a change in pull request #12962:
URL: https://github.com/apache/beam/pull/12962#discussion_r497362928



##
File path: vendor/calcite-1_25_0/build.gradle
##
@@ -18,29 +18,33 @@
 
 plugins { id 'org.apache.beam.vendor-java' }
 
-description = "Apache Beam :: Vendored Dependencies :: Calcite 1.20.0"
+description = "Apache Beam :: Vendored Dependencies :: Calcite 1.25.0"
 
 group = "org.apache.beam"
 version = "0.2"

Review comment:
   > Can you publish a build scan by running your second step with `--scan` 
? Then we can see what is broken.
   
   https://scans.gradle.com/s/ofcfnu774du4y





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] nielsbasjes commented on a change in pull request #12962: [WIP][BEAM-9379] Update calcite to 1.25

[GitBox]

nielsbasjes commented on a change in pull request #12962:
URL: https://github.com/apache/beam/pull/12962#discussion_r497363201



##
File path: vendor/calcite-1_25_0/build.gradle
##
@@ -18,29 +18,33 @@
 
 plugins { id 'org.apache.beam.vendor-java' }
 
-description = "Apache Beam :: Vendored Dependencies :: Calcite 1.20.0"
+description = "Apache Beam :: Vendored Dependencies :: Calcite 1.25.0"
 
 group = "org.apache.beam"
 version = "0.2"

Review comment:
   Yes





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] aromanenko-dev commented on pull request #12572: [BEAM-10123] Add Kafka Commit transform.

[GitBox]

aromanenko-dev commented on pull request #12572:
URL: https://github.com/apache/beam/pull/12572#issuecomment-701311000


   > Yeah I also want to have some tests around this but it may not be possible 
to involve actual Kafka consumer to do so. The way I can come up is to use mock 
but I'm not sure whether it's feasible.
   
   Can we add "SDF Read"-related tests to `KafkaIOIT` in this case?



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] runningcode edited a comment on pull request #12969: [BEAM-10986] Rollback to shadow 4.0.3

[GitBox]

runningcode edited a comment on pull request #12969:
URL: https://github.com/apache/beam/pull/12969#issuecomment-701301646


   Hi, I'm not sure that this fix helps. The 
[expansion-service/build.gradle](https://github.com/apache/beam/blob/master/sdks/java/expansion-service/build.gradle#L22)
 does not define a `shadowClosure` unlike the 
[core/build.gradle](https://github.com/apache/beam/blob/master/sdks/java/core/build.gradle#L24).
 
   
   If the `shadowClosure` is not present, the shadow plugin is not applied in 
the 
[`BeamModulePlugin`](https://github.com/apache/beam/blob/master/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L985)



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] nielsbasjes commented on pull request #12962: [WIP][BEAM-9379] Update calcite to 1.25

[GitBox]

nielsbasjes commented on pull request #12962:
URL: https://github.com/apache/beam/pull/12962#issuecomment-701269350


   > Can you publish a build scan by running your second step with `--scan` ? 
Then we can see what is broken.
   
   https://scans.gradle.com/s/ofcfnu774du4y



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #12962: [WIP][BEAM-9379] Update calcite to 1.25

[GitBox]

codecov[bot] edited a comment on pull request #12962:
URL: https://github.com/apache/beam/pull/12962#issuecomment-700648248


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12962?src=pr=h1) Report
   > Merging 
[#12962](https://codecov.io/gh/apache/beam/pull/12962?src=pr=desc) into 
[master](https://codecov.io/gh/apache/beam/commit/cee7388365970774969e63fe90fdf3cdd804e79f?el=desc)
 will **decrease** coverage by `0.00%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/beam/pull/12962/graphs/tree.svg?width=650=150=pr=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12962?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master   #12962  +/-   ##
   ==
   - Coverage   82.34%   82.33%   -0.01% 
   ==
 Files 455  455  
 Lines   5465054647   -3 
   ==
   - Hits4500244995   -7 
   - Misses   9648 9652   +4 
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/beam/pull/12962?src=pr=tree) | Coverage 
Δ | |
   |---|---|---|
   | 
[...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/12962/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==)
 | `88.68% <0.00%> (-1.23%)` | :arrow_down: |
   | 
[sdks/python/apache\_beam/io/localfilesystem.py](https://codecov.io/gh/apache/beam/pull/12962/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vbG9jYWxmaWxlc3lzdGVtLnB5)
 | `90.90% <0.00%> (-0.76%)` | :arrow_down: |
   | 
[...nners/portability/fn\_api\_runner/worker\_handlers.py](https://codecov.io/gh/apache/beam/pull/12962/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL3dvcmtlcl9oYW5kbGVycy5weQ==)
 | `80.57% <0.00%> (-0.18%)` | :arrow_down: |
   | 
[...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/12962/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==)
 | `89.98% <0.00%> (-0.17%)` | :arrow_down: |
   | 
[sdks/python/apache\_beam/io/gcp/bigquery.py](https://codecov.io/gh/apache/beam/pull/12962/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5LnB5)
 | `80.56% <0.00%> (-0.15%)` | :arrow_down: |
   | 
[...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/12962/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==)
 | `94.31% <0.00%> (-0.14%)` | :arrow_down: |
   | 
[sdks/python/apache\_beam/io/fileio.py](https://codecov.io/gh/apache/beam/pull/12962/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZmlsZWlvLnB5)
 | `95.80% <0.00%> (-0.05%)` | :arrow_down: |
   | 
[...ache\_beam/runners/interactive/recording\_manager.py](https://codecov.io/gh/apache/beam/pull/12962/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9yZWNvcmRpbmdfbWFuYWdlci5weQ==)
 | `98.90% <0.00%> (+0.02%)` | :arrow_up: |
   | 
[sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/12962/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=)
 | `89.20% <0.00%> (+0.44%)` | :arrow_up: |
   | 
[...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/12962/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5)
 | `79.53% <0.00%> (+0.67%)` | :arrow_up: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/beam/pull/12962?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/beam/pull/12962?src=pr=footer). Last 
update 
[cee7388...19d11b7](https://codecov.io/gh/apache/beam/pull/12962?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] runningcode commented on pull request #12969: [BEAM-10986] Rollback to shadow 4.0.3

[GitBox]

runningcode commented on pull request #12969:
URL: https://github.com/apache/beam/pull/12969#issuecomment-701301646


   Hi, I'm not sure that this fix helps. The 
[expansion-service/build.gradle](https://github.com/apache/beam/blob/master/sdks/java/expansion-service/build.gradle#L22)
 does not define a `shadowClosure` unlike say the 
[core/build.gradle](https://github.com/apache/beam/blob/master/sdks/java/core/build.gradle#L24).
 
   
   If the `shadowClosure` is not present, the shadow plugin is not applied in 
the 
[`BeamModulePlugin`](https://github.com/apache/beam/blob/master/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L985)



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #12576: [BEAM-10671] Add environment configuration fields as first-class pipeline options.

[GitBox]

codecov[bot] edited a comment on pull request #12576:
URL: https://github.com/apache/beam/pull/12576#issuecomment-692353567


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12576?src=pr=h1) Report
   > Merging 
[#12576](https://codecov.io/gh/apache/beam/pull/12576?src=pr=desc) into 
[master](https://codecov.io/gh/apache/beam/commit/8bedfd402ccd1336f2b40fad58cd5833cecb9638?el=desc)
 will **increase** coverage by `0.06%`.
   > The diff coverage is `86.66%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/beam/pull/12576/graphs/tree.svg?width=650=150=pr=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12576?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master   #12576  +/-   ##
   ==
   + Coverage   82.35%   82.42%   +0.06% 
   ==
 Files 455  455  
 Lines   5465054978 +328 
   ==
   + Hits4500745315 +308 
   - Misses   9643 9663  +20 
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/beam/pull/12576?src=pr=tree) | Coverage 
Δ | |
   |---|---|---|
   | 
[...dks/python/apache\_beam/options/pipeline\_options.py](https://codecov.io/gh/apache/beam/pull/12576/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vb3B0aW9ucy9waXBlbGluZV9vcHRpb25zLnB5)
 | `93.76% <70.58%> (-1.03%)` | :arrow_down: |
   | 
[sdks/python/apache\_beam/transforms/environments.py](https://codecov.io/gh/apache/beam/pull/12576/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9lbnZpcm9ubWVudHMucHk=)
 | `83.73% <83.33%> (+0.19%)` | :arrow_up: |
   | 
[.../apache\_beam/options/pipeline\_options\_validator.py](https://codecov.io/gh/apache/beam/pull/12576/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vb3B0aW9ucy9waXBlbGluZV9vcHRpb25zX3ZhbGlkYXRvci5weQ==)
 | `98.69% <100.00%> (+0.25%)` | :arrow_up: |
   | 
[...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/12576/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==)
 | `89.90% <0.00%> (-1.84%)` | :arrow_down: |
   | 
[...hon/apache\_beam/runners/direct/test\_stream\_impl.py](https://codecov.io/gh/apache/beam/pull/12576/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvdGVzdF9zdHJlYW1faW1wbC5weQ==)
 | `93.38% <0.00%> (-0.74%)` | :arrow_down: |
   | 
[...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/12576/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==)
 | `89.98% <0.00%> (-0.17%)` | :arrow_down: |
   | 
[...ache\_beam/runners/interactive/recording\_manager.py](https://codecov.io/gh/apache/beam/pull/12576/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9yZWNvcmRpbmdfbWFuYWdlci5weQ==)
 | `98.89% <0.00%> (+0.01%)` | :arrow_up: |
   | 
[sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/12576/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=)
 | `89.20% <0.00%> (+0.44%)` | :arrow_up: |
   | 
[...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/12576/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5)
 | `79.72% <0.00%> (+0.86%)` | :arrow_up: |
   | 
[sdks/python/apache\_beam/io/fileio.py](https://codecov.io/gh/apache/beam/pull/12576/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZmlsZWlvLnB5)
 | `97.16% <0.00%> (+1.32%)` | :arrow_up: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/beam/pull/12576?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/beam/pull/12576?src=pr=footer). Last 
update 
[8bedfd4...c032963](https://codecov.io/gh/apache/beam/pull/12576?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] nielsbasjes commented on a change in pull request #12962: [WIP][BEAM-9379] Update calcite to 1.25

[GitBox]

nielsbasjes commented on a change in pull request #12962:
URL: https://github.com/apache/beam/pull/12962#discussion_r497365708



##
File path: vendor/calcite-1_25_0/build.gradle
##
@@ -18,29 +18,33 @@
 
 plugins { id 'org.apache.beam.vendor-java' }
 
-description = "Apache Beam :: Vendored Dependencies :: Calcite 1.20.0"
+description = "Apache Beam :: Vendored Dependencies :: Calcite 1.25.0"
 
 group = "org.apache.beam"
 version = "0.2"
 
-def calcite_version = "1.20.0"
+def calcite_version = "1.25.0"
 def avatica_version = "1.16.0"
-def prefix = "org.apache.beam.vendor.calcite.v1_20_0"
+def prefix = "org.apache.beam.vendor.calcite.v1_25_0"
 
 List packagesToRelocate = [
 "com.esri",
+"com.fasterxml",
 "com.google.common",
-"com.google.thirdparty",
 "com.google.protobuf",
-"com.fasterxml",
+"com.google.thirdparty",
 "com.jayway",
 "com.yahoo",
+"javax.annotation",
+"net.minidev",
 "org.apache.calcite",
 "org.apache.commons",
 "org.apache.http",
+"org.apiguardian.api",
 "org.codehaus",
+"org.objectweb",
 "org.pentaho",
-"org.yaml"
+"org.yaml",
 ]
 
 vendorJava(

Review comment:
   I have copied (+ updated version) of the runtime dependencies part of 
this.
   As this is uncharted territory for me I'm unsure if this is correct now.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #12962: [WIP][BEAM-9379] Update calcite to 1.25

[GitBox]

codecov[bot] edited a comment on pull request #12962:
URL: https://github.com/apache/beam/pull/12962#issuecomment-700648248


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12962?src=pr=h1) Report
   > Merging 
[#12962](https://codecov.io/gh/apache/beam/pull/12962?src=pr=desc) into 
[master](https://codecov.io/gh/apache/beam/commit/cee7388365970774969e63fe90fdf3cdd804e79f?el=desc)
 will **decrease** coverage by `0.01%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/beam/pull/12962/graphs/tree.svg?width=650=150=pr=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12962?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master   #12962  +/-   ##
   ==
   - Coverage   82.34%   82.33%   -0.02% 
   ==
 Files 455  455  
 Lines   5465054650  
   ==
   - Hits4500244996   -6 
   - Misses   9648 9654   +6 
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/beam/pull/12962?src=pr=tree) | Coverage 
Δ | |
   |---|---|---|
   | 
[...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/12962/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==)
 | `88.68% <0.00%> (-1.23%)` | :arrow_down: |
   | 
[sdks/python/apache\_beam/io/localfilesystem.py](https://codecov.io/gh/apache/beam/pull/12962/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vbG9jYWxmaWxlc3lzdGVtLnB5)
 | `90.90% <0.00%> (-0.76%)` | :arrow_down: |
   | 
[...nners/portability/fn\_api\_runner/worker\_handlers.py](https://codecov.io/gh/apache/beam/pull/12962/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL3dvcmtlcl9oYW5kbGVycy5weQ==)
 | `80.57% <0.00%> (-0.18%)` | :arrow_down: |
   | 
[...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/12962/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==)
 | `89.98% <0.00%> (-0.17%)` | :arrow_down: |
   | 
[...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/12962/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==)
 | `94.58% <0.00%> (+0.13%)` | :arrow_up: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/beam/pull/12962?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/beam/pull/12962?src=pr=footer). Last 
update 
[cee7388...19d11b7](https://codecov.io/gh/apache/beam/pull/12962?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] aromanenko-dev commented on a change in pull request #12572: [BEAM-10123] Add Kafka Commit transform.

[GitBox]

aromanenko-dev commented on a change in pull request #12572:
URL: https://github.com/apache/beam/pull/12572#discussion_r497414578



##
File path: 
sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaCommitOffset.java
##
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.kafka;
+
+import static 
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
+
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.beam.sdk.coders.KvCoder;
+import org.apache.beam.sdk.coders.VarLongCoder;
+import org.apache.beam.sdk.coders.VoidCoder;
+import org.apache.beam.sdk.schemas.NoSuchSchemaException;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.MapElements;
+import org.apache.beam.sdk.transforms.Max;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.SerializableFunction;
+import org.apache.beam.sdk.transforms.windowing.FixedWindows;
+import org.apache.beam.sdk.transforms.windowing.Window;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.TypeDescriptor;
+import org.apache.kafka.clients.consumer.Consumer;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.OffsetAndMetadata;
+import org.joda.time.Duration;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** A {@link PTransform} that commits offsets of {@link KafkaRecord}. */
+public class KafkaCommitOffset
+extends PTransform<
+PCollection>>, 
PCollection> {
+  private final KafkaIO.ReadSourceDescriptors readSourceDescriptors;
+
+  KafkaCommitOffset(KafkaIO.ReadSourceDescriptors readSourceDescriptors) 
{
+this.readSourceDescriptors = readSourceDescriptors;
+  }
+
+  static class CommitOffsetDoFn extends DoFn, 
Void> {
+private static final Logger LOG = 
LoggerFactory.getLogger(CommitOffsetDoFn.class);
+private final Map offsetConsumerConfig;
+private final Map consumerConfig;
+private final SerializableFunction, Consumer>
+consumerFactoryFn;
+
+private transient ConsumerSpEL consumerSpEL = null;
+
+CommitOffsetDoFn(KafkaIO.ReadSourceDescriptors readSourceDescriptors) {
+  offsetConsumerConfig = readSourceDescriptors.getOffsetConsumerConfig();
+  consumerConfig = readSourceDescriptors.getConsumerConfig();
+  consumerFactoryFn = readSourceDescriptors.getConsumerFactoryFn();
+}
+
+@ProcessElement
+public void processElement(@Element KV 
element) {
+  Map updatedConsumerConfig =
+  overrideBootstrapServersConfig(consumerConfig, element.getKey());
+  try (Consumer offsetConsumer =
+  consumerFactoryFn.apply(
+  KafkaIOUtils.getOffsetConsumerConfig(
+  "commitOffset", offsetConsumerConfig, 
updatedConsumerConfig))) {
+try {
+  offsetConsumer.commitSync(
+  Collections.singletonMap(
+  element.getKey().getTopicPartition(),
+  new OffsetAndMetadata(element.getValue() + 1)));
+} catch (Exception e) {
+  // TODO: consider retrying.
+  LOG.warn("Getting exception when committing offset: {}", 
e.getMessage());
+}
+  }
+}
+
+private Map overrideBootstrapServersConfig(
+Map currentConfig, KafkaSourceDescriptor description) {
+  checkState(
+  currentConfig.containsKey(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG)
+  || description.getBootStrapServers() != null);
+  Map config = new HashMap<>(currentConfig);
+  if (description.getBootStrapServers() != null
+  && description.getBootStrapServers().size() > 0) {
+config.put(
+ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG,
+String.join(",", description.getBootStrapServers()));
+  }
+  return config;
+}
+  }
+
+  @Override
+  public PCollection expand(PCollection>> input) {
+try {
+  return input
+  .apply(
+  MapElements.into(new 

[GitHub] [beam] rezarokni commented on pull request #12973: [BEAM-10234] Create ApproximateDistinct using HLL Impl

[GitBox]

rezarokni commented on pull request #12973:
URL: https://github.com/apache/beam/pull/12973#issuecomment-701316654


   @zfraa



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] runningcode opened a new pull request #12974: [BEAM-10986] Add shadow jar configuration to expansion-service.

[GitBox]

runningcode opened a new pull request #12974:
URL: https://github.com/apache/beam/pull/12974


   The shadowJar configuration was never specified so it should have never been 
applying the shadow plugin.
   
   The 
[expansion-service/build.gradle](https://github.com/apache/beam/blob/master/sdks/java/expansion-service/build.gradle#L22)
 does not define a `shadowClosure` unlike the 
[core/build.gradle](https://github.com/apache/beam/blob/master/sdks/java/core/build.gradle#L24).
   
   If the `shadowClosure` is not present, the shadow plugin is not applied in 
the 
[`BeamModulePlugin`](https://github.com/apache/beam/blob/master/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L985)
   
   **Please** add a meaningful description for your change here
   
   
   
   Thank you for your contribution! Follow this checklist to help us 
incorporate your contribution quickly and easily:
   
- [ ] [**Choose 
reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and 
mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more 
tips on [how to make review process 
smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
   --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/)
 | --- | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/)
 | --- | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
 | ---
   Java | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/)
 | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[![Build
 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/)
 | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[![Build
 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/badge/i
 
con)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)[![Build
 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[![Build
 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/)
 | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/)
 | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[![Build
 Status](htt
 
ps://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[![Build
 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
 | 

[GitHub] [beam] runningcode commented on pull request #12974: [BEAM-10986] Add shadow jar configuration to expansion-service.

[GitBox]

runningcode commented on pull request #12974:
URL: https://github.com/apache/beam/pull/12974#issuecomment-701337357


   Replaces https://github.com/apache/beam/pull/12969 
   R: @robinyqiu R: @TheNeuralBit R: @mxm 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] piotr-szuberski opened a new pull request #12975: Update Beam's io dependencies

[GitBox]

piotr-szuberski opened a new pull request #12975:
URL: https://github.com/apache/beam/pull/12975


   [BEAM-10370] redis.clients:jedis from 3.0.1 to 3.3.0
   
   
   
   Thank you for your contribution! Follow this checklist to help us 
incorporate your contribution quickly and easily:
   
- [ ] [**Choose 
reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and 
mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more 
tips on [how to make review process 
smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
   --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/)
 | --- | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/)
 | --- | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
 | ---
   Java | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/)
 | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[![Build
 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/)
 | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[![Build
 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/badge/i
 
con)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)[![Build
 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[![Build
 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/)
 | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/)
 | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[![Build
 Status](htt
 
ps://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[![Build
 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
 | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
   Python | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[![Build
 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)[![Build
 

[GitHub] [beam] codecov[bot] edited a comment on pull request #12964: [BEAM-10639] Integration test that exercises --setup_file flag on Dataflow

[GitBox]

codecov[bot] edited a comment on pull request #12964:
URL: https://github.com/apache/beam/pull/12964#issuecomment-701370267


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12964?src=pr=h1) Report
   > Merging 
[#12964](https://codecov.io/gh/apache/beam/pull/12964?src=pr=desc) into 
[master](https://codecov.io/gh/apache/beam/commit/321ac13a28906788507eea2e2eee7b1c3229fc29?el=desc)
 will **decrease** coverage by `0.05%`.
   > The diff coverage is `0.00%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/beam/pull/12964/graphs/tree.svg?width=650=150=pr=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12964?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master   #12964  +/-   ##
   ==
   - Coverage   82.34%   82.29%   -0.06% 
   ==
 Files 455  456   +1 
 Lines   5465054672  +22 
   ==
   - Hits4500244991  -11 
   - Misses   9648 9681  +33 
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/beam/pull/12964?src=pr=tree) | Coverage 
Δ | |
   |---|---|---|
   | 
[...les/complete/juliaset/juliaset/juliaset\_test\_it.py](https://codecov.io/gh/apache/beam/pull/12964/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29tcGxldGUvanVsaWFzZXQvanVsaWFzZXQvanVsaWFzZXRfdGVzdF9pdC5weQ==)
 | `0.00% <0.00%> (ø)` | |
   | 
[...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/12964/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==)
 | `88.68% <0.00%> (-2.45%)` | :arrow_down: |
   | 
[...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/12964/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==)
 | `94.18% <0.00%> (-0.27%)` | :arrow_down: |
   | 
[sdks/python/apache\_beam/io/fileio.py](https://codecov.io/gh/apache/beam/pull/12964/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZmlsZWlvLnB5)
 | `95.80% <0.00%> (-0.05%)` | :arrow_down: |
   | 
[...ache\_beam/runners/interactive/recording\_manager.py](https://codecov.io/gh/apache/beam/pull/12964/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9yZWNvcmRpbmdfbWFuYWdlci5weQ==)
 | `98.90% <0.00%> (+0.02%)` | :arrow_up: |
   | 
[...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/12964/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5)
 | `79.53% <0.00%> (+0.67%)` | :arrow_up: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/beam/pull/12964?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/beam/pull/12964?src=pr=footer). Last 
update 
[321ac13...ffd072f](https://codecov.io/gh/apache/beam/pull/12964?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #12964: [BEAM-10639] Integration test that exercises --setup_file flag on Dataflow

[GitBox]

codecov[bot] edited a comment on pull request #12964:
URL: https://github.com/apache/beam/pull/12964#issuecomment-701370267


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12964?src=pr=h1) Report
   > Merging 
[#12964](https://codecov.io/gh/apache/beam/pull/12964?src=pr=desc) into 
[master](https://codecov.io/gh/apache/beam/commit/321ac13a28906788507eea2e2eee7b1c3229fc29?el=desc)
 will **decrease** coverage by `0.03%`.
   > The diff coverage is `0.00%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/beam/pull/12964/graphs/tree.svg?width=650=150=pr=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12964?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master   #12964  +/-   ##
   ==
   - Coverage   82.34%   82.31%   -0.04% 
   ==
 Files 455  456   +1 
 Lines   5465054672  +22 
   ==
   - Hits4500245001   -1 
   - Misses   9648 9671  +23 
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/beam/pull/12964?src=pr=tree) | Coverage 
Δ | |
   |---|---|---|
   | 
[...les/complete/juliaset/juliaset/juliaset\_test\_it.py](https://codecov.io/gh/apache/beam/pull/12964/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29tcGxldGUvanVsaWFzZXQvanVsaWFzZXQvanVsaWFzZXRfdGVzdF9pdC5weQ==)
 | `0.00% <0.00%> (ø)` | |
   | 
[...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/12964/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==)
 | `89.90% <0.00%> (-1.23%)` | :arrow_down: |
   | 
[...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/12964/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==)
 | `94.18% <0.00%> (-0.27%)` | :arrow_down: |
   | 
[sdks/python/apache\_beam/io/gcp/bigquery.py](https://codecov.io/gh/apache/beam/pull/12964/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5LnB5)
 | `80.56% <0.00%> (-0.15%)` | :arrow_down: |
   | 
[sdks/python/apache\_beam/io/fileio.py](https://codecov.io/gh/apache/beam/pull/12964/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZmlsZWlvLnB5)
 | `95.80% <0.00%> (-0.05%)` | :arrow_down: |
   | 
[...ache\_beam/runners/interactive/recording\_manager.py](https://codecov.io/gh/apache/beam/pull/12964/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9yZWNvcmRpbmdfbWFuYWdlci5weQ==)
 | `98.90% <0.00%> (+0.02%)` | :arrow_up: |
   | 
[...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/12964/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==)
 | `90.14% <0.00%> (+0.16%)` | :arrow_up: |
   | 
[...nners/portability/fn\_api\_runner/worker\_handlers.py](https://codecov.io/gh/apache/beam/pull/12964/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL3dvcmtlcl9oYW5kbGVycy5weQ==)
 | `80.75% <0.00%> (+0.17%)` | :arrow_up: |
   | 
[sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/12964/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=)
 | `89.20% <0.00%> (+0.44%)` | :arrow_up: |
   | ... and [2 
more](https://codecov.io/gh/apache/beam/pull/12964/diff?src=pr=tree-more) | |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/beam/pull/12964?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/beam/pull/12964?src=pr=footer). Last 
update 
[321ac13...8a617f6](https://codecov.io/gh/apache/beam/pull/12964?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] kamilwu commented on pull request #12964: [BEAM-10639] Integration test that exercises --setup_file flag on Dataflow

[GitBox]

kamilwu commented on pull request #12964:
URL: https://github.com/apache/beam/pull/12964#issuecomment-701399340


   Run Python 3.8 PostCommit



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] codecov[bot] commented on pull request #12964: [BEAM-10639] Integration test that exercises --setup_file flag on Dataflow

[GitBox]

codecov[bot] commented on pull request #12964:
URL: https://github.com/apache/beam/pull/12964#issuecomment-701370267


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12964?src=pr=h1) Report
   > Merging 
[#12964](https://codecov.io/gh/apache/beam/pull/12964?src=pr=desc) into 
[master](https://codecov.io/gh/apache/beam/commit/321ac13a28906788507eea2e2eee7b1c3229fc29?el=desc)
 will **decrease** coverage by `0.05%`.
   > The diff coverage is `0.00%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/beam/pull/12964/graphs/tree.svg?width=650=150=pr=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12964?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master   #12964  +/-   ##
   ==
   - Coverage   82.34%   82.29%   -0.06% 
   ==
 Files 455  456   +1 
 Lines   5465054672  +22 
   ==
   - Hits4500244991  -11 
   - Misses   9648 9681  +33 
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/beam/pull/12964?src=pr=tree) | Coverage 
Δ | |
   |---|---|---|
   | 
[...les/complete/juliaset/juliaset/juliaset\_test\_it.py](https://codecov.io/gh/apache/beam/pull/12964/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29tcGxldGUvanVsaWFzZXQvanVsaWFzZXQvanVsaWFzZXRfdGVzdF9pdC5weQ==)
 | `0.00% <0.00%> (ø)` | |
   | 
[...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/12964/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==)
 | `88.68% <0.00%> (-2.45%)` | :arrow_down: |
   | 
[...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/12964/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==)
 | `94.18% <0.00%> (-0.27%)` | :arrow_down: |
   | 
[sdks/python/apache\_beam/io/fileio.py](https://codecov.io/gh/apache/beam/pull/12964/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZmlsZWlvLnB5)
 | `95.80% <0.00%> (-0.05%)` | :arrow_down: |
   | 
[...ache\_beam/runners/interactive/recording\_manager.py](https://codecov.io/gh/apache/beam/pull/12964/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9yZWNvcmRpbmdfbWFuYWdlci5weQ==)
 | `98.90% <0.00%> (+0.02%)` | :arrow_up: |
   | 
[...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/12964/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5)
 | `79.53% <0.00%> (+0.67%)` | :arrow_up: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/beam/pull/12964?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/beam/pull/12964?src=pr=footer). Last 
update 
[321ac13...ffd072f](https://codecov.io/gh/apache/beam/pull/12964?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] kamilwu commented on a change in pull request #12927: Update some of the Python dependencies from Beam Dependency Check Report

[GitBox]

kamilwu commented on a change in pull request #12927:
URL: https://github.com/apache/beam/pull/12927#discussion_r497520564



##
File path: sdks/python/apache_beam/io/avroio.py
##
@@ -627,11 +627,19 @@ def write_record(self, writer, value):
 writer.append(value)
 
 
+class _FastAvroWriter(Writer):
+  """An adapter class which exposes a file handle so that it can be closed
+  by the sink. """
+  def __init__(self, file_handle, schema, codec):
+super(_FastAvroWriter, self).__init__(file_handle, schema, codec)

Review comment:
   Thanks. We've already got an answer, let's wait until we know if it's 
possible for the `fo` attribute to become available.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] kamilwu commented on pull request #12964: [BEAM-10639] Integration test that exercises --setup_file flag on Dataflow

[GitBox]

kamilwu commented on pull request #12964:
URL: https://github.com/apache/beam/pull/12964#issuecomment-701358974


   Run Python 3.8 PostCommit



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #12964: [BEAM-10639] Integration test that exercises --setup_file flag on Dataflow

[GitBox]

codecov[bot] edited a comment on pull request #12964:
URL: https://github.com/apache/beam/pull/12964#issuecomment-701370267


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12964?src=pr=h1) Report
   > Merging 
[#12964](https://codecov.io/gh/apache/beam/pull/12964?src=pr=desc) into 
[master](https://codecov.io/gh/apache/beam/commit/321ac13a28906788507eea2e2eee7b1c3229fc29?el=desc)
 will **decrease** coverage by `0.05%`.
   > The diff coverage is `0.00%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/beam/pull/12964/graphs/tree.svg?width=650=150=pr=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12964?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master   #12964  +/-   ##
   ==
   - Coverage   82.34%   82.29%   -0.06% 
   ==
 Files 455  456   +1 
 Lines   5465054672  +22 
   ==
   - Hits4500244991  -11 
   - Misses   9648 9681  +33 
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/beam/pull/12964?src=pr=tree) | Coverage 
Δ | |
   |---|---|---|
   | 
[...les/complete/juliaset/juliaset/juliaset\_test\_it.py](https://codecov.io/gh/apache/beam/pull/12964/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29tcGxldGUvanVsaWFzZXQvanVsaWFzZXQvanVsaWFzZXRfdGVzdF9pdC5weQ==)
 | `0.00% <0.00%> (ø)` | |
   | 
[...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/12964/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==)
 | `88.68% <0.00%> (-2.45%)` | :arrow_down: |
   | 
[...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/12964/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==)
 | `94.18% <0.00%> (-0.27%)` | :arrow_down: |
   | 
[sdks/python/apache\_beam/io/fileio.py](https://codecov.io/gh/apache/beam/pull/12964/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZmlsZWlvLnB5)
 | `95.80% <0.00%> (-0.05%)` | :arrow_down: |
   | 
[...ache\_beam/runners/interactive/recording\_manager.py](https://codecov.io/gh/apache/beam/pull/12964/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9yZWNvcmRpbmdfbWFuYWdlci5weQ==)
 | `98.90% <0.00%> (+0.02%)` | :arrow_up: |
   | 
[...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/12964/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5)
 | `79.53% <0.00%> (+0.67%)` | :arrow_up: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/beam/pull/12964?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/beam/pull/12964?src=pr=footer). Last 
update 
[321ac13...ffd072f](https://codecov.io/gh/apache/beam/pull/12964?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] piotr-szuberski commented on pull request #12975: Update Beam's io dependencies

[GitBox]

piotr-szuberski commented on pull request #12975:
URL: https://github.com/apache/beam/pull/12975#issuecomment-701392442


   Run Java PreCommit



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #12976: [release-2.25.0][BEAM-10986] Rollback to shadow 4.0.3

[GitBox]

codecov[bot] edited a comment on pull request #12976:
URL: https://github.com/apache/beam/pull/12976#issuecomment-701505415


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12976?src=pr=h1) Report
   > Merging 
[#12976](https://codecov.io/gh/apache/beam/pull/12976?src=pr=desc) into 
[release-2.25.0](https://codecov.io/gh/apache/beam/commit/0a507a1127a33aa954a4e13268de73eff1298550?el=desc)
 will **decrease** coverage by `0.00%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/beam/pull/12976/graphs/tree.svg?width=650=150=pr=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12976?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   release-2.25.0   #12976  +/-   ##
   ==
   - Coverage   82.31%   82.30%   -0.01% 
   ==
 Files 455  455  
 Lines   5458954589  
   ==
   - Hits4493344931   -2 
   - Misses   9656 9658   +2 
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/beam/pull/12976?src=pr=tree) | Coverage 
Δ | |
   |---|---|---|
   | 
[...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/12976/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==)
 | `88.68% <0.00%> (-1.23%)` | :arrow_down: |
   | 
[...nners/portability/fn\_api\_runner/worker\_handlers.py](https://codecov.io/gh/apache/beam/pull/12976/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL3dvcmtlcl9oYW5kbGVycy5weQ==)
 | `80.57% <0.00%> (-0.18%)` | :arrow_down: |
   | 
[...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/12976/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==)
 | `94.45% <0.00%> (-0.14%)` | :arrow_down: |
   | 
[sdks/python/apache\_beam/io/iobase.py](https://codecov.io/gh/apache/beam/pull/12976/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vaW9iYXNlLnB5)
 | `84.03% <0.00%> (+0.28%)` | :arrow_up: |
   | 
[sdks/python/apache\_beam/io/gcp/bigquery.py](https://codecov.io/gh/apache/beam/pull/12976/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5LnB5)
 | `80.71% <0.00%> (+0.44%)` | :arrow_up: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/beam/pull/12976?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/beam/pull/12976?src=pr=footer). Last 
update 
[8d91b9c...741dbad](https://codecov.io/gh/apache/beam/pull/12976?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] mxm commented on pull request #12974: [BEAM-10986] Add shadow jar configuration to expansion-service.

[GitBox]

mxm commented on pull request #12974:
URL: https://github.com/apache/beam/pull/12974#issuecomment-701516486


   Should it be `io/expansion-service` instead of `sdks/java/expansion-service`?



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] pabloem merged pull request #12956: Add nexmark python query 10 to choices

[GitBox]

pabloem merged pull request #12956:
URL: https://github.com/apache/beam/pull/12956


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #12976: [release-2.25.0][BEAM-10986] Rollback to shadow 4.0.3

[GitBox]

codecov[bot] edited a comment on pull request #12976:
URL: https://github.com/apache/beam/pull/12976#issuecomment-701505415


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12976?src=pr=h1) Report
   > Merging 
[#12976](https://codecov.io/gh/apache/beam/pull/12976?src=pr=desc) into 
[release-2.25.0](https://codecov.io/gh/apache/beam/commit/0a507a1127a33aa954a4e13268de73eff1298550?el=desc)
 will **decrease** coverage by `0.00%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/beam/pull/12976/graphs/tree.svg?width=650=150=pr=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12976?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   release-2.25.0   #12976  +/-   ##
   ==
   - Coverage   82.31%   82.30%   -0.01% 
   ==
 Files 455  455  
 Lines   5458954589  
   ==
   - Hits4493344931   -2 
   - Misses   9656 9658   +2 
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/beam/pull/12976?src=pr=tree) | Coverage 
Δ | |
   |---|---|---|
   | 
[...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/12976/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==)
 | `88.68% <0.00%> (-1.23%)` | :arrow_down: |
   | 
[...nners/portability/fn\_api\_runner/worker\_handlers.py](https://codecov.io/gh/apache/beam/pull/12976/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL3dvcmtlcl9oYW5kbGVycy5weQ==)
 | `80.57% <0.00%> (-0.18%)` | :arrow_down: |
   | 
[...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/12976/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==)
 | `94.45% <0.00%> (-0.14%)` | :arrow_down: |
   | 
[sdks/python/apache\_beam/io/iobase.py](https://codecov.io/gh/apache/beam/pull/12976/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vaW9iYXNlLnB5)
 | `84.03% <0.00%> (+0.28%)` | :arrow_up: |
   | 
[sdks/python/apache\_beam/io/gcp/bigquery.py](https://codecov.io/gh/apache/beam/pull/12976/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5LnB5)
 | `80.71% <0.00%> (+0.44%)` | :arrow_up: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/beam/pull/12976?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/beam/pull/12976?src=pr=footer). Last 
update 
[8d91b9c...741dbad](https://codecov.io/gh/apache/beam/pull/12976?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #12957: [BEAM-10114] Upgrade version of pubsublite to 0.4.1.

[GitBox]

codecov[bot] edited a comment on pull request #12957:
URL: https://github.com/apache/beam/pull/12957#issuecomment-700276403


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12957?src=pr=h1) Report
   > Merging 
[#12957](https://codecov.io/gh/apache/beam/pull/12957?src=pr=desc) into 
[master](https://codecov.io/gh/apache/beam/commit/321ac13a28906788507eea2e2eee7b1c3229fc29?el=desc)
 will **decrease** coverage by `0.00%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/beam/pull/12957/graphs/tree.svg?width=650=150=pr=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12957?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master   #12957  +/-   ##
   ==
   - Coverage   82.34%   82.34%   -0.01% 
   ==
 Files 455  455  
 Lines   5465054650  
   ==
   - Hits4500244999   -3 
   - Misses   9648 9651   +3 
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/beam/pull/12957?src=pr=tree) | Coverage 
Δ | |
   |---|---|---|
   | 
[...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/12957/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==)
 | `88.68% <0.00%> (-2.45%)` | :arrow_down: |
   | 
[...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/12957/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==)
 | `94.58% <0.00%> (+0.13%)` | :arrow_up: |
   | 
[sdks/python/apache\_beam/io/iobase.py](https://codecov.io/gh/apache/beam/pull/12957/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vaW9iYXNlLnB5)
 | `84.03% <0.00%> (+0.28%)` | :arrow_up: |
   | 
[sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/12957/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=)
 | `89.20% <0.00%> (+0.44%)` | :arrow_up: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/beam/pull/12957?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/beam/pull/12957?src=pr=footer). Last 
update 
[321ac13...e0c8ddf](https://codecov.io/gh/apache/beam/pull/12957?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] kennknowles commented on pull request #12868: Fix up-to-date checking mechanism on Tox Task.

[GitBox]

kennknowles commented on pull request #12868:
URL: https://github.com/apache/beam/pull/12868#issuecomment-701604539


   Is it just that the XML report that Jenkins is looking for is not registered 
as an output?



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #12941: [BEAM-9681] Add textio.Read lesson to Go SDK katas

[GitBox]

codecov[bot] edited a comment on pull request #12941:
URL: https://github.com/apache/beam/pull/12941#issuecomment-698987193


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12941?src=pr=h1) Report
   > Merging 
[#12941](https://codecov.io/gh/apache/beam/pull/12941?src=pr=desc) into 
[master](https://codecov.io/gh/apache/beam/commit/aa58e1e5db4af2a6f97520756831e87aa1d3e3fb?el=desc)
 will **decrease** coverage by `0.13%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/beam/pull/12941/graphs/tree.svg?width=650=150=pr=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12941?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master   #12941  +/-   ##
   ==
   - Coverage   82.43%   82.30%   -0.14% 
   ==
 Files 454  455   +1 
 Lines   5434555166 +821 
   ==
   + Hits4479945404 +605 
   - Misses   9546 9762 +216 
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/beam/pull/12941?src=pr=tree) | Coverage 
Δ | |
   |---|---|---|
   | 
[sdks/python/apache\_beam/io/gcp/pubsub.py](https://codecov.io/gh/apache/beam/pull/12941/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3B1YnN1Yi5weQ==)
 | `92.30% <0.00%> (-1.28%)` | :arrow_down: |
   | 
[...dks/python/apache\_beam/options/pipeline\_options.py](https://codecov.io/gh/apache/beam/pull/12941/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vb3B0aW9ucy9waXBlbGluZV9vcHRpb25zLnB5)
 | `93.76% <0.00%> (-0.99%)` | :arrow_down: |
   | 
[...on/apache\_beam/runners/dataflow/dataflow\_runner.py](https://codecov.io/gh/apache/beam/pull/12941/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9kYXRhZmxvd19ydW5uZXIucHk=)
 | `76.93% <0.00%> (-0.31%)` | :arrow_down: |
   | 
[sdks/python/apache\_beam/io/iobase.py](https://codecov.io/gh/apache/beam/pull/12941/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vaW9iYXNlLnB5)
 | `84.03% <0.00%> (-0.30%)` | :arrow_down: |
   | 
[sdks/python/apache\_beam/transforms/util.py](https://codecov.io/gh/apache/beam/pull/12941/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy91dGlsLnB5)
 | `95.44% <0.00%> (-0.19%)` | :arrow_down: |
   | 
[sdks/python/apache\_beam/transforms/environments.py](https://codecov.io/gh/apache/beam/pull/12941/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9lbnZpcm9ubWVudHMucHk=)
 | `83.73% <0.00%> (-0.14%)` | :arrow_down: |
   | 
[...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/12941/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==)
 | `94.45% <0.00%> (-0.14%)` | :arrow_down: |
   | 
[sdks/python/apache\_beam/coders/coder\_impl.py](https://codecov.io/gh/apache/beam/pull/12941/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL2NvZGVyX2ltcGwucHk=)
 | `95.13% <0.00%> (-0.12%)` | :arrow_down: |
   | 
[sdks/python/apache\_beam/dataframe/io.py](https://codecov.io/gh/apache/beam/pull/12941/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2lvLnB5)
 | `92.00% <0.00%> (-0.10%)` | :arrow_down: |
   | 
[sdks/python/apache\_beam/io/fileio.py](https://codecov.io/gh/apache/beam/pull/12941/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZmlsZWlvLnB5)
 | `95.80% <0.00%> (-0.05%)` | :arrow_down: |
   | ... and [25 
more](https://codecov.io/gh/apache/beam/pull/12941/diff?src=pr=tree-more) | |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/beam/pull/12941?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/beam/pull/12941?src=pr=footer). Last 
update 
[aa58e1e...832cbde](https://codecov.io/gh/apache/beam/pull/12941?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] tysonjh commented on a change in pull request #11856: [BEAM-7505] SideInput Python Load Test job

[GitBox]

tysonjh commented on a change in pull request #11856:
URL: https://github.com/apache/beam/pull/11856#discussion_r497786859



##
File path: sdks/python/apache_beam/testing/load_tests/sideinput_test.py
##
@@ -122,85 +122,95 @@ def process(self, unused_element, side_input):
 # No-op. We only make sure that the element is accessed.
 next(it)
   except StopIteration:
-return
+break
 
 class MappingSideInputTestDoFn(beam.DoFn):

Review comment:
   Add typehints here?

##
File path: sdks/python/apache_beam/testing/load_tests/sideinput_test.py
##
@@ -122,85 +122,95 @@ def process(self, unused_element, side_input):
 # No-op. We only make sure that the element is accessed.
 next(it)
   except StopIteration:
-return
+break
 
 class MappingSideInputTestDoFn(beam.DoFn):
-  """Take a sequence of keys as an additional side input and for each
-  key in the sequence checks the value for key in the dictionary."""
-  def process(self, unused_element, dict_side_input, keys_to_check):
-for key in keys_to_check:
-  # No-op. We only make sure that the element is accessed.
-  dict_side_input[key]
-
-class GetRandomKeys(beam.DoFn):
-  def __init__(self, n):
-self._n = n
+  """Iterates over first n keys in the dictionary and checks the value."""
+  def __init__(self, first_n):
+self._first_n = first_n
 
   def process(self, unused_element, dict_side_input):
-import random
-n = min(self._n, len(dict_side_input))
-return random.sample(dict_side_input.keys(), n)
+i = 0
+for key in dict_side_input:
+  if i == self._first_n:
+break
+  # No-op. We only make sure that the element is accessed.
+  dict_side_input[key]
+  i += 1
 
-class AddEventTimestamps(beam.DoFn):
-  """Assign timestamp to each element of PCollection."""
-  def setup(self):
-self._timestamp = 0
+@typehints.with_input_types(int)
+@typehints.with_output_types(int)
+class AssignTimestamps(beam.DoFn):
+  """Produces timestamped values. Timestamps are equal to the value of the
+  element."""
+  def __init__(self):
+# Avoid having to use save_main_session
+self.window = window

Review comment:
   What does this do? 

##
File path: sdks/python/apache_beam/testing/load_tests/sideinput_test.py
##
@@ -122,85 +122,95 @@ def process(self, unused_element, side_input):
 # No-op. We only make sure that the element is accessed.
 next(it)
   except StopIteration:
-return
+break
 
 class MappingSideInputTestDoFn(beam.DoFn):
-  """Take a sequence of keys as an additional side input and for each
-  key in the sequence checks the value for key in the dictionary."""
-  def process(self, unused_element, dict_side_input, keys_to_check):
-for key in keys_to_check:
-  # No-op. We only make sure that the element is accessed.
-  dict_side_input[key]
-
-class GetRandomKeys(beam.DoFn):
-  def __init__(self, n):
-self._n = n
+  """Iterates over first n keys in the dictionary and checks the value."""
+  def __init__(self, first_n):
+self._first_n = first_n
 
   def process(self, unused_element, dict_side_input):
-import random
-n = min(self._n, len(dict_side_input))
-return random.sample(dict_side_input.keys(), n)
+i = 0
+for key in dict_side_input:
+  if i == self._first_n:
+break
+  # No-op. We only make sure that the element is accessed.
+  dict_side_input[key]
+  i += 1
 
-class AddEventTimestamps(beam.DoFn):
-  """Assign timestamp to each element of PCollection."""
-  def setup(self):
-self._timestamp = 0
+@typehints.with_input_types(int)
+@typehints.with_output_types(int)
+class AssignTimestamps(beam.DoFn):
+  """Produces timestamped values. Timestamps are equal to the value of the
+  element."""
+  def __init__(self):
+# Avoid having to use save_main_session
+self.window = window
 
   def process(self, element):
-from apache_beam.transforms.combiners import window
-yield window.TimestampedValue(element, self._timestamp)
-self._timestamp += 1
-
-input_pc = (
-self.pipeline
-| 'Read synthetic' >> beam.io.Read(
-SyntheticSource(self.parse_synthetic_source_options()))
-| 'Collect start time metrics' >> beam.ParDo(
-MeasureTime(self.metrics_namespace)))
-
-if self.side_input_size != self.input_options.get('num_records'):
-  side_input = (
-  input_pc
-  | 'Sample {} elements'.format(self.side_input_size) >>
-  

[GitHub] [beam] lostluck merged pull request #12941: [BEAM-9681] Add textio.Read lesson to Go SDK katas

[GitBox]

lostluck merged pull request #12941:
URL: https://github.com/apache/beam/pull/12941


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] kennknowles opened a new pull request #12980: [BEAM-10991] Revert "[BEAM-8543] Dataflow streaming timers are not strictly time ordered when set earlier mid-bundle (#11924)"

[GitBox]

kennknowles opened a new pull request #12980:
URL: https://github.com/apache/beam/pull/12980


   This reverts commit 88acc5267f759d81e9836a9db17b9e0ee521c785.
   
   There is a bug in this commit that causes deleted timers to not clear their
   watermark holds, resulting in stuck pipelines. See
   https://issues.apache.org/jira/browse/BEAM-10991
   
   
   
   Thank you for your contribution! Follow this checklist to help us 
incorporate your contribution quickly and easily:
   
- [x] [**Choose 
reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and 
mention them in a comment (`R: @username`).
- [x] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [x] Update `CHANGES.md` with noteworthy changes.
- [x] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more 
tips on [how to make review process 
smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
   --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/)
 | --- | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/)
 | --- | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
 | ---
   Java | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/)
 | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[![Build
 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/)
 | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[![Build
 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/badge/i
 
con)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)[![Build
 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[![Build
 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/)
 | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/)
 | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[![Build
 Status](htt
 
ps://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[![Build
 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
 | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
   Python | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[![Build
 

[GitHub] [beam] lostluck commented on pull request #12505: [BEAM-8106] Add version to java container image name

[GitBox]

lostluck commented on pull request #12505:
URL: https://github.com/apache/beam/pull/12505#issuecomment-701684906


   Ack. Thanks!
   
   On Tue, Sep 29, 2020, 5:38 PM Brian Hulette 
   wrote:
   
   > Thank you!
   >
   > —
   > You are receiving this because you were mentioned.
   > Reply to this email directly, view it on GitHub
   > , or
   > unsubscribe
   > 

   > .
   >
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] kennknowles commented on pull request #12980: [BEAM-10991] Revert "[BEAM-8543] Dataflow streaming timers are not strictly time ordered when set earlier mid-bundle (#11924)"

[GitBox]

kennknowles commented on pull request #12980:
URL: https://github.com/apache/beam/pull/12980#issuecomment-701684994


   CC @robinyqiu we will want to cherrypick this



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] aaltay commented on pull request #12980: [BEAM-10991] Revert "[BEAM-8543] Dataflow streaming timers are not strictly time ordered when set earlier mid-bundle (#11924)"

[GitBox]

aaltay commented on pull request #12980:
URL: https://github.com/apache/beam/pull/12980#issuecomment-701691856


   /cc @robinyqiu - This change also needs to go to the release branch.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] rohdesamuel opened a new pull request #12983: [BEAM-10994] Add hotKeyLoggingEnabled to the Java DataflowPipelineOptions

[GitBox]

rohdesamuel opened a new pull request #12983:
URL: https://github.com/apache/beam/pull/12983


   This will ultimately control whether or not the literal hot key content is 
logged to Cloud Logging.
   
   
   
   Thank you for your contribution! Follow this checklist to help us 
incorporate your contribution quickly and easily:
   
- [ ] [**Choose 
reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and 
mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more 
tips on [how to make review process 
smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
   --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/)
 | --- | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/)
 | --- | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
 | ---
   Java | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/)
 | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[![Build
 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/)
 | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[![Build
 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/badge/i
 
con)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)[![Build
 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[![Build
 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/)
 | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/)
 | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[![Build
 Status](htt
 
ps://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[![Build
 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
 | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
   Python | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[![Build
 

[GitHub] [beam] kileys commented on a change in pull request #12970: [BEAM-8024] Add dataflow and flink runner to JPMS test

[GitBox]

kileys commented on a change in pull request #12970:
URL: https://github.com/apache/beam/pull/12970#discussion_r497962184



##
File path: sdks/java/testing/jpms-tests/build.gradle
##
@@ -31,13 +34,66 @@ enableJavaPerformanceTesting()
 description = "Apache Beam :: SDKs :: Java :: Testing :: JPMS Tests"
 ext.summary = "E2E test for Java 9 modules"
 
+/*
+ * List of runners to run post commit tests on.
+ */
+def postCommitRunnerClass = [
+directRunner: "org.apache.beam.runners.direct.DirectRunner",
+flinkRunner: "org.apache.beam.runners.flink.TestFlinkRunner",
+dataflowRunner: "org.apache.beam.runners.dataflow.TestDataflowRunner",
+]
+for (String runner : postCommitRunnerClass.keySet()) {
+  configurations.create(runner + "PostCommit")
+}
+
 dependencies {
   compile project(path: ":sdks:java:core", configuration: "shadow")
-  compile project(path: ":runners:direct-java")
   compile project(path: ":sdks:java:extensions:google-cloud-platform-core")
 
   testCompile library.java.junit
   testCompile library.java.hamcrest_core
+
+  // Add dependencies for the PostCommit configurations
+  // For each runner a project level dependency on the test project.
+  for (String runner : postCommitRunnerClass.keySet()) {
+delegate.add(runner + "PostCommit", 
project(":sdks:java:testing:jpms-tests"))
+delegate.add(runner + "PostCommit", project(path: 
":sdks:java:testing:jpms-tests", configuration: "testRuntime"))
+  }
+  directRunnerPostCommit project(":runners:direct-java")
+  flinkRunnerPostCommit project(":runners:flink:1.10")
+  dataflowRunnerPostCommit project(":runners:google-cloud-dataflow-java")
+}
+
+/*
+ * Create a ${runner}PostCommit task for each runner which runs a set
+ * of integration tests for WordCount and WindowedWordCount.
+ */
+def gcpProject = project.findProperty('gcpProject') ?: 'apache-beam-testing'
+def gcpRegion = project.findProperty('gcpRegion') ?: 'us-central1'
+def gcsTempRoot = project.findProperty('gcsTempRoot') ?: 
'gs://temp-storage-for-end-to-end-tests/'
+
+for (String runner : postCommitRunnerClass.keySet()) {
+  tasks.create(name: runner + "PostCommit", type: Test) {
+def postCommitBeamTestPipelineOptions = [
+"--project=${gcpProject}",
+"--tempRoot=${gcsTempRoot}",
+"--runner=" + postCommitRunnerClass[runner],
+]
+if ("dataflowRunner".equals(runner)) {
+  postCommitBeamTestPipelineOptions.add("--region=${gcpRegion}")
+}
+classpath = configurations."${runner}PostCommit"
+include "**/*IT.class"
+maxParallelForks 4
+systemProperty "beamTestPipelineOptions", 
JsonOutput.toJson(postCommitBeamTestPipelineOptions)
+  }
+}
+
+/* Define a common postcommit task which depends on all the individual 
postcommits. */

Review comment:
   Makes sense, changed the post commit references to integration test





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] y1chi commented on pull request #12562: [BEAM-10200] Respect profile_memory option and add memory profiler to…

[GitBox]

y1chi commented on pull request #12562:
URL: https://github.com/apache/beam/pull/12562#issuecomment-701672888


   @angoenka I think this PR can be merged



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] lostluck commented on pull request #12941: [BEAM-9681] Add textio.Read lesson to Go SDK katas

[GitBox]

lostluck commented on pull request #12941:
URL: https://github.com/apache/beam/pull/12941#issuecomment-701672348


   The only reason the python tests (and java tests) are running are because of 
the testdata expansion which isn't something that would have caused the tests 
to break. Given the common failure at head, I'm inclined to ignore it. 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] kennknowles commented on pull request #12980: [BEAM-10991] Revert "[BEAM-8543] Dataflow streaming timers are not strictly time ordered when set earlier mid-bundle (#11924)"

[GitBox]

kennknowles commented on pull request #12980:
URL: https://github.com/apache/beam/pull/12980#issuecomment-701692388


   run dataflow validatesrunner



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] kennknowles commented on pull request #12981: [BEAM-10435] Add LOOPBACK mode Java ValidatesRunner test suite for ULR

[GitBox]

kennknowles commented on pull request #12981:
URL: https://github.com/apache/beam/pull/12981#issuecomment-701701451


   Pushing whitespace-only fixup



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] kennknowles commented on pull request #12981: [BEAM-10435] Add LOOPBACK mode Java ValidatesRunner test suite for ULR

[GitBox]

kennknowles commented on pull request #12981:
URL: https://github.com/apache/beam/pull/12981#issuecomment-701701366


   Noting that the whitespace check is pulled in by `applyPythonNature` which 
is too big a hammer but refactoring will be some work. And it seems `./gradlew 
applySpotless` does not eliminate it even though we could turn it on for the 
groovy.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] codecov[bot] commented on pull request #12385: [BEAM-10527] Migrate Flink and Spark tests to pytest.

[GitBox]

codecov[bot] commented on pull request #12385:
URL: https://github.com/apache/beam/pull/12385#issuecomment-701745199


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12385?src=pr=h1) Report
   > Merging 
[#12385](https://codecov.io/gh/apache/beam/pull/12385?src=pr=desc) into 
[master](https://codecov.io/gh/apache/beam/commit/7c91a63aa746c25eccea1d296a0b8ea948a0ed73?el=desc)
 will **increase** coverage by `0.11%`.
   > The diff coverage is `79.48%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/beam/pull/12385/graphs/tree.svg?width=650=150=pr=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12385?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master   #12385  +/-   ##
   ==
   + Coverage   82.39%   82.50%   +0.11% 
   ==
 Files 453  453  
 Lines   5462354612  -11 
   ==
   + Hits4500545059  +54 
   + Misses   9618 9553  -65 
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/beam/pull/12385?src=pr=tree) | Coverage 
Δ | |
   |---|---|---|
   | 
[sdks/python/apache\_beam/io/fileio.py](https://codecov.io/gh/apache/beam/pull/12385/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZmlsZWlvLnB5)
 | `95.80% <ø> (ø)` | |
   | 
[sdks/python/apache\_beam/io/gcp/bigquery.py](https://codecov.io/gh/apache/beam/pull/12385/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5LnB5)
 | `80.23% <0.00%> (+0.14%)` | :arrow_up: |
   | 
[sdks/python/apache\_beam/io/gcp/bigquery\_tools.py](https://codecov.io/gh/apache/beam/pull/12385/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X3Rvb2xzLnB5)
 | `88.35% <0.00%> (ø)` | |
   | 
[...eam/testing/benchmarks/nexmark/nexmark\_launcher.py](https://codecov.io/gh/apache/beam/pull/12385/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvbmV4bWFya19sYXVuY2hlci5weQ==)
 | `0.00% <0.00%> (ø)` | |
   | 
[...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/12385/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5)
 | `79.53% <66.66%> (ø)` | |
   | 
[...dks/python/apache\_beam/options/pipeline\_options.py](https://codecov.io/gh/apache/beam/pull/12385/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vb3B0aW9ucy9waXBlbGluZV9vcHRpb25zLnB5)
 | `93.76% <70.58%> (ø)` | |
   | 
[sdks/python/apache\_beam/transforms/environments.py](https://codecov.io/gh/apache/beam/pull/12385/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9lbnZpcm9ubWVudHMucHk=)
 | `83.73% <83.33%> (ø)` | |
   | 
[.../apache\_beam/options/pipeline\_options\_validator.py](https://codecov.io/gh/apache/beam/pull/12385/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vb3B0aW9ucy9waXBlbGluZV9vcHRpb25zX3ZhbGlkYXRvci5weQ==)
 | `98.69% <100.00%> (ø)` | |
   | 
[...ache\_beam/runners/interactive/recording\_manager.py](https://codecov.io/gh/apache/beam/pull/12385/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9yZWNvcmRpbmdfbWFuYWdlci5weQ==)
 | `98.90% <100.00%> (ø)` | |
   | ... and [15 
more](https://codecov.io/gh/apache/beam/pull/12385/diff?src=pr=tree-more) | |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/beam/pull/12385?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/beam/pull/12385?src=pr=footer). Last 
update 
[4161140...7219836](https://codecov.io/gh/apache/beam/pull/12385?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #12385: [BEAM-10527] Migrate Flink and Spark tests to pytest.

[GitBox]

codecov[bot] edited a comment on pull request #12385:
URL: https://github.com/apache/beam/pull/12385#issuecomment-701745199


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12385?src=pr=h1) Report
   > Merging 
[#12385](https://codecov.io/gh/apache/beam/pull/12385?src=pr=desc) into 
[master](https://codecov.io/gh/apache/beam/commit/7c91a63aa746c25eccea1d296a0b8ea948a0ed73?el=desc)
 will **increase** coverage by `0.11%`.
   > The diff coverage is `79.48%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/beam/pull/12385/graphs/tree.svg?width=650=150=pr=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12385?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master   #12385  +/-   ##
   ==
   + Coverage   82.39%   82.50%   +0.11% 
   ==
 Files 453  453  
 Lines   5462354612  -11 
   ==
   + Hits4500545059  +54 
   + Misses   9618 9553  -65 
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/beam/pull/12385?src=pr=tree) | Coverage 
Δ | |
   |---|---|---|
   | 
[sdks/python/apache\_beam/io/fileio.py](https://codecov.io/gh/apache/beam/pull/12385/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZmlsZWlvLnB5)
 | `95.80% <ø> (ø)` | |
   | 
[sdks/python/apache\_beam/io/gcp/bigquery.py](https://codecov.io/gh/apache/beam/pull/12385/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5LnB5)
 | `80.23% <0.00%> (+0.14%)` | :arrow_up: |
   | 
[sdks/python/apache\_beam/io/gcp/bigquery\_tools.py](https://codecov.io/gh/apache/beam/pull/12385/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X3Rvb2xzLnB5)
 | `88.35% <0.00%> (ø)` | |
   | 
[...eam/testing/benchmarks/nexmark/nexmark\_launcher.py](https://codecov.io/gh/apache/beam/pull/12385/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvbmV4bWFya19sYXVuY2hlci5weQ==)
 | `0.00% <0.00%> (ø)` | |
   | 
[...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/12385/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5)
 | `79.53% <66.66%> (ø)` | |
   | 
[...dks/python/apache\_beam/options/pipeline\_options.py](https://codecov.io/gh/apache/beam/pull/12385/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vb3B0aW9ucy9waXBlbGluZV9vcHRpb25zLnB5)
 | `93.76% <70.58%> (ø)` | |
   | 
[sdks/python/apache\_beam/transforms/environments.py](https://codecov.io/gh/apache/beam/pull/12385/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9lbnZpcm9ubWVudHMucHk=)
 | `83.73% <83.33%> (ø)` | |
   | 
[.../apache\_beam/options/pipeline\_options\_validator.py](https://codecov.io/gh/apache/beam/pull/12385/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vb3B0aW9ucy9waXBlbGluZV9vcHRpb25zX3ZhbGlkYXRvci5weQ==)
 | `98.69% <100.00%> (ø)` | |
   | 
[...ache\_beam/runners/interactive/recording\_manager.py](https://codecov.io/gh/apache/beam/pull/12385/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9yZWNvcmRpbmdfbWFuYWdlci5weQ==)
 | `98.90% <100.00%> (ø)` | |
   | ... and [15 
more](https://codecov.io/gh/apache/beam/pull/12385/diff?src=pr=tree-more) | |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/beam/pull/12385?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/beam/pull/12385?src=pr=footer). Last 
update 
[4161140...7219836](https://codecov.io/gh/apache/beam/pull/12385?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #12385: [BEAM-10527] Migrate Flink and Spark tests to pytest.

[GitBox]

codecov[bot] edited a comment on pull request #12385:
URL: https://github.com/apache/beam/pull/12385#issuecomment-701745199


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12385?src=pr=h1) Report
   > Merging 
[#12385](https://codecov.io/gh/apache/beam/pull/12385?src=pr=desc) into 
[master](https://codecov.io/gh/apache/beam/commit/7c91a63aa746c25eccea1d296a0b8ea948a0ed73?el=desc)
 will **increase** coverage by `0.11%`.
   > The diff coverage is `79.48%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/beam/pull/12385/graphs/tree.svg?width=650=150=pr=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12385?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master   #12385  +/-   ##
   ==
   + Coverage   82.39%   82.50%   +0.11% 
   ==
 Files 453  453  
 Lines   5462354612  -11 
   ==
   + Hits4500545059  +54 
   + Misses   9618 9553  -65 
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/beam/pull/12385?src=pr=tree) | Coverage 
Δ | |
   |---|---|---|
   | 
[sdks/python/apache\_beam/io/fileio.py](https://codecov.io/gh/apache/beam/pull/12385/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZmlsZWlvLnB5)
 | `95.80% <ø> (ø)` | |
   | 
[sdks/python/apache\_beam/io/gcp/bigquery.py](https://codecov.io/gh/apache/beam/pull/12385/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5LnB5)
 | `80.23% <0.00%> (+0.14%)` | :arrow_up: |
   | 
[sdks/python/apache\_beam/io/gcp/bigquery\_tools.py](https://codecov.io/gh/apache/beam/pull/12385/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X3Rvb2xzLnB5)
 | `88.35% <0.00%> (ø)` | |
   | 
[...eam/testing/benchmarks/nexmark/nexmark\_launcher.py](https://codecov.io/gh/apache/beam/pull/12385/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvbmV4bWFya19sYXVuY2hlci5weQ==)
 | `0.00% <0.00%> (ø)` | |
   | 
[...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/12385/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5)
 | `79.53% <66.66%> (ø)` | |
   | 
[...dks/python/apache\_beam/options/pipeline\_options.py](https://codecov.io/gh/apache/beam/pull/12385/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vb3B0aW9ucy9waXBlbGluZV9vcHRpb25zLnB5)
 | `93.76% <70.58%> (ø)` | |
   | 
[sdks/python/apache\_beam/transforms/environments.py](https://codecov.io/gh/apache/beam/pull/12385/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9lbnZpcm9ubWVudHMucHk=)
 | `83.73% <83.33%> (ø)` | |
   | 
[.../apache\_beam/options/pipeline\_options\_validator.py](https://codecov.io/gh/apache/beam/pull/12385/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vb3B0aW9ucy9waXBlbGluZV9vcHRpb25zX3ZhbGlkYXRvci5weQ==)
 | `98.69% <100.00%> (ø)` | |
   | 
[...ache\_beam/runners/interactive/recording\_manager.py](https://codecov.io/gh/apache/beam/pull/12385/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9yZWNvcmRpbmdfbWFuYWdlci5weQ==)
 | `98.90% <100.00%> (ø)` | |
   | ... and [15 
more](https://codecov.io/gh/apache/beam/pull/12385/diff?src=pr=tree-more) | |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/beam/pull/12385?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/beam/pull/12385?src=pr=footer). Last 
update 
[4161140...7219836](https://codecov.io/gh/apache/beam/pull/12385?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] damondouglas commented on pull request #12941: [BEAM-9681] Add textio.Read lesson to Go SDK katas

[GitBox]

damondouglas commented on pull request #12941:
URL: https://github.com/apache/beam/pull/12941#issuecomment-701647073


   Hello @pabloem the [Python ("Run Python 
PreCommit")](https://ci-beam.apache.org/job/beam_PreCommit_Python_Commit/15464/)
 is what I was referring to that failed after 832cbde



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] tvalentyn commented on a change in pull request #12927: Update some of the Python dependencies from Beam Dependency Check Report

[GitBox]

tvalentyn commented on a change in pull request #12927:
URL: https://github.com/apache/beam/pull/12927#discussion_r497830780



##
File path: sdks/python/apache_beam/io/avroio.py
##
@@ -627,11 +627,19 @@ def write_record(self, writer, value):
 writer.append(value)
 
 
+class _FastAvroWriter(Writer):
+  """An adapter class which exposes a file handle so that it can be closed
+  by the sink. """
+  def __init__(self, file_handle, schema, codec):
+super(_FastAvroWriter, self).__init__(file_handle, schema, codec)

Review comment:
   Let's do the following:
   
   1. File a JIRA for this issue, mention in the the error message that you 
observed. Let's mention in the JIRA  the workaround (install cython or 
downgrade fastavro to 0.21.24) in case users discover this error.
   
   2. Add a workaround in code:
   ```
   try:
 writer.fo.close()
   except AttributeError:
 # We can clean this up after fastavro lower bound includes  a fix for 
https://github.com/fastavro/fastavro/issues/472.
 writer.encoder.close()
   ```
   3. Test that the workaround works.
   
   Thanks for spotting this.  How did you catch it by the way? Was it a unit 
test failure that occurred locally? I wonder why our non-cython test suite does 
not catch it.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] tvalentyn commented on a change in pull request #12927: Update some of the Python dependencies from Beam Dependency Check Report

[GitBox]

tvalentyn commented on a change in pull request #12927:
URL: https://github.com/apache/beam/pull/12927#discussion_r497830780



##
File path: sdks/python/apache_beam/io/avroio.py
##
@@ -627,11 +627,19 @@ def write_record(self, writer, value):
 writer.append(value)
 
 
+class _FastAvroWriter(Writer):
+  """An adapter class which exposes a file handle so that it can be closed
+  by the sink. """
+  def __init__(self, file_handle, schema, codec):
+super(_FastAvroWriter, self).__init__(file_handle, schema, codec)

Review comment:
   Let's do the following:
   
   1. File a JIRA for this issue, mention in the Jira the error message that 
you observed. Let's mention in the JIRA  the workaround (install cython or 
downgrade fastavro to 0.21.24) in case users discover this error.
   
   2. Add a workaround in code:
   ```
   try:
 writer.fo.close()
   except AttributeError:
 # We can clean this up after fastavro lower bound includes  a fix for 
https://github.com/fastavro/fastavro/issues/472.
 writer.encoder.close()
   ```
   3. Test that the workaround works.
   
   Thanks for spotting this.  How did you catch it by the way? Was it a unit 
test failure that occurred locally? I wonder why our non-cython test suite does 
not catch it.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] reuvenlax commented on pull request #12980: [BEAM-10991] Revert "[BEAM-8543] Dataflow streaming timers are not strictly time ordered when set earlier mid-bundle (#11924)"

[GitBox]

reuvenlax commented on pull request #12980:
URL: https://github.com/apache/beam/pull/12980#issuecomment-701690047


   LGTM



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] kennknowles commented on a change in pull request #12981: [BEAM-10435] Add LOOPBACK mode Java ValidatesRunner test suite for ULR

[GitBox]

kennknowles commented on a change in pull request #12981:
URL: https://github.com/apache/beam/pull/12981#discussion_r497849873



##
File path: runners/portability/java/build.gradle
##
@@ -50,27 +50,32 @@ dependencies {
   validatesRunner project(path: project.path, configuration: "testRuntime")
 }
 
-
 project.evaluationDependsOn(":sdks:java:core")
+project.evaluationDependsOn(":sdks:python")
 project.evaluationDependsOn(":runners:core-java")
 
-ext.localJobServicePidFile = "${project.buildDir}/local_job_service_pid"
+ext.pythonSdkDir = 
project.findProject(":sdks:python").getProjectDir().getAbsoluteFile().toString()
+
+// If this is set via -P then we assume it is already running and will not 
start it
+def needsLocalJobService = !project.hasProperty("localJobServicePortFile")

Review comment:
   Noting that my previous approach did not work. The property was always 
set by the time the conditional was checked. I had only verified it against a 
job service that I launched externally so did not catch that the logic was 
flawed. This approach does work, by setting `needsLocalJobService` earlier.

##
File path: runners/portability/java/build.gradle
##
@@ -50,27 +50,32 @@ dependencies {
   validatesRunner project(path: project.path, configuration: "testRuntime")
 }
 
-
 project.evaluationDependsOn(":sdks:java:core")
+project.evaluationDependsOn(":sdks:python")
 project.evaluationDependsOn(":runners:core-java")
 
-ext.localJobServicePidFile = "${project.buildDir}/local_job_service_pid"
+ext.pythonSdkDir = 
project.findProject(":sdks:python").getProjectDir().getAbsoluteFile().toString()
+
+// If this is set via -P then we assume it is already running and will not 
start it
+def needsLocalJobService = !project.hasProperty("localJobServicePortFile")
+
 ext.localJobServicePortFile = project.hasProperty("localJobServicePortFile") ? 
project.property("localJobServicePortFile") : 
"${project.buildDir}/local_job_service_port"
+
 ext.localJobServiceStdoutFile = "${project.buildDir}/local_job_service_stdout"
+ext.localJobServicePidFile = "${project.buildDir}/local_job_service_pid"
 
 void execInVirtualenv(String... args) {
   String shellCommand = ". ${project.ext.envdir}/bin/activate && " + 
args.collect { arg -> "'" + arg.replaceAll("'", "\\'") + "'" }.join(" ")
   exec {
-workingDir pythonSdkDir
+workingDir project.ext.pythonSdkDir

Review comment:
   The prior reference to `pythonSdkDir` was actually an unbound variable.

##
File path: runners/portability/java/build.gradle
##
@@ -108,95 +120,113 @@ startLocalJobService.finalizedBy stopLocalJobService
 
 /**
  * Runs Java ValidatesRunner tests against the Universal Local Runner (ULR) 
aka local_job_service_main
- * with subprocess SDK harness environments.
+ * with the specified environment type.
  */
-task ulrValidatesRunnerTests(type: Test) {
-  dependsOn ":sdks:java:container:docker"
-
-  if (!project.hasProperty("localJobServicePortFile")) {
-dependsOn startLocalJobService
+def createUlrValidatesRunnerTask = { name, environmentType ->
+  Task vrTask = tasks.create(name: name, type: Test, group: "Verification") {
+description "PortableRunner Java docker ValidatesRunner suite"
+classpath = configurations.validatesRunner
+systemProperty "beamTestPipelineOptions", JsonOutput.toJson([
+"--runner=TestUniversalRunner",
+"--experiments=beam_fn_api",
+"--defaultEnvironmentType=${environmentType}",
+"--localJobServicePortFile=${localJobServicePortFile}"
+])
+testClassesDirs = 
files(project(":sdks:java:core").sourceSets.test.output.classesDirs)
+useJUnit {
+  includeCategories 'org.apache.beam.sdk.testing.ValidatesRunner'
+  excludeCategories 'org.apache.beam.sdk.testing.UsesGaugeMetrics'
+  excludeCategories 'org.apache.beam.sdk.testing.UsesOnWindowExpiration'
+  excludeCategories 'org.apache.beam.sdk.testing.UsesBundleFinalizer'
+  excludeCategories 'org.apache.beam.sdk.testing.UsesMapState'
+  excludeCategories 'org.apache.beam.sdk.testing.UsesSetState'
+  excludeCategories 'org.apache.beam.sdk.testing.UsesOrderedListState'
+  excludeCategories 'org.apache.beam.sdk.testing.UsesTestStream'

Review comment:
   Previously, only a special case of `UsesTestStreamWithMultipleStages` 
was sickbayed. But I got a failure of being unable to understand the URN 
`beam:transforms:teststream:v1` so I interpret that to mean it is not supported 
at all.

##
File path: runners/portability/java/build.gradle
##
@@ -82,8 +87,15 @@ void execBackgroundInVirtualenv(String... args) {
   proc.waitFor();
 }
 
-task startLocalJobService {
+task installBeamPythonInVirtualenv {
   dependsOn setupVirtualenv
+  doLast {
+execInVirtualenv "pip", "install", "-e", "."

Review comment:
   I did not understand previously that the Beam SDK and requirements were 
not installed into the virtualenv. Again, masked by the fact 

[GitHub] [beam] tvalentyn commented on a change in pull request #12981: [BEAM-10435] Add LOOPBACK mode Java ValidatesRunner test suite for ULR

[GitBox]

tvalentyn commented on a change in pull request #12981:
URL: https://github.com/apache/beam/pull/12981#discussion_r497854932



##
File path: runners/portability/java/build.gradle
##
@@ -108,95 +120,113 @@ startLocalJobService.finalizedBy stopLocalJobService
 
 /**
  * Runs Java ValidatesRunner tests against the Universal Local Runner (ULR) 
aka local_job_service_main
- * with subprocess SDK harness environments.
+ * with the specified environment type.
  */
-task ulrValidatesRunnerTests(type: Test) {
-  dependsOn ":sdks:java:container:docker"
-
-  if (!project.hasProperty("localJobServicePortFile")) {
-dependsOn startLocalJobService
+def createUlrValidatesRunnerTask = { name, environmentType ->
+  Task vrTask = tasks.create(name: name, type: Test, group: "Verification") {
+description "PortableRunner Java docker ValidatesRunner suite"
+classpath = configurations.validatesRunner
+systemProperty "beamTestPipelineOptions", JsonOutput.toJson([
+"--runner=TestUniversalRunner",
+"--experiments=beam_fn_api",
+"--defaultEnvironmentType=${environmentType}",
+"--localJobServicePortFile=${localJobServicePortFile}"
+])
+testClassesDirs = 
files(project(":sdks:java:core").sourceSets.test.output.classesDirs)
+useJUnit {
+  includeCategories 'org.apache.beam.sdk.testing.ValidatesRunner'
+  excludeCategories 'org.apache.beam.sdk.testing.UsesGaugeMetrics'
+  excludeCategories 'org.apache.beam.sdk.testing.UsesOnWindowExpiration'
+  excludeCategories 'org.apache.beam.sdk.testing.UsesBundleFinalizer'

Review comment:
   Do we need a TODO for any of these exclusions?





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] robertwb opened a new pull request #12982: [BEAM-9547] Dataframe covariance and correlation.

[GitBox]

robertwb opened a new pull request #12982:
URL: https://github.com/apache/beam/pull/12982


   This implements correlation, covariance, and standard deviation for series 
and dataframes.
   
   It is split up into three commits. The first is the math-heavy one that 
implements the distributed algorithm, the second uses it everywhere, and the 
third adds an apply method to scalars to simplify things. 
   
   
   
   Thank you for your contribution! Follow this checklist to help us 
incorporate your contribution quickly and easily:
   
- [ ] [**Choose 
reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and 
mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more 
tips on [how to make review process 
smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
   --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/)
 | --- | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/)
 | --- | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
 | ---
   Java | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/)
 | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[![Build
 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/)
 | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[![Build
 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/badge/i
 
con)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)[![Build
 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[![Build
 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/)
 | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/)
 | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[![Build
 Status](htt
 
ps://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[![Build
 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
 | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
   Python | [![Build 

[GitHub] [beam] robertwb commented on pull request #12982: [BEAM-9547] Dataframe covariance and correlation.

[GitBox]

robertwb commented on pull request #12982:
URL: https://github.com/apache/beam/pull/12982#issuecomment-701708339


   R: @TheNeuralBit 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] boyuanzz commented on pull request #12930: [BEAM-10962] Add Multiple PubSub reader to Python SDK

[GitBox]

boyuanzz commented on pull request #12930:
URL: https://github.com/apache/beam/pull/12930#issuecomment-701708484


   Expect what Cham has mentioned, another thing is current implementation of 
`MultipleReadFromPubSub` only can configure multiple `ReadFromPubSub` with the 
same attribute, like the same `with_attributes`, `timestamp_label`, `id_label`, 
which is not ideal. Given that `ReadPubSub` is a native transform for Dataflow, 
having `MultipleReadFromPubSub` seems like the only solution for now. I'm 
thinking we could create a `PubSubSourceDescriptor` which includes `topic`, 
`subscription` and other attributes. And we expose `add` API from 
`MultipleReadFromPubSub` to allow end users to add a new Read.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] boyuanzz edited a comment on pull request #12930: [BEAM-10962] Add Multiple PubSub reader to Python SDK

[GitBox]

boyuanzz edited a comment on pull request #12930:
URL: https://github.com/apache/beam/pull/12930#issuecomment-701708484


   Except what Cham has mentioned, another thing is current implementation of 
`MultipleReadFromPubSub` only can configure multiple `ReadFromPubSub` with the 
same attribute, like the same `with_attributes`, `timestamp_label`, `id_label`, 
which is not ideal. Given that `ReadPubSub` is a native transform for Dataflow, 
having `MultipleReadFromPubSub` seems like the only solution for now. I'm 
thinking we could create a `PubSubSourceDescriptor` which includes `topic`, 
`subscription` and other attributes. And we expose `add` API from 
`MultipleReadFromPubSub` to allow end users to add a new Read.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] robertwb commented on pull request #12819: [BEAM-9561] Initial framework for testing pandas website docs.

[GitBox]

robertwb commented on pull request #12819:
URL: https://github.com/apache/beam/pull/12819#issuecomment-701720434


   Run Python PreCommit



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #12819: [BEAM-9561] Initial framework for testing pandas website docs.

[GitBox]

codecov[bot] edited a comment on pull request #12819:
URL: https://github.com/apache/beam/pull/12819#issuecomment-691336298


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12819?src=pr=h1) Report
   > Merging 
[#12819](https://codecov.io/gh/apache/beam/pull/12819?src=pr=desc) into 
[master](https://codecov.io/gh/apache/beam/commit/ed60540875a79fe2ba3442d5f87f8c88223be6b7?el=desc)
 will **increase** coverage by `0.13%`.
   > The diff coverage is `96.22%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/beam/pull/12819/graphs/tree.svg?width=650=150=pr=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12819?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master   #12819  +/-   ##
   ==
   + Coverage   82.31%   82.44%   +0.13% 
   ==
 Files 455  455  
 Lines   5459654864 +268 
   ==
   + Hits4494145233 +292 
   + Misses   9655 9631  -24 
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/beam/pull/12819?src=pr=tree) | Coverage 
Δ | |
   |---|---|---|
   | 
[sdks/python/apache\_beam/dataframe/doctests.py](https://codecov.io/gh/apache/beam/pull/12819/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2RvY3Rlc3RzLnB5)
 | `96.75% <96.22%> (-0.45%)` | :arrow_down: |
   | 
[...dks/python/apache\_beam/options/pipeline\_options.py](https://codecov.io/gh/apache/beam/pull/12819/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vb3B0aW9ucy9waXBlbGluZV9vcHRpb25zLnB5)
 | `93.76% <0.00%> (-1.03%)` | :arrow_down: |
   | 
[sdks/python/apache\_beam/io/gcp/bigquery.py](https://codecov.io/gh/apache/beam/pull/12819/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5LnB5)
 | `80.08% <0.00%> (-0.63%)` | :arrow_down: |
   | 
[sdks/python/apache\_beam/io/fileio.py](https://codecov.io/gh/apache/beam/pull/12819/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZmlsZWlvLnB5)
 | `95.80% <0.00%> (-0.05%)` | :arrow_down: |
   | 
[...e\_beam/portability/api/beam\_runner\_api\_pb2\_urns.py](https://codecov.io/gh/apache/beam/pull/12819/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjJfdXJucy5weQ==)
 | `100.00% <0.00%> (ø)` | |
   | 
[...\_beam/runners/portability/sdk\_container\_builder.py](https://codecov.io/gh/apache/beam/pull/12819/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9zZGtfY29udGFpbmVyX2J1aWxkZXIucHk=)
 | `31.21% <0.00%> (ø)` | |
   | 
[...eam/testing/benchmarks/nexmark/nexmark\_launcher.py](https://codecov.io/gh/apache/beam/pull/12819/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvbmV4bWFya19sYXVuY2hlci5weQ==)
 | `0.00% <0.00%> (ø)` | |
   | 
[...ache\_beam/runners/interactive/recording\_manager.py](https://codecov.io/gh/apache/beam/pull/12819/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9yZWNvcmRpbmdfbWFuYWdlci5weQ==)
 | `98.90% <0.00%> (+0.02%)` | :arrow_up: |
   | 
[sdks/python/apache\_beam/transforms/environments.py](https://codecov.io/gh/apache/beam/pull/12819/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9lbnZpcm9ubWVudHMucHk=)
 | `83.73% <0.00%> (+0.19%)` | :arrow_up: |
   | ... and [12 
more](https://codecov.io/gh/apache/beam/pull/12819/diff?src=pr=tree-more) | |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/beam/pull/12819?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/beam/pull/12819?src=pr=footer). Last 
update 
[c38fa68...a0a3099](https://codecov.io/gh/apache/beam/pull/12819?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #12979: [BEAM-10545] Add 2 show options

[GitBox]

codecov[bot] edited a comment on pull request #12979:
URL: https://github.com/apache/beam/pull/12979#issuecomment-701667582


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12979?src=pr=h1) Report
   > Merging 
[#12979](https://codecov.io/gh/apache/beam/pull/12979?src=pr=desc) into 
[master](https://codecov.io/gh/apache/beam/commit/cee7388365970774969e63fe90fdf3cdd804e79f?el=desc)
 will **increase** coverage by `0.04%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/beam/pull/12979/graphs/tree.svg?width=650=150=pr=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12979?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master   #12979  +/-   ##
   ==
   + Coverage   82.34%   82.39%   +0.04% 
   ==
 Files 455  453   -2 
 Lines   5465054623  -27 
   ==
   + Hits4500245006   +4 
   + Misses   9648 9617  -31 
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/beam/pull/12979?src=pr=tree) | Coverage 
Δ | |
   |---|---|---|
   | 
[...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/12979/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==)
 | `88.68% <0.00%> (-1.23%)` | :arrow_down: |
   | 
[...dks/python/apache\_beam/options/pipeline\_options.py](https://codecov.io/gh/apache/beam/pull/12979/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vb3B0aW9ucy9waXBlbGluZV9vcHRpb25zLnB5)
 | `93.76% <0.00%> (-1.03%)` | :arrow_down: |
   | 
[sdks/python/apache\_beam/io/localfilesystem.py](https://codecov.io/gh/apache/beam/pull/12979/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vbG9jYWxmaWxlc3lzdGVtLnB5)
 | `90.90% <0.00%> (-0.76%)` | :arrow_down: |
   | 
[sdks/python/apache\_beam/io/gcp/bigquery.py](https://codecov.io/gh/apache/beam/pull/12979/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5LnB5)
 | `80.23% <0.00%> (-0.48%)` | :arrow_down: |
   | 
[...nners/portability/fn\_api\_runner/worker\_handlers.py](https://codecov.io/gh/apache/beam/pull/12979/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL3dvcmtlcl9oYW5kbGVycy5weQ==)
 | `80.57% <0.00%> (-0.18%)` | :arrow_down: |
   | 
[sdks/python/apache\_beam/io/gcp/bigquery\_tools.py](https://codecov.io/gh/apache/beam/pull/12979/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X3Rvb2xzLnB5)
 | `88.35% <0.00%> (-0.15%)` | :arrow_down: |
   | 
[...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/12979/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==)
 | `94.31% <0.00%> (-0.14%)` | :arrow_down: |
   | 
[sdks/python/apache\_beam/io/fileio.py](https://codecov.io/gh/apache/beam/pull/12979/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZmlsZWlvLnB5)
 | `95.80% <0.00%> (-0.05%)` | :arrow_down: |
   | 
[...e\_beam/portability/api/beam\_runner\_api\_pb2\_urns.py](https://codecov.io/gh/apache/beam/pull/12979/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjJfdXJucy5weQ==)
 | `100.00% <0.00%> (ø)` | |
   | 
[...eam/testing/benchmarks/nexmark/nexmark\_launcher.py](https://codecov.io/gh/apache/beam/pull/12979/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvbmV4bWFya19sYXVuY2hlci5weQ==)
 | `0.00% <0.00%> (ø)` | |
   | ... and [6 
more](https://codecov.io/gh/apache/beam/pull/12979/diff?src=pr=tree-more) | |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/beam/pull/12979?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/beam/pull/12979?src=pr=footer). Last 
update 
[cee7388...98bb062](https://codecov.io/gh/apache/beam/pull/12979?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] pabloem commented on pull request #12941: [BEAM-9681] Add textio.Read lesson to Go SDK katas

[GitBox]

pabloem commented on pull request #12941:
URL: https://github.com/apache/beam/pull/12941#issuecomment-701673167


   I'm also okay to ignore. Feel free to merge~



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] KevinGG commented on pull request #12979: [BEAM-10545] Add 2 show options

[GitBox]

KevinGG commented on pull request #12979:
URL: https://github.com/apache/beam/pull/12979#issuecomment-701673028


   R: @prodonjs
   PTAL, thx!



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] kennknowles commented on a change in pull request #12981: [BEAM-10435] Add LOOPBACK mode Java ValidatesRunner test suite for ULR

[GitBox]

kennknowles commented on a change in pull request #12981:
URL: https://github.com/apache/beam/pull/12981#discussion_r497856286



##
File path: runners/portability/java/build.gradle
##
@@ -82,8 +87,15 @@ void execBackgroundInVirtualenv(String... args) {
   proc.waitFor();
 }
 
-task startLocalJobService {
+task installBeamPythonInVirtualenv {
   dependsOn setupVirtualenv
+  doLast {
+execInVirtualenv "pip", "install", "-e", "."

Review comment:
   So you mean more of an approach like this?
   
   ```
  dependsOn ':sdks:python:sdist'
   doLast {
 def distTarBall = "${pythonRootDir}/build/apache-beam.tar.gz"
 project.exec {
   executable 'sh'
   args '-c', ". ${project.ext.envdir}/bin/activate && pip install 
--retries 10 ${distTarBall}"
 }
   }
 }
   ```
   
   I am not familiar with the install variants. For ValidatesRunner we should 
not need any cloud-specific things. I do not know what `[test]` changes.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] kennknowles commented on a change in pull request #12981: [BEAM-10435] Add LOOPBACK mode Java ValidatesRunner test suite for ULR

[GitBox]

kennknowles commented on a change in pull request #12981:
URL: https://github.com/apache/beam/pull/12981#discussion_r497856286



##
File path: runners/portability/java/build.gradle
##
@@ -82,8 +87,15 @@ void execBackgroundInVirtualenv(String... args) {
   proc.waitFor();
 }
 
-task startLocalJobService {
+task installBeamPythonInVirtualenv {
   dependsOn setupVirtualenv
+  doLast {
+execInVirtualenv "pip", "install", "-e", "."

Review comment:
   So you mean more of an approach like this?
   
   ```
  dependsOn ':sdks:python:sdist'
   doLast {
 def distTarBall = "${pythonRootDir}/build/apache-beam.tar.gz"
 project.exec {
   executable 'sh'
   args '-c', ". ${project.ext.envdir}/bin/activate && pip install 
--retries 10 ${distTarBall}"
 }
   }
 }
   ```
   
   I am not familiar with the install variants. For ValidatesRunner we should 
not need any cloud-specific things. I do not know what `test` changes.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] kennknowles commented on a change in pull request #12981: [BEAM-10435] Add LOOPBACK mode Java ValidatesRunner test suite for ULR

[GitBox]

kennknowles commented on a change in pull request #12981:
URL: https://github.com/apache/beam/pull/12981#discussion_r497856286



##
File path: runners/portability/java/build.gradle
##
@@ -82,8 +87,15 @@ void execBackgroundInVirtualenv(String... args) {
   proc.waitFor();
 }
 
-task startLocalJobService {
+task installBeamPythonInVirtualenv {
   dependsOn setupVirtualenv
+  doLast {
+execInVirtualenv "pip", "install", "-e", "."

Review comment:
   So you mean more of an approach like this?
   
   ```
  dependsOn ':sdks:python:sdist'
   doLast {
 def distTarBall = "${pythonRootDir}/build/apache-beam.tar.gz"
 execInVirtualenv "pip", "install", "--retries", "10", distTarBall
 }
   }
 }
   ```
   
   I am not familiar with the install variants. For ValidatesRunner we should 
not need any cloud-specific things. I do not know what `[test]` changes.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] tvalentyn commented on a change in pull request #12981: [BEAM-10435] Add LOOPBACK mode Java ValidatesRunner test suite for ULR

[GitBox]

tvalentyn commented on a change in pull request #12981:
URL: https://github.com/apache/beam/pull/12981#discussion_r497856592



##
File path: runners/portability/java/build.gradle
##
@@ -82,8 +87,15 @@ void execBackgroundInVirtualenv(String... args) {
   proc.waitFor();
 }
 
-task startLocalJobService {
+task installBeamPythonInVirtualenv {
   dependsOn setupVirtualenv
+  doLast {
+execInVirtualenv "pip", "install", "-e", "."

Review comment:
   it currently installs all `extra` dependencies (test deps + deps for all 
of the supported clouds) 





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] kennknowles commented on pull request #12962: [WIP][BEAM-9379] Update calcite to 1.25

[GitBox]

kennknowles commented on pull request #12962:
URL: https://github.com/apache/beam/pull/12962#issuecomment-701703218


   
https://scans.gradle.com/s/ofcfnu774du4y/tests/:sdks:java:extensions:sql:zetasql:test/org.apache.beam.sdk.extensions.sql.zetasql.ZetaSqlTimeFunctionsTest/testFormatDate#1
   
   this one looks easier - just a change of capital to lowercase CC @robinyqiu 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] kennknowles commented on pull request #12962: [WIP][BEAM-9379] Update calcite to 1.25

[GitBox]

kennknowles commented on pull request #12962:
URL: https://github.com/apache/beam/pull/12962#issuecomment-701703585


   
https://scans.gradle.com/s/ofcfnu774du4y/tests/:sdks:java:extensions:sql:test/org.apache.beam.sdk.extensions.sql.impl.rel.BeamMatchRelTest/matchPrevFunctionTest#1
   
   Seems there is something special and missing from `BeamMatchRel`



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] rohdesamuel opened a new pull request #12986: [BEAM-10994] Adds ability to log literal hot key in Streaming Dataflow

[GitBox]

rohdesamuel opened a new pull request #12986:
URL: https://github.com/apache/beam/pull/12986


   The logging of the key is controlled by the DataflowPipelineOption 
`hotKeyLoggingEnabled`. When true and a hot key is detected from Dataflow, the 
literal hot key content will be logged to Cloud Logging. Otherwise, the default 
message without a key will be displayed.
   
   
   
   
   
   Thank you for your contribution! Follow this checklist to help us 
incorporate your contribution quickly and easily:
   
- [ ] [**Choose 
reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and 
mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more 
tips on [how to make review process 
smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
   --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/)
 | --- | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/)
 | --- | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
 | ---
   Java | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/)
 | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[![Build
 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/)
 | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[![Build
 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/badge/i
 
con)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)[![Build
 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[![Build
 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/)
 | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/)
 | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[![Build
 Status](htt
 
ps://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[![Build
 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
 | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
   Python | [![Build 

[GitHub] [beam] rohdesamuel opened a new pull request #12985: [BEAM-10994] Adds ability to log literal hot key in Batch Dataflow

[GitBox]

rohdesamuel opened a new pull request #12985:
URL: https://github.com/apache/beam/pull/12985


   The logging of the key is controlled by the DataflowPipelineOption 
`hotKeyLoggingEnabled`. When true and a hot key is detected from Dataflow, the 
literal hot key content will be logged to Cloud Logging. Otherwise, the default 
message without a key will be displayed.
   
   
   
   Thank you for your contribution! Follow this checklist to help us 
incorporate your contribution quickly and easily:
   
- [ ] [**Choose 
reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and 
mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more 
tips on [how to make review process 
smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
   --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/)
 | --- | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/)
 | --- | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
 | ---
   Java | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/)
 | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[![Build
 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/)
 | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[![Build
 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/badge/i
 
con)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)[![Build
 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[![Build
 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/)
 | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/)
 | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[![Build
 Status](htt
 
ps://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[![Build
 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
 | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
   Python | [![Build 

[GitHub] [beam] veblush commented on pull request #12965: [BEAM-5748] Upgrade conscrypt to latest version

[GitBox]

veblush commented on pull request #12965:
URL: https://github.com/apache/beam/pull/12965#issuecomment-701646445


   Awesome! I'm about to make the same PR but you did it. Thanks! 
   
   FYI, this is required for gRPC ALTS to have better throughput by saving CPU 
time significantly. Notable PRs
   - https://github.com/google/conscrypt/pull/859: Removing unnecessary memory 
copies between gRPC and Conscrypt
   - https://github.com/google/conscrypt/pull/867: Addressing breaking change 
introduced from Conscrypt 2.1, which prevented gRPC from working properly with 
Conscrypt.
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #12941: [BEAM-9681] Add textio.Read lesson to Go SDK katas

[GitBox]

codecov[bot] edited a comment on pull request #12941:
URL: https://github.com/apache/beam/pull/12941#issuecomment-698987193


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12941?src=pr=h1) Report
   > Merging 
[#12941](https://codecov.io/gh/apache/beam/pull/12941?src=pr=desc) into 
[master](https://codecov.io/gh/apache/beam/commit/aa58e1e5db4af2a6f97520756831e87aa1d3e3fb?el=desc)
 will **decrease** coverage by `0.13%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/beam/pull/12941/graphs/tree.svg?width=650=150=pr=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12941?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master   #12941  +/-   ##
   ==
   - Coverage   82.43%   82.30%   -0.14% 
   ==
 Files 454  455   +1 
 Lines   5434555166 +821 
   ==
   + Hits4479945404 +605 
   - Misses   9546 9762 +216 
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/beam/pull/12941?src=pr=tree) | Coverage 
Δ | |
   |---|---|---|
   | 
[sdks/python/apache\_beam/io/gcp/pubsub.py](https://codecov.io/gh/apache/beam/pull/12941/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3B1YnN1Yi5weQ==)
 | `92.30% <0.00%> (-1.28%)` | :arrow_down: |
   | 
[...dks/python/apache\_beam/options/pipeline\_options.py](https://codecov.io/gh/apache/beam/pull/12941/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vb3B0aW9ucy9waXBlbGluZV9vcHRpb25zLnB5)
 | `93.76% <0.00%> (-0.99%)` | :arrow_down: |
   | 
[...on/apache\_beam/runners/dataflow/dataflow\_runner.py](https://codecov.io/gh/apache/beam/pull/12941/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9kYXRhZmxvd19ydW5uZXIucHk=)
 | `76.93% <0.00%> (-0.31%)` | :arrow_down: |
   | 
[sdks/python/apache\_beam/io/iobase.py](https://codecov.io/gh/apache/beam/pull/12941/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vaW9iYXNlLnB5)
 | `84.03% <0.00%> (-0.30%)` | :arrow_down: |
   | 
[sdks/python/apache\_beam/transforms/util.py](https://codecov.io/gh/apache/beam/pull/12941/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy91dGlsLnB5)
 | `95.44% <0.00%> (-0.19%)` | :arrow_down: |
   | 
[sdks/python/apache\_beam/transforms/environments.py](https://codecov.io/gh/apache/beam/pull/12941/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9lbnZpcm9ubWVudHMucHk=)
 | `83.73% <0.00%> (-0.14%)` | :arrow_down: |
   | 
[...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/12941/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==)
 | `94.45% <0.00%> (-0.14%)` | :arrow_down: |
   | 
[sdks/python/apache\_beam/coders/coder\_impl.py](https://codecov.io/gh/apache/beam/pull/12941/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL2NvZGVyX2ltcGwucHk=)
 | `95.13% <0.00%> (-0.12%)` | :arrow_down: |
   | 
[sdks/python/apache\_beam/dataframe/io.py](https://codecov.io/gh/apache/beam/pull/12941/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2lvLnB5)
 | `92.00% <0.00%> (-0.10%)` | :arrow_down: |
   | 
[sdks/python/apache\_beam/io/fileio.py](https://codecov.io/gh/apache/beam/pull/12941/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZmlsZWlvLnB5)
 | `95.80% <0.00%> (-0.05%)` | :arrow_down: |
   | ... and [25 
more](https://codecov.io/gh/apache/beam/pull/12941/diff?src=pr=tree-more) | |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/beam/pull/12941?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/beam/pull/12941?src=pr=footer). Last 
update 
[aa58e1e...832cbde](https://codecov.io/gh/apache/beam/pull/12941?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #12979: [BEAM-10545] Add 2 show options

[GitBox]

codecov[bot] edited a comment on pull request #12979:
URL: https://github.com/apache/beam/pull/12979#issuecomment-701667582


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12979?src=pr=h1) Report
   > Merging 
[#12979](https://codecov.io/gh/apache/beam/pull/12979?src=pr=desc) into 
[master](https://codecov.io/gh/apache/beam/commit/cee7388365970774969e63fe90fdf3cdd804e79f?el=desc)
 will **increase** coverage by `0.04%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/beam/pull/12979/graphs/tree.svg?width=650=150=pr=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12979?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master   #12979  +/-   ##
   ==
   + Coverage   82.34%   82.39%   +0.04% 
   ==
 Files 455  453   -2 
 Lines   5465054623  -27 
   ==
   + Hits4500245006   +4 
   + Misses   9648 9617  -31 
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/beam/pull/12979?src=pr=tree) | Coverage 
Δ | |
   |---|---|---|
   | 
[...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/12979/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==)
 | `88.68% <0.00%> (-1.23%)` | :arrow_down: |
   | 
[...dks/python/apache\_beam/options/pipeline\_options.py](https://codecov.io/gh/apache/beam/pull/12979/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vb3B0aW9ucy9waXBlbGluZV9vcHRpb25zLnB5)
 | `93.76% <0.00%> (-1.03%)` | :arrow_down: |
   | 
[sdks/python/apache\_beam/io/localfilesystem.py](https://codecov.io/gh/apache/beam/pull/12979/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vbG9jYWxmaWxlc3lzdGVtLnB5)
 | `90.90% <0.00%> (-0.76%)` | :arrow_down: |
   | 
[sdks/python/apache\_beam/io/gcp/bigquery.py](https://codecov.io/gh/apache/beam/pull/12979/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5LnB5)
 | `80.23% <0.00%> (-0.48%)` | :arrow_down: |
   | 
[...nners/portability/fn\_api\_runner/worker\_handlers.py](https://codecov.io/gh/apache/beam/pull/12979/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL3dvcmtlcl9oYW5kbGVycy5weQ==)
 | `80.57% <0.00%> (-0.18%)` | :arrow_down: |
   | 
[sdks/python/apache\_beam/io/gcp/bigquery\_tools.py](https://codecov.io/gh/apache/beam/pull/12979/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X3Rvb2xzLnB5)
 | `88.35% <0.00%> (-0.15%)` | :arrow_down: |
   | 
[...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/12979/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==)
 | `94.31% <0.00%> (-0.14%)` | :arrow_down: |
   | 
[sdks/python/apache\_beam/io/fileio.py](https://codecov.io/gh/apache/beam/pull/12979/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZmlsZWlvLnB5)
 | `95.80% <0.00%> (-0.05%)` | :arrow_down: |
   | 
[...e\_beam/portability/api/beam\_runner\_api\_pb2\_urns.py](https://codecov.io/gh/apache/beam/pull/12979/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjJfdXJucy5weQ==)
 | `100.00% <0.00%> (ø)` | |
   | 
[...eam/testing/benchmarks/nexmark/nexmark\_launcher.py](https://codecov.io/gh/apache/beam/pull/12979/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvbmV4bWFya19sYXVuY2hlci5weQ==)
 | `0.00% <0.00%> (ø)` | |
   | ... and [6 
more](https://codecov.io/gh/apache/beam/pull/12979/diff?src=pr=tree-more) | |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/beam/pull/12979?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/beam/pull/12979?src=pr=footer). Last 
update 
[cee7388...98bb062](https://codecov.io/gh/apache/beam/pull/12979?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] kennknowles opened a new pull request #12981: [BEAM-10435] Add LOOPBACK mode Java ValidatesRunner test suite for ULR

[GitBox]

kennknowles opened a new pull request #12981:
URL: https://github.com/apache/beam/pull/12981


   This just adds the additional gradle target. TODO: add continuous Jenkins 
runs. It looks like it takes long enough it has to be a postcommit.
   
   
   
   Thank you for your contribution! Follow this checklist to help us 
incorporate your contribution quickly and easily:
   
- [x] [**Choose 
reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and 
mention them in a comment (`R: @username`).
- [x] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [x] Update `CHANGES.md` with noteworthy changes.
- [x] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more 
tips on [how to make review process 
smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
   --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/)
 | --- | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/)
 | --- | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
 | ---
   Java | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/)
 | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[![Build
 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/)
 | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[![Build
 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/badge/i
 
con)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)[![Build
 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[![Build
 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/)
 | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/)
 | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[![Build
 Status](htt
 
ps://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[![Build
 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
 | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
   Python | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[![Build
 

[GitHub] [beam] tvalentyn commented on a change in pull request #12981: [BEAM-10435] Add LOOPBACK mode Java ValidatesRunner test suite for ULR

[GitBox]

tvalentyn commented on a change in pull request #12981:
URL: https://github.com/apache/beam/pull/12981#discussion_r497854011



##
File path: runners/portability/java/build.gradle
##
@@ -82,8 +87,15 @@ void execBackgroundInVirtualenv(String... args) {
   proc.waitFor();
 }
 
-task startLocalJobService {
+task installBeamPythonInVirtualenv {
   dependsOn setupVirtualenv
+  doLast {
+execInVirtualenv "pip", "install", "-e", "."

Review comment:
   We had issues in the past with installing SDK in developer mode in 
tests. Not sure if this still relevant. If you want to install from tarball, 
you can consider using 
https://github.com/apache/beam/blob/943a948e5a6dd7ec81760359c198c3fb2b3c3dcd/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L1999
   
   (nameing is for that target is not very mnemonic..)





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] kennknowles commented on pull request #12962: [WIP][BEAM-9379] Update calcite to 1.25

[GitBox]

kennknowles commented on pull request #12962:
URL: https://github.com/apache/beam/pull/12962#issuecomment-701702697


   
https://scans.gradle.com/s/ofcfnu774du4y/tests/:sdks:java:extensions:sql:test/org.apache.beam.sdk.extensions.sql.BeamSqlDslUdfUdafTest/testUdfWithListInput#1
   
   Seems parsing of arrays changed?



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] codecov[bot] commented on pull request #12981: [BEAM-10435] Add LOOPBACK mode Java ValidatesRunner test suite for ULR

[GitBox]

codecov[bot] commented on pull request #12981:
URL: https://github.com/apache/beam/pull/12981#issuecomment-701714122


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12981?src=pr=h1) Report
   > Merging 
[#12981](https://codecov.io/gh/apache/beam/pull/12981?src=pr=desc) into 
[master](https://codecov.io/gh/apache/beam/commit/7c91a63aa746c25eccea1d296a0b8ea948a0ed73?el=desc)
 will **increase** coverage by `0.11%`.
   > The diff coverage is `79.68%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/beam/pull/12981/graphs/tree.svg?width=650=150=pr=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12981?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master   #12981  +/-   ##
   ==
   + Coverage   82.39%   82.51%   +0.11% 
   ==
 Files 453  453  
 Lines   5462354610  -13 
   ==
   + Hits4500545059  +54 
   + Misses   9618 9551  -67 
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/beam/pull/12981?src=pr=tree) | Coverage 
Δ | |
   |---|---|---|
   | 
[sdks/python/apache\_beam/io/fileio.py](https://codecov.io/gh/apache/beam/pull/12981/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZmlsZWlvLnB5)
 | `95.80% <ø> (ø)` | |
   | 
[sdks/python/apache\_beam/io/gcp/bigquery.py](https://codecov.io/gh/apache/beam/pull/12981/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5LnB5)
 | `80.23% <0.00%> (+0.14%)` | :arrow_up: |
   | 
[sdks/python/apache\_beam/io/gcp/bigquery\_tools.py](https://codecov.io/gh/apache/beam/pull/12981/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X3Rvb2xzLnB5)
 | `88.35% <0.00%> (ø)` | |
   | 
[...eam/testing/benchmarks/nexmark/nexmark\_launcher.py](https://codecov.io/gh/apache/beam/pull/12981/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvbmV4bWFya19sYXVuY2hlci5weQ==)
 | `0.00% <0.00%> (ø)` | |
   | 
[...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/12981/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5)
 | `79.53% <66.66%> (ø)` | |
   | 
[...dks/python/apache\_beam/options/pipeline\_options.py](https://codecov.io/gh/apache/beam/pull/12981/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vb3B0aW9ucy9waXBlbGluZV9vcHRpb25zLnB5)
 | `93.76% <70.58%> (ø)` | |
   | 
[sdks/python/apache\_beam/utils/profiler.py](https://codecov.io/gh/apache/beam/pull/12981/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvcHJvZmlsZXIucHk=)
 | `86.45% <80.00%> (+54.34%)` | :arrow_up: |
   | 
[sdks/python/apache\_beam/transforms/environments.py](https://codecov.io/gh/apache/beam/pull/12981/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9lbnZpcm9ubWVudHMucHk=)
 | `83.73% <83.33%> (ø)` | |
   | 
[.../apache\_beam/options/pipeline\_options\_validator.py](https://codecov.io/gh/apache/beam/pull/12981/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vb3B0aW9ucy9waXBlbGluZV9vcHRpb25zX3ZhbGlkYXRvci5weQ==)
 | `98.69% <100.00%> (ø)` | |
   | 
[...ache\_beam/runners/interactive/recording\_manager.py](https://codecov.io/gh/apache/beam/pull/12981/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9yZWNvcmRpbmdfbWFuYWdlci5weQ==)
 | `98.90% <100.00%> (ø)` | |
   | ... and [15 
more](https://codecov.io/gh/apache/beam/pull/12981/diff?src=pr=tree-more) | |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/beam/pull/12981?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/beam/pull/12981?src=pr=footer). Last 
update 
[443cf83...ec613a7](https://codecov.io/gh/apache/beam/pull/12981?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] frankzhao commented on pull request #12960: [BEAM-9804] Allow user configuration of BigQuery temporary dataset

[GitBox]

frankzhao commented on pull request #12960:
URL: https://github.com/apache/beam/pull/12960#issuecomment-701738038


   R: @pabloem



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] pabloem commented on pull request #12941: [BEAM-9681] Add textio.Read lesson to Go SDK katas

[GitBox]

pabloem commented on pull request #12941:
URL: https://github.com/apache/beam/pull/12941#issuecomment-701649639


   Run Python PreCommit



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] pabloem commented on pull request #12941: [BEAM-9681] Add textio.Read lesson to Go SDK katas

[GitBox]

pabloem commented on pull request #12941:
URL: https://github.com/apache/beam/pull/12941#issuecomment-701649895


   a similar -though not the same- test is failing on the main branch: 
https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/ 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] codecov[bot] commented on pull request #12979: [BEAM-10545] Add 2 show options

[GitBox]

codecov[bot] commented on pull request #12979:
URL: https://github.com/apache/beam/pull/12979#issuecomment-701667582


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12979?src=pr=h1) Report
   > Merging 
[#12979](https://codecov.io/gh/apache/beam/pull/12979?src=pr=desc) into 
[master](https://codecov.io/gh/apache/beam/commit/cee7388365970774969e63fe90fdf3cdd804e79f?el=desc)
 will **increase** coverage by `0.04%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/beam/pull/12979/graphs/tree.svg?width=650=150=pr=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12979?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master   #12979  +/-   ##
   ==
   + Coverage   82.34%   82.39%   +0.04% 
   ==
 Files 455  453   -2 
 Lines   5465054623  -27 
   ==
   + Hits4500245006   +4 
   + Misses   9648 9617  -31 
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/beam/pull/12979?src=pr=tree) | Coverage 
Δ | |
   |---|---|---|
   | 
[...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/12979/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==)
 | `88.68% <0.00%> (-1.23%)` | :arrow_down: |
   | 
[...dks/python/apache\_beam/options/pipeline\_options.py](https://codecov.io/gh/apache/beam/pull/12979/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vb3B0aW9ucy9waXBlbGluZV9vcHRpb25zLnB5)
 | `93.76% <0.00%> (-1.03%)` | :arrow_down: |
   | 
[sdks/python/apache\_beam/io/localfilesystem.py](https://codecov.io/gh/apache/beam/pull/12979/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vbG9jYWxmaWxlc3lzdGVtLnB5)
 | `90.90% <0.00%> (-0.76%)` | :arrow_down: |
   | 
[sdks/python/apache\_beam/io/gcp/bigquery.py](https://codecov.io/gh/apache/beam/pull/12979/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5LnB5)
 | `80.23% <0.00%> (-0.48%)` | :arrow_down: |
   | 
[...nners/portability/fn\_api\_runner/worker\_handlers.py](https://codecov.io/gh/apache/beam/pull/12979/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL3dvcmtlcl9oYW5kbGVycy5weQ==)
 | `80.57% <0.00%> (-0.18%)` | :arrow_down: |
   | 
[sdks/python/apache\_beam/io/gcp/bigquery\_tools.py](https://codecov.io/gh/apache/beam/pull/12979/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X3Rvb2xzLnB5)
 | `88.35% <0.00%> (-0.15%)` | :arrow_down: |
   | 
[...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/12979/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==)
 | `94.31% <0.00%> (-0.14%)` | :arrow_down: |
   | 
[sdks/python/apache\_beam/io/fileio.py](https://codecov.io/gh/apache/beam/pull/12979/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZmlsZWlvLnB5)
 | `95.80% <0.00%> (-0.05%)` | :arrow_down: |
   | 
[...e\_beam/portability/api/beam\_runner\_api\_pb2\_urns.py](https://codecov.io/gh/apache/beam/pull/12979/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjJfdXJucy5weQ==)
 | `100.00% <0.00%> (ø)` | |
   | 
[...eam/testing/benchmarks/nexmark/nexmark\_launcher.py](https://codecov.io/gh/apache/beam/pull/12979/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvbmV4bWFya19sYXVuY2hlci5weQ==)
 | `0.00% <0.00%> (ø)` | |
   | ... and [6 
more](https://codecov.io/gh/apache/beam/pull/12979/diff?src=pr=tree-more) | |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/beam/pull/12979?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/beam/pull/12979?src=pr=footer). Last 
update 
[cee7388...98bb062](https://codecov.io/gh/apache/beam/pull/12979?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] kennknowles commented on pull request #12981: [BEAM-10435] Add LOOPBACK mode Java ValidatesRunner test suite for ULR

[GitBox]

kennknowles commented on pull request #12981:
URL: https://github.com/apache/beam/pull/12981#issuecomment-701699000


   https://gradle.com/s/r6phv7qxmg6so



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] kennknowles commented on a change in pull request #12981: [BEAM-10435] Add LOOPBACK mode Java ValidatesRunner test suite for ULR

[GitBox]

kennknowles commented on a change in pull request #12981:
URL: https://github.com/apache/beam/pull/12981#discussion_r497855759



##
File path: runners/portability/java/build.gradle
##
@@ -108,95 +120,113 @@ startLocalJobService.finalizedBy stopLocalJobService
 
 /**
  * Runs Java ValidatesRunner tests against the Universal Local Runner (ULR) 
aka local_job_service_main
- * with subprocess SDK harness environments.
+ * with the specified environment type.
  */
-task ulrValidatesRunnerTests(type: Test) {
-  dependsOn ":sdks:java:container:docker"
-
-  if (!project.hasProperty("localJobServicePortFile")) {
-dependsOn startLocalJobService
+def createUlrValidatesRunnerTask = { name, environmentType ->
+  Task vrTask = tasks.create(name: name, type: Test, group: "Verification") {
+description "PortableRunner Java docker ValidatesRunner suite"
+classpath = configurations.validatesRunner
+systemProperty "beamTestPipelineOptions", JsonOutput.toJson([
+"--runner=TestUniversalRunner",
+"--experiments=beam_fn_api",
+"--defaultEnvironmentType=${environmentType}",
+"--localJobServicePortFile=${localJobServicePortFile}"
+])
+testClassesDirs = 
files(project(":sdks:java:core").sourceSets.test.output.classesDirs)
+useJUnit {
+  includeCategories 'org.apache.beam.sdk.testing.ValidatesRunner'
+  excludeCategories 'org.apache.beam.sdk.testing.UsesGaugeMetrics'
+  excludeCategories 'org.apache.beam.sdk.testing.UsesOnWindowExpiration'
+  excludeCategories 'org.apache.beam.sdk.testing.UsesBundleFinalizer'

Review comment:
   I don't believe so. I think all excluded categories are essentially 
feature enhancements for the roadmap. For individual test cases that do not 
correspond to a category they each have a Jira, since we should aim to support 
features entirely or not at all. We should also avoid adding fine-grained 
categories if they do not correspond to real general concepts.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] kennknowles commented on a change in pull request #12981: [BEAM-10435] Add LOOPBACK mode Java ValidatesRunner test suite for ULR

[GitBox]

kennknowles commented on a change in pull request #12981:
URL: https://github.com/apache/beam/pull/12981#discussion_r497855885



##
File path: runners/portability/java/build.gradle
##
@@ -82,8 +87,15 @@ void execBackgroundInVirtualenv(String... args) {
   proc.waitFor();
 }
 
-task startLocalJobService {
+task installBeamPythonInVirtualenv {
   dependsOn setupVirtualenv
+  doLast {
+execInVirtualenv "pip", "install", "-e", "."

Review comment:
   I did not use that because I read the name and it seemed tied to GCP.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] kennknowles commented on pull request #12962: [WIP][BEAM-9379] Update calcite to 1.25

[GitBox]

kennknowles commented on pull request #12962:
URL: https://github.com/apache/beam/pull/12962#issuecomment-701703858


   
https://scans.gradle.com/s/ofcfnu774du4y/tests/:sdks:java:extensions:sql:test/org.apache.beam.sdk.extensions.sql.BeamSqlDslAggregationTest/testWindowOnNonTimestampField#1
   
   This is just a bad test. It checks the exact string instead of the key 
elements of the string so it is sensitive to irrelevant changes.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] kennknowles commented on pull request #12962: [WIP][BEAM-9379] Update calcite to 1.25

[GitBox]

kennknowles commented on pull request #12962:
URL: https://github.com/apache/beam/pull/12962#issuecomment-701704059


   
https://scans.gradle.com/s/ofcfnu774du4y/tests/:sdks:java:extensions:sql:test/org.apache.beam.sdk.extensions.sql.impl.udf.BeamSqlUdfExpressionTest/testSHA1?anchor=e30=0-1-0#1
   
   Switch from `byte[]` to avatica's `ByteString`



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] rohdesamuel opened a new pull request #12984: [BEAM-10994] Add ability for HotKeyLogger to log a key

[GitBox]

rohdesamuel opened a new pull request #12984:
URL: https://github.com/apache/beam/pull/12984


   Adds interface for HotKeyLogger to take an object key.
   
   
   
   Thank you for your contribution! Follow this checklist to help us 
incorporate your contribution quickly and easily:
   
- [ ] [**Choose 
reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and 
mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more 
tips on [how to make review process 
smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
   --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/)
 | --- | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/)
 | --- | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
 | ---
   Java | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/)
 | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[![Build
 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/)
 | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[![Build
 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/badge/i
 
con)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)[![Build
 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[![Build
 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/)
 | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/)
 | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[![Build
 Status](htt
 
ps://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[![Build
 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
 | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
   Python | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[![Build
 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)[![Build
 

[GitHub] [beam] frankzhao removed a comment on pull request #12960: [BEAM-9804] Allow user configuration of BigQuery temporary dataset

[GitBox]

frankzhao removed a comment on pull request #12960:
URL: https://github.com/apache/beam/pull/12960#issuecomment-700713637


   retest this please



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #12819: [BEAM-9561] Initial framework for testing pandas website docs.

[GitBox]

codecov[bot] edited a comment on pull request #12819:
URL: https://github.com/apache/beam/pull/12819#issuecomment-691336298


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12819?src=pr=h1) Report
   > Merging 
[#12819](https://codecov.io/gh/apache/beam/pull/12819?src=pr=desc) into 
[master](https://codecov.io/gh/apache/beam/commit/ed60540875a79fe2ba3442d5f87f8c88223be6b7?el=desc)
 will **increase** coverage by `0.13%`.
   > The diff coverage is `96.22%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/beam/pull/12819/graphs/tree.svg?width=650=150=pr=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12819?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master   #12819  +/-   ##
   ==
   + Coverage   82.31%   82.44%   +0.13% 
   ==
 Files 455  455  
 Lines   5459654864 +268 
   ==
   + Hits4494145233 +292 
   + Misses   9655 9631  -24 
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/beam/pull/12819?src=pr=tree) | Coverage 
Δ | |
   |---|---|---|
   | 
[sdks/python/apache\_beam/dataframe/doctests.py](https://codecov.io/gh/apache/beam/pull/12819/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2RvY3Rlc3RzLnB5)
 | `96.75% <96.22%> (-0.45%)` | :arrow_down: |
   | 
[...dks/python/apache\_beam/options/pipeline\_options.py](https://codecov.io/gh/apache/beam/pull/12819/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vb3B0aW9ucy9waXBlbGluZV9vcHRpb25zLnB5)
 | `93.76% <0.00%> (-1.03%)` | :arrow_down: |
   | 
[sdks/python/apache\_beam/io/gcp/bigquery.py](https://codecov.io/gh/apache/beam/pull/12819/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5LnB5)
 | `80.08% <0.00%> (-0.63%)` | :arrow_down: |
   | 
[sdks/python/apache\_beam/io/fileio.py](https://codecov.io/gh/apache/beam/pull/12819/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZmlsZWlvLnB5)
 | `95.80% <0.00%> (-0.05%)` | :arrow_down: |
   | 
[...e\_beam/portability/api/beam\_runner\_api\_pb2\_urns.py](https://codecov.io/gh/apache/beam/pull/12819/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjJfdXJucy5weQ==)
 | `100.00% <0.00%> (ø)` | |
   | 
[...\_beam/runners/portability/sdk\_container\_builder.py](https://codecov.io/gh/apache/beam/pull/12819/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9zZGtfY29udGFpbmVyX2J1aWxkZXIucHk=)
 | `31.21% <0.00%> (ø)` | |
   | 
[...eam/testing/benchmarks/nexmark/nexmark\_launcher.py](https://codecov.io/gh/apache/beam/pull/12819/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvbmV4bWFya19sYXVuY2hlci5weQ==)
 | `0.00% <0.00%> (ø)` | |
   | 
[...ache\_beam/runners/interactive/recording\_manager.py](https://codecov.io/gh/apache/beam/pull/12819/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9yZWNvcmRpbmdfbWFuYWdlci5weQ==)
 | `98.90% <0.00%> (+0.02%)` | :arrow_up: |
   | 
[sdks/python/apache\_beam/transforms/environments.py](https://codecov.io/gh/apache/beam/pull/12819/diff?src=pr=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9lbnZpcm9ubWVudHMucHk=)
 | `83.73% <0.00%> (+0.19%)` | :arrow_up: |
   | ... and [12 
more](https://codecov.io/gh/apache/beam/pull/12819/diff?src=pr=tree-more) | |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/beam/pull/12819?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/beam/pull/12819?src=pr=footer). Last 
update 
[c38fa68...a0a3099](https://codecov.io/gh/apache/beam/pull/12819?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] KevinGG opened a new pull request #12979: [BEAM-10545] Add 2 show options

[GitBox]

KevinGG opened a new pull request #12979:
URL: https://github.com/apache/beam/pull/12979


   1. Added 2 show API options in the side panel: duration and n.
   2. Added an `apply` button to alter the behavior of setting show API
  options: now the configuration change only takes effect after
  clicking the button; previously, the configuration is automatically
  applied when clicking a checkbox, but this behavior does not apply
  when the configuration now has text inputs.
   3. Added `interruptKernelIfNotDone` to the InspectableViewModel and
  always execute it when querying the kernel from such model. This made
  switching selected item and applying new show API options very
  snappy.
   
   
   
   Thank you for your contribution! Follow this checklist to help us 
incorporate your contribution quickly and easily:
   
- [ ] [**Choose 
reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and 
mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more 
tips on [how to make review process 
smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
   --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/)
 | --- | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/)
 | --- | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
 | ---
   Java | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/)
 | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[![Build
 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/)
 | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[![Build
 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/badge/i
 
con)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)[![Build
 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[![Build
 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/)
 | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/)
 | [![Build 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[![Build
 Status](htt
 
ps://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[![Build
 
Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
 | [![Build 

[GitHub] [beam] angoenka merged pull request #12562: [BEAM-10200] Respect profile_memory option and add memory profiler to…

[GitBox]

angoenka merged pull request #12562:
URL: https://github.com/apache/beam/pull/12562


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] boyuanzz closed pull request #12926: [WIP] Clean up with_attributes from PubsubSource

[GitBox]

boyuanzz closed pull request #12926:
URL: https://github.com/apache/beam/pull/12926


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] kennknowles commented on pull request #12962: [WIP][BEAM-9379] Update calcite to 1.25

[GitBox]

kennknowles commented on pull request #12962:
URL: https://github.com/apache/beam/pull/12962#issuecomment-701702361


   
https://scans.gradle.com/s/ofcfnu774du4y/tests/:sdks:java:extensions:sql:test/org.apache.beam.sdk.extensions.sql.BeamSqlDslSqlStdOperatorsTest/testArithmeticOperator#1
   
   Looks like Calcite switched to `java.math.BigDecimal`?



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] boyuanzz commented on pull request #12926: [WIP] Clean up with_attributes from PubsubSource

[GitBox]

boyuanzz commented on pull request #12926:
URL: https://github.com/apache/beam/pull/12926#issuecomment-701702337


   I'm going to close this PR since it brings performance regression.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] tvalentyn commented on a change in pull request #12981: [BEAM-10435] Add LOOPBACK mode Java ValidatesRunner test suite for ULR

[GitBox]

tvalentyn commented on a change in pull request #12981:
URL: https://github.com/apache/beam/pull/12981#discussion_r497864103



##
File path: runners/portability/java/build.gradle
##
@@ -82,8 +87,15 @@ void execBackgroundInVirtualenv(String... args) {
   proc.waitFor();
 }
 
-task startLocalJobService {
+task installBeamPythonInVirtualenv {
   dependsOn setupVirtualenv
+  doLast {
+execInVirtualenv "pip", "install", "-e", "."

Review comment:
   Race condition between comment updates, did not see your prior comments.
   
   [test] installs additional dependencies mentioned in 
https://github.com/apache/beam/blob/e6b3cf1da5f17a3a15f2a48986a1b84266f3a64c/sdks/python/setup.py#L176
   Same applies for other 'extra' dependencies.
   
   Your proposal sounds good to me. I would  move :
   installBeamPythonInVirtualenv into BeamModulePlugin.groovy -> 
installBeamPython, but feel free to keep here if you feel strongly about it.
   
   In a future change we can rename installGcpTest to 
installBeamPythonWithAllExtraDeps
   
   





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org