[GitHub] [beam] angoenka commented on pull request #12042: WIP: Jenkins Plugin

2020-06-30 Thread GitBox
angoenka commented on pull request #12042: URL: https://github.com/apache/beam/pull/12042#issuecomment-651573889 R: @tvalentyn Would you also take a look at the PR from Jenkins point of view. This is an automated message

[GitHub] [beam] amaliujia merged pull request #12110: [BEAM-9890] Try fixing BIT_AND(NULL) issue

2020-06-30 Thread GitBox
amaliujia merged pull request #12110: URL: https://github.com/apache/beam/pull/12110 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [beam] piotr-szuberski commented on a change in pull request #12023: [BEAM-10135] Add Python wrapper for Jdbc Write external transform

2020-06-30 Thread GitBox
piotr-szuberski commented on a change in pull request #12023: URL: https://github.com/apache/beam/pull/12023#discussion_r447596365 ## File path: sdks/python/apache_beam/io/external/jdbc.py ## @@ -0,0 +1,116 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or

[GitHub] [beam] piotr-szuberski commented on a change in pull request #12022: [BEAM-10135] Add Jdbc Write external transform

2020-06-30 Thread GitBox
piotr-szuberski commented on a change in pull request #12022: URL: https://github.com/apache/beam/pull/12022#discussion_r447616009 ## File path: sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/external/JdbcExternalWrite.java ## @@ -0,0 +1,155 @@ +/* + * Licensed

[GitHub] [beam] kamilwu commented on pull request #12132: [BEAM-10371] Run dependency check script with Python 3

2020-06-30 Thread GitBox
kamilwu commented on pull request #12132: URL: https://github.com/apache/beam/pull/12132#issuecomment-651671162 R: @TheNeuralBit cc: @tvalentyn This is an automated message from the Apache Git Service. To respond to the

[GitHub] [beam] kamilwu commented on pull request #12132: [BEAM-10371] Run dependency check script with Python 3

2020-06-30 Thread GitBox
kamilwu commented on pull request #12132: URL: https://github.com/apache/beam/pull/12132#issuecomment-651681684 Run Python2_PVR_Flink PreCommit This is an automated message from the Apache Git Service. To respond to the

[GitHub] [beam] piotr-szuberski commented on a change in pull request #12022: [BEAM-10135] Add Jdbc Write external transform

2020-06-30 Thread GitBox
piotr-szuberski commented on a change in pull request #12022: URL: https://github.com/apache/beam/pull/12022#discussion_r447616560 ## File path: sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/external/JdbcExternalWrite.java ## @@ -0,0 +1,155 @@ +/* + * Licensed

[GitHub] [beam] kamilwu opened a new pull request #12132: [BEAM-10371] Run dependency check script with Python 3

2020-06-30 Thread GitBox
kamilwu opened a new pull request #12132: URL: https://github.com/apache/beam/pull/12132 Python 2 is incompatible with some dependencies, which prevents Beam's community from getting the full picture of dependencies. Thank you for your contribution!

[GitHub] [beam] piotr-szuberski commented on a change in pull request #12023: [BEAM-10135] Add Python wrapper for Jdbc Write external transform

2020-06-30 Thread GitBox
piotr-szuberski commented on a change in pull request #12023: URL: https://github.com/apache/beam/pull/12023#discussion_r447596365 ## File path: sdks/python/apache_beam/io/external/jdbc.py ## @@ -0,0 +1,116 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or

[GitHub] [beam] piotr-szuberski commented on a change in pull request #12023: [BEAM-10135] Add Python wrapper for Jdbc Write external transform

2020-06-30 Thread GitBox
piotr-szuberski commented on a change in pull request #12023: URL: https://github.com/apache/beam/pull/12023#discussion_r447596365 ## File path: sdks/python/apache_beam/io/external/jdbc.py ## @@ -0,0 +1,116 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or

[GitHub] [beam] piotr-szuberski commented on a change in pull request #12023: [BEAM-10135] Add Python wrapper for Jdbc Write external transform

2020-06-30 Thread GitBox
piotr-szuberski commented on a change in pull request #12023: URL: https://github.com/apache/beam/pull/12023#discussion_r447604523 ## File path: sdks/python/apache_beam/io/external/xlang_jdbcio_it_test.py ## @@ -0,0 +1,138 @@ +# +# Licensed to the Apache Software Foundation

[GitHub] [beam] piotr-szuberski commented on a change in pull request #12022: [BEAM-10135] Add Jdbc Write external transform

2020-06-30 Thread GitBox
piotr-szuberski commented on a change in pull request #12022: URL: https://github.com/apache/beam/pull/12022#discussion_r447648168 ## File path: sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/external/JdbcExternalWrite.java ## @@ -0,0 +1,155 @@ +/* + * Licensed

[GitHub] [beam] piotr-szuberski commented on a change in pull request #12022: [BEAM-10135] Add Jdbc Write external transform

2020-06-30 Thread GitBox
piotr-szuberski commented on a change in pull request #12022: URL: https://github.com/apache/beam/pull/12022#discussion_r447648913 ## File path: sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/external/package-info.java ## @@ -0,0 +1,24 @@ +/* + * Licensed to the

[GitHub] [beam] piotr-szuberski commented on a change in pull request #12022: [BEAM-10135] Add Jdbc Write external transform

2020-06-30 Thread GitBox
piotr-szuberski commented on a change in pull request #12022: URL: https://github.com/apache/beam/pull/12022#discussion_r447647773 ## File path: runners/flink/job-server/flink_job_server.gradle ## @@ -88,6 +88,7 @@ dependencies { // TODO: Enable AWS and HDFS file system.

[GitHub] [beam] piotr-szuberski commented on a change in pull request #12023: [BEAM-10135] Add Python wrapper for Jdbc Write external transform

2020-06-30 Thread GitBox
piotr-szuberski commented on a change in pull request #12023: URL: https://github.com/apache/beam/pull/12023#discussion_r447688596 ## File path: sdks/python/apache_beam/io/external/jdbc.py ## @@ -0,0 +1,134 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or

[GitHub] [beam] piotr-szuberski commented on a change in pull request #12023: [BEAM-10135] Add Python wrapper for Jdbc Write external transform

2020-06-30 Thread GitBox
piotr-szuberski commented on a change in pull request #12023: URL: https://github.com/apache/beam/pull/12023#discussion_r447688596 ## File path: sdks/python/apache_beam/io/external/jdbc.py ## @@ -0,0 +1,134 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or

[GitHub] [beam] piotr-szuberski commented on a change in pull request #12022: [BEAM-10135] Add Jdbc Write external transform

2020-06-30 Thread GitBox
piotr-szuberski commented on a change in pull request #12022: URL: https://github.com/apache/beam/pull/12022#discussion_r447655698 ## File path: sdks/java/container/Dockerfile ## @@ -27,6 +27,7 @@ ADD target/beam-sdks-java-harness.jar /opt/apache/beam/jars/ # TODO May be

[GitHub] [beam] piotr-szuberski commented on a change in pull request #12023: [BEAM-10135] Add Python wrapper for Jdbc Write external transform

2020-06-30 Thread GitBox
piotr-szuberski commented on a change in pull request #12023: URL: https://github.com/apache/beam/pull/12023#discussion_r447688596 ## File path: sdks/python/apache_beam/io/external/jdbc.py ## @@ -0,0 +1,134 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or

[GitHub] [beam] piotr-szuberski commented on a change in pull request #12023: [BEAM-10135] Add Python wrapper for Jdbc Write external transform

2020-06-30 Thread GitBox
piotr-szuberski commented on a change in pull request #12023: URL: https://github.com/apache/beam/pull/12023#discussion_r447688596 ## File path: sdks/python/apache_beam/io/external/jdbc.py ## @@ -0,0 +1,134 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or

[GitHub] [beam] piotr-szuberski commented on a change in pull request #12023: [BEAM-10135] Add Python wrapper for Jdbc Write external transform

2020-06-30 Thread GitBox
piotr-szuberski commented on a change in pull request #12023: URL: https://github.com/apache/beam/pull/12023#discussion_r447688596 ## File path: sdks/python/apache_beam/io/external/jdbc.py ## @@ -0,0 +1,134 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or

[GitHub] [beam] aaltay commented on pull request #12129: Change GroupIntoBatches to group for real

2020-06-30 Thread GitBox
aaltay commented on pull request #12129: URL: https://github.com/apache/beam/pull/12129#issuecomment-651979803 Run Python2_PVR_Flink PreCommit This is an automated message from the Apache Git Service. To respond to the

[GitHub] [beam] TheNeuralBit commented on pull request #12067: [BEAM-10308] Make component ID assignments consistent across PipelineContext instances

2020-06-30 Thread GitBox
TheNeuralBit commented on pull request #12067: URL: https://github.com/apache/beam/pull/12067#issuecomment-651988106 retest this please This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] pabloem commented on pull request #12119: Add nexmark option to allow cancel streaming query job after complete

2020-06-30 Thread GitBox
pabloem commented on pull request #12119: URL: https://github.com/apache/beam/pull/12119#issuecomment-652000592 retest this please This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [beam] pabloem commented on pull request #12082: Standardizing BigQuery job names in Beam Python and Java SDKs

2020-06-30 Thread GitBox
pabloem commented on pull request #12082: URL: https://github.com/apache/beam/pull/12082#issuecomment-65147 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [beam] pabloem commented on pull request #12082: Standardizing BigQuery job names in Beam Python and Java SDKs

2020-06-30 Thread GitBox
pabloem commented on pull request #12082: URL: https://github.com/apache/beam/pull/12082#issuecomment-65251 Run Python 3.7 PostCommit This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] pabloem commented on pull request #12125: Reducing how much we call BigQuery Get Table API. The BQ API is queri…

2020-06-30 Thread GitBox
pabloem commented on pull request #12125: URL: https://github.com/apache/beam/pull/12125#issuecomment-651999756 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [beam] aaltay commented on pull request #12130: [BEAM-10381] Increase retries in pull_licenses_java.py from 3 to 9.

2020-06-30 Thread GitBox
aaltay commented on pull request #12130: URL: https://github.com/apache/beam/pull/12130#issuecomment-651999466 retest this please This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [beam] pabloem commented on pull request #12123: MetricFilter matching by Metric objects.

2020-06-30 Thread GitBox
pabloem commented on pull request #12123: URL: https://github.com/apache/beam/pull/12123#issuecomment-651999437 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [beam] aaltay commented on a change in pull request #12130: [BEAM-10381] Increase retries in pull_licenses_java.py from 3 to 9.

2020-06-30 Thread GitBox
aaltay commented on a change in pull request #12130: URL: https://github.com/apache/beam/pull/12130#discussion_r447930638 ## File path: sdks/java/container/license_scripts/pull_licenses_java.py ## @@ -39,7 +39,7 @@ from urllib.request import urlopen, URLError, HTTPError

[GitHub] [beam] pabloem commented on pull request #12107: Interactive Environment Inspector for messaging

2020-06-30 Thread GitBox
pabloem commented on pull request #12107: URL: https://github.com/apache/beam/pull/12107#issuecomment-651993513 Run Python2_PVR_Flink PreCommit This is an automated message from the Apache Git Service. To respond to the

[GitHub] [beam] pabloem commented on pull request #12107: Interactive Environment Inspector for messaging

2020-06-30 Thread GitBox
pabloem commented on pull request #12107: URL: https://github.com/apache/beam/pull/12107#issuecomment-651993429 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [beam] pabloem commented on pull request #12103: Bug fix: non 200 response streams are read twice which cause "Cannot …

2020-06-30 Thread GitBox
pabloem commented on pull request #12103: URL: https://github.com/apache/beam/pull/12103#issuecomment-651993623 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [beam] udim commented on pull request #11939: [BEAM-10197] Support typehints for Python's frozenset

2020-06-30 Thread GitBox
udim commented on pull request #11939: URL: https://github.com/apache/beam/pull/11939#issuecomment-651976446 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[GitHub] [beam] ajamato commented on a change in pull request #12130: [BEAM-10381] Increase retries in pull_licenses_java.py from 3 to 9.

2020-06-30 Thread GitBox
ajamato commented on a change in pull request #12130: URL: https://github.com/apache/beam/pull/12130#discussion_r447916417 ## File path: sdks/java/container/license_scripts/pull_licenses_java.py ## @@ -39,7 +39,7 @@ from urllib.request import urlopen, URLError, HTTPError

[GitHub] [beam] rohdesamuel commented on pull request #11765: [BEAM-9322] Remove passthrough_pcollection_output_ids and force_generated_pcollection_output_ids flags

2020-06-30 Thread GitBox
rohdesamuel commented on pull request #11765: URL: https://github.com/apache/beam/pull/11765#issuecomment-652015699 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] iemejia commented on pull request #12064: [BEAM-10284] Add option to pass configuration into ParquetIO.Sink

2020-06-30 Thread GitBox
iemejia commented on pull request #12064: URL: https://github.com/apache/beam/pull/12064#issuecomment-652048238 Thanks @JozoVilcek we should probably add some doc explaining that decision in the future, but not worth to block the PR for this.

[GitHub] [beam] ibzib commented on pull request #12138: Update ZetaSQL to 2020.06.1 release.

2020-06-30 Thread GitBox
ibzib commented on pull request #12138: URL: https://github.com/apache/beam/pull/12138#issuecomment-652048626 Looks like the new release hasn't been uploaded to maven central yet. This is an automated message from the Apache

[GitHub] [beam] iemejia merged pull request #12064: [BEAM-10284] Add option to pass configuration into ParquetIO.Sink

2020-06-30 Thread GitBox
iemejia merged pull request #12064: URL: https://github.com/apache/beam/pull/12064 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [beam] pabloem commented on pull request #12125: Reducing how much we call BigQuery Get Table API. The BQ API is queri…

2020-06-30 Thread GitBox
pabloem commented on pull request #12125: URL: https://github.com/apache/beam/pull/12125#issuecomment-652064977 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [beam] tweise opened a new pull request #12140: Remove misleading unsupported features mention for Python streaming

2020-06-30 Thread GitBox
tweise opened a new pull request #12140: URL: https://github.com/apache/beam/pull/12140 This was pointed out by a user. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ]

[GitHub] [beam] pabloem commented on pull request #12035: [BEAM-10220] Add support for REQUIRE_MISSING in RowJsonDeserializer, Make ACCEPT_MISSING_OR_NULL the default behavior.

2020-06-30 Thread GitBox
pabloem commented on pull request #12035: URL: https://github.com/apache/beam/pull/12035#issuecomment-652065015 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] TheNeuralBit commented on pull request #12067: [BEAM-10308] Make component ID assignments consistent across PipelineContext instances

2020-06-30 Thread GitBox
TheNeuralBit commented on pull request #12067: URL: https://github.com/apache/beam/pull/12067#issuecomment-652002592 Run XVR_Spark PostCommit This is an automated message from the Apache Git Service. To respond to the

[GitHub] [beam] TheNeuralBit commented on pull request #12067: [BEAM-10308] Make component ID assignments consistent across PipelineContext instances

2020-06-30 Thread GitBox
TheNeuralBit commented on pull request #12067: URL: https://github.com/apache/beam/pull/12067#issuecomment-652002495 Run XVR_Flink PostCommit This is an automated message from the Apache Git Service. To respond to the

[GitHub] [beam] aaltay merged pull request #12130: [BEAM-10381] Increase retries in pull_licenses_java.py from 3 to 9.

2020-06-30 Thread GitBox
aaltay merged pull request #12130: URL: https://github.com/apache/beam/pull/12130 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [beam] TheNeuralBit commented on pull request #12136: [BEAM-10391] Add attribute map null check in dataflow worker

2020-06-30 Thread GitBox
TheNeuralBit commented on pull request #12136: URL: https://github.com/apache/beam/pull/12136#issuecomment-652067240 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] amaliujia commented on pull request #12135: Move Dataflow runner to new beam-master-20200629 container

2020-06-30 Thread GitBox
amaliujia commented on pull request #12135: URL: https://github.com/apache/beam/pull/12135#issuecomment-652068919 It doesn't seem to be related though, but that Java Precommit didn't finish and it had a failed test. This is

[GitHub] [beam] rohdesamuel commented on pull request #11765: [BEAM-9322] Remove passthrough_pcollection_output_ids and force_generated_pcollection_output_ids flags

2020-06-30 Thread GitBox
rohdesamuel commented on pull request #11765: URL: https://github.com/apache/beam/pull/11765#issuecomment-652070641 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] rohdesamuel commented on pull request #11765: [BEAM-9322] Remove passthrough_pcollection_output_ids and force_generated_pcollection_output_ids flags

2020-06-30 Thread GitBox
rohdesamuel commented on pull request #11765: URL: https://github.com/apache/beam/pull/11765#issuecomment-652071096 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] lukecwik opened a new pull request #12137: [BEAM-10392] Attempt to fix/enable detection of rabbitmq:test stuckness.

2020-06-30 Thread GitBox
lukecwik opened a new pull request #12137: URL: https://github.com/apache/beam/pull/12137 * Ensure that the Consumer uses a thread safe object to pass the messages through (Consumer is invoked on a separate thread as per

[GitHub] [beam] davidyan74 commented on pull request #12047: [BEAM-10291] Adding full thread dump upon lull detection

2020-06-30 Thread GitBox
davidyan74 commented on pull request #12047: URL: https://github.com/apache/beam/pull/12047#issuecomment-652022287 @tvalentyn Added a unit test. PTAL. Thank you! This is an automated message from the Apache Git Service. To

[GitHub] [beam] lukecwik commented on pull request #12137: [BEAM-10392] Attempt to fix/enable detection of rabbitmq:test stuckness.

2020-06-30 Thread GitBox
lukecwik commented on pull request #12137: URL: https://github.com/apache/beam/pull/12137#issuecomment-652021896 R: @jbonofre @ajamato CC: @iemejia @mxm This is an automated message from the Apache Git Service. To

[GitHub] [beam] ibzib opened a new pull request #12138: Update ZetaSQL to 2020.06.1 release.

2020-06-30 Thread GitBox
ibzib opened a new pull request #12138: URL: https://github.com/apache/beam/pull/12138 R: @apilloud cc: @robinyqiu @amaliujia Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[GitHub] [beam] Ardagan commented on pull request #12139: [DO NOT REVIEW] Scd py bq test

2020-06-30 Thread GitBox
Ardagan commented on pull request #12139: URL: https://github.com/apache/beam/pull/12139#issuecomment-652059471 Run Python 2 PostCommit This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] Ardagan opened a new pull request #12139: [DO NOT REVIEW] Scd py bq test

2020-06-30 Thread GitBox
Ardagan opened a new pull request #12139: URL: https://github.com/apache/beam/pull/12139 **Please** add a meaningful description for your change here Thank you for your contribution! Follow this checklist to help us incorporate your contribution

[GitHub] [beam] pabloem commented on pull request #12125: Reducing how much we call BigQuery Get Table API. The BQ API is queri…

2020-06-30 Thread GitBox
pabloem commented on pull request #12125: URL: https://github.com/apache/beam/pull/12125#issuecomment-652064746 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [beam] pabloem commented on pull request #12084: [BEAM-10317] Python - Update BigQueryIO to tag BigQuery Jobs with the Dataflow Job ID

2020-06-30 Thread GitBox
pabloem commented on pull request #12084: URL: https://github.com/apache/beam/pull/12084#issuecomment-652066216 Run Python 3.5 PostCommit This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] TheNeuralBit edited a comment on pull request #12067: [BEAM-10308] Make component ID assignments consistent across PipelineContext instances

2020-06-30 Thread GitBox
TheNeuralBit edited a comment on pull request #12067: URL: https://github.com/apache/beam/pull/12067#issuecomment-652067737 The only failure in Python precommit seems to be a flake due to BEAM-10007. I think this is safe to merge after an LGTM

[GitHub] [beam] TheNeuralBit commented on pull request #12067: [BEAM-10308] Make component ID assignments consistent across PipelineContext instances

2020-06-30 Thread GitBox
TheNeuralBit commented on pull request #12067: URL: https://github.com/apache/beam/pull/12067#issuecomment-652067737 The only failure in Python precommit seems to be due to BEAM-10007. I think this is safe to merge after an LGTM

[GitHub] [beam] TheNeuralBit commented on pull request #12035: [BEAM-10220] Add support for REQUIRE_MISSING in RowJsonDeserializer, Make ACCEPT_MISSING_OR_NULL the default behavior.

2020-06-30 Thread GitBox
TheNeuralBit commented on pull request #12035: URL: https://github.com/apache/beam/pull/12035#issuecomment-652067854 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] TheNeuralBit commented on pull request #12035: [BEAM-10220] Add support for REQUIRE_MISSING in RowJsonDeserializer, Make ACCEPT_MISSING_OR_NULL the default behavior.

2020-06-30 Thread GitBox
TheNeuralBit commented on pull request #12035: URL: https://github.com/apache/beam/pull/12035#issuecomment-652029350 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] aaltay commented on pull request #12130: [BEAM-10381] Increase retries in pull_licenses_java.py from 3 to 9.

2020-06-30 Thread GitBox
aaltay commented on pull request #12130: URL: https://github.com/apache/beam/pull/12130#issuecomment-652041807 retest this please This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [beam] apilloud commented on pull request #12138: Update ZetaSQL to 2020.06.1 release.

2020-06-30 Thread GitBox
apilloud commented on pull request #12138: URL: https://github.com/apache/beam/pull/12138#issuecomment-652069731 test this please This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [beam] lostluck opened a new pull request #12141: [BEAM-9615] Add Row coder functions.

2020-06-30 Thread GitBox
lostluck opened a new pull request #12141: URL: https://github.com/apache/beam/pull/12141 Adds row encoder and decoder function generators, using reflection. Doesn't provide single element convenience functions for encoding and decoding struct values. Doesn't support looking up

[GitHub] [beam] aaltay commented on a change in pull request #12140: Remove misleading unsupported features mention for Python streaming

2020-06-30 Thread GitBox
aaltay commented on a change in pull request #12140: URL: https://github.com/apache/beam/pull/12140#discussion_r448012845 ## File path: website/www/site/content/en/documentation/sdks/python-streaming.md ## @@ -168,9 +168,7 @@ Python streaming execution does not currently

[GitHub] [beam] n3lsok commented on a change in pull request #11405: [BEAM-9747] Deprecate RedisIO.readAll() and add RedisIO.readKeyPatterns as a replacement

2020-06-30 Thread GitBox
n3lsok commented on a change in pull request #11405: URL: https://github.com/apache/beam/pull/11405#discussion_r448014725 ## File path: sdks/java/io/redis/src/main/java/org/apache/beam/sdk/io/redis/RedisIO.java ## @@ -123,8 +123,24 @@ public static Read read() { * Like

[GitHub] [beam] aaltay commented on pull request #12126: Update docs for Reshuffle to remove the reference for deduplication

2020-06-30 Thread GitBox
aaltay commented on pull request #12126: URL: https://github.com/apache/beam/pull/12126#issuecomment-652078379 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [beam] youngoli commented on pull request #12124: [BEAM-10312] Sub-element progress accounted for during splitting.

2020-06-30 Thread GitBox
youngoli commented on pull request #12124: URL: https://github.com/apache/beam/pull/12124#issuecomment-652092483 Run Go PostCommit This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [beam] youngoli commented on a change in pull request #12124: [BEAM-10312] Sub-element progress accounted for during splitting.

2020-06-30 Thread GitBox
youngoli commented on a change in pull request #12124: URL: https://github.com/apache/beam/pull/12124#discussion_r448027552 ## File path: sdks/go/pkg/beam/core/runtime/exec/datasource.go ## @@ -302,12 +302,23 @@ func (n *DataSource) Split(splits []int64, frac float64, bufSize

[GitHub] [beam] Ardagan commented on pull request #12139: [DO NOT REVIEW] Scd py bq test

2020-06-30 Thread GitBox
Ardagan commented on pull request #12139: URL: https://github.com/apache/beam/pull/12139#issuecomment-652098133 Run Python 3.7 PostCommit This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] Ardagan commented on pull request #12139: [DO NOT REVIEW] Scd py bq test

2020-06-30 Thread GitBox
Ardagan commented on pull request #12139: URL: https://github.com/apache/beam/pull/12139#issuecomment-652098053 Run Python 2 PostCommit This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] tweise commented on a change in pull request #12140: Remove misleading unsupported features mention for Python streaming

2020-06-30 Thread GitBox
tweise commented on a change in pull request #12140: URL: https://github.com/apache/beam/pull/12140#discussion_r448034061 ## File path: website/www/site/content/en/documentation/sdks/python-streaming.md ## @@ -168,9 +168,7 @@ Python streaming execution does not currently

[GitHub] [beam] corvin-quodai opened a new pull request #12142: [BEAM-10004] Fixes occasional ZeroDivisionError when bundle size weights become 0 for MongoDB IO

2020-06-30 Thread GitBox
corvin-quodai opened a new pull request #12142: URL: https://github.com/apache/beam/pull/12142 Depending on runner and the mongo collection to read, the bundle size can fall below 1mb which resulted in Source Bundles with weight=0. This lead to a zero division error (raised from

[GitHub] [beam] aaltay commented on a change in pull request #12047: [BEAM-10291] Adding full thread dump upon lull detection

2020-06-30 Thread GitBox
aaltay commented on a change in pull request #12047: URL: https://github.com/apache/beam/pull/12047#discussion_r448052626 ## File path: sdks/python/apache_beam/runners/worker/sdk_worker.py ## @@ -570,6 +576,23 @@ def _log_lull_in_bundle_processor(self, processor):

[GitHub] [beam] lastomato commented on pull request #12103: Bug fix: non 200 response streams are read twice which cause "Cannot …

2020-06-30 Thread GitBox
lastomato commented on pull request #12103: URL: https://github.com/apache/beam/pull/12103#issuecomment-652124759 Thanks, Pablo. Is this something I should look into, or is it transient? This is an automated message from the

[GitHub] [beam] davidyan74 commented on a change in pull request #12047: [BEAM-10291] Adding full thread dump upon lull detection

2020-06-30 Thread GitBox
davidyan74 commented on a change in pull request #12047: URL: https://github.com/apache/beam/pull/12047#discussion_r448058760 ## File path: sdks/python/apache_beam/runners/worker/sdk_worker.py ## @@ -570,6 +576,23 @@ def _log_lull_in_bundle_processor(self, processor):

[GitHub] [beam] milantracy commented on pull request #10487: [BEAM-5504] Introduce PubsubAvroTable

2020-06-30 Thread GitBox
milantracy commented on pull request #10487: URL: https://github.com/apache/beam/pull/10487#issuecomment-652128780 @TheNeuralBit @amaliujia @aaltay sorry for updating the PR after such a long time , feel free to leave your comments.

[GitHub] [beam] steveniemitz opened a new pull request #12144: [BEAM-10395] Deduplicate uploads by destinations before uploading

2020-06-30 Thread GitBox
steveniemitz opened a new pull request #12144: URL: https://github.com/apache/beam/pull/12144 If a job is configured to upload multiple files to the same destination, they'll end up attempting to overwrite each other in parallel, causing the runner to need to retry multiple times until

[GitHub] [beam] lostluck commented on pull request #12141: [BEAM-9615] Add Row coder functions.

2020-06-30 Thread GitBox
lostluck commented on pull request #12141: URL: https://github.com/apache/beam/pull/12141#issuecomment-652076911 R: @youngoli @tysonjh This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] TheNeuralBit commented on pull request #12067: [BEAM-10308] Make component ID assignments consistent across PipelineContext instances

2020-06-30 Thread GitBox
TheNeuralBit commented on pull request #12067: URL: https://github.com/apache/beam/pull/12067#issuecomment-652090261 @lukecwik do you happen to have time to review this? Looks like cham is OOO I'd like to get this in before the release cut tomorrow since it fixes a significant issue

[GitHub] [beam] youngoli merged pull request #12124: [BEAM-10312] Sub-element progress accounted for during splitting.

2020-06-30 Thread GitBox
youngoli merged pull request #12124: URL: https://github.com/apache/beam/pull/12124 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [beam] tweise commented on a change in pull request #12140: Remove misleading unsupported features mention for Python streaming

2020-06-30 Thread GitBox
tweise commented on a change in pull request #12140: URL: https://github.com/apache/beam/pull/12140#discussion_r448033172 ## File path: website/www/site/content/en/documentation/sdks/python-streaming.md ## @@ -168,9 +168,7 @@ Python streaming execution does not currently

[GitHub] [beam] milantracy commented on a change in pull request #10487: [BEAM-5504] Introduce PubsubAvroTable

2020-06-30 Thread GitBox
milantracy commented on a change in pull request #10487: URL: https://github.com/apache/beam/pull/10487#discussion_r448052137 ## File path: sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/AvroUtils.java ## @@ -339,6 +350,35 @@ public static Schema

[GitHub] [beam] boyuanzz commented on a change in pull request #12016: [BEAM-10341] Support drain in python and java SDF

2020-06-30 Thread GitBox
boyuanzz commented on a change in pull request #12016: URL: https://github.com/apache/beam/pull/12016#discussion_r448060178 ## File path: sdks/python/apache_beam/transforms/core.py ## @@ -320,6 +320,22 @@ def split_and_size(self, element, restriction): for part in

[GitHub] [beam] udim commented on pull request #12129: Change GroupIntoBatches to group for real

2020-06-30 Thread GitBox
udim commented on pull request #12129: URL: https://github.com/apache/beam/pull/12129#issuecomment-652140583 Also, please add a JIRA :) This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] udim commented on pull request #12009: [BEAM-10258] Support type hint annotations on PTransform's expand()

2020-06-30 Thread GitBox
udim commented on pull request #12009: URL: https://github.com/apache/beam/pull/12009#issuecomment-652079902 @saavan-google-intern I haven't taken a close look, but the failing tests could be due changes in this PR. PTAL

[GitHub] [beam] y1chi commented on a change in pull request #12140: Remove misleading unsupported features mention for Python streaming

2020-06-30 Thread GitBox
y1chi commented on a change in pull request #12140: URL: https://github.com/apache/beam/pull/12140#discussion_r448021758 ## File path: website/www/site/content/en/documentation/sdks/python-streaming.md ## @@ -168,9 +168,7 @@ Python streaming execution does not currently

[GitHub] [beam] corvin-quodai commented on pull request #12142: [BEAM-10004] Fixes occasional ZeroDivisionError when bundle size weights become 0 for MongoDB IO

2020-06-30 Thread GitBox
corvin-quodai commented on pull request #12142: URL: https://github.com/apache/beam/pull/12142#issuecomment-652120114 R: @aaltay R: @y1chi I am not sure how rounding up the source bundle size would affect other parts of beam. Would appreciate your input. Happy to change this PR

[GitHub] [beam] pabloem commented on pull request #12103: Bug fix: non 200 response streams are read twice which cause "Cannot …

2020-06-30 Thread GitBox
pabloem commented on pull request #12103: URL: https://github.com/apache/beam/pull/12103#issuecomment-652085431 I'll merge once this passes This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] TheNeuralBit merged pull request #12035: [BEAM-10220] Add support for REQUIRE_MISSING in RowJsonDeserializer, Make ACCEPT_MISSING_OR_NULL the default behavior.

2020-06-30 Thread GitBox
TheNeuralBit merged pull request #12035: URL: https://github.com/apache/beam/pull/12035 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [beam] youngoli commented on a change in pull request #12127: [BEAM-9615] Add bytes, bool, and iterable coders

2020-06-30 Thread GitBox
youngoli commented on a change in pull request #12127: URL: https://github.com/apache/beam/pull/12127#discussion_r448028531 ## File path: sdks/go/pkg/beam/core/graph/coder/bytes_test.go ## @@ -0,0 +1,61 @@ +// Licensed to the Apache Software Foundation (ASF) under one or more

[GitHub] [beam] tweise merged pull request #12140: Remove misleading unsupported features mention for Python streaming

2020-06-30 Thread GitBox
tweise merged pull request #12140: URL: https://github.com/apache/beam/pull/12140 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [beam] milantracy commented on a change in pull request #10487: [BEAM-5504] Introduce PubsubAvroTable

2020-06-30 Thread GitBox
milantracy commented on a change in pull request #10487: URL: https://github.com/apache/beam/pull/10487#discussion_r448052689 ## File path: sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/AvroPubsubMessageToRow.java ## @@ -0,0

[GitHub] [beam] aaltay commented on pull request #12142: [BEAM-10004] Fixes occasional ZeroDivisionError when bundle size weights become 0 for MongoDB IO

2020-06-30 Thread GitBox
aaltay commented on pull request #12142: URL: https://github.com/apache/beam/pull/12142#issuecomment-652120930 R: @chamikaramj - any concerns about this bundle size change? This is an automated message from the Apache Git

[GitHub] [beam] davidyan74 opened a new pull request #12143: [BEAM-10291] Adding full thread dump upon lull detection for Dataflow…

2020-06-30 Thread GitBox
davidyan74 opened a new pull request #12143: URL: https://github.com/apache/beam/pull/12143 Adding full thread dump upon lull detection for Dataflow Java worker. Thank you for your contribution! Follow this checklist to help us incorporate your

[GitHub] [beam] milantracy commented on pull request #10487: [BEAM-5504] Introduce PubsubAvroTable

2020-06-30 Thread GitBox
milantracy commented on pull request #10487: URL: https://github.com/apache/beam/pull/10487#issuecomment-652203139 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] pabloem merged pull request #12103: Bug fix: non 200 response streams are read twice which cause "Cannot …

2020-06-30 Thread GitBox
pabloem merged pull request #12103: URL: https://github.com/apache/beam/pull/12103 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [beam] pabloem commented on pull request #12103: Bug fix: non 200 response streams are read twice which cause "Cannot …

2020-06-30 Thread GitBox
pabloem commented on pull request #12103: URL: https://github.com/apache/beam/pull/12103#issuecomment-652204041 thanks @lastomato ! This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [beam] pabloem merged pull request #12123: MetricFilter matching by Metric objects.

2020-06-30 Thread GitBox
pabloem merged pull request #12123: URL: https://github.com/apache/beam/pull/12123 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [beam] pabloem commented on a change in pull request #12123: MetricFilter matching by Metric objects.

2020-06-30 Thread GitBox
pabloem commented on a change in pull request #12123: URL: https://github.com/apache/beam/pull/12123#discussion_r448128821 ## File path: sdks/python/apache_beam/metrics/metric_test.py ## @@ -159,15 +163,26 @@ def process(self, element): res = pipeline.run()

[GitHub] [beam] pabloem commented on pull request #12085: [BEAM-10318] fix uninitialized grpc_server in FnApiRunner

2020-06-30 Thread GitBox
pabloem commented on pull request #12085: URL: https://github.com/apache/beam/pull/12085#issuecomment-652206523 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

  1   2   >