[GitHub] [beam] purbanow commented on a change in pull request #12117: [BEAM-10343] Add dispositions for SnowflakeIO.write

2020-07-06 Thread GitBox
purbanow commented on a change in pull request #12117: URL: https://github.com/apache/beam/pull/12117#discussion_r450012687 ## File path: sdks/java/io/snowflake/src/main/java/org/apache/beam/sdk/io/snowflake/services/SnowflakeServiceImpl.java ## @@ -169,6 +191,54 @@ private

[GitHub] [beam] piotr-szuberski commented on pull request #12145: [BEAM-10136] [BEAM-10135] Add JdbcIO for cross-language with python wrapper

2020-07-06 Thread GitBox
piotr-szuberski commented on pull request #12145: URL: https://github.com/apache/beam/pull/12145#issuecomment-654116484 Run XVR_Flink PostCommit This is an automated message from the Apache Git Service. To respond to the

[GitHub] [beam] purbanow commented on a change in pull request #12117: [BEAM-10343] Add dispositions for SnowflakeIO.write

2020-07-06 Thread GitBox
purbanow commented on a change in pull request #12117: URL: https://github.com/apache/beam/pull/12117#discussion_r450011282 ## File path: sdks/java/io/snowflake/src/main/java/org/apache/beam/sdk/io/snowflake/SnowflakeIO.java ## @@ -155,7 +157,7 @@ * items.apply( *

[GitHub] [beam] DariuszAniszewski opened a new pull request #12176: [BEAM-9895] Added integration test for SnowflakeIO

2020-07-06 Thread GitBox
DariuszAniszewski opened a new pull request #12176: URL: https://github.com/apache/beam/pull/12176 Added integration test for SnowflakeIO. To run it, a real Snowflake instance has to be created and configured. Once [this thread on

[GitHub] [beam] kamilwu commented on pull request #11788: [BEAM-9785] Add Python 3.8 postcommit tests

2020-07-06 Thread GitBox
kamilwu commented on pull request #11788: URL: https://github.com/apache/beam/pull/11788#issuecomment-654111381 I see that PostCommit suite has passed on Python 3.8, so it seems all IO libraries are compatible (unless we skipped some of tests, but I don't think so)

[GitHub] [beam] kamilwu commented on pull request #11788: [BEAM-9785] Add Python 3.8 postcommit tests

2020-07-06 Thread GitBox
kamilwu commented on pull request #11788: URL: https://github.com/apache/beam/pull/11788#issuecomment-654111796 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [beam] purbanow commented on a change in pull request #12117: [BEAM-10343] Add dispositions for SnowflakeIO.write

2020-07-06 Thread GitBox
purbanow commented on a change in pull request #12117: URL: https://github.com/apache/beam/pull/12117#discussion_r450015267 ## File path: sdks/java/io/snowflake/src/main/java/org/apache/beam/sdk/io/snowflake/services/SnowflakeServiceImpl.java ## @@ -25,7 +27,9 @@ import

[GitHub] [beam] purbanow commented on a change in pull request #12117: [BEAM-10343] Add dispositions for SnowflakeIO.write

2020-07-06 Thread GitBox
purbanow commented on a change in pull request #12117: URL: https://github.com/apache/beam/pull/12117#discussion_r450018715 ## File path: sdks/java/io/snowflake/src/main/java/org/apache/beam/sdk/io/snowflake/data/text/SnowflakeBinary.java ## @@ -0,0 +1,61 @@ +/* + * Licensed

[GitHub] [beam] kkucharc commented on pull request #12146: [BEAM-10383] Update Snowflake JDBC dependency

2020-07-06 Thread GitBox
kkucharc commented on pull request #12146: URL: https://github.com/apache/beam/pull/12146#issuecomment-654062448 I can see all tests passed and review passed, so I'm merging. Thanks a lot all! :) This is an automated

[GitHub] [beam] purbanow commented on a change in pull request #12117: [BEAM-10343] Add dispositions for SnowflakeIO.write

2020-07-06 Thread GitBox
purbanow commented on a change in pull request #12117: URL: https://github.com/apache/beam/pull/12117#discussion_r450011779 ## File path: sdks/java/io/snowflake/src/main/java/org/apache/beam/sdk/io/snowflake/SnowflakeIO.java ## @@ -593,7 +606,7 @@ public void

[GitHub] [beam] mwalenia merged pull request #12176: [BEAM-9895] Added integration test for SnowflakeIO

2020-07-06 Thread GitBox
mwalenia merged pull request #12176: URL: https://github.com/apache/beam/pull/12176 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [beam] piotr-szuberski commented on pull request #12145: [BEAM-10136] [BEAM-10135] Add JdbcIO for cross-language with python wrapper

2020-07-06 Thread GitBox
piotr-szuberski commented on pull request #12145: URL: https://github.com/apache/beam/pull/12145#issuecomment-654134208 Run XVR_Flink PostCommit This is an automated message from the Apache Git Service. To respond to the

[GitHub] [beam] purbanow commented on a change in pull request #12117: [BEAM-10343] Add dispositions for SnowflakeIO.write

2020-07-06 Thread GitBox
purbanow commented on a change in pull request #12117: URL: https://github.com/apache/beam/pull/12117#discussion_r450018838 ## File path: sdks/java/io/snowflake/src/main/java/org/apache/beam/sdk/io/snowflake/data/text/SnowflakeVarchar.java ## @@ -0,0 +1,59 @@ +/* + * Licensed

[GitHub] [beam] kkucharc merged pull request #12146: [BEAM-10383] Update Snowflake JDBC dependency

2020-07-06 Thread GitBox
kkucharc merged pull request #12146: URL: https://github.com/apache/beam/pull/12146 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [beam] kamilwu commented on pull request #12132: [BEAM-10371] Run dependency check script with Python 3

2020-07-06 Thread GitBox
kamilwu commented on pull request #12132: URL: https://github.com/apache/beam/pull/12132#issuecomment-654112297 @TheNeuralBit Could you take a look? This is an automated message from the Apache Git Service. To respond to the

[GitHub] [beam] chamikaramj commented on pull request #12188: [BEAM-10411] Adds an example that use Python cross-language Kafka transforms.

2020-07-06 Thread GitBox
chamikaramj commented on pull request #12188: URL: https://github.com/apache/beam/pull/12188#issuecomment-654607544 R: @davidcavazos @ihji This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] amaliujia commented on a change in pull request #12169: [BEAM-9953[SQL][ZetaSQL] Support Pure SQL user-defined table-valued function.

2020-07-06 Thread GitBox
amaliujia commented on a change in pull request #12169: URL: https://github.com/apache/beam/pull/12169#discussion_r450616651 ## File path: sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLPlannerImpl.java ## @@ -211,7 +212,7 @@

[GitHub] [beam] amaliujia commented on a change in pull request #12169: [BEAM-9953[SQL][ZetaSQL] Support Pure SQL user-defined table-valued function.

2020-07-06 Thread GitBox
amaliujia commented on a change in pull request #12169: URL: https://github.com/apache/beam/pull/12169#discussion_r450616694 ## File path: sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLPlannerImpl.java ## @@ -171,6 +178,18 @@

[GitHub] [beam] chamikaramj opened a new pull request #12188: [BEAM-10411] Adds an example that use Python cross-language Kafka transforms.

2020-07-06 Thread GitBox
chamikaramj opened a new pull request #12188: URL: https://github.com/apache/beam/pull/12188 **Please** add a meaningful description for your change here Thank you for your contribution! Follow this checklist to help us incorporate your contribution

[GitHub] [beam] amaliujia commented on a change in pull request #12169: [BEAM-9953[SQL][ZetaSQL] Support Pure SQL user-defined table-valued function.

2020-07-06 Thread GitBox
amaliujia commented on a change in pull request #12169: URL: https://github.com/apache/beam/pull/12169#discussion_r450616472 ## File path: sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ConversionContext.java ## @@ -33,7

[GitHub] [beam] piotr-szuberski commented on pull request #12145: [BEAM-10136] [BEAM-10135] Add JdbcIO for cross-language with python wrapper

2020-07-06 Thread GitBox
piotr-szuberski commented on pull request #12145: URL: https://github.com/apache/beam/pull/12145#issuecomment-654164768 Run XVR_Flink PostCommit This is an automated message from the Apache Git Service. To respond to the

[GitHub] [beam] purbanow commented on a change in pull request #12117: [BEAM-10343] Add dispositions for SnowflakeIO.write

2020-07-06 Thread GitBox
purbanow commented on a change in pull request #12117: URL: https://github.com/apache/beam/pull/12117#discussion_r450218823 ## File path: sdks/java/io/snowflake/src/test/java/org/apache/beam/sdk/io/snowflake/test/unit/data/SnowflakeBooleanTest.java ## @@ -0,0 +1,32 @@ +/* + *

[GitHub] [beam] piotr-szuberski commented on pull request #12145: [BEAM-10136] [BEAM-10135] Add JdbcIO for cross-language with python wrapper

2020-07-06 Thread GitBox
piotr-szuberski commented on pull request #12145: URL: https://github.com/apache/beam/pull/12145#issuecomment-654229168 Run XVR_Flink PostCommit This is an automated message from the Apache Git Service. To respond to the

[GitHub] [beam] purbanow commented on pull request #12117: [BEAM-10343] Add dispositions for SnowflakeIO.write

2020-07-06 Thread GitBox
purbanow commented on pull request #12117: URL: https://github.com/apache/beam/pull/12117#issuecomment-654253476 @lgajowy thanks a lot for CR. I respond for your all comments. Will you find moment to check again? This is

[GitHub] [beam] purbanow edited a comment on pull request #12117: [BEAM-10343] Add dispositions for SnowflakeIO.write

2020-07-06 Thread GitBox
purbanow edited a comment on pull request #12117: URL: https://github.com/apache/beam/pull/12117#issuecomment-654253476 @lgajowy thanks a lot for CR. I respond to all comments. Will you find moment to check again? This is

[GitHub] [beam] mxm commented on a change in pull request #12133: [BEAM-10385] Integrate SQL expansion into Flink job server

2020-07-06 Thread GitBox
mxm commented on a change in pull request #12133: URL: https://github.com/apache/beam/pull/12133#discussion_r450162277 ## File path: sdks/python/apache_beam/transforms/sql.py ## @@ -67,10 +66,9 @@ class SqlTransform(ExternalTransform): """ URN =

[GitHub] [beam] piotr-szuberski commented on pull request #12145: [BEAM-10136] [BEAM-10135] Add JdbcIO for cross-language with python wrapper

2020-07-06 Thread GitBox
piotr-szuberski commented on pull request #12145: URL: https://github.com/apache/beam/pull/12145#issuecomment-654260584 Run XVR_Flink PostCommit This is an automated message from the Apache Git Service. To respond to the

[GitHub] [beam] kennknowles commented on pull request #12162: [BEAM-10402] Enable checker framework, disabled everywhere since every module has errors. Eliminate some errors (but no complete modules)

2020-07-06 Thread GitBox
kennknowles commented on pull request #12162: URL: https://github.com/apache/beam/pull/12162#issuecomment-654367733 Ah, those failures are a good catch, actually. The versions of the analyzer are not pinned, and they released 3.5.0 on July 1. As of this moment it is not yet present at

[GitHub] [beam] sclukas77 commented on a change in pull request #12090: [BEAM-10336,BEAM-10337] Add SchemaIO abstraction and implement for PubSub

2020-07-06 Thread GitBox
sclukas77 commented on a change in pull request #12090: URL: https://github.com/apache/beam/pull/12090#discussion_r450389176 ## File path: sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubIOJsonTable.java ## @@ -122,14

[GitHub] [beam] pabloem merged pull request #12084: [BEAM-10317] Python - Update BigQueryIO to tag BigQuery Jobs with the Dataflow Job ID

2020-07-06 Thread GitBox
pabloem merged pull request #12084: URL: https://github.com/apache/beam/pull/12084 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [beam] ajamato opened a new pull request #12177: [BEAM-10317] Add requests to python deps

2020-07-06 Thread GitBox
ajamato opened a new pull request #12177: URL: https://github.com/apache/beam/pull/12177 [BEAM-10317] Add requests to python deps Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[GitHub] [beam] pabloem commented on pull request #12083: [BEAM-10317] Java - Update BigQueryIO to tag BigQuery Jobs with the Dataflow Job ID

2020-07-06 Thread GitBox
pabloem commented on pull request #12083: URL: https://github.com/apache/beam/pull/12083#issuecomment-654402994 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [beam] ajamato commented on pull request #12177: [BEAM-10317] Add requests to python deps

2020-07-06 Thread GitBox
ajamato commented on pull request #12177: URL: https://github.com/apache/beam/pull/12177#issuecomment-654403556 @pabloem @robertwb This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [beam] kennknowles commented on pull request #12162: [BEAM-10402] Enable checker framework, disabled everywhere since every module has errors. Eliminate some errors (but no complete modules)

2020-07-06 Thread GitBox
kennknowles commented on pull request #12162: URL: https://github.com/apache/beam/pull/12162#issuecomment-654365801 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [beam] piotr-szuberski edited a comment on pull request #12145: [BEAM-10136] [BEAM-10135] Add JdbcIO for cross-language with python wrapper

2020-07-06 Thread GitBox
piotr-szuberski edited a comment on pull request #12145: URL: https://github.com/apache/beam/pull/12145#issuecomment-654333554 @TheNeuralBit For some reason "Flink CrossLanguageValidatesRunner Tests" timeouts on xlang_jdbc test and on the other hand the same test passes on Python

[GitHub] [beam] piotr-szuberski edited a comment on pull request #12145: [BEAM-10136] [BEAM-10135] Add JdbcIO for cross-language with python wrapper

2020-07-06 Thread GitBox
piotr-szuberski edited a comment on pull request #12145: URL: https://github.com/apache/beam/pull/12145#issuecomment-654333554 @TheNeuralBit For some reason "Flink CrossLanguageValidatesRunner Tests" timeouts on xlang_jdbc test and on the other hand the same test passes on Python

[GitHub] [beam] ibzib commented on a change in pull request #12163: [BEAM-9712] Refactor ZetaSQL planner code and support setting default timezone

2020-07-06 Thread GitBox
ibzib commented on a change in pull request #12163: URL: https://github.com/apache/beam/pull/12163#discussion_r450384095 ## File path: sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLQueryPlanner.java ## @@ -65,9 +65,12 @@

[GitHub] [beam] Ardagan commented on pull request #12167: [BEAM-9889] Populate local instance of InfluxDB with data

2020-07-06 Thread GitBox
Ardagan commented on pull request #12167: URL: https://github.com/apache/beam/pull/12167#issuecomment-654381327 I'm not well familiar with influx DB operations. It might be worth another pair of eyes taking a look at overall backup approach.

[GitHub] [beam] Ardagan commented on a change in pull request #12167: [BEAM-9889] Populate local instance of InfluxDB with data

2020-07-06 Thread GitBox
Ardagan commented on a change in pull request #12167: URL: https://github.com/apache/beam/pull/12167#discussion_r450384812 ## File path: .test-infra/metrics/influxdb/Dockerfile ## @@ -0,0 +1,35 @@

[GitHub] [beam] aaltay commented on pull request #12057: Use bz2 to compress pickler output for better compression.

2020-07-06 Thread GitBox
aaltay commented on pull request #12057: URL: https://github.com/apache/beam/pull/12057#issuecomment-654380953 R: @tvalentyn - Could you review this once you get a chance? This is an automated message from the Apache Git

[GitHub] [beam] tvalentyn commented on pull request #12057: Use bz2 to compress pickler output for better compression.

2020-07-06 Thread GitBox
tvalentyn commented on pull request #12057: URL: https://github.com/apache/beam/pull/12057#issuecomment-654387707 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [beam] boyuanzz commented on pull request #12143: [BEAM-10291] Adding full thread dump upon lull detection for Dataflow…

2020-07-06 Thread GitBox
boyuanzz commented on pull request #12143: URL: https://github.com/apache/beam/pull/12143#issuecomment-654395602 @pabloem Could you please help review this PR since you have done some excellent work related to lull detction.

[GitHub] [beam] pabloem commented on pull request #12084: [BEAM-10317] Python - Update BigQueryIO to tag BigQuery Jobs with the Dataflow Job ID

2020-07-06 Thread GitBox
pabloem commented on pull request #12084: URL: https://github.com/apache/beam/pull/12084#issuecomment-654395407 Thanks @ajamato - LGTM This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] pabloem merged pull request #12075: Remove usage of jdk-internal NotNull annotation

2020-07-06 Thread GitBox
pabloem merged pull request #12075: URL: https://github.com/apache/beam/pull/12075 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [beam] tvalentyn commented on pull request #11788: [BEAM-9785] Add Python 3.8 postcommit tests

2020-07-06 Thread GitBox
tvalentyn commented on pull request #11788: URL: https://github.com/apache/beam/pull/11788#issuecomment-654406007 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [beam] tvalentyn commented on a change in pull request #11788: [BEAM-9785] Add Python 3.8 postcommit tests

2020-07-06 Thread GitBox
tvalentyn commented on a change in pull request #11788: URL: https://github.com/apache/beam/pull/11788#discussion_r450417158 ## File path: sdks/python/test-suites/direct/py38/build.gradle ## @@ -0,0 +1,35 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one +

[GitHub] [beam] aaltay commented on pull request #12057: Use bz2 to compress pickler output for better compression.

2020-07-06 Thread GitBox
aaltay commented on pull request #12057: URL: https://github.com/apache/beam/pull/12057#issuecomment-654408159 Please do not merge yet. Running a few more tests. This is an automated message from the Apache Git Service. To

[GitHub] [beam] pabloem commented on pull request #12143: [BEAM-10291] Adding full thread dump upon lull detection for Dataflow…

2020-07-06 Thread GitBox
pabloem commented on pull request #12143: URL: https://github.com/apache/beam/pull/12143#issuecomment-654408145 sure, I'll take a look! This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] kennknowles commented on pull request #12162: [BEAM-10402] Enable checker framework, disabled everywhere since every module has errors. Eliminate some errors (but no complete modules)

2020-07-06 Thread GitBox
kennknowles commented on pull request #12162: URL: https://github.com/apache/beam/pull/12162#issuecomment-654366722 Run Python2_PVR_Flink PreCommit This is an automated message from the Apache Git Service. To respond to the

[GitHub] [beam] kennknowles commented on pull request #12162: [BEAM-10402] Enable checker framework, disabled everywhere since every module has errors. Eliminate some errors (but no complete modules)

2020-07-06 Thread GitBox
kennknowles commented on pull request #12162: URL: https://github.com/apache/beam/pull/12162#issuecomment-654366549 Maybe wait a week or two after getting this more established, then we can remove the spotbugs legacy annotations and turn off spotbugs nullness since it is a heuristic with

[GitHub] [beam] mxm commented on pull request #12133: [BEAM-10385] Integrate SQL expansion into Flink job server

2020-07-06 Thread GitBox
mxm commented on pull request #12133: URL: https://github.com/apache/beam/pull/12133#issuecomment-654371175 Run Python2_PVR_Flink PreCommit This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] piotr-szuberski edited a comment on pull request #12145: [BEAM-10136] [BEAM-10135] Add JdbcIO for cross-language with python wrapper

2020-07-06 Thread GitBox
piotr-szuberski edited a comment on pull request #12145: URL: https://github.com/apache/beam/pull/12145#issuecomment-654333554 @TheNeuralBit For some reason "Flink CrossLanguageValidatesRunner Tests" timeouts on xlang_jdbc test and on the other hand the same test passes on Python

[GitHub] [beam] ibzib commented on pull request #12157: [BEAM-7587] Spark portable streaming

2020-07-06 Thread GitBox
ibzib commented on pull request #12157: URL: https://github.com/apache/beam/pull/12157#issuecomment-654405149 Please fix compilation errors: ``` > Task :runners:spark:compileJava FAILED

[GitHub] [beam] annaqin418 commented on a change in pull request #12157: [BEAM-7587] Spark portable streaming

2020-07-06 Thread GitBox
annaqin418 commented on a change in pull request #12157: URL: https://github.com/apache/beam/pull/12157#discussion_r450281739 ## File path: runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkTranslationContext.java ## @@ -35,15 +35,15 @@ * compute

[GitHub] [beam] aaltay merged pull request #12165: fix formatting for pull request post-commit status table

2020-07-06 Thread GitBox
aaltay merged pull request #12165: URL: https://github.com/apache/beam/pull/12165 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [beam] jmcomets commented on pull request #12173: [BEAM-XXX] JdbcIO - Allow customization of the PoolableDataSource's pool

2020-07-06 Thread GitBox
jmcomets commented on pull request #12173: URL: https://github.com/apache/beam/pull/12173#issuecomment-654291827 R: @jbonofre Should be all good now! ;) This is an automated message from the Apache Git Service. To respond to

[GitHub] [beam] TheNeuralBit commented on pull request #12162: [BEAM-10402] Enable checker framework, disabled everywhere since every module has errors. Eliminate some errors (but no complete modules)

2020-07-06 Thread GitBox
TheNeuralBit commented on pull request #12162: URL: https://github.com/apache/beam/pull/12162#issuecomment-654321921 does the @DefaultAnnotation(NonNull.class) still work? This is an automated message from the Apache Git

[GitHub] [beam] piotr-szuberski commented on pull request #12145: [BEAM-10136] [BEAM-10135] Add JdbcIO for cross-language with python wrapper

2020-07-06 Thread GitBox
piotr-szuberski commented on pull request #12145: URL: https://github.com/apache/beam/pull/12145#issuecomment-654333554 @TheNeuralBit For some reason "Flink CrossLanguageValidatesRunner Tests" timeouts on xlang_jdbc test and on the other hand the same test passes on Python precommit

[GitHub] [beam] piotr-szuberski edited a comment on pull request #12145: [BEAM-10136] [BEAM-10135] Add JdbcIO for cross-language with python wrapper

2020-07-06 Thread GitBox
piotr-szuberski edited a comment on pull request #12145: URL: https://github.com/apache/beam/pull/12145#issuecomment-654333554 @TheNeuralBit For some reason "Flink CrossLanguageValidatesRunner Tests" timeouts on xlang_jdbc test and on the other hand the same test passes on Python

[GitHub] [beam] piotr-szuberski edited a comment on pull request #12145: [BEAM-10136] [BEAM-10135] Add JdbcIO for cross-language with python wrapper

2020-07-06 Thread GitBox
piotr-szuberski edited a comment on pull request #12145: URL: https://github.com/apache/beam/pull/12145#issuecomment-654333554 @TheNeuralBit For some reason "Flink CrossLanguageValidatesRunner Tests" timeouts on xlang_jdbc test and on the other hand the same test passes on Python

[GitHub] [beam] y1chi commented on pull request #12142: [BEAM-10004] Fixes occasional ZeroDivisionError when bundle size weights become 0 for MongoDB IO

2020-07-06 Thread GitBox
y1chi commented on pull request #12142: URL: https://github.com/apache/beam/pull/12142#issuecomment-654335421 Run Python2_PVR_Flink PreCommit This is an automated message from the Apache Git Service. To respond to the

[GitHub] [beam] davidyan74 commented on pull request #12143: [BEAM-10291] Adding full thread dump upon lull detection for Dataflow…

2020-07-06 Thread GitBox
davidyan74 commented on pull request #12143: URL: https://github.com/apache/beam/pull/12143#issuecomment-654341406 Latest test failure was on org.apache.beam.sdk.io.jdbc.JdbcIOTest.testCustomFluentBackOffConfiguration and again seems unrelated with this PR: ```

[GitHub] [beam] purbanow edited a comment on pull request #12117: [BEAM-10343] Add dispositions for SnowflakeIO.write

2020-07-06 Thread GitBox
purbanow edited a comment on pull request #12117: URL: https://github.com/apache/beam/pull/12117#issuecomment-654253476 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [beam] ibzib commented on a change in pull request #12169: [BEAM-9953[SQL][ZetaSQL] Support Pure SQL user-defined table-valued function.

2020-07-06 Thread GitBox
ibzib commented on a change in pull request #12169: URL: https://github.com/apache/beam/pull/12169#discussion_r450526186 ## File path: sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLPlannerImpl.java ## @@ -171,6 +178,18 @@

[GitHub] [beam] ibzib commented on a change in pull request #12169: [BEAM-9953[SQL][ZetaSQL] Support Pure SQL user-defined table-valued function.

2020-07-06 Thread GitBox
ibzib commented on a change in pull request #12169: URL: https://github.com/apache/beam/pull/12169#discussion_r450526675 ## File path: sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLPlannerImpl.java ## @@ -211,7 +212,7 @@

[GitHub] [beam] ibzib commented on a change in pull request #12169: [BEAM-9953[SQL][ZetaSQL] Support Pure SQL user-defined table-valued function.

2020-07-06 Thread GitBox
ibzib commented on a change in pull request #12169: URL: https://github.com/apache/beam/pull/12169#discussion_r450527335 ## File path: sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ConversionContext.java ## @@ -33,7

[GitHub] [beam] amaliujia commented on a change in pull request #12169: [BEAM-9953[SQL][ZetaSQL] Support Pure SQL user-defined table-valued function.

2020-07-06 Thread GitBox
amaliujia commented on a change in pull request #12169: URL: https://github.com/apache/beam/pull/12169#discussion_r450529634 ## File path: sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ConversionContext.java ## @@ -33,7

[GitHub] [beam] pabloem merged pull request #12083: [BEAM-10317] Java - Update BigQueryIO to tag BigQuery Jobs with the Dataflow Job ID

2020-07-06 Thread GitBox
pabloem merged pull request #12083: URL: https://github.com/apache/beam/pull/12083 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [beam] pabloem commented on pull request #12083: [BEAM-10317] Java - Update BigQueryIO to tag BigQuery Jobs with the Dataflow Job ID

2020-07-06 Thread GitBox
pabloem commented on pull request #12083: URL: https://github.com/apache/beam/pull/12083#issuecomment-654513954 merged. Thanks Alex! This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [beam] kennknowles commented on pull request #12162: [BEAM-10402] Enable checker framework, disabled everywhere since every module has errors. Eliminate some errors (but no complete modules)

2020-07-06 Thread GitBox
kennknowles commented on pull request #12162: URL: https://github.com/apache/beam/pull/12162#issuecomment-654515881 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [beam] angoenka commented on pull request #12042: [BEAM-9962] Jenkins Plugin

2020-07-06 Thread GitBox
angoenka commented on pull request #12042: URL: https://github.com/apache/beam/pull/12042#issuecomment-654516184 Thanks @TobKed for the input. 1. Yes, I agree we should have a document on beam website for the instructions. 2. I plan to move the plugin to Gradle later if possible

[GitHub] [beam] kennknowles commented on pull request #12162: [BEAM-10402] Enable checker framework, disabled everywhere since every module has errors. Eliminate some errors (but no complete modules)

2020-07-06 Thread GitBox
kennknowles commented on pull request #12162: URL: https://github.com/apache/beam/pull/12162#issuecomment-654523685 run python precommit This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] lukecwik commented on a change in pull request #12089: [BEAM-10283] Add new overloads of withKeyRanges and withRowFilter met…

2020-07-06 Thread GitBox
lukecwik commented on a change in pull request #12089: URL: https://github.com/apache/beam/pull/12089#discussion_r450541065 ## File path: sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java ## @@ -410,13 +417,22 @@ public Read

[GitHub] [beam] lukecwik commented on a change in pull request #12089: [BEAM-10283] Add new overloads of withKeyRanges and withRowFilter met…

2020-07-06 Thread GitBox
lukecwik commented on a change in pull request #12089: URL: https://github.com/apache/beam/pull/12089#discussion_r450541889 ## File path: sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableReadOptions.java ## @@ -0,0 +1,73 @@ +package

[GitHub] [beam] rezarokni commented on a change in pull request #12175: [BEAM-10201] Reduce scope of ParseResult AutoValue properties.

2020-07-06 Thread GitBox
rezarokni commented on a change in pull request #12175: URL: https://github.com/apache/beam/pull/12175#discussion_r450542229 ## File path: sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/JsonToRow.java ## @@ -334,27 +335,27 @@ private ObjectMapper objectMapper() {

[GitHub] [beam] lukecwik commented on a change in pull request #12089: [BEAM-10283] Add new overloads of withKeyRanges and withRowFilter met…

2020-07-06 Thread GitBox
lukecwik commented on a change in pull request #12089: URL: https://github.com/apache/beam/pull/12089#discussion_r450541973 ## File path: sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIOTest.java ## @@ -1100,7 +1141,7 @@ public

[GitHub] [beam] aaltay commented on pull request #12178: Remove the non-actional warning message

2020-07-06 Thread GitBox
aaltay commented on pull request #12178: URL: https://github.com/apache/beam/pull/12178#issuecomment-654525781 Run Python2_PVR_Flink PreCommit This is an automated message from the Apache Git Service. To respond to the

[GitHub] [beam] TheNeuralBit commented on a change in pull request #12090: [BEAM-10336,BEAM-10337] Add SchemaIO abstraction and implement for PubSub

2020-07-06 Thread GitBox
TheNeuralBit commented on a change in pull request #12090: URL: https://github.com/apache/beam/pull/12090#discussion_r450535581 ## File path: sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubSchemaCapableIOProvider.java ## @@ -0,0

[GitHub] [beam] aaltay commented on pull request #12178: Remove the non-actional warning message

2020-07-06 Thread GitBox
aaltay commented on pull request #12178: URL: https://github.com/apache/beam/pull/12178#issuecomment-654525757 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [beam] aaltay commented on pull request #12179: [BEAM-5414] - Update grpcio-tools to its latest version.

2020-07-06 Thread GitBox
aaltay commented on pull request #12179: URL: https://github.com/apache/beam/pull/12179#issuecomment-654526140 Run Python 3.7 PostCommit This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] aaltay commented on pull request #12179: [BEAM-5414] - Update grpcio-tools to its latest version.

2020-07-06 Thread GitBox
aaltay commented on pull request #12179: URL: https://github.com/apache/beam/pull/12179#issuecomment-654526206 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [beam] pabloem commented on pull request #11765: [BEAM-9322] Remove passthrough_pcollection_output_ids and force_generated_pcollection_output_ids flags

2020-07-06 Thread GitBox
pabloem commented on pull request #11765: URL: https://github.com/apache/beam/pull/11765#issuecomment-654409876 thanks. Sam has confirmed the import works. This is an automated message from the Apache Git Service. To respond

[GitHub] [beam] rohdesamuel commented on pull request #11765: [BEAM-9322] Remove passthrough_pcollection_output_ids and force_generated_pcollection_output_ids flags

2020-07-06 Thread GitBox
rohdesamuel commented on pull request #11765: URL: https://github.com/apache/beam/pull/11765#issuecomment-654409890 > just to confirm: have you verified that this imports (and tests) well into google repository? > @rohdesamuel Yes, the last test successfully imported (6/24).

[GitHub] [beam] pabloem merged pull request #11765: [BEAM-9322] Remove passthrough_pcollection_output_ids and force_generated_pcollection_output_ids flags

2020-07-06 Thread GitBox
pabloem merged pull request #11765: URL: https://github.com/apache/beam/pull/11765 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [beam] robertwb commented on a change in pull request #12177: [BEAM-10317] Add requests to python deps

2020-07-06 Thread GitBox
robertwb commented on a change in pull request #12177: URL: https://github.com/apache/beam/pull/12177#discussion_r450424896 ## File path: sdks/python/setup.py ## @@ -168,6 +168,7 @@ def get_version(): # [BEAM-5628] Beam VCF IO is not supported in Python 3.

[GitHub] [beam] tvalentyn commented on pull request #11788: [BEAM-9785] Add Python 3.8 postcommit tests

2020-07-06 Thread GitBox
tvalentyn commented on pull request #11788: URL: https://github.com/apache/beam/pull/11788#issuecomment-654419277 @epicfaace could you please address the merge conflict in .github/PULL_REQUEST_TEMPLATE.md? This is an

[GitHub] [beam] ajamato commented on a change in pull request #12070: Add some metrics for BigQueryStorageStreamReader

2020-07-06 Thread GitBox
ajamato commented on a change in pull request #12070: URL: https://github.com/apache/beam/pull/12070#discussion_r450433163 ## File path: sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryStorageStreamSource.java ## @@ -219,7 +232,15

[GitHub] [beam] robinyqiu commented on a change in pull request #12163: [BEAM-9712] Refactor ZetaSQL planner code and support setting default timezone

2020-07-06 Thread GitBox
robinyqiu commented on a change in pull request #12163: URL: https://github.com/apache/beam/pull/12163#discussion_r450447457 ## File path: sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/CalciteQueryPlanner.java ## @@ -75,7 +75,9 @@ private

[GitHub] [beam] tvalentyn commented on pull request #12179: [BEAM-5414] - Update grpcio-tools to its latest version.

2020-07-06 Thread GitBox
tvalentyn commented on pull request #12179: URL: https://github.com/apache/beam/pull/12179#issuecomment-654436338 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [beam] tvalentyn commented on pull request #12180: Adds Python 3.8 ValidatesRunner tests for Dataflow.

2020-07-06 Thread GitBox
tvalentyn commented on pull request #12180: URL: https://github.com/apache/beam/pull/12180#issuecomment-654438165 Run Python Dataflow ValidatesRunner This is an automated message from the Apache Git Service. To respond to

[GitHub] [beam] tvalentyn commented on pull request #12180: Adds Python 3.8 ValidatesRunner tests for Dataflow.

2020-07-06 Thread GitBox
tvalentyn commented on pull request #12180: URL: https://github.com/apache/beam/pull/12180#issuecomment-654438046 Run Dataflow Python ValidatesRunner This is an automated message from the Apache Git Service. To respond to

[GitHub] [beam] ibzib commented on a change in pull request #12169: [BEAM-9953[SQL][ZetaSQL] Support Pure SQL user-defined table-valued function.

2020-07-06 Thread GitBox
ibzib commented on a change in pull request #12169: URL: https://github.com/apache/beam/pull/12169#discussion_r450436253 ## File path: sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLDialectSpecTest.java ## @@ -2916,6 +2917,57

[GitHub] [beam] kennknowles commented on a change in pull request #12155: [BEAM-10400] Fix race condition in DirectRunner watermark update

2020-07-06 Thread GitBox
kennknowles commented on a change in pull request #12155: URL: https://github.com/apache/beam/pull/12155#discussion_r450454617 ## File path: runners/direct-java/src/main/java/org/apache/beam/runners/direct/WatermarkManager.java ## @@ -1110,17 +1109,7 @@ public void

[GitHub] [beam] kennknowles commented on pull request #12155: [BEAM-10400] Fix race condition in DirectRunner watermark update

2020-07-06 Thread GitBox
kennknowles commented on pull request #12155: URL: https://github.com/apache/beam/pull/12155#issuecomment-654451164 (but I don't know if apply N updates logic is necessary - it may be overkill, but it may be from static analysis making sure that the work done while holding a lock is

[GitHub] [beam] kennknowles commented on a change in pull request #11243: [BEAM-9136]Add licenses for dependencies for Java

2020-07-06 Thread GitBox
kennknowles commented on a change in pull request #11243: URL: https://github.com/apache/beam/pull/11243#discussion_r450468077 ## File path: sdks/java/container/build.gradle ## @@ -68,6 +73,28 @@ golang { } } +// this is a workaround to call generateLicenseReport task

[GitHub] [beam] robertwb commented on a change in pull request #12023: [BEAM-10135] Add Python wrapper for Jdbc Write external transform

2020-07-06 Thread GitBox
robertwb commented on a change in pull request #12023: URL: https://github.com/apache/beam/pull/12023#discussion_r450467294 ## File path: sdks/python/apache_beam/io/external/jdbc.py ## @@ -0,0 +1,116 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more

[GitHub] [beam] robertwb opened a new pull request #12182: [BEAM-4605] Remove portable runner GBK hack.

2020-07-06 Thread GitBox
robertwb opened a new pull request #12182: URL: https://github.com/apache/beam/pull/12182 This is no longer needed now that https://github.com/apache/beam/pull/11503 was submitted. Thank you for your contribution! Follow this checklist to help us

[GitHub] [beam] robertwb commented on pull request #12182: [BEAM-4605] Remove portable runner GBK hack.

2020-07-06 Thread GitBox
robertwb commented on pull request #12182: URL: https://github.com/apache/beam/pull/12182#issuecomment-654457842 R: @rohdesamuel Someone ran into this recently and was confused by it. Thanks to your cleanup, we can now get rid of this hack altogether.

[GitHub] [beam] lostluck commented on pull request #11979: [BEAM-9679] Add Partition task to Core Transform katas

2020-07-06 Thread GitBox
lostluck commented on pull request #11979: URL: https://github.com/apache/beam/pull/11979#issuecomment-654457555 I can't speak to the Kata generated code, but the go.mod and go.sum files are strongly recommended to be submitted as is to ensure full transitive dependency of the right and

[GitHub] [beam] lostluck merged pull request #12141: [BEAM-9615] Add Row coder functions.

2020-07-06 Thread GitBox
lostluck merged pull request #12141: URL: https://github.com/apache/beam/pull/12141 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

  1   2   >