[GitHub] [beam] TheNeuralBit commented on pull request #12399: [BEAM-10559] Add apache_beam.examples.sql_taxi

2020-07-30 Thread GitBox
TheNeuralBit commented on pull request #12399: URL: https://github.com/apache/beam/pull/12399#issuecomment-665898624 Run PythonDocker PreCommit This is an automated message from the Apache Git Service. To respond to the

[GitHub] [beam] amaliujia merged pull request #12400: [BEAM-10595] Remove rules that not work well with Java UDF

2020-07-30 Thread GitBox
amaliujia merged pull request #12400: URL: https://github.com/apache/beam/pull/12400 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [beam] robertwb merged pull request #11974: [BEAM-9547] Add more methods to deferred dataframes.

2020-07-30 Thread GitBox
robertwb merged pull request #11974: URL: https://github.com/apache/beam/pull/11974 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [beam] chamikaramj commented on a change in pull request #12399: [BEAM-10559] Add apache_beam.examples.sql_taxi

2020-07-30 Thread GitBox
chamikaramj commented on a change in pull request #12399: URL: https://github.com/apache/beam/pull/12399#discussion_r462658530 ## File path: sdks/python/apache_beam/examples/sql_taxi.py ## @@ -0,0 +1,97 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or

[GitHub] [beam] boyuanzz opened a new pull request #12419: [WIP] [BEAM-10303] Handle split when truncate observes windows.

2020-07-30 Thread GitBox
boyuanzz opened a new pull request #12419: URL: https://github.com/apache/beam/pull/12419 **Please** add a meaningful description for your change here Thank you for your contribution! Follow this checklist to help us incorporate your contribution

[GitHub] [beam] jiyongjung0 commented on pull request #12404: [BEAM-10598] Bumps google cloud bigquery to 1.26.1

2020-07-30 Thread GitBox
jiyongjung0 commented on pull request #12404: URL: https://github.com/apache/beam/pull/12404#issuecomment-666007314 R: @aaltay Could you please take a look? This is an automated message from the Apache Git Service.

[GitHub] [beam] rohdesamuel opened a new pull request #12414: Fix BCJ to stop caching when the cache signature has changed.

2020-07-30 Thread GitBox
rohdesamuel opened a new pull request #12414: URL: https://github.com/apache/beam/pull/12414 Change-Id: I52a4f36b09e6f6899d7c59756c9702ba983e083b PR 5/7 The logic to stop reading from cache was incorrect and did not properly stop if the cache changed.

[GitHub] [beam] pabloem commented on pull request #12418: Fix lint issue with ml transforms

2020-07-30 Thread GitBox
pabloem commented on pull request #12418: URL: https://github.com/apache/beam/pull/12418#issuecomment-665868369 oddly I got this to pass on my machine. This is an automated message from the Apache Git Service. To respond to

[GitHub] [beam] tvalentyn commented on a change in pull request #12212: Prepare a website update for 2.23.0

2020-07-30 Thread GitBox
tvalentyn commented on a change in pull request #12212: URL: https://github.com/apache/beam/pull/12212#discussion_r462566471 ## File path: website/www/site/content/en/get-started/downloads.md ## @@ -87,6 +87,13 @@ versions denoted `0.x.y`. ## Releases +### 2.23.0

[GitHub] [beam] iindyk commented on pull request #12420: Extending ApproximateQuantiles functionality to deal with non-uniform weights.

2020-07-30 Thread GitBox
iindyk commented on pull request #12420: URL: https://github.com/apache/beam/pull/12420#issuecomment-666062605 R: @chrisgorgo This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [beam] rohdesamuel commented on pull request #12410: Gracefully stop the TestStream RPC if the job stops.

2020-07-30 Thread GitBox
rohdesamuel commented on pull request #12410: URL: https://github.com/apache/beam/pull/12410#issuecomment-665827616 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [beam] kennknowles commented on pull request #12416: Update google-api-services versions.

2020-07-30 Thread GitBox
kennknowles commented on pull request #12416: URL: https://github.com/apache/beam/pull/12416#issuecomment-665951261 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [beam] saavannanavati commented on pull request #12009: [BEAM-10258] Support type hint annotations on PTransform's expand()

2020-07-30 Thread GitBox
saavannanavati commented on pull request #12009: URL: https://github.com/apache/beam/pull/12009#issuecomment-666071126 run python precommit This is an automated message from the Apache Git Service. To respond to the

[GitHub] [beam] pabloem opened a new pull request #12418: Fix lint issue with ml transforms

2020-07-30 Thread GitBox
pabloem opened a new pull request #12418: URL: https://github.com/apache/beam/pull/12418 **Please** add a meaningful description for your change here Thank you for your contribution! Follow this checklist to help us incorporate your contribution

[GitHub] [beam] robinyqiu commented on a change in pull request #12348: [BEAM-10240] Support ZetaSQL DATETIME functions in BeamSQL

2020-07-30 Thread GitBox
robinyqiu commented on a change in pull request #12348: URL: https://github.com/apache/beam/pull/12348#discussion_r462568818 ## File path: sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SupportedZetaSqlBuiltinFunctions.java ## @@

[GitHub] [beam] KevinGG commented on pull request #12372: [BEAM-10545] KernelModel and jest tests

2020-07-30 Thread GitBox
KevinGG commented on pull request #12372: URL: https://github.com/apache/beam/pull/12372#issuecomment-665888592 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [beam] tvalentyn merged pull request #12212: Prepare a website update for 2.23.0

2020-07-30 Thread GitBox
tvalentyn merged pull request #12212: URL: https://github.com/apache/beam/pull/12212 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [beam] tysonjh opened a new pull request #12416: Update google-api-services versions.

2020-07-30 Thread GitBox
tysonjh opened a new pull request #12416: URL: https://github.com/apache/beam/pull/12416 Update google-api-services* versions and google-client version to latest. Thank you for your contribution! Follow this checklist to help us incorporate your

[GitHub] [beam] lostluck merged pull request #12393: Support creation of empty PCollection in beam CreateList

2020-07-30 Thread GitBox
lostluck merged pull request #12393: URL: https://github.com/apache/beam/pull/12393 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [beam] rohdesamuel opened a new pull request #12411: Add ElementLimiters which allows the cache to prematurely based on read elements.

2020-07-30 Thread GitBox
rohdesamuel opened a new pull request #12411: URL: https://github.com/apache/beam/pull/12411 Change-Id: I9f7030999f2ce38073ca72512c71794dbdbc301e PR 2/7 This will allow for the cache to stop reading based on elements. Currently, the cache can only stop based on global

[GitHub] [beam] lostluck commented on pull request #12393: Support creation of empty PCollection in beam CreateList

2020-07-30 Thread GitBox
lostluck commented on pull request #12393: URL: https://github.com/apache/beam/pull/12393#issuecomment-665916062 Thank you for the contribution! This is an automated message from the Apache Git Service. To respond to the

[GitHub] [beam] henryken commented on pull request #12362: Convert katas to use with syntax rather than explicit run call.

2020-07-30 Thread GitBox
henryken commented on pull request #12362: URL: https://github.com/apache/beam/pull/12362#issuecomment-666093915 Thanks for the change @robertwb! LGTM. After the merge, I'll help to update the course on Stepik. This is an

[GitHub] [beam] iindyk opened a new pull request #12420: Extending ApproximateQuantiles functionality to deal with non-uniform weights.

2020-07-30 Thread GitBox
iindyk opened a new pull request #12420: URL: https://github.com/apache/beam/pull/12420 Extending functionality of ApproximateQuantiles PTransform to calculate quantiles of a data stream with custom weights. In case of weighted quantiles, input PCollection is expected to contain tuples of

[GitHub] [beam] rohdesamuel opened a new pull request #12412: Add ElementLimiters to all Cache Managers.

2020-07-30 Thread GitBox
rohdesamuel opened a new pull request #12412: URL: https://github.com/apache/beam/pull/12412 Change-Id: I093e1dc1f99c31b1e0bb868fc3bcfc507bd24c8d PR 3/7 This builds on https://github.com/apache/beam/pull/12411 to utilize the ElementLimiters to stop reading from cache. This

[GitHub] [beam] rohdesamuel opened a new pull request #12410: Gracefully stop the TestStream RPC if the job stops.

2020-07-30 Thread GitBox
rohdesamuel opened a new pull request #12410: URL: https://github.com/apache/beam/pull/12410 Change-Id: I1b25694c2f8f52d5335da889610ae1871deb0917 PR 1/7 When the DirectRunner gets canceled, there is no signal to pass to the evaluation context to gracefully shut it down. The

[GitHub] [beam] piotr-szuberski commented on a change in pull request #12406: [BEAM-10591] Fix flaky xlang jdbc test

2020-07-30 Thread GitBox
piotr-szuberski commented on a change in pull request #12406: URL: https://github.com/apache/beam/pull/12406#discussion_r462771399 ## File path: sdks/python/apache_beam/io/external/xlang_jdbcio_it_test.py ## @@ -91,7 +90,12 @@ def setUp(self): self.host, self.port,

[GitHub] [beam] ibzib commented on a change in pull request #12212: Prepare a website update for 2.23.0

2020-07-30 Thread GitBox
ibzib commented on a change in pull request #12212: URL: https://github.com/apache/beam/pull/12212#discussion_r462571597 ## File path: website/www/site/content/en/get-started/downloads.md ## @@ -87,6 +87,13 @@ versions denoted `0.x.y`. ## Releases +### 2.23.0 (2020-07-16)

[GitHub] [beam] chamikaramj commented on a change in pull request #12406: [BEAM-10591] Fix flaky xlang jdbc test

2020-07-30 Thread GitBox
chamikaramj commented on a change in pull request #12406: URL: https://github.com/apache/beam/pull/12406#discussion_r462656463 ## File path: sdks/python/apache_beam/io/external/xlang_jdbcio_it_test.py ## @@ -91,7 +90,12 @@ def setUp(self): self.host, self.port,

[GitHub] [beam] prodonjs commented on a change in pull request #12372: [BEAM-10545] KernelModel and jest tests

2020-07-30 Thread GitBox
prodonjs commented on a change in pull request #12372: URL: https://github.com/apache/beam/pull/12372#discussion_r462515880 ## File path: sdks/python/apache_beam/runners/interactive/extensions/apache-beam-jupyterlab-sidepanel/src/kernel/KernelModel.ts ## @@ -0,0 +1,161 @@ +//

[GitHub] [beam] aaltay commented on pull request #12118: [BEAM-7705] Add BigQuery Java samples

2020-07-30 Thread GitBox
aaltay commented on pull request #12118: URL: https://github.com/apache/beam/pull/12118#issuecomment-665835545 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [beam] ihji commented on pull request #12106: [BEAM-9792] Fixing the IOException handling in InsertAll for BigQuery

2020-07-30 Thread GitBox
ihji commented on pull request #12106: URL: https://github.com/apache/beam/pull/12106#issuecomment-665922796 LGTM. Thanks. @chamikaramj Any additional comment? This is an automated message from the Apache Git

[GitHub] [beam] KevinGG commented on a change in pull request #12372: [BEAM-10545] KernelModel and jest tests

2020-07-30 Thread GitBox
KevinGG commented on a change in pull request #12372: URL: https://github.com/apache/beam/pull/12372#discussion_r462542550 ## File path: sdks/python/apache_beam/runners/interactive/extensions/apache-beam-jupyterlab-sidepanel/src/kernel/KernelModel.ts ## @@ -0,0 +1,161 @@ +//

[GitHub] [beam] kennknowles commented on pull request #12400: [BEAM-10595] Remove rules that not work well with Java UDF

2020-07-30 Thread GitBox
kennknowles commented on pull request #12400: URL: https://github.com/apache/beam/pull/12400#issuecomment-666032661 It is a good point that `CalcMergeRule` and `FilterToCalcRule` should be enough so this rule is not needed. You might ask on dev@calcite why they both exist. But how

[GitHub] [beam] aaltay commented on pull request #12409: [BEAM-7390] Add max code snippets

2020-07-30 Thread GitBox
aaltay commented on pull request #12409: URL: https://github.com/apache/beam/pull/12409#issuecomment-665836821 LGTM. Ping me when ready. This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] aaltay merged pull request #12394: [BEAM-7390] Add latest code snippets

2020-07-30 Thread GitBox
aaltay merged pull request #12394: URL: https://github.com/apache/beam/pull/12394 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [beam] ihji commented on a change in pull request #12106: [BEAM-9792] Fixing the IOException handling in InsertAll for BigQuery

2020-07-30 Thread GitBox
ihji commented on a change in pull request #12106: URL: https://github.com/apache/beam/pull/12106#discussion_r462534513 ## File path: sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImplTest.java ## @@ -733,26 +734,29 @@

[GitHub] [beam] kennknowles merged pull request #11924: [BEAM-8543] Dataflow streaming timers are not strictly time ordered when set earlier mid-bundle

2020-07-30 Thread GitBox
kennknowles merged pull request #11924: URL: https://github.com/apache/beam/pull/11924 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [beam] rohdesamuel opened a new pull request #12413: Add max count to utils.to_element_list

2020-07-30 Thread GitBox
rohdesamuel opened a new pull request #12413: URL: https://github.com/apache/beam/pull/12413 Change-Id: I9a2fbf1532b3d22a612e7a09f4f1fb2b9635c40b PR 4/7 Reading elements from cache can yield TestStreamFileRecords which can have many elements that exceeds the requested element

[GitHub] [beam] tvalentyn merged pull request #12213: Add a blogpost for Apache Beam 2.23.0.

2020-07-30 Thread GitBox
tvalentyn merged pull request #12213: URL: https://github.com/apache/beam/pull/12213 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [beam] tvalentyn commented on pull request #12212: Prepare a website update for 2.23.0

2020-07-30 Thread GitBox
tvalentyn commented on pull request #12212: URL: https://github.com/apache/beam/pull/12212#issuecomment-665910356 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [beam] davidyan74 commented on a change in pull request #12411: [BEAM-10603] Add ElementLimiters which allows the cache to prematurely based on read elements.

2020-07-30 Thread GitBox
davidyan74 commented on a change in pull request #12411: URL: https://github.com/apache/beam/pull/12411#discussion_r462561306 ## File path: sdks/python/apache_beam/runners/interactive/options/capture_limiters.py ## @@ -71,3 +87,62 @@ def _trigger(self): def

[GitHub] [beam] tysonjh commented on pull request #12416: Update google-api-services versions.

2020-07-30 Thread GitBox
tysonjh commented on pull request #12416: URL: https://github.com/apache/beam/pull/12416#issuecomment-665843586 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [beam] damondouglas commented on pull request #12402: [BEAM-9679] Add Composite transforms to Go SDK katas

2020-07-30 Thread GitBox
damondouglas commented on pull request #12402: URL: https://github.com/apache/beam/pull/12402#issuecomment-665916576 @lostluck The `*-remote.yaml` files are now committed from updating the [Stepik course](https://stepik.org/course/70387) and this PR is ready to merge.

[GitHub] [beam] simonepri commented on pull request #12421: Fix go count on an empty pcollection

2020-07-30 Thread GitBox
simonepri commented on pull request #12421: URL: https://github.com/apache/beam/pull/12421#issuecomment-666285522 R: @youngoli, @lostluck This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] piotr-szuberski opened a new pull request #12422: [BEAM-601] Run KinesisIOIT with localstack

2020-07-30 Thread GitBox
piotr-szuberski opened a new pull request #12422: URL: https://github.com/apache/beam/pull/12422 I've done this by the way of writing cross-language integration tests. Thank you for your contribution! Follow this checklist to help us incorporate your

[GitHub] [beam] piotr-szuberski commented on pull request #12422: [BEAM-601] Run KinesisIOIT with localstack

2020-07-30 Thread GitBox
piotr-szuberski commented on pull request #12422: URL: https://github.com/apache/beam/pull/12422#issuecomment-666291814 R: @TheNeuralBit This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] rworley-monster removed a comment on pull request #12389: [BEAM-10587] Support Maps in BigQuery

2020-07-30 Thread GitBox
rworley-monster removed a comment on pull request #12389: URL: https://github.com/apache/beam/pull/12389#issuecomment-664974133 retest this please This is an automated message from the Apache Git Service. To respond to the

[GitHub] [beam] damgad commented on pull request #12407: Enable all Jenkins jobs triggering for committers

2020-07-30 Thread GitBox
damgad commented on pull request #12407: URL: https://github.com/apache/beam/pull/12407#issuecomment-666297321 run seed job This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [beam] damgad commented on pull request #12407: Enable all Jenkins jobs triggering for committers

2020-07-30 Thread GitBox
damgad commented on pull request #12407: URL: https://github.com/apache/beam/pull/12407#issuecomment-666303215 Verified that it works as expected: ``` INFO:root:Generating Committers.groovy INFO:root:70 committers' GitHub usernames fetched correctly

[GitHub] [beam] kkucharc commented on pull request #12151: [BEAM-9896] Add streaming for SnowflakeIO.Write to Java SDK

2020-07-30 Thread GitBox
kkucharc commented on pull request #12151: URL: https://github.com/apache/beam/pull/12151#issuecomment-666319639 Friendly pinging after 3 days @pabloem :) This is an automated message from the Apache Git Service. To respond

[GitHub] [beam] simonepri opened a new pull request #12421: Fix go count on an empty pcollection

2020-07-30 Thread GitBox
simonepri opened a new pull request #12421: URL: https://github.com/apache/beam/pull/12421 The documentation says > CountElms counts the number of elements in a collection. > It expects a PCollection as input and returns a PCollection of **one element** containing the count.

[GitHub] [beam] TobKed commented on a change in pull request #12405: [BEAM-10599] Add documentation about CI on GitHub Action

2020-07-30 Thread GitBox
TobKed commented on a change in pull request #12405: URL: https://github.com/apache/beam/pull/12405#discussion_r462946005 ## File path: CI.md ## @@ -0,0 +1,104 @@ + + +# Apache Beam + +## CI Environment + +Continuous Integration is important component of making Apache Beam

[GitHub] [beam] damgad commented on pull request #12407: Enable all Jenkins jobs triggering for committers

2020-07-30 Thread GitBox
damgad commented on pull request #12407: URL: https://github.com/apache/beam/pull/12407#issuecomment-666304130 R: @aaltay @udim This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [beam] piotr-szuberski commented on pull request #12422: [BEAM-601] Run KinesisIOIT with localstack

2020-07-30 Thread GitBox
piotr-szuberski commented on pull request #12422: URL: https://github.com/apache/beam/pull/12422#issuecomment-666312635 Run Java PostCommit This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] TobKed commented on a change in pull request #12405: [BEAM-10599] Add documentation about CI on GitHub Action

2020-07-30 Thread GitBox
TobKed commented on a change in pull request #12405: URL: https://github.com/apache/beam/pull/12405#discussion_r462952380 ## File path: CI.md ## @@ -0,0 +1,104 @@ + + +# Apache Beam + +## CI Environment + +Continuous Integration is important component of making Apache Beam

[GitHub] [beam] piotr-szuberski edited a comment on pull request #12297: [BEAM-10137] Add KinesisIO.Read external transform

2020-07-30 Thread GitBox
piotr-szuberski edited a comment on pull request #12297: URL: https://github.com/apache/beam/pull/12297#issuecomment-666279110 @TheNeuralBit Sorry for lots of messages but every day brings something new. I've managed to modify existing KinesisIOIT to work with localstack (I'll push

[GitHub] [beam] piotr-szuberski commented on pull request #12297: [BEAM-10137] Add KinesisIO.Read external transform

2020-07-30 Thread GitBox
piotr-szuberski commented on pull request #12297: URL: https://github.com/apache/beam/pull/12297#issuecomment-666279110 @TheNeuralBit Sorry for lots of messages but every day brings something new. I've managed to modify existing KinesisIOIT to work with localstack (I'll push it in a

[GitHub] [beam] piotr-szuberski edited a comment on pull request #12297: [BEAM-10137] Add KinesisIO.Read external transform

2020-07-30 Thread GitBox
piotr-szuberski edited a comment on pull request #12297: URL: https://github.com/apache/beam/pull/12297#issuecomment-666279110 @TheNeuralBit Sorry for lots of messages but every day brings something new. I've managed to modify existing KinesisIOIT to work with localstack (I'll push

[GitHub] [beam] TobKed commented on pull request #12405: [BEAM-10599] Add documentation about CI on GitHub Action

2020-07-30 Thread GitBox
TobKed commented on pull request #12405: URL: https://github.com/apache/beam/pull/12405#issuecomment-666329030 @aaltay thanks for review. I fixed typo and replied to your comments. This is an automated message from the

[GitHub] [beam] TobKed commented on a change in pull request #12405: [BEAM-10599] Add documentation about CI on GitHub Action

2020-07-30 Thread GitBox
TobKed commented on a change in pull request #12405: URL: https://github.com/apache/beam/pull/12405#discussion_r462953052 ## File path: CI.md ## @@ -0,0 +1,104 @@ + + +# Apache Beam + +## CI Environment + +Continuous Integration is important component of making Apache Beam

[GitHub] [beam] DariuszAniszewski commented on pull request #12099: [BEAM-10331] Add SnowflakeIO to list of built-in IO transforms.

2020-07-30 Thread GitBox
DariuszAniszewski commented on pull request #12099: URL: https://github.com/apache/beam/pull/12099#issuecomment-666260179 Hey @TheNeuralBit @pabloem The `2.23.0` version is released, so this PR can be merged ;) This

[GitHub] [beam] rworley-monster commented on a change in pull request #12389: [BEAM-10587] Support Maps in BigQuery

2020-07-30 Thread GitBox
rworley-monster commented on a change in pull request #12389: URL: https://github.com/apache/beam/pull/12389#discussion_r462906573 ## File path: sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryUtils.java ## @@ -442,6 +450,17 @@

[GitHub] [beam] damgad commented on pull request #12407: Enable all Jenkins jobs triggering for committers

2020-07-30 Thread GitBox
damgad commented on pull request #12407: URL: https://github.com/apache/beam/pull/12407#issuecomment-666299886 run seed job This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [beam] TobKed commented on a change in pull request #12150: [BEAM-10398] Use GitHub Actions in wheels release process for Python

2020-07-30 Thread GitBox
TobKed commented on a change in pull request #12150: URL: https://github.com/apache/beam/pull/12150#discussion_r462956031 ## File path: release/src/main/scripts/download_github_actions_artifacts.py ## @@ -170,7 +170,7 @@ def get_last_run_id( f"GCS location corresponding

[GitHub] [beam] TheNeuralBit merged pull request #12399: [BEAM-10559] Add apache_beam.examples.sql_taxi

2020-07-30 Thread GitBox
TheNeuralBit merged pull request #12399: URL: https://github.com/apache/beam/pull/12399 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [beam] tysonjh commented on pull request #12416: Update google-api-services versions.

2020-07-30 Thread GitBox
tysonjh commented on pull request #12416: URL: https://github.com/apache/beam/pull/12416#issuecomment-666812699 No, my mistake. It fails locally too, I was running it incorrectly. This is an automated message from the Apache

[GitHub] [beam] tysonjh commented on pull request #12416: Update google-api-services versions.

2020-07-30 Thread GitBox
tysonjh commented on pull request #12416: URL: https://github.com/apache/beam/pull/12416#issuecomment-666812322 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [beam] ibzib commented on a change in pull request #12385: [BEAM-10527] Migrate Flink and Spark tests to pytest.

2020-07-30 Thread GitBox
ibzib commented on a change in pull request #12385: URL: https://github.com/apache/beam/pull/12385#discussion_r463345888 ## File path: sdks/python/apache_beam/runners/portability/flink_runner_test.py ## @@ -53,361 +53,380 @@ from apache_beam.transforms import userstate from

[GitHub] [beam] pabloem commented on pull request #12203: [BEAM-6928] Make Python SDK custom Sink the default Sink for BigQuery

2020-07-30 Thread GitBox
pabloem commented on pull request #12203: URL: https://github.com/apache/beam/pull/12203#issuecomment-666851408 only PullLicenses is failing @chamikaramj PTAL This is an automated message from the Apache Git Service. To

[GitHub] [beam] lostluck merged pull request #12425: [BEAM-10610][BEAM-9982] Support Loopback mode with universal runners in the Go SDK.

2020-07-30 Thread GitBox
lostluck merged pull request #12425: URL: https://github.com/apache/beam/pull/12425 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [beam] lostluck closed pull request #12429: Clean ExtractOutput of mean transform

2020-07-30 Thread GitBox
lostluck closed pull request #12429: URL: https://github.com/apache/beam/pull/12429 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [beam] TheNeuralBit commented on a change in pull request #12422: [BEAM-601] Run KinesisIOIT with localstack

2020-07-30 Thread GitBox
TheNeuralBit commented on a change in pull request #12422: URL: https://github.com/apache/beam/pull/12422#discussion_r463325571 ## File path: sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/KinesisIOIT.java ## @@ -34,34 +35,65 @@ import

[GitHub] [beam] tysonjh commented on pull request #12416: Update google-api-services versions.

2020-07-30 Thread GitBox
tysonjh commented on pull request #12416: URL: https://github.com/apache/beam/pull/12416#issuecomment-666808017 More test failures. NPEs in what seems like unrelated code, but notably the pubsub one again which passes locally when I run it for some reason.

[GitHub] [beam] chamikaramj commented on a change in pull request #12149: [BEAM-9897] Add cross-language support to SnowflakeIO.Read

2020-07-30 Thread GitBox
chamikaramj commented on a change in pull request #12149: URL: https://github.com/apache/beam/pull/12149#discussion_r463341503 ## File path: sdks/python/apache_beam/io/external/snowflake.py ## @@ -0,0 +1,185 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one

[GitHub] [beam] ibzib commented on a change in pull request #12385: [BEAM-10527] Migrate Flink and Spark tests to pytest.

2020-07-30 Thread GitBox
ibzib commented on a change in pull request #12385: URL: https://github.com/apache/beam/pull/12385#discussion_r463346402 ## File path: sdks/python/apache_beam/runners/portability/flink_runner_test.py ## @@ -53,361 +53,380 @@ from apache_beam.transforms import userstate from

[GitHub] [beam] ibzib commented on a change in pull request #12385: [BEAM-10527] Migrate Flink and Spark tests to pytest.

2020-07-30 Thread GitBox
ibzib commented on a change in pull request #12385: URL: https://github.com/apache/beam/pull/12385#discussion_r463348975 ## File path: sdks/python/apache_beam/runners/portability/flink_runner_test.py ## @@ -53,361 +53,380 @@ from apache_beam.transforms import userstate from

[GitHub] [beam] ibzib commented on a change in pull request #12385: [BEAM-10527] Migrate Flink and Spark tests to pytest.

2020-07-30 Thread GitBox
ibzib commented on a change in pull request #12385: URL: https://github.com/apache/beam/pull/12385#discussion_r463348975 ## File path: sdks/python/apache_beam/runners/portability/flink_runner_test.py ## @@ -53,361 +53,380 @@ from apache_beam.transforms import userstate from

[GitHub] [beam] ibzib commented on pull request #12385: [BEAM-10527] Migrate Flink and Spark tests to pytest.

2020-07-30 Thread GitBox
ibzib commented on pull request #12385: URL: https://github.com/apache/beam/pull/12385#issuecomment-666850930 > I have already spent a long time trying to fix quotes, so I can't help but wondering: why do we need flinkCompatibilityMatrixPROCESS in the first place, when it is not being run

[GitHub] [beam] pabloem merged pull request #12082: Standardizing BigQuery job names in Beam Python and Java SDKs

2020-07-30 Thread GitBox
pabloem merged pull request #12082: URL: https://github.com/apache/beam/pull/12082 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [beam] lostluck commented on a change in pull request #12350: [BEAM-10289] Dynamic splitting implementation.

2020-07-30 Thread GitBox
lostluck commented on a change in pull request #12350: URL: https://github.com/apache/beam/pull/12350#discussion_r462445798 ## File path: sdks/go/pkg/beam/core/runtime/exec/datasource.go ## @@ -302,31 +308,76 @@ func (n *DataSource) Split(splits []int64, frac float64, bufSize

[GitHub] [beam] kennknowles merged pull request #12387: Upgrade spotbugs from 3.1.12 to 4.0.6, spotbugs gradle plugin from 2.0.0 to 3.0.0

2020-07-30 Thread GitBox
kennknowles merged pull request #12387: URL: https://github.com/apache/beam/pull/12387 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [beam] chamikaramj commented on pull request #12198: Widen ranges for GCP libraries

2020-07-30 Thread GitBox
chamikaramj commented on pull request #12198: URL: https://github.com/apache/beam/pull/12198#issuecomment-666825962 Thanks for clarifying. LGTM. This is an automated message from the Apache Git Service. To respond to the

[GitHub] [beam] pabloem commented on pull request #12203: [BEAM-6928] Make Python SDK custom Sink the default Sink for BigQuery

2020-07-30 Thread GitBox
pabloem commented on pull request #12203: URL: https://github.com/apache/beam/pull/12203#issuecomment-666858821 Run Python2_PVR_Flink PreCommit This is an automated message from the Apache Git Service. To respond to the

[GitHub] [beam] ZijieSong946 commented on a change in pull request #12348: [BEAM-10240] Support ZetaSQL DATETIME functions in BeamSQL

2020-07-30 Thread GitBox
ZijieSong946 commented on a change in pull request #12348: URL: https://github.com/apache/beam/pull/12348#discussion_r463375064 ## File path: sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSqlBeamTranslationUtils.java ## @@

[GitHub] [beam] George-Wu commented on a change in pull request #12331: [BEAM-10601] DICOM API Beam IO connector

2020-07-30 Thread GitBox
George-Wu commented on a change in pull request #12331: URL: https://github.com/apache/beam/pull/12331#discussion_r463382501 ## File path: sdks/python/apache_beam/io/gcp/dicomio.py ## @@ -114,55 +115,60 @@ class DicomSearch(PTransform): """A PTransform used for

[GitHub] [beam] boyuanzz opened a new pull request #12430: [WIP][BEAM-10303] Scale progress with respect to windows observation.

2020-07-30 Thread GitBox
boyuanzz opened a new pull request #12430: URL: https://github.com/apache/beam/pull/12430 **Please** add a meaningful description for your change here Thank you for your contribution! Follow this checklist to help us incorporate your contribution

[GitHub] [beam] simonepri commented on pull request #12429: Clean ExtractOutput of mean transform

2020-07-30 Thread GitBox
simonepri commented on pull request #12429: URL: https://github.com/apache/beam/pull/12429#issuecomment-666770785 R: @youngoli, @lostluck Feel free to close it without merging if you don't feel it is needed. This is

[GitHub] [beam] jiyongjung0 commented on pull request #12404: [BEAM-10598] Bumps google cloud bigquery to 1.26.1

2020-07-30 Thread GitBox
jiyongjung0 commented on pull request #12404: URL: https://github.com/apache/beam/pull/12404#issuecomment-666770943 Thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [beam] pabloem commented on pull request #12082: Standardizing BigQuery job names in Beam Python and Java SDKs

2020-07-30 Thread GitBox
pabloem commented on pull request #12082: URL: https://github.com/apache/beam/pull/12082#issuecomment-666774552 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [beam] pabloem commented on pull request #12203: [BEAM-6928] Make Python SDK custom Sink the default Sink for BigQuery

2020-07-30 Thread GitBox
pabloem commented on pull request #12203: URL: https://github.com/apache/beam/pull/12203#issuecomment-666830620 Run Python2_PVR_Flink PreCommit This is an automated message from the Apache Git Service. To respond to the

[GitHub] [beam] ibzib commented on pull request #12385: [BEAM-10527] Migrate Flink and Spark tests to pytest.

2020-07-30 Thread GitBox
ibzib commented on pull request #12385: URL: https://github.com/apache/beam/pull/12385#issuecomment-666846231 > I couldn't find the published test results in Jenkins. Do we have to add this separately? https://ci-beam.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Commit/6126/

[GitHub] [beam] ibzib commented on pull request #12385: [BEAM-10527] Migrate Flink and Spark tests to pytest.

2020-07-30 Thread GitBox
ibzib commented on pull request #12385: URL: https://github.com/apache/beam/pull/12385#issuecomment-666849394 I got `flinkCompatibilityMatrixPROCESS` to pass on my machine by escaping the arguments via `${1@Q}`. Apparently whatever shell Jenkins is using does not support this. I will have

[GitHub] [beam] George-Wu commented on a change in pull request #12331: [BEAM-10601] DICOM API Beam IO connector

2020-07-30 Thread GitBox
George-Wu commented on a change in pull request #12331: URL: https://github.com/apache/beam/pull/12331#discussion_r463382633 ## File path: sdks/python/apache_beam/io/gcp/dicomio.py ## @@ -372,52 +420,63 @@ def __init__(self, destination_dict, input_type, credential=None):

[GitHub] [beam] George-Wu commented on a change in pull request #12331: [BEAM-10601] DICOM API Beam IO connector

2020-07-30 Thread GitBox
George-Wu commented on a change in pull request #12331: URL: https://github.com/apache/beam/pull/12331#discussion_r463382681 ## File path: sdks/python/apache_beam/io/gcp/dicomio.py ## @@ -426,6 +485,44 @@ def process(self, element): out = {} out['status'] =

[GitHub] [beam] George-Wu commented on a change in pull request #12331: [BEAM-10601] DICOM API Beam IO connector

2020-07-30 Thread GitBox
George-Wu commented on a change in pull request #12331: URL: https://github.com/apache/beam/pull/12331#discussion_r463382756 ## File path: sdks/python/apache_beam/io/gcp/dicomio.py ## @@ -372,52 +420,63 @@ def __init__(self, destination_dict, input_type, credential=None):

[GitHub] [beam] George-Wu commented on a change in pull request #12331: [BEAM-10601] DICOM API Beam IO connector

2020-07-30 Thread GitBox
George-Wu commented on a change in pull request #12331: URL: https://github.com/apache/beam/pull/12331#discussion_r463383155 ## File path: sdks/python/apache_beam/io/gcp/dicomio.py ## @@ -164,70 +167,109 @@ class DicomSearch(PTransform): } """ - def __init__(self,

[GitHub] [beam] lostluck commented on pull request #12429: Clean ExtractOutput of mean transform

2020-07-30 Thread GitBox
lostluck commented on pull request #12429: URL: https://github.com/apache/beam/pull/12429#issuecomment-666901638 Going to close it, largely because while in terms of Beam Semantics it's unlikely to trigger, not guarding a divide by zero seems fundamentally wrong. Something could change

[GitHub] [beam] piotr-szuberski commented on pull request #12297: [BEAM-10137] Add KinesisIO for cross-language usage with python wrapper

2020-07-30 Thread GitBox
piotr-szuberski commented on pull request #12297: URL: https://github.com/apache/beam/pull/12297#issuecomment-666458687 Run Python 3.8 PostCommit This is an automated message from the Apache Git Service. To respond to the

[GitHub] [beam] TheNeuralBit commented on pull request #12099: [BEAM-10331] Add SnowflakeIO to list of built-in IO transforms.

2020-07-30 Thread GitBox
TheNeuralBit commented on pull request #12099: URL: https://github.com/apache/beam/pull/12099#issuecomment-666476133 Thanks for the ping! Merging now :) This is an automated message from the Apache Git Service. To respond to

[GitHub] [beam] mxm commented on a change in pull request #12385: [BEAM-10527] Migrate Flink and Spark tests to pytest.

2020-07-30 Thread GitBox
mxm commented on a change in pull request #12385: URL: https://github.com/apache/beam/pull/12385#discussion_r463133822 ## File path: sdks/python/apache_beam/runners/portability/spark_runner_test.py ## @@ -21,141 +21,158 @@ import argparse import logging -import sys import

  1   2   3   >