[GitHub] [beam] chamikaramj commented on pull request #11846: [BEAM-9869] adding self-contained Kafka service jar for testing

2020-05-28 Thread GitBox
chamikaramj commented on pull request #11846: URL: https://github.com/apache/beam/pull/11846#issuecomment-635752368 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] chamikaramj commented on pull request #11846: [BEAM-9869] adding self-contained Kafka service jar for testing

2020-05-28 Thread GitBox
chamikaramj commented on pull request #11846: URL: https://github.com/apache/beam/pull/11846#issuecomment-635752434 Run Python2_PVR_Flink PreCommit This is an automated message from the Apache Git Service. To respond to the

[GitHub] [beam] chamikaramj commented on pull request #11847: [BEAM-10125] adding cross-language KafkaIO integration test

2020-05-28 Thread GitBox
chamikaramj commented on pull request #11847: URL: https://github.com/apache/beam/pull/11847#issuecomment-635752517 Retest this please This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] chamikaramj commented on pull request #11847: [BEAM-10125] adding cross-language KafkaIO integration test

2020-05-28 Thread GitBox
chamikaramj commented on pull request #11847: URL: https://github.com/apache/beam/pull/11847#issuecomment-635752197 Retest this please This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] chamikaramj commented on pull request #11847: [BEAM-10125] adding cross-language KafkaIO integration test

2020-05-28 Thread GitBox
chamikaramj commented on pull request #11847: URL: https://github.com/apache/beam/pull/11847#issuecomment-635752138 Retest this please This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] darshanj commented on pull request #11855: [BEAM-10005] | combinefn for ApproximateQuantiles and ApproximateUnique

2020-05-28 Thread GitBox
darshanj commented on pull request #11855: URL: https://github.com/apache/beam/pull/11855#issuecomment-635734663 R: @kennknowles This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [beam] darshanj opened a new pull request #11855: [BEAM-10005] | combinefn for ApproximateQuantiles and ApproximateUnique

2020-05-28 Thread GitBox
darshanj opened a new pull request #11855: URL: https://github.com/apache/beam/pull/11855 Added api combineFn that can be used for inputs windowed with non global windows **Please** add a meaningful description for your change here Thank you

[GitHub] [beam] boyuanzz commented on a change in pull request #11642: Replace call to .checkpoint() in SDF direct runner to .try_claim(0)

2020-05-28 Thread GitBox
boyuanzz commented on a change in pull request #11642: URL: https://github.com/apache/beam/pull/11642#discussion_r432230809 ## File path: sdks/python/apache_beam/runners/direct/sdf_direct_runner.py ## @@ -464,7 +464,7 @@ def initiate_checkpoint(): with

[GitHub] [beam] lukecwik commented on pull request #11821: [BEAM-10097, BEAM-5982, BEAM-3080] Use primitive views directly instead of transforming KV> to the view type via a naive ma

2020-05-28 Thread GitBox
lukecwik commented on pull request #11821: URL: https://github.com/apache/beam/pull/11821#issuecomment-635722845 Run Spark ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] aaltay commented on pull request #11642: Replace call to .checkpoint() in SDF direct runner to .try_claim(0)

2020-05-28 Thread GitBox
aaltay commented on pull request #11642: URL: https://github.com/apache/beam/pull/11642#issuecomment-635705379 > > This is a single line change and passing all the tests. If the change make sense can we merge it? (question to @boyuanzz ) > > I don't think the change is correct. I

[GitHub] [beam] chamikaramj commented on pull request #11846: [BEAM-9869] adding self-contained Kafka service jar for testing

2020-05-28 Thread GitBox
chamikaramj commented on pull request #11846: URL: https://github.com/apache/beam/pull/11846#issuecomment-635702404 Retest this please This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] chamikaramj commented on pull request #11846: [BEAM-9869] adding self-contained Kafka service jar for testing

2020-05-28 Thread GitBox
chamikaramj commented on pull request #11846: URL: https://github.com/apache/beam/pull/11846#issuecomment-635702360 Retest this please This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] chamikaramj commented on pull request #11846: [BEAM-9869] adding self-contained Kafka service jar for testing

2020-05-28 Thread GitBox
chamikaramj commented on pull request #11846: URL: https://github.com/apache/beam/pull/11846#issuecomment-635702305 LGTM. Thanks. This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [beam] steveniemitz commented on pull request #11849: [BEAM-9964] Move workerCacheMb to a user-visible place

2020-05-28 Thread GitBox
steveniemitz commented on pull request #11849: URL: https://github.com/apache/beam/pull/11849#issuecomment-635697264 =/ looks like the dataflow precommit succeeded but the API call to update it here failed. This is an

[GitHub] [beam] tvalentyn commented on pull request #11788: [BEAM-9785] Add Python 3.8 postcommit tests

2020-05-28 Thread GitBox
tvalentyn commented on pull request #11788: URL: https://github.com/apache/beam/pull/11788#issuecomment-635696787 @epicfaace Thanks for your initiative to help with Python 3.8. Please see the discussion on introducing high-priority/low priority versions:

[GitHub] [beam] steveniemitz commented on pull request #11849: [BEAM-9964] Move workerCacheMb to a user-visible place

2020-05-28 Thread GitBox
steveniemitz commented on pull request #11849: URL: https://github.com/apache/beam/pull/11849#issuecomment-635696194 > gahh so sorry that I missed this. I guess you did have to end up contributing this : ) heh no problem, teamwork! :highfive:

[GitHub] [beam] chamikaramj commented on pull request #11814: [BEAM-10078] uniquify Dataflow specific jars when staging

2020-05-28 Thread GitBox
chamikaramj commented on pull request #11814: URL: https://github.com/apache/beam/pull/11814#issuecomment-635695779 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] chamikaramj commented on pull request #11847: [BEAM-10125] adding cross-language KafkaIO integration test

2020-05-28 Thread GitBox
chamikaramj commented on pull request #11847: URL: https://github.com/apache/beam/pull/11847#issuecomment-635695687 Retest this please This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] robertwb commented on a change in pull request #11835: Various fixes to allow Java PAssert to run on Python

2020-05-28 Thread GitBox
robertwb commented on a change in pull request #11835: URL: https://github.com/apache/beam/pull/11835#discussion_r432198245 ## File path: sdks/python/apache_beam/transforms/trigger_test.py ## @@ -518,6 +519,28 @@ def format_result(k_v): 'B-3': {10, 15, 16},

[GitHub] [beam] boyuanzz commented on pull request #11642: Replace call to .checkpoint() in SDF direct runner to .try_claim(0)

2020-05-28 Thread GitBox
boyuanzz commented on pull request #11642: URL: https://github.com/apache/beam/pull/11642#issuecomment-635680691 > This is a single line change and passing all the tests. If the change make sense can we merge it? (question to @boyuanzz ) I don't think the change is correct. I can

[GitHub] [beam] chamikaramj commented on pull request #11814: [BEAM-10078] uniquify Dataflow specific jars when staging

2020-05-28 Thread GitBox
chamikaramj commented on pull request #11814: URL: https://github.com/apache/beam/pull/11814#issuecomment-635677737 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [beam] chamikaramj commented on pull request #11847: [BEAM-10125] adding cross-language KafkaIO integration test

2020-05-28 Thread GitBox
chamikaramj commented on pull request #11847: URL: https://github.com/apache/beam/pull/11847#issuecomment-635676513 Retest this please This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] chamikaramj commented on pull request #11847: [BEAM-10125] adding cross-language KafkaIO integration test

2020-05-28 Thread GitBox
chamikaramj commented on pull request #11847: URL: https://github.com/apache/beam/pull/11847#issuecomment-635676452 Retest this please This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] aaltay commented on pull request #11788: [BEAM-9785] Add Python 3.8 postcommit tests

2020-05-28 Thread GitBox
aaltay commented on pull request #11788: URL: https://github.com/apache/beam/pull/11788#issuecomment-635675870 /cc @tvalentyn This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [beam] aaltay commented on pull request #11642: Replace call to .checkpoint() in SDF direct runner to .try_claim(0)

2020-05-28 Thread GitBox
aaltay commented on pull request #11642: URL: https://github.com/apache/beam/pull/11642#issuecomment-635675425 This is a single line change and passing all the tests. If the change make sense can we merge it? (question to @boyuanzz )

[GitHub] [beam] aaltay commented on pull request #11758: Old Fastjson has a serious security problem

2020-05-28 Thread GitBox
aaltay commented on pull request #11758: URL: https://github.com/apache/beam/pull/11758#issuecomment-635675525 retest this please This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [beam] aaltay commented on pull request #11181: [BEAM-9500] [WIP] Refactor load tests

2020-05-28 Thread GitBox
aaltay commented on pull request #11181: URL: https://github.com/apache/beam/pull/11181#issuecomment-635674912 @piotr-szuberski - what is the next step for this PR? Is it still active? Should we close it? This is an

[GitHub] [beam] chamikaramj commented on pull request #11814: [BEAM-10078] uniquify Dataflow specific jars when staging

2020-05-28 Thread GitBox
chamikaramj commented on pull request #11814: URL: https://github.com/apache/beam/pull/11814#issuecomment-635674009 I don't think so. We changed from using "key=value" strings to StagedFile objects in https://github.com/apache/beam/pull/11039/files.

[GitHub] [beam] aaltay commented on pull request #11706: [BEAM-8451] annotate python only sections

2020-05-28 Thread GitBox
aaltay commented on pull request #11706: URL: https://github.com/apache/beam/pull/11706#issuecomment-635672892 R: @rosetn This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [beam] TheNeuralBit commented on pull request #11814: [BEAM-10078] uniquify Dataflow specific jars when staging

2020-05-28 Thread GitBox
TheNeuralBit commented on pull request #11814: URL: https://github.com/apache/beam/pull/11814#issuecomment-635672193 @chamikaramj, @ihji is the `filesToStage` change a problem? This is an automated message from the Apache

[GitHub] [beam] aaltay commented on pull request #11758: Old Fastjson has a serious security problem

2020-05-28 Thread GitBox
aaltay commented on pull request #11758: URL: https://github.com/apache/beam/pull/11758#issuecomment-635672368 LGTM. I can merge it if tests pass. This is an automated message from the Apache Git Service. To respond to the

[GitHub] [beam] aaltay commented on pull request #11610: [BEAM-9825] | Implement Intersect,Union,Except transforms

2020-05-28 Thread GitBox
aaltay commented on pull request #11610: URL: https://github.com/apache/beam/pull/11610#issuecomment-635672074 All tests passed. I do not see a LGTM, maybe I am missing. Is this ready to be merged? This is an automated

[GitHub] [beam] aaltay commented on pull request #11779: [BEAM-10055] Add --region to python examples where it was missing

2020-05-28 Thread GitBox
aaltay commented on pull request #11779: URL: https://github.com/apache/beam/pull/11779#issuecomment-635671668 LGTM. I will merge after tests pass. Thank you @tedromer This is an automated message from the Apache Git

[GitHub] [beam] aaltay commented on pull request #11779: [BEAM-10055] Add --region to python examples where it was missing

2020-05-28 Thread GitBox
aaltay commented on pull request #11779: URL: https://github.com/apache/beam/pull/11779#issuecomment-635671566 retest this please This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [beam] chamikaramj commented on pull request #11847: [BEAM-10125] adding cross-language KafkaIO integration test

2020-05-28 Thread GitBox
chamikaramj commented on pull request #11847: URL: https://github.com/apache/beam/pull/11847#issuecomment-635670461 LGTM. Thanks. This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [beam] aaltay commented on pull request #11819: [BEAM-8371] Remove support for Python 2

2020-05-28 Thread GitBox
aaltay commented on pull request #11819: URL: https://github.com/apache/beam/pull/11819#issuecomment-635669412 Should we close this pull request for now? This is an automated message from the Apache Git Service. To respond

[GitHub] [beam] aaltay commented on pull request #11682: [BEAM-9946] | added new api in Partition Transform

2020-05-28 Thread GitBox
aaltay commented on pull request #11682: URL: https://github.com/apache/beam/pull/11682#issuecomment-635668922 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [beam] lukecwik commented on pull request #11821: [BEAM-10097, BEAM-5982, BEAM-3080] Use primitive views directly instead of transforming KV> to the view type via a naive ma

2020-05-28 Thread GitBox
lukecwik commented on pull request #11821: URL: https://github.com/apache/beam/pull/11821#issuecomment-635664997 Run Spark ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] rohdesamuel commented on pull request #11594: [BEAM-9692] Replace apply_WriteToBigQuery with PTransformOverride

2020-05-28 Thread GitBox
rohdesamuel commented on pull request #11594: URL: https://github.com/apache/beam/pull/11594#issuecomment-635664357 Looks like the PreCommit failed with "Exception: Dataflow only supports Python versions 2 and 3.5+, got: (3, 8)". Is that a known failure?

[GitHub] [beam] pabloem commented on pull request #11849: [BEAM-9964] Move workerCacheMb to a user-visible place

2020-05-28 Thread GitBox
pabloem commented on pull request #11849: URL: https://github.com/apache/beam/pull/11849#issuecomment-635663865 gahh so sorry that I missed this. I guess you did have to end up contributing this : ) This is an automated

[GitHub] [beam] robertwb commented on a change in pull request #11632: [BEAM-7746] Fix type errors and enable checks for apache_beam.dataframe.*

2020-05-28 Thread GitBox
robertwb commented on a change in pull request #11632: URL: https://github.com/apache/beam/pull/11632#discussion_r432170796 ## File path: sdks/python/apache_beam/dataframe/transforms.py ## @@ -16,13 +16,28 @@ from __future__ import absolute_import +import typing +from

[GitHub] [beam] chamikaramj opened a new pull request #11854: [BEAM-8019] Cherry pick 11844

2020-05-28 Thread GitBox
chamikaramj opened a new pull request #11854: URL: https://github.com/apache/beam/pull/11854 Without this cross-language KafkaIO users may have to do pipeline.run(False) instead of pipeline.run() when executing a pipeline using Dataflow. @TheNeuralBit this should

[GitHub] [beam] chamikaramj commented on pull request #11854: [BEAM-8019] Cherry pick 11844

2020-05-28 Thread GitBox
chamikaramj commented on pull request #11854: URL: https://github.com/apache/beam/pull/11854#issuecomment-635657611 R: @TheNeuralBit This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] robertwb merged pull request #11853: Update multi-language roadmap status.

2020-05-28 Thread GitBox
robertwb merged pull request #11853: URL: https://github.com/apache/beam/pull/11853 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [beam] chamikaramj commented on pull request #11844: [BEAM-8019] Enables proto holders when 'test_runner_api' is True.

2020-05-28 Thread GitBox
chamikaramj commented on pull request #11844: URL: https://github.com/apache/beam/pull/11844#issuecomment-635651784 Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [beam] robertwb merged pull request #11844: [BEAM-8019] Enables proto holders when 'test_runner_api' is True.

2020-05-28 Thread GitBox
robertwb merged pull request #11844: URL: https://github.com/apache/beam/pull/11844 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [beam] aaltay commented on pull request #11851: [BEAM-10144] Update PipelineOptions snippets for best practices

2020-05-28 Thread GitBox
aaltay commented on pull request #11851: URL: https://github.com/apache/beam/pull/11851#issuecomment-635648807 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [beam] iemejia commented on a change in pull request #11853: Update multi-language roadmap status.

2020-05-28 Thread GitBox
iemejia commented on a change in pull request #11853: URL: https://github.com/apache/beam/pull/11853#discussion_r432163908 ## File path: website/www/site/content/en/roadmap/connectors-multi-sdk.md ## @@ -62,27 +62,29 @@ Work related to making cross-language transforms

[GitHub] [beam] damondouglas commented on pull request #11803: [BEAM-9679] Add a CoGroupByKey lesson to the Core Transforms section

2020-05-28 Thread GitBox
damondouglas commented on pull request #11803: URL: https://github.com/apache/beam/pull/11803#issuecomment-635644834 I updated [the stepik course](https://stepik.org/course/70387) and commited the updated `*-remote.yaml` files to this PR. It is ready to merge into master. Thank you,

[GitHub] [beam] lukecwik commented on pull request #11821: [BEAM-10097, BEAM-5982, BEAM-3080] Use primitive views directly instead of transforming KV> to the view type via a naive ma

2020-05-28 Thread GitBox
lukecwik commented on pull request #11821: URL: https://github.com/apache/beam/pull/11821#issuecomment-635644599 Run Spark ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] pabloem commented on pull request #11594: [BEAM-9692] Replace apply_WriteToBigQuery with PTransformOverride

2020-05-28 Thread GitBox
pabloem commented on pull request #11594: URL: https://github.com/apache/beam/pull/11594#issuecomment-635640799 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [beam] pabloem commented on pull request #11594: [BEAM-9692] Replace apply_WriteToBigQuery with PTransformOverride

2020-05-28 Thread GitBox
pabloem commented on pull request #11594: URL: https://github.com/apache/beam/pull/11594#issuecomment-635640892 Run Python 2 PostCommit This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] rohdesamuel commented on pull request #11594: [BEAM-9692] Replace apply_WriteToBigQuery with PTransformOverride

2020-05-28 Thread GitBox
rohdesamuel commented on pull request #11594: URL: https://github.com/apache/beam/pull/11594#issuecomment-635640073 R: @pabloem This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [beam] udim merged pull request #11070: [BEAM-8280] Blog post: Python typing changes

2020-05-28 Thread GitBox
udim merged pull request #11070: URL: https://github.com/apache/beam/pull/11070 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [beam] lukecwik commented on pull request #11821: [BEAM-10097, BEAM-5982, BEAM-3080] Use primitive views directly instead of transforming KV> to the view type via a naive ma

2020-05-28 Thread GitBox
lukecwik commented on pull request #11821: URL: https://github.com/apache/beam/pull/11821#issuecomment-635630717 Run Spark ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] ihji commented on a change in pull request #11847: [BEAM-10125] adding cross-language KafkaIO integration test

2020-05-28 Thread GitBox
ihji commented on a change in pull request #11847: URL: https://github.com/apache/beam/pull/11847#discussion_r432142814 ## File path: sdks/python/apache_beam/io/external/xlang_kafkaio_it_test.py ## @@ -0,0 +1,145 @@ +"""Integration test for Python cross-language pipelines for

[GitHub] [beam] ihji commented on a change in pull request #11847: [BEAM-10125] adding cross-language KafkaIO integration test

2020-05-28 Thread GitBox
ihji commented on a change in pull request #11847: URL: https://github.com/apache/beam/pull/11847#discussion_r432142992 ## File path: sdks/python/apache_beam/io/external/xlang_kafkaio_it_test.py ## @@ -0,0 +1,145 @@ +"""Integration test for Python cross-language pipelines for

[GitHub] [beam] ihji commented on a change in pull request #11847: [BEAM-10125] adding cross-language KafkaIO integration test

2020-05-28 Thread GitBox
ihji commented on a change in pull request #11847: URL: https://github.com/apache/beam/pull/11847#discussion_r432142695 ## File path: sdks/python/apache_beam/io/external/xlang_kafkaio_it_test.py ## @@ -0,0 +1,145 @@ +"""Integration test for Python cross-language pipelines for

[GitHub] [beam] jhnmora000 commented on pull request #11845: [BEAM-9198] BeamSQL aggregation analytics functionality

2020-05-28 Thread GitBox
jhnmora000 commented on pull request #11845: URL: https://github.com/apache/beam/pull/11845#issuecomment-635624778 Thanks for your help @amaliujia . I will close this PR and continue experimenting with BeamSQL/Calcite.

[GitHub] [beam] jhnmora000 closed pull request #11845: [BEAM-9198] BeamSQL aggregation analytics functionality

2020-05-28 Thread GitBox
jhnmora000 closed pull request #11845: URL: https://github.com/apache/beam/pull/11845 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [beam] lukecwik commented on pull request #11821: [BEAM-10097, BEAM-5982, BEAM-3080] Use primitive views directly instead of transforming KV> to the view type via a naive ma

2020-05-28 Thread GitBox
lukecwik commented on pull request #11821: URL: https://github.com/apache/beam/pull/11821#issuecomment-635621052 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [beam] lukecwik commented on pull request #11821: [BEAM-10097, BEAM-5982, BEAM-3080] Use primitive views directly instead of transforming KV> to the view type via a naive ma

2020-05-28 Thread GitBox
lukecwik commented on pull request #11821: URL: https://github.com/apache/beam/pull/11821#issuecomment-635620991 Run Java Flink PortableValidatesRunner Streaming This is an automated message from the Apache Git Service. To

[GitHub] [beam] lukecwik commented on pull request #11821: [BEAM-10097, BEAM-5982, BEAM-3080] Use primitive views directly instead of transforming KV> to the view type via a naive ma

2020-05-28 Thread GitBox
lukecwik commented on pull request #11821: URL: https://github.com/apache/beam/pull/11821#issuecomment-635620885 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [beam] chamikaramj commented on pull request #11853: Update multi-language roadmap status.

2020-05-28 Thread GitBox
chamikaramj commented on pull request #11853: URL: https://github.com/apache/beam/pull/11853#issuecomment-635620838 LGTM. Thanks for updating. This is an automated message from the Apache Git Service. To respond to the

[GitHub] [beam] youngoli merged pull request #11791: [BEAM-9935] Respect allowed split points and fraction in Go.

2020-05-28 Thread GitBox
youngoli merged pull request #11791: URL: https://github.com/apache/beam/pull/11791 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [beam] davidcavazos commented on pull request #11851: [BEAM-10144] Update PipelineOptions snippets for best practices

2020-05-28 Thread GitBox
davidcavazos commented on pull request #11851: URL: https://github.com/apache/beam/pull/11851#issuecomment-635618216 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] TheNeuralBit commented on pull request #11814: [BEAM-10078] uniquify Dataflow specific jars when staging

2020-05-28 Thread GitBox
TheNeuralBit commented on pull request #11814: URL: https://github.com/apache/beam/pull/11814#issuecomment-635615410 retest this please This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] youngoli commented on pull request #11791: [BEAM-9935] Respect allowed split points and fraction in Go.

2020-05-28 Thread GitBox
youngoli commented on pull request #11791: URL: https://github.com/apache/beam/pull/11791#issuecomment-635615190 Run Go PostCommit This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [beam] robertwb opened a new pull request #11853: Update multi-language roadmap status.

2020-05-28 Thread GitBox
robertwb opened a new pull request #11853: URL: https://github.com/apache/beam/pull/11853 Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose

[GitHub] [beam] lukecwik commented on pull request #11821: [BEAM-10097, BEAM-5982, BEAM-3080] Use primitive views directly instead of transforming KV> to the view type via a naive ma

2020-05-28 Thread GitBox
lukecwik commented on pull request #11821: URL: https://github.com/apache/beam/pull/11821#issuecomment-635613803 Run Spark ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] tvalentyn commented on a change in pull request #11086: [BEAM-8910] Make custom BQ source read from Avro

2020-05-28 Thread GitBox
tvalentyn commented on a change in pull request #11086: URL: https://github.com/apache/beam/pull/11086#discussion_r432112443 ## File path: sdks/python/apache_beam/io/gcp/big_query_query_to_table_it_test.py ## @@ -254,11 +256,36 @@ def test_big_query_new_types(self):

[GitHub] [beam] tvalentyn commented on pull request #11661: [BEAM-7774] Remove perfkit benchmarking tool from python performance …

2020-05-28 Thread GitBox
tvalentyn commented on pull request #11661: URL: https://github.com/apache/beam/pull/11661#issuecomment-635593321 @kamilwu - please merge once this looks good to you, I don't have other input here. This is an automated

[GitHub] [beam] tvalentyn commented on a change in pull request #11661: [BEAM-7774] Remove perfkit benchmarking tool from python performance …

2020-05-28 Thread GitBox
tvalentyn commented on a change in pull request #11661: URL: https://github.com/apache/beam/pull/11661#discussion_r432107251 ## File path: .test-infra/metrics/grafana/dashboards/perftests_metrics/Python_Performance_Tests.json ## @@ -77,7 +77,7 @@ ],

[GitHub] [beam] chamikaramj commented on pull request #11814: [BEAM-10078] uniquify Dataflow specific jars when staging

2020-05-28 Thread GitBox
chamikaramj commented on pull request #11814: URL: https://github.com/apache/beam/pull/11814#issuecomment-635591331 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [beam] tvalentyn commented on a change in pull request #11661: [BEAM-7774] Remove perfkit benchmarking tool from python performance …

2020-05-28 Thread GitBox
tvalentyn commented on a change in pull request #11661: URL: https://github.com/apache/beam/pull/11661#discussion_r432106093 ## File path: sdks/python/apache_beam/examples/wordcount_it_test.py ## @@ -84,11 +87,45 @@ def _run_wordcount_it(self, run_wordcount, **opts): #

[GitHub] [beam] lostluck merged pull request #11207: [BEAM-9220] Go Dataflow jobs to use runner v2

2020-05-28 Thread GitBox
lostluck merged pull request #11207: URL: https://github.com/apache/beam/pull/11207 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [beam] rionmonster commented on a change in pull request #11761: [BEAM-10027] Support for Kotlin-based Beam Katas

2020-05-28 Thread GitBox
rionmonster commented on a change in pull request #11761: URL: https://github.com/apache/beam/pull/11761#discussion_r432104332 ## File path: learning/katas/kotlin/Windowing/Fixed Time Window/Fixed Time Window/test/org/apache/beam/learning/katas/windowing/fixedwindow/WindowedEvent.kt

[GitHub] [beam] rionmonster commented on a change in pull request #11761: [BEAM-10027] Support for Kotlin-based Beam Katas

2020-05-28 Thread GitBox
rionmonster commented on a change in pull request #11761: URL: https://github.com/apache/beam/pull/11761#discussion_r432103074 ## File path: learning/katas/kotlin/Core Transforms/Combine/CombineFn/src/org/apache/beam/learning/katas/coretransforms/combine/combinefn/Task.kt ##

[GitHub] [beam] aaltay commented on pull request #11851: [BEAM-10144] Update PipelineOptions snippets for best practices

2020-05-28 Thread GitBox
aaltay commented on pull request #11851: URL: https://github.com/apache/beam/pull/11851#issuecomment-635587627 retest this please This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [beam] tvalentyn commented on a change in pull request #11661: [BEAM-7774] Remove perfkit benchmarking tool from python performance …

2020-05-28 Thread GitBox
tvalentyn commented on a change in pull request #11661: URL: https://github.com/apache/beam/pull/11661#discussion_r432101289 ## File path: .test-infra/metrics/grafana/dashboards/perftests_metrics/Python_Performance_Tests.json ## @@ -0,0 +1,297 @@ +{ Review comment: >

[GitHub] [beam] ihji commented on a change in pull request #11814: [BEAM-10078] uniquify Dataflow specific jars when staging

2020-05-28 Thread GitBox
ihji commented on a change in pull request #11814: URL: https://github.com/apache/beam/pull/11814#discussion_r432099933 ## File path: runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/PackageUtil.java ## @@ -397,10 +397,21 @@ public static

[GitHub] [beam] ibzib opened a new pull request #11852: [BEAM-10107] Remove outdated instructions for website updates in rele…

2020-05-28 Thread GitBox
ibzib opened a new pull request #11852: URL: https://github.com/apache/beam/pull/11852 …ase guide. Just some minor docs cleanup. R: @TheNeuralBit Thank you for your contribution! Follow this checklist to help us incorporate your

[GitHub] [beam] chamikaramj commented on a change in pull request #11847: [BEAM-10125] adding cross-language KafkaIO integration test

2020-05-28 Thread GitBox
chamikaramj commented on a change in pull request #11847: URL: https://github.com/apache/beam/pull/11847#discussion_r432083261 ## File path: sdks/python/apache_beam/io/external/xlang_kafkaio_it_test.py ## @@ -0,0 +1,145 @@ +"""Integration test for Python cross-language

[GitHub] [beam] robertwb commented on a change in pull request #11835: Various fixes to allow Java PAssert to run on Python

2020-05-28 Thread GitBox
robertwb commented on a change in pull request #11835: URL: https://github.com/apache/beam/pull/11835#discussion_r432093237 ## File path: sdks/python/apache_beam/transforms/trigger_test.py ## @@ -518,6 +519,28 @@ def format_result(k_v): 'B-3': {10, 15, 16},

[GitHub] [beam] pabloem removed a comment on pull request #11086: [BEAM-8910] Make custom BQ source read from Avro

2020-05-28 Thread GitBox
pabloem removed a comment on pull request #11086: URL: https://github.com/apache/beam/pull/11086#issuecomment-626066297 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [beam] lostluck merged pull request #11806: [BEAM-9679] Flatten Kata for Go

2020-05-28 Thread GitBox
lostluck merged pull request #11806: URL: https://github.com/apache/beam/pull/11806 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [beam] lostluck commented on pull request #11806: [BEAM-9679] Flatten Kata for Go

2020-05-28 Thread GitBox
lostluck commented on pull request #11806: URL: https://github.com/apache/beam/pull/11806#issuecomment-635568870 @damondouglas That sounds correct to me as well, in order to avoid colliding stepik updates. This is an

[GitHub] [beam] pabloem removed a comment on pull request #11086: [BEAM-8910] Make custom BQ source read from Avro

2020-05-28 Thread GitBox
pabloem removed a comment on pull request #11086: URL: https://github.com/apache/beam/pull/11086#issuecomment-634947671 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [beam] damondouglas commented on pull request #11806: [BEAM-9679] Flatten Kata for Go

2020-05-28 Thread GitBox
damondouglas commented on pull request #11806: URL: https://github.com/apache/beam/pull/11806#issuecomment-635563478 @henryken Just confirming these steps: 1. @lostluck merges this PR #11806 to master 1. @damondouglas merges new changes from PR #11806 to PR #11803 1.

[GitHub] [beam] TheNeuralBit commented on pull request #11777: [BEAM-10054] Fix watermark hold for on_time_pane

2020-05-28 Thread GitBox
TheNeuralBit commented on pull request #11777: URL: https://github.com/apache/beam/pull/11777#issuecomment-635549457 Are those tests sufficient though if they're passing before this PR? This is an automated message from the

[GitHub] [beam] kennknowles commented on a change in pull request #11835: Various fixes to allow Java PAssert to run on Python

2020-05-28 Thread GitBox
kennknowles commented on a change in pull request #11835: URL: https://github.com/apache/beam/pull/11835#discussion_r432064924 ## File path: sdks/python/apache_beam/transforms/trigger_test.py ## @@ -518,6 +519,28 @@ def format_result(k_v): 'B-3': {10, 15,

[GitHub] [beam] piotr-szuberski commented on a change in pull request #11661: [BEAM-7774] Remove perfkit benchmarking tool from python performance …

2020-05-28 Thread GitBox
piotr-szuberski commented on a change in pull request #11661: URL: https://github.com/apache/beam/pull/11661#discussion_r432061809 ## File path: sdks/python/apache_beam/examples/wordcount_it_test.py ## @@ -84,11 +87,45 @@ def _run_wordcount_it(self, run_wordcount, **opts):

[GitHub] [beam] piotr-szuberski commented on a change in pull request #11661: [BEAM-7774] Remove perfkit benchmarking tool from python performance …

2020-05-28 Thread GitBox
piotr-szuberski commented on a change in pull request #11661: URL: https://github.com/apache/beam/pull/11661#discussion_r432061809 ## File path: sdks/python/apache_beam/examples/wordcount_it_test.py ## @@ -84,11 +87,45 @@ def _run_wordcount_it(self, run_wordcount, **opts):

[GitHub] [beam] davidcavazos commented on a change in pull request #11851: [BEAM-10144] Update PipelineOptions snippets for best practices

2020-05-28 Thread GitBox
davidcavazos commented on a change in pull request #11851: URL: https://github.com/apache/beam/pull/11851#discussion_r432049685 ## File path: sdks/python/apache_beam/examples/snippets/snippets.py ## @@ -226,35 +227,33 @@ def _add_argparse_args(cls, parser): # [END

[GitHub] [beam] lukecwik commented on a change in pull request #11792: WIP: Add ValidatesRunner task for local_job_service and Java SDK harness

2020-05-28 Thread GitBox
lukecwik commented on a change in pull request #11792: URL: https://github.com/apache/beam/pull/11792#discussion_r432050850 ## File path: runners/portability/java/build.gradle ## @@ -31,9 +45,123 @@ dependencies { compile project(path: ":sdks:java:harness", configuration:

[GitHub] [beam] davidcavazos commented on a change in pull request #11851: [BEAM-10144] Update PipelineOptions snippets for best practices

2020-05-28 Thread GitBox
davidcavazos commented on a change in pull request #11851: URL: https://github.com/apache/beam/pull/11851#discussion_r432049685 ## File path: sdks/python/apache_beam/examples/snippets/snippets.py ## @@ -226,35 +227,33 @@ def _add_argparse_args(cls, parser): # [END

[GitHub] [beam] davidcavazos commented on a change in pull request #11851: [BEAM-10144] Update PipelineOptions snippets for best practices

2020-05-28 Thread GitBox
davidcavazos commented on a change in pull request #11851: URL: https://github.com/apache/beam/pull/11851#discussion_r432049685 ## File path: sdks/python/apache_beam/examples/snippets/snippets.py ## @@ -226,35 +227,33 @@ def _add_argparse_args(cls, parser): # [END

[GitHub] [beam] davidcavazos commented on a change in pull request #11851: [BEAM-10144] Update PipelineOptions snippets for best practices

2020-05-28 Thread GitBox
davidcavazos commented on a change in pull request #11851: URL: https://github.com/apache/beam/pull/11851#discussion_r432049685 ## File path: sdks/python/apache_beam/examples/snippets/snippets.py ## @@ -226,35 +227,33 @@ def _add_argparse_args(cls, parser): # [END

[GitHub] [beam] davidcavazos opened a new pull request #11851: [BEAM-10144] Update PipelineOptions snippets for best practices

2020-05-28 Thread GitBox
davidcavazos opened a new pull request #11851: URL: https://github.com/apache/beam/pull/11851 Updating `PipelineOptions` snippets to mirror best practices. Making setting pipeline options programmatically less cumbersome, and adding the necessary imports to each snippet. R: @aaltay

[GitHub] [beam] pabloem opened a new pull request #11850: [BEAM-1438] Allow 0 shards on WriteFiles streaming

2020-05-28 Thread GitBox
pabloem opened a new pull request #11850: URL: https://github.com/apache/beam/pull/11850 **Please** add a meaningful description for your change here Thank you for your contribution! Follow this checklist to help us incorporate your contribution

  1   2   >