[GitHub] [beam] codecov[bot] edited a comment on pull request #12576: [BEAM-10671] Add environment configuration fields as first-class pipeline options.

2020-09-30 Thread GitBox
codecov[bot] edited a comment on pull request #12576: URL: https://github.com/apache/beam/pull/12576#issuecomment-692353567 # [Codecov](https://codecov.io/gh/apache/beam/pull/12576?src=pr=h1) Report > Merging [#12576](https://codecov.io/gh/apache/beam/pull/12576?src=pr=desc) into

[GitHub] [beam] rezarokni commented on pull request #12973: [BEAM-10234] Create ApproximateDistinct using HLL Impl

2020-09-30 Thread GitBox
rezarokni commented on pull request #12973: URL: https://github.com/apache/beam/pull/12973#issuecomment-701219619 @robinyqiu This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [beam] TobKed commented on a change in pull request #12318: [BEAM-8665] Add test suites to run Beam tests on Linux/Windows/Mac platforms.

2020-09-30 Thread GitBox
TobKed commented on a change in pull request #12318: URL: https://github.com/apache/beam/pull/12318#discussion_r497319272 ## File path: sdks/python/apache_beam/runners/portability/stager_test.py ## @@ -167,6 +167,9 @@ def test_no_main_session(self): # xdist adds

[GitHub] [beam] mxm commented on pull request #12576: [BEAM-10671] Add environment configuration fields as first-class pipeline options.

2020-09-30 Thread GitBox
mxm commented on pull request #12576: URL: https://github.com/apache/beam/pull/12576#issuecomment-701244754 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] rezarokni opened a new pull request #12973: [BEAM-10234] Create ApproximateDistinct using HLL Impl

2020-09-30 Thread GitBox
rezarokni opened a new pull request #12973: URL: https://github.com/apache/beam/pull/12973 [BEAM-10234] Create ApproximateDistinct using HLL Impl Create ApproximateDistinct which uses HLL Impl Marked ApproximateUnique as deprecated in preparation for removal in favour of

[GitHub] [beam] mxm commented on pull request #12576: [BEAM-10671] Add environment configuration fields as first-class pipeline options.

2020-09-30 Thread GitBox
mxm commented on pull request #12576: URL: https://github.com/apache/beam/pull/12576#issuecomment-701245561 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [beam] nielsbasjes commented on a change in pull request #12962: [WIP][BEAM-9379] Update calcite to 1.25

2020-09-30 Thread GitBox
nielsbasjes commented on a change in pull request #12962: URL: https://github.com/apache/beam/pull/12962#discussion_r497362928 ## File path: vendor/calcite-1_25_0/build.gradle ## @@ -18,29 +18,33 @@ plugins { id 'org.apache.beam.vendor-java' } -description = "Apache Beam

[GitHub] [beam] nielsbasjes commented on a change in pull request #12962: [WIP][BEAM-9379] Update calcite to 1.25

2020-09-30 Thread GitBox
nielsbasjes commented on a change in pull request #12962: URL: https://github.com/apache/beam/pull/12962#discussion_r497363201 ## File path: vendor/calcite-1_25_0/build.gradle ## @@ -18,29 +18,33 @@ plugins { id 'org.apache.beam.vendor-java' } -description = "Apache Beam

[GitHub] [beam] aromanenko-dev commented on pull request #12572: [BEAM-10123] Add Kafka Commit transform.

2020-09-30 Thread GitBox
aromanenko-dev commented on pull request #12572: URL: https://github.com/apache/beam/pull/12572#issuecomment-701311000 > Yeah I also want to have some tests around this but it may not be possible to involve actual Kafka consumer to do so. The way I can come up is to use mock but I'm not

[GitHub] [beam] runningcode edited a comment on pull request #12969: [BEAM-10986] Rollback to shadow 4.0.3

2020-09-30 Thread GitBox
runningcode edited a comment on pull request #12969: URL: https://github.com/apache/beam/pull/12969#issuecomment-701301646 Hi, I'm not sure that this fix helps. The [expansion-service/build.gradle](https://github.com/apache/beam/blob/master/sdks/java/expansion-service/build.gradle#L22)

[GitHub] [beam] nielsbasjes commented on pull request #12962: [WIP][BEAM-9379] Update calcite to 1.25

2020-09-30 Thread GitBox
nielsbasjes commented on pull request #12962: URL: https://github.com/apache/beam/pull/12962#issuecomment-701269350 > Can you publish a build scan by running your second step with `--scan` ? Then we can see what is broken. https://scans.gradle.com/s/ofcfnu774du4y

[GitHub] [beam] codecov[bot] edited a comment on pull request #12962: [WIP][BEAM-9379] Update calcite to 1.25

2020-09-30 Thread GitBox
codecov[bot] edited a comment on pull request #12962: URL: https://github.com/apache/beam/pull/12962#issuecomment-700648248 # [Codecov](https://codecov.io/gh/apache/beam/pull/12962?src=pr=h1) Report > Merging [#12962](https://codecov.io/gh/apache/beam/pull/12962?src=pr=desc) into

[GitHub] [beam] runningcode commented on pull request #12969: [BEAM-10986] Rollback to shadow 4.0.3

2020-09-30 Thread GitBox
runningcode commented on pull request #12969: URL: https://github.com/apache/beam/pull/12969#issuecomment-701301646 Hi, I'm not sure that this fix helps. The [expansion-service/build.gradle](https://github.com/apache/beam/blob/master/sdks/java/expansion-service/build.gradle#L22) does not

[GitHub] [beam] codecov[bot] edited a comment on pull request #12576: [BEAM-10671] Add environment configuration fields as first-class pipeline options.

2020-09-30 Thread GitBox
codecov[bot] edited a comment on pull request #12576: URL: https://github.com/apache/beam/pull/12576#issuecomment-692353567 # [Codecov](https://codecov.io/gh/apache/beam/pull/12576?src=pr=h1) Report > Merging [#12576](https://codecov.io/gh/apache/beam/pull/12576?src=pr=desc) into

[GitHub] [beam] nielsbasjes commented on a change in pull request #12962: [WIP][BEAM-9379] Update calcite to 1.25

2020-09-30 Thread GitBox
nielsbasjes commented on a change in pull request #12962: URL: https://github.com/apache/beam/pull/12962#discussion_r497365708 ## File path: vendor/calcite-1_25_0/build.gradle ## @@ -18,29 +18,33 @@ plugins { id 'org.apache.beam.vendor-java' } -description = "Apache Beam

[GitHub] [beam] codecov[bot] edited a comment on pull request #12962: [WIP][BEAM-9379] Update calcite to 1.25

2020-09-30 Thread GitBox
codecov[bot] edited a comment on pull request #12962: URL: https://github.com/apache/beam/pull/12962#issuecomment-700648248 # [Codecov](https://codecov.io/gh/apache/beam/pull/12962?src=pr=h1) Report > Merging [#12962](https://codecov.io/gh/apache/beam/pull/12962?src=pr=desc) into

[GitHub] [beam] aromanenko-dev commented on a change in pull request #12572: [BEAM-10123] Add Kafka Commit transform.

2020-09-30 Thread GitBox
aromanenko-dev commented on a change in pull request #12572: URL: https://github.com/apache/beam/pull/12572#discussion_r497414578 ## File path: sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaCommitOffset.java ## @@ -0,0 +1,130 @@ +/* + * Licensed to the

[GitHub] [beam] rezarokni commented on pull request #12973: [BEAM-10234] Create ApproximateDistinct using HLL Impl

2020-09-30 Thread GitBox
rezarokni commented on pull request #12973: URL: https://github.com/apache/beam/pull/12973#issuecomment-701316654 @zfraa This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [beam] runningcode opened a new pull request #12974: [BEAM-10986] Add shadow jar configuration to expansion-service.

2020-09-30 Thread GitBox
runningcode opened a new pull request #12974: URL: https://github.com/apache/beam/pull/12974 The shadowJar configuration was never specified so it should have never been applying the shadow plugin. The

[GitHub] [beam] runningcode commented on pull request #12974: [BEAM-10986] Add shadow jar configuration to expansion-service.

2020-09-30 Thread GitBox
runningcode commented on pull request #12974: URL: https://github.com/apache/beam/pull/12974#issuecomment-701337357 Replaces https://github.com/apache/beam/pull/12969 R: @robinyqiu R: @TheNeuralBit R: @mxm This is an

[GitHub] [beam] piotr-szuberski opened a new pull request #12975: Update Beam's io dependencies

2020-09-30 Thread GitBox
piotr-szuberski opened a new pull request #12975: URL: https://github.com/apache/beam/pull/12975 [BEAM-10370] redis.clients:jedis from 3.0.1 to 3.3.0 Thank you for your contribution! Follow this checklist to help us incorporate your contribution

[GitHub] [beam] codecov[bot] edited a comment on pull request #12964: [BEAM-10639] Integration test that exercises --setup_file flag on Dataflow

2020-09-30 Thread GitBox
codecov[bot] edited a comment on pull request #12964: URL: https://github.com/apache/beam/pull/12964#issuecomment-701370267 # [Codecov](https://codecov.io/gh/apache/beam/pull/12964?src=pr=h1) Report > Merging [#12964](https://codecov.io/gh/apache/beam/pull/12964?src=pr=desc) into

[GitHub] [beam] codecov[bot] edited a comment on pull request #12964: [BEAM-10639] Integration test that exercises --setup_file flag on Dataflow

2020-09-30 Thread GitBox
codecov[bot] edited a comment on pull request #12964: URL: https://github.com/apache/beam/pull/12964#issuecomment-701370267 # [Codecov](https://codecov.io/gh/apache/beam/pull/12964?src=pr=h1) Report > Merging [#12964](https://codecov.io/gh/apache/beam/pull/12964?src=pr=desc) into

[GitHub] [beam] kamilwu commented on pull request #12964: [BEAM-10639] Integration test that exercises --setup_file flag on Dataflow

2020-09-30 Thread GitBox
kamilwu commented on pull request #12964: URL: https://github.com/apache/beam/pull/12964#issuecomment-701399340 Run Python 3.8 PostCommit This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] codecov[bot] commented on pull request #12964: [BEAM-10639] Integration test that exercises --setup_file flag on Dataflow

2020-09-30 Thread GitBox
codecov[bot] commented on pull request #12964: URL: https://github.com/apache/beam/pull/12964#issuecomment-701370267 # [Codecov](https://codecov.io/gh/apache/beam/pull/12964?src=pr=h1) Report > Merging [#12964](https://codecov.io/gh/apache/beam/pull/12964?src=pr=desc) into

[GitHub] [beam] kamilwu commented on a change in pull request #12927: Update some of the Python dependencies from Beam Dependency Check Report

2020-09-30 Thread GitBox
kamilwu commented on a change in pull request #12927: URL: https://github.com/apache/beam/pull/12927#discussion_r497520564 ## File path: sdks/python/apache_beam/io/avroio.py ## @@ -627,11 +627,19 @@ def write_record(self, writer, value): writer.append(value) +class

[GitHub] [beam] kamilwu commented on pull request #12964: [BEAM-10639] Integration test that exercises --setup_file flag on Dataflow

2020-09-30 Thread GitBox
kamilwu commented on pull request #12964: URL: https://github.com/apache/beam/pull/12964#issuecomment-701358974 Run Python 3.8 PostCommit This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] codecov[bot] edited a comment on pull request #12964: [BEAM-10639] Integration test that exercises --setup_file flag on Dataflow

2020-09-30 Thread GitBox
codecov[bot] edited a comment on pull request #12964: URL: https://github.com/apache/beam/pull/12964#issuecomment-701370267 # [Codecov](https://codecov.io/gh/apache/beam/pull/12964?src=pr=h1) Report > Merging [#12964](https://codecov.io/gh/apache/beam/pull/12964?src=pr=desc) into

[GitHub] [beam] piotr-szuberski commented on pull request #12975: Update Beam's io dependencies

2020-09-30 Thread GitBox
piotr-szuberski commented on pull request #12975: URL: https://github.com/apache/beam/pull/12975#issuecomment-701392442 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] codecov[bot] edited a comment on pull request #12976: [release-2.25.0][BEAM-10986] Rollback to shadow 4.0.3

2020-09-30 Thread GitBox
codecov[bot] edited a comment on pull request #12976: URL: https://github.com/apache/beam/pull/12976#issuecomment-701505415 # [Codecov](https://codecov.io/gh/apache/beam/pull/12976?src=pr=h1) Report > Merging [#12976](https://codecov.io/gh/apache/beam/pull/12976?src=pr=desc) into

[GitHub] [beam] mxm commented on pull request #12974: [BEAM-10986] Add shadow jar configuration to expansion-service.

2020-09-30 Thread GitBox
mxm commented on pull request #12974: URL: https://github.com/apache/beam/pull/12974#issuecomment-701516486 Should it be `io/expansion-service` instead of `sdks/java/expansion-service`? This is an automated message from the

[GitHub] [beam] pabloem merged pull request #12956: Add nexmark python query 10 to choices

2020-09-30 Thread GitBox
pabloem merged pull request #12956: URL: https://github.com/apache/beam/pull/12956 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [beam] codecov[bot] edited a comment on pull request #12976: [release-2.25.0][BEAM-10986] Rollback to shadow 4.0.3

2020-09-30 Thread GitBox
codecov[bot] edited a comment on pull request #12976: URL: https://github.com/apache/beam/pull/12976#issuecomment-701505415 # [Codecov](https://codecov.io/gh/apache/beam/pull/12976?src=pr=h1) Report > Merging [#12976](https://codecov.io/gh/apache/beam/pull/12976?src=pr=desc) into

[GitHub] [beam] codecov[bot] edited a comment on pull request #12957: [BEAM-10114] Upgrade version of pubsublite to 0.4.1.

2020-09-30 Thread GitBox
codecov[bot] edited a comment on pull request #12957: URL: https://github.com/apache/beam/pull/12957#issuecomment-700276403 # [Codecov](https://codecov.io/gh/apache/beam/pull/12957?src=pr=h1) Report > Merging [#12957](https://codecov.io/gh/apache/beam/pull/12957?src=pr=desc) into

[GitHub] [beam] kennknowles commented on pull request #12868: Fix up-to-date checking mechanism on Tox Task.

2020-09-30 Thread GitBox
kennknowles commented on pull request #12868: URL: https://github.com/apache/beam/pull/12868#issuecomment-701604539 Is it just that the XML report that Jenkins is looking for is not registered as an output? This is an

[GitHub] [beam] codecov[bot] edited a comment on pull request #12941: [BEAM-9681] Add textio.Read lesson to Go SDK katas

2020-09-30 Thread GitBox
codecov[bot] edited a comment on pull request #12941: URL: https://github.com/apache/beam/pull/12941#issuecomment-698987193 # [Codecov](https://codecov.io/gh/apache/beam/pull/12941?src=pr=h1) Report > Merging [#12941](https://codecov.io/gh/apache/beam/pull/12941?src=pr=desc) into

[GitHub] [beam] tysonjh commented on a change in pull request #11856: [BEAM-7505] SideInput Python Load Test job

2020-09-30 Thread GitBox
tysonjh commented on a change in pull request #11856: URL: https://github.com/apache/beam/pull/11856#discussion_r497786859 ## File path: sdks/python/apache_beam/testing/load_tests/sideinput_test.py ## @@ -122,85 +122,95 @@ def process(self, unused_element, side_input):

[GitHub] [beam] lostluck merged pull request #12941: [BEAM-9681] Add textio.Read lesson to Go SDK katas

2020-09-30 Thread GitBox
lostluck merged pull request #12941: URL: https://github.com/apache/beam/pull/12941 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [beam] kennknowles opened a new pull request #12980: [BEAM-10991] Revert "[BEAM-8543] Dataflow streaming timers are not strictly time ordered when set earlier mid-bundle (#11924)"

2020-09-30 Thread GitBox
kennknowles opened a new pull request #12980: URL: https://github.com/apache/beam/pull/12980 This reverts commit 88acc5267f759d81e9836a9db17b9e0ee521c785. There is a bug in this commit that causes deleted timers to not clear their watermark holds, resulting in stuck pipelines. See

[GitHub] [beam] lostluck commented on pull request #12505: [BEAM-8106] Add version to java container image name

2020-09-30 Thread GitBox
lostluck commented on pull request #12505: URL: https://github.com/apache/beam/pull/12505#issuecomment-701684906 Ack. Thanks! On Tue, Sep 29, 2020, 5:38 PM Brian Hulette wrote: > Thank you! > > — > You are receiving this because you were mentioned. > Reply to

[GitHub] [beam] kennknowles commented on pull request #12980: [BEAM-10991] Revert "[BEAM-8543] Dataflow streaming timers are not strictly time ordered when set earlier mid-bundle (#11924)"

2020-09-30 Thread GitBox
kennknowles commented on pull request #12980: URL: https://github.com/apache/beam/pull/12980#issuecomment-701684994 CC @robinyqiu we will want to cherrypick this This is an automated message from the Apache Git Service. To

[GitHub] [beam] aaltay commented on pull request #12980: [BEAM-10991] Revert "[BEAM-8543] Dataflow streaming timers are not strictly time ordered when set earlier mid-bundle (#11924)"

2020-09-30 Thread GitBox
aaltay commented on pull request #12980: URL: https://github.com/apache/beam/pull/12980#issuecomment-701691856 /cc @robinyqiu - This change also needs to go to the release branch. This is an automated message from the Apache

[GitHub] [beam] rohdesamuel opened a new pull request #12983: [BEAM-10994] Add hotKeyLoggingEnabled to the Java DataflowPipelineOptions

2020-09-30 Thread GitBox
rohdesamuel opened a new pull request #12983: URL: https://github.com/apache/beam/pull/12983 This will ultimately control whether or not the literal hot key content is logged to Cloud Logging. Thank you for your contribution! Follow this checklist to

[GitHub] [beam] kileys commented on a change in pull request #12970: [BEAM-8024] Add dataflow and flink runner to JPMS test

2020-09-30 Thread GitBox
kileys commented on a change in pull request #12970: URL: https://github.com/apache/beam/pull/12970#discussion_r497962184 ## File path: sdks/java/testing/jpms-tests/build.gradle ## @@ -31,13 +34,66 @@ enableJavaPerformanceTesting() description = "Apache Beam :: SDKs :: Java

[GitHub] [beam] y1chi commented on pull request #12562: [BEAM-10200] Respect profile_memory option and add memory profiler to…

2020-09-30 Thread GitBox
y1chi commented on pull request #12562: URL: https://github.com/apache/beam/pull/12562#issuecomment-701672888 @angoenka I think this PR can be merged This is an automated message from the Apache Git Service. To respond to

[GitHub] [beam] lostluck commented on pull request #12941: [BEAM-9681] Add textio.Read lesson to Go SDK katas

2020-09-30 Thread GitBox
lostluck commented on pull request #12941: URL: https://github.com/apache/beam/pull/12941#issuecomment-701672348 The only reason the python tests (and java tests) are running are because of the testdata expansion which isn't something that would have caused the tests to break. Given the

[GitHub] [beam] kennknowles commented on pull request #12980: [BEAM-10991] Revert "[BEAM-8543] Dataflow streaming timers are not strictly time ordered when set earlier mid-bundle (#11924)"

2020-09-30 Thread GitBox
kennknowles commented on pull request #12980: URL: https://github.com/apache/beam/pull/12980#issuecomment-701692388 run dataflow validatesrunner This is an automated message from the Apache Git Service. To respond to the

[GitHub] [beam] kennknowles commented on pull request #12981: [BEAM-10435] Add LOOPBACK mode Java ValidatesRunner test suite for ULR

2020-09-30 Thread GitBox
kennknowles commented on pull request #12981: URL: https://github.com/apache/beam/pull/12981#issuecomment-701701451 Pushing whitespace-only fixup This is an automated message from the Apache Git Service. To respond to the

[GitHub] [beam] kennknowles commented on pull request #12981: [BEAM-10435] Add LOOPBACK mode Java ValidatesRunner test suite for ULR

2020-09-30 Thread GitBox
kennknowles commented on pull request #12981: URL: https://github.com/apache/beam/pull/12981#issuecomment-701701366 Noting that the whitespace check is pulled in by `applyPythonNature` which is too big a hammer but refactoring will be some work. And it seems `./gradlew applySpotless` does

[GitHub] [beam] codecov[bot] commented on pull request #12385: [BEAM-10527] Migrate Flink and Spark tests to pytest.

2020-09-30 Thread GitBox
codecov[bot] commented on pull request #12385: URL: https://github.com/apache/beam/pull/12385#issuecomment-701745199 # [Codecov](https://codecov.io/gh/apache/beam/pull/12385?src=pr=h1) Report > Merging [#12385](https://codecov.io/gh/apache/beam/pull/12385?src=pr=desc) into

[GitHub] [beam] codecov[bot] edited a comment on pull request #12385: [BEAM-10527] Migrate Flink and Spark tests to pytest.

2020-09-30 Thread GitBox
codecov[bot] edited a comment on pull request #12385: URL: https://github.com/apache/beam/pull/12385#issuecomment-701745199 # [Codecov](https://codecov.io/gh/apache/beam/pull/12385?src=pr=h1) Report > Merging [#12385](https://codecov.io/gh/apache/beam/pull/12385?src=pr=desc) into

[GitHub] [beam] codecov[bot] edited a comment on pull request #12385: [BEAM-10527] Migrate Flink and Spark tests to pytest.

2020-09-30 Thread GitBox
codecov[bot] edited a comment on pull request #12385: URL: https://github.com/apache/beam/pull/12385#issuecomment-701745199 # [Codecov](https://codecov.io/gh/apache/beam/pull/12385?src=pr=h1) Report > Merging [#12385](https://codecov.io/gh/apache/beam/pull/12385?src=pr=desc) into

[GitHub] [beam] damondouglas commented on pull request #12941: [BEAM-9681] Add textio.Read lesson to Go SDK katas

2020-09-30 Thread GitBox
damondouglas commented on pull request #12941: URL: https://github.com/apache/beam/pull/12941#issuecomment-701647073 Hello @pabloem the [Python ("Run Python PreCommit")](https://ci-beam.apache.org/job/beam_PreCommit_Python_Commit/15464/) is what I was referring to that failed after

[GitHub] [beam] tvalentyn commented on a change in pull request #12927: Update some of the Python dependencies from Beam Dependency Check Report

2020-09-30 Thread GitBox
tvalentyn commented on a change in pull request #12927: URL: https://github.com/apache/beam/pull/12927#discussion_r497830780 ## File path: sdks/python/apache_beam/io/avroio.py ## @@ -627,11 +627,19 @@ def write_record(self, writer, value): writer.append(value) +class

[GitHub] [beam] tvalentyn commented on a change in pull request #12927: Update some of the Python dependencies from Beam Dependency Check Report

2020-09-30 Thread GitBox
tvalentyn commented on a change in pull request #12927: URL: https://github.com/apache/beam/pull/12927#discussion_r497830780 ## File path: sdks/python/apache_beam/io/avroio.py ## @@ -627,11 +627,19 @@ def write_record(self, writer, value): writer.append(value) +class

[GitHub] [beam] reuvenlax commented on pull request #12980: [BEAM-10991] Revert "[BEAM-8543] Dataflow streaming timers are not strictly time ordered when set earlier mid-bundle (#11924)"

2020-09-30 Thread GitBox
reuvenlax commented on pull request #12980: URL: https://github.com/apache/beam/pull/12980#issuecomment-701690047 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [beam] kennknowles commented on a change in pull request #12981: [BEAM-10435] Add LOOPBACK mode Java ValidatesRunner test suite for ULR

2020-09-30 Thread GitBox
kennknowles commented on a change in pull request #12981: URL: https://github.com/apache/beam/pull/12981#discussion_r497849873 ## File path: runners/portability/java/build.gradle ## @@ -50,27 +50,32 @@ dependencies { validatesRunner project(path: project.path,

[GitHub] [beam] tvalentyn commented on a change in pull request #12981: [BEAM-10435] Add LOOPBACK mode Java ValidatesRunner test suite for ULR

2020-09-30 Thread GitBox
tvalentyn commented on a change in pull request #12981: URL: https://github.com/apache/beam/pull/12981#discussion_r497854932 ## File path: runners/portability/java/build.gradle ## @@ -108,95 +120,113 @@ startLocalJobService.finalizedBy stopLocalJobService /** * Runs Java

[GitHub] [beam] robertwb opened a new pull request #12982: [BEAM-9547] Dataframe covariance and correlation.

2020-09-30 Thread GitBox
robertwb opened a new pull request #12982: URL: https://github.com/apache/beam/pull/12982 This implements correlation, covariance, and standard deviation for series and dataframes. It is split up into three commits. The first is the math-heavy one that implements the distributed

[GitHub] [beam] robertwb commented on pull request #12982: [BEAM-9547] Dataframe covariance and correlation.

2020-09-30 Thread GitBox
robertwb commented on pull request #12982: URL: https://github.com/apache/beam/pull/12982#issuecomment-701708339 R: @TheNeuralBit This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [beam] boyuanzz commented on pull request #12930: [BEAM-10962] Add Multiple PubSub reader to Python SDK

2020-09-30 Thread GitBox
boyuanzz commented on pull request #12930: URL: https://github.com/apache/beam/pull/12930#issuecomment-701708484 Expect what Cham has mentioned, another thing is current implementation of `MultipleReadFromPubSub` only can configure multiple `ReadFromPubSub` with the same attribute, like

[GitHub] [beam] boyuanzz edited a comment on pull request #12930: [BEAM-10962] Add Multiple PubSub reader to Python SDK

2020-09-30 Thread GitBox
boyuanzz edited a comment on pull request #12930: URL: https://github.com/apache/beam/pull/12930#issuecomment-701708484 Except what Cham has mentioned, another thing is current implementation of `MultipleReadFromPubSub` only can configure multiple `ReadFromPubSub` with the same attribute,

[GitHub] [beam] robertwb commented on pull request #12819: [BEAM-9561] Initial framework for testing pandas website docs.

2020-09-30 Thread GitBox
robertwb commented on pull request #12819: URL: https://github.com/apache/beam/pull/12819#issuecomment-701720434 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] codecov[bot] edited a comment on pull request #12819: [BEAM-9561] Initial framework for testing pandas website docs.

2020-09-30 Thread GitBox
codecov[bot] edited a comment on pull request #12819: URL: https://github.com/apache/beam/pull/12819#issuecomment-691336298 # [Codecov](https://codecov.io/gh/apache/beam/pull/12819?src=pr=h1) Report > Merging [#12819](https://codecov.io/gh/apache/beam/pull/12819?src=pr=desc) into

[GitHub] [beam] codecov[bot] edited a comment on pull request #12979: [BEAM-10545] Add 2 show options

2020-09-30 Thread GitBox
codecov[bot] edited a comment on pull request #12979: URL: https://github.com/apache/beam/pull/12979#issuecomment-701667582 # [Codecov](https://codecov.io/gh/apache/beam/pull/12979?src=pr=h1) Report > Merging [#12979](https://codecov.io/gh/apache/beam/pull/12979?src=pr=desc) into

[GitHub] [beam] pabloem commented on pull request #12941: [BEAM-9681] Add textio.Read lesson to Go SDK katas

2020-09-30 Thread GitBox
pabloem commented on pull request #12941: URL: https://github.com/apache/beam/pull/12941#issuecomment-701673167 I'm also okay to ignore. Feel free to merge~ This is an automated message from the Apache Git Service. To

[GitHub] [beam] KevinGG commented on pull request #12979: [BEAM-10545] Add 2 show options

2020-09-30 Thread GitBox
KevinGG commented on pull request #12979: URL: https://github.com/apache/beam/pull/12979#issuecomment-701673028 R: @prodonjs PTAL, thx! This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] kennknowles commented on a change in pull request #12981: [BEAM-10435] Add LOOPBACK mode Java ValidatesRunner test suite for ULR

2020-09-30 Thread GitBox
kennknowles commented on a change in pull request #12981: URL: https://github.com/apache/beam/pull/12981#discussion_r497856286 ## File path: runners/portability/java/build.gradle ## @@ -82,8 +87,15 @@ void execBackgroundInVirtualenv(String... args) { proc.waitFor(); }

[GitHub] [beam] kennknowles commented on a change in pull request #12981: [BEAM-10435] Add LOOPBACK mode Java ValidatesRunner test suite for ULR

2020-09-30 Thread GitBox
kennknowles commented on a change in pull request #12981: URL: https://github.com/apache/beam/pull/12981#discussion_r497856286 ## File path: runners/portability/java/build.gradle ## @@ -82,8 +87,15 @@ void execBackgroundInVirtualenv(String... args) { proc.waitFor(); }

[GitHub] [beam] kennknowles commented on a change in pull request #12981: [BEAM-10435] Add LOOPBACK mode Java ValidatesRunner test suite for ULR

2020-09-30 Thread GitBox
kennknowles commented on a change in pull request #12981: URL: https://github.com/apache/beam/pull/12981#discussion_r497856286 ## File path: runners/portability/java/build.gradle ## @@ -82,8 +87,15 @@ void execBackgroundInVirtualenv(String... args) { proc.waitFor(); }

[GitHub] [beam] tvalentyn commented on a change in pull request #12981: [BEAM-10435] Add LOOPBACK mode Java ValidatesRunner test suite for ULR

2020-09-30 Thread GitBox
tvalentyn commented on a change in pull request #12981: URL: https://github.com/apache/beam/pull/12981#discussion_r497856592 ## File path: runners/portability/java/build.gradle ## @@ -82,8 +87,15 @@ void execBackgroundInVirtualenv(String... args) { proc.waitFor(); }

[GitHub] [beam] kennknowles commented on pull request #12962: [WIP][BEAM-9379] Update calcite to 1.25

2020-09-30 Thread GitBox
kennknowles commented on pull request #12962: URL: https://github.com/apache/beam/pull/12962#issuecomment-701703218 https://scans.gradle.com/s/ofcfnu774du4y/tests/:sdks:java:extensions:sql:zetasql:test/org.apache.beam.sdk.extensions.sql.zetasql.ZetaSqlTimeFunctionsTest/testFormatDate#1

[GitHub] [beam] kennknowles commented on pull request #12962: [WIP][BEAM-9379] Update calcite to 1.25

2020-09-30 Thread GitBox
kennknowles commented on pull request #12962: URL: https://github.com/apache/beam/pull/12962#issuecomment-701703585 https://scans.gradle.com/s/ofcfnu774du4y/tests/:sdks:java:extensions:sql:test/org.apache.beam.sdk.extensions.sql.impl.rel.BeamMatchRelTest/matchPrevFunctionTest#1

[GitHub] [beam] rohdesamuel opened a new pull request #12986: [BEAM-10994] Adds ability to log literal hot key in Streaming Dataflow

2020-09-30 Thread GitBox
rohdesamuel opened a new pull request #12986: URL: https://github.com/apache/beam/pull/12986 The logging of the key is controlled by the DataflowPipelineOption `hotKeyLoggingEnabled`. When true and a hot key is detected from Dataflow, the literal hot key content will be logged to Cloud

[GitHub] [beam] rohdesamuel opened a new pull request #12985: [BEAM-10994] Adds ability to log literal hot key in Batch Dataflow

2020-09-30 Thread GitBox
rohdesamuel opened a new pull request #12985: URL: https://github.com/apache/beam/pull/12985 The logging of the key is controlled by the DataflowPipelineOption `hotKeyLoggingEnabled`. When true and a hot key is detected from Dataflow, the literal hot key content will be logged to Cloud

[GitHub] [beam] veblush commented on pull request #12965: [BEAM-5748] Upgrade conscrypt to latest version

2020-09-30 Thread GitBox
veblush commented on pull request #12965: URL: https://github.com/apache/beam/pull/12965#issuecomment-701646445 Awesome! I'm about to make the same PR but you did it. Thanks! FYI, this is required for gRPC ALTS to have better throughput by saving CPU time significantly. Notable PRs

[GitHub] [beam] codecov[bot] edited a comment on pull request #12941: [BEAM-9681] Add textio.Read lesson to Go SDK katas

2020-09-30 Thread GitBox
codecov[bot] edited a comment on pull request #12941: URL: https://github.com/apache/beam/pull/12941#issuecomment-698987193 # [Codecov](https://codecov.io/gh/apache/beam/pull/12941?src=pr=h1) Report > Merging [#12941](https://codecov.io/gh/apache/beam/pull/12941?src=pr=desc) into

[GitHub] [beam] codecov[bot] edited a comment on pull request #12979: [BEAM-10545] Add 2 show options

2020-09-30 Thread GitBox
codecov[bot] edited a comment on pull request #12979: URL: https://github.com/apache/beam/pull/12979#issuecomment-701667582 # [Codecov](https://codecov.io/gh/apache/beam/pull/12979?src=pr=h1) Report > Merging [#12979](https://codecov.io/gh/apache/beam/pull/12979?src=pr=desc) into

[GitHub] [beam] kennknowles opened a new pull request #12981: [BEAM-10435] Add LOOPBACK mode Java ValidatesRunner test suite for ULR

2020-09-30 Thread GitBox
kennknowles opened a new pull request #12981: URL: https://github.com/apache/beam/pull/12981 This just adds the additional gradle target. TODO: add continuous Jenkins runs. It looks like it takes long enough it has to be a postcommit. Thank you for

[GitHub] [beam] tvalentyn commented on a change in pull request #12981: [BEAM-10435] Add LOOPBACK mode Java ValidatesRunner test suite for ULR

2020-09-30 Thread GitBox
tvalentyn commented on a change in pull request #12981: URL: https://github.com/apache/beam/pull/12981#discussion_r497854011 ## File path: runners/portability/java/build.gradle ## @@ -82,8 +87,15 @@ void execBackgroundInVirtualenv(String... args) { proc.waitFor(); }

[GitHub] [beam] kennknowles commented on pull request #12962: [WIP][BEAM-9379] Update calcite to 1.25

2020-09-30 Thread GitBox
kennknowles commented on pull request #12962: URL: https://github.com/apache/beam/pull/12962#issuecomment-701702697 https://scans.gradle.com/s/ofcfnu774du4y/tests/:sdks:java:extensions:sql:test/org.apache.beam.sdk.extensions.sql.BeamSqlDslUdfUdafTest/testUdfWithListInput#1 Seems

[GitHub] [beam] codecov[bot] commented on pull request #12981: [BEAM-10435] Add LOOPBACK mode Java ValidatesRunner test suite for ULR

2020-09-30 Thread GitBox
codecov[bot] commented on pull request #12981: URL: https://github.com/apache/beam/pull/12981#issuecomment-701714122 # [Codecov](https://codecov.io/gh/apache/beam/pull/12981?src=pr=h1) Report > Merging [#12981](https://codecov.io/gh/apache/beam/pull/12981?src=pr=desc) into

[GitHub] [beam] frankzhao commented on pull request #12960: [BEAM-9804] Allow user configuration of BigQuery temporary dataset

2020-09-30 Thread GitBox
frankzhao commented on pull request #12960: URL: https://github.com/apache/beam/pull/12960#issuecomment-701738038 R: @pabloem This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [beam] pabloem commented on pull request #12941: [BEAM-9681] Add textio.Read lesson to Go SDK katas

2020-09-30 Thread GitBox
pabloem commented on pull request #12941: URL: https://github.com/apache/beam/pull/12941#issuecomment-701649639 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [beam] pabloem commented on pull request #12941: [BEAM-9681] Add textio.Read lesson to Go SDK katas

2020-09-30 Thread GitBox
pabloem commented on pull request #12941: URL: https://github.com/apache/beam/pull/12941#issuecomment-701649895 a similar -though not the same- test is failing on the main branch: https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/

[GitHub] [beam] codecov[bot] commented on pull request #12979: [BEAM-10545] Add 2 show options

2020-09-30 Thread GitBox
codecov[bot] commented on pull request #12979: URL: https://github.com/apache/beam/pull/12979#issuecomment-701667582 # [Codecov](https://codecov.io/gh/apache/beam/pull/12979?src=pr=h1) Report > Merging [#12979](https://codecov.io/gh/apache/beam/pull/12979?src=pr=desc) into

[GitHub] [beam] kennknowles commented on pull request #12981: [BEAM-10435] Add LOOPBACK mode Java ValidatesRunner test suite for ULR

2020-09-30 Thread GitBox
kennknowles commented on pull request #12981: URL: https://github.com/apache/beam/pull/12981#issuecomment-701699000 https://gradle.com/s/r6phv7qxmg6so This is an automated message from the Apache Git Service. To respond to

[GitHub] [beam] kennknowles commented on a change in pull request #12981: [BEAM-10435] Add LOOPBACK mode Java ValidatesRunner test suite for ULR

2020-09-30 Thread GitBox
kennknowles commented on a change in pull request #12981: URL: https://github.com/apache/beam/pull/12981#discussion_r497855759 ## File path: runners/portability/java/build.gradle ## @@ -108,95 +120,113 @@ startLocalJobService.finalizedBy stopLocalJobService /** * Runs

[GitHub] [beam] kennknowles commented on a change in pull request #12981: [BEAM-10435] Add LOOPBACK mode Java ValidatesRunner test suite for ULR

2020-09-30 Thread GitBox
kennknowles commented on a change in pull request #12981: URL: https://github.com/apache/beam/pull/12981#discussion_r497855885 ## File path: runners/portability/java/build.gradle ## @@ -82,8 +87,15 @@ void execBackgroundInVirtualenv(String... args) { proc.waitFor(); }

[GitHub] [beam] kennknowles commented on pull request #12962: [WIP][BEAM-9379] Update calcite to 1.25

2020-09-30 Thread GitBox
kennknowles commented on pull request #12962: URL: https://github.com/apache/beam/pull/12962#issuecomment-701703858 https://scans.gradle.com/s/ofcfnu774du4y/tests/:sdks:java:extensions:sql:test/org.apache.beam.sdk.extensions.sql.BeamSqlDslAggregationTest/testWindowOnNonTimestampField#1

[GitHub] [beam] kennknowles commented on pull request #12962: [WIP][BEAM-9379] Update calcite to 1.25

2020-09-30 Thread GitBox
kennknowles commented on pull request #12962: URL: https://github.com/apache/beam/pull/12962#issuecomment-701704059 https://scans.gradle.com/s/ofcfnu774du4y/tests/:sdks:java:extensions:sql:test/org.apache.beam.sdk.extensions.sql.impl.udf.BeamSqlUdfExpressionTest/testSHA1?anchor=e30=0-1-0#1

[GitHub] [beam] rohdesamuel opened a new pull request #12984: [BEAM-10994] Add ability for HotKeyLogger to log a key

2020-09-30 Thread GitBox
rohdesamuel opened a new pull request #12984: URL: https://github.com/apache/beam/pull/12984 Adds interface for HotKeyLogger to take an object key. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly

[GitHub] [beam] frankzhao removed a comment on pull request #12960: [BEAM-9804] Allow user configuration of BigQuery temporary dataset

2020-09-30 Thread GitBox
frankzhao removed a comment on pull request #12960: URL: https://github.com/apache/beam/pull/12960#issuecomment-700713637 retest this please This is an automated message from the Apache Git Service. To respond to the

[GitHub] [beam] codecov[bot] edited a comment on pull request #12819: [BEAM-9561] Initial framework for testing pandas website docs.

2020-09-30 Thread GitBox
codecov[bot] edited a comment on pull request #12819: URL: https://github.com/apache/beam/pull/12819#issuecomment-691336298 # [Codecov](https://codecov.io/gh/apache/beam/pull/12819?src=pr=h1) Report > Merging [#12819](https://codecov.io/gh/apache/beam/pull/12819?src=pr=desc) into

[GitHub] [beam] KevinGG opened a new pull request #12979: [BEAM-10545] Add 2 show options

2020-09-30 Thread GitBox
KevinGG opened a new pull request #12979: URL: https://github.com/apache/beam/pull/12979 1. Added 2 show API options in the side panel: duration and n. 2. Added an `apply` button to alter the behavior of setting show API options: now the configuration change only takes effect after

[GitHub] [beam] angoenka merged pull request #12562: [BEAM-10200] Respect profile_memory option and add memory profiler to…

2020-09-30 Thread GitBox
angoenka merged pull request #12562: URL: https://github.com/apache/beam/pull/12562 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [beam] boyuanzz closed pull request #12926: [WIP] Clean up with_attributes from PubsubSource

2020-09-30 Thread GitBox
boyuanzz closed pull request #12926: URL: https://github.com/apache/beam/pull/12926 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [beam] kennknowles commented on pull request #12962: [WIP][BEAM-9379] Update calcite to 1.25

2020-09-30 Thread GitBox
kennknowles commented on pull request #12962: URL: https://github.com/apache/beam/pull/12962#issuecomment-701702361 https://scans.gradle.com/s/ofcfnu774du4y/tests/:sdks:java:extensions:sql:test/org.apache.beam.sdk.extensions.sql.BeamSqlDslSqlStdOperatorsTest/testArithmeticOperator#1

[GitHub] [beam] boyuanzz commented on pull request #12926: [WIP] Clean up with_attributes from PubsubSource

2020-09-30 Thread GitBox
boyuanzz commented on pull request #12926: URL: https://github.com/apache/beam/pull/12926#issuecomment-701702337 I'm going to close this PR since it brings performance regression. This is an automated message from the Apache

[GitHub] [beam] tvalentyn commented on a change in pull request #12981: [BEAM-10435] Add LOOPBACK mode Java ValidatesRunner test suite for ULR

2020-09-30 Thread GitBox
tvalentyn commented on a change in pull request #12981: URL: https://github.com/apache/beam/pull/12981#discussion_r497864103 ## File path: runners/portability/java/build.gradle ## @@ -82,8 +87,15 @@ void execBackgroundInVirtualenv(String... args) { proc.waitFor(); }

  1   2   >