[GitHub] [beam] lukecwik commented on a change in pull request #12511: Upgrade to Netty 4.1.50.Final and Netty_tcnative 2.0.31.Final

lukecwik commented on a change in pull request #12511: URL: https://github.com/apache/beam/pull/12511#discussion_r468747932 ## File path: buildSrc/src/main/groovy/org/apache/beam/gradle/GrpcVendoring_1_26_0.groovy ## @@ -27,7 +27,7 @@ class GrpcVendoring_1_26_0 { static

[GitHub] [beam] KevinGG commented on a change in pull request #12460: [BEAM-10545] HtmlView module

KevinGG commented on a change in pull request #12460: URL: https://github.com/apache/beam/pull/12460#discussion_r468740290 ## File path: sdks/python/apache_beam/runners/interactive/extensions/apache-beam-jupyterlab-sidepanel/src/common/HtmlView.tsx ## @@ -0,0 +1,119 @@ +//

[GitHub] [beam] TheNeuralBit commented on a change in pull request #12526: [BEAM-10663] Disable python kafka integration tests

TheNeuralBit commented on a change in pull request #12526: URL: https://github.com/apache/beam/pull/12526#discussion_r468789977 ## File path: sdks/python/apache_beam/io/external/xlang_kafkaio_it_test.py ## @@ -94,6 +94,7 @@ def run_xlang_kafkaio(self, pipeline):

[GitHub] [beam] TheNeuralBit merged pull request #12530: [BEAM-7996] Add Python SqlTransform test that includes a MAP input and output

TheNeuralBit merged pull request #12530: URL: https://github.com/apache/beam/pull/12530 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [beam] kkucharc commented on a change in pull request #12435: [BEAM-10616] Added Python Pardo load tests for streaming on Dataflow

kkucharc commented on a change in pull request #12435: URL: https://github.com/apache/beam/pull/12435#discussion_r468807769 ## File path: .test-infra/jenkins/job_LoadTests_ParDo_Python.groovy ## @@ -151,3 +151,35 @@

[GitHub] [beam] emilymye removed a comment on pull request #12505: [WIP][BEAM-8106] Add version to java container image name

emilymye removed a comment on pull request #12505: URL: https://github.com/apache/beam/pull/12505#issuecomment-671632378 Run Python2_PVR_Flink PreCommit This is an automated message from the Apache Git Service. To

[GitHub] [beam] emilymye commented on pull request #12505: [WIP][BEAM-8106] Add version to java container image name

emilymye commented on pull request #12505: URL: https://github.com/apache/beam/pull/12505#issuecomment-672207756 retest this please This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [beam] TheNeuralBit commented on pull request #12526: [BEAM-10663] Disable python kafka integration tests

TheNeuralBit commented on pull request #12526: URL: https://github.com/apache/beam/pull/12526#issuecomment-672235923 Looks like the pending jobs are actually complete, they just haven't updated the PR. Merging now. This is

[GitHub] [beam] robertwb commented on a change in pull request #12516: [BEAM-9547] Implement dataframes top, join, merge.

robertwb commented on a change in pull request #12516: URL: https://github.com/apache/beam/pull/12516#discussion_r468862256 ## File path: sdks/python/apache_beam/dataframe/frames.py ## @@ -54,6 +54,42 @@ def agg(self, *args, **kwargs): 'order-sensitive') diff =

[GitHub] [beam] TobKed commented on a change in pull request #12452: [BEAM-10623] Add workflow to run Beam python tests on Linux/Windows/Mac platforms

TobKed commented on a change in pull request #12452: URL: https://github.com/apache/beam/pull/12452#discussion_r468877557 ## File path: CI.md ## @@ -75,8 +75,28 @@ run categories. Here is a summary of the run categories with regards of the jobs Those jobs often have matrix

[GitHub] [beam] tvalentyn edited a comment on pull request #12452: [BEAM-10623] Add workflow to run Beam python tests on Linux/Windows/Mac platforms

tvalentyn edited a comment on pull request #12452: URL: https://github.com/apache/beam/pull/12452#issuecomment-672296133 Thanks, @TobKed , I don't have additional comments. Once all reviewers, whose feedback you expect, give their LGTM, please squash fixup commits, and leave only the

[GitHub] [beam] tvalentyn commented on pull request #12452: [BEAM-10623] Add workflow to run Beam python tests on Linux/Windows/Mac platforms

tvalentyn commented on pull request #12452: URL: https://github.com/apache/beam/pull/12452#issuecomment-672296133 Thanks, @TobKed , I don't have additional comments. Once all reviewers, whose feedback you expect, gave their LGTM, please squash fixup commits, and leave only the commits

[GitHub] [beam] sclukas77 commented on a change in pull request #12498: [BEAM-10654] Implemented ExternalSchemaIOTransformRegistrar for jdbc

sclukas77 commented on a change in pull request #12498: URL: https://github.com/apache/beam/pull/12498#discussion_r468720134 ## File path:

[GitHub] [beam] mxm commented on pull request #12481: [BEAM-10571] Use schemas in ExternalConfigurationPayload

mxm commented on pull request #12481: URL: https://github.com/apache/beam/pull/12481#issuecomment-672083732 Run XVR_Direct PostCommit This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] danielxjd commented on pull request #12223: [Beam-4379] Make ParquetIO read splittable

danielxjd commented on pull request #12223: URL: https://github.com/apache/beam/pull/12223#issuecomment-672092490 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [beam] TheNeuralBit commented on pull request #12481: [BEAM-10571] Use schemas in ExternalConfigurationPayload

TheNeuralBit commented on pull request #12481: URL: https://github.com/apache/beam/pull/12481#issuecomment-672106456 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [beam] aaltay merged pull request #12528: Extending archiveJunit Jenkins post-commit task with stability history

aaltay merged pull request #12528: URL: https://github.com/apache/beam/pull/12528 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [beam] boyuanzz edited a comment on pull request #12518: [BEAM-10663] Workaround of AutoValueSchema doesn't work with SchemaFieldName

boyuanzz edited a comment on pull request #12518: URL: https://github.com/apache/beam/pull/12518#issuecomment-672158339 > Will it be possible to re-enable the Kafka test on direct runner? I don't think we have Kafka test on direct runner. Here is the only place I can find to run

[GitHub] [beam] KevinGG commented on a change in pull request #12460: [BEAM-10545] HtmlView module

KevinGG commented on a change in pull request #12460: URL: https://github.com/apache/beam/pull/12460#discussion_r468778647 ## File path: sdks/python/apache_beam/runners/interactive/extensions/apache-beam-jupyterlab-sidepanel/src/common/HtmlView.tsx ## @@ -0,0 +1,119 @@ +//

[GitHub] [beam] kennknowles commented on a change in pull request #12521: [BEAM-8125] Add verifyDeterministic test to SchemaCoderTest

kennknowles commented on a change in pull request #12521: URL: https://github.com/apache/beam/pull/12521#discussion_r468783212 ## File path: sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/SchemaCoderTest.java ## @@ -290,5 +329,31 @@ public void

[GitHub] [beam] TheNeuralBit commented on pull request #12522: Fix format string in PipelineValidator

TheNeuralBit commented on pull request #12522: URL: https://github.com/apache/beam/pull/12522#issuecomment-672195554 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] robertwb commented on pull request #12516: [BEAM-9547] Implement dataframes top, join, merge.

robertwb commented on pull request #12516: URL: https://github.com/apache/beam/pull/12516#issuecomment-672277371 > This is going to take me a while to review since I have very little Python experience and even less Panda experience. If you want a more thorough, faster review, I suggest

[GitHub] [beam] KevinGG commented on a change in pull request #12460: [BEAM-10545] HtmlView module

KevinGG commented on a change in pull request #12460: URL: https://github.com/apache/beam/pull/12460#discussion_r468764276 ## File path: sdks/python/apache_beam/runners/interactive/extensions/apache-beam-jupyterlab-sidepanel/src/common/HtmlView.tsx ## @@ -0,0 +1,119 @@ +//

[GitHub] [beam] damgad commented on pull request #12529: Moving /tmp directory cleanup of CI workers to Inventory Jenkins job

damgad commented on pull request #12529: URL: https://github.com/apache/beam/pull/12529#issuecomment-672145185 R: @aaltay This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [beam] TheNeuralBit commented on pull request #12518: [BEAM-10663] Workaround of AutoValueSchema doesn't work with SchemaFieldName

TheNeuralBit commented on pull request #12518: URL: https://github.com/apache/beam/pull/12518#issuecomment-672156135 Will it be possible to re-enable the Kafka test on direct runner? This is an automated message from the

[GitHub] [beam] boyuanzz commented on pull request #12518: [BEAM-10663] Workaround of AutoValueSchema doesn't work with SchemaFieldName

boyuanzz commented on pull request #12518: URL: https://github.com/apache/beam/pull/12518#issuecomment-672201149 > Is bundle finalization not an issue for DirectRunner ? If so we should be able to run the Kafka test on DirectRunner. Within https://github.com/apache/beam/pull/12488,

[GitHub] [beam] ihji commented on pull request #12533: [BEAM-10679] improving XLang KafkaIO streaming test

ihji commented on pull request #12533: URL: https://github.com/apache/beam/pull/12533#issuecomment-672209854 R: @chamikaramj This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [beam] TheNeuralBit commented on a change in pull request #12521: [BEAM-8125] Add verifyDeterministic test to SchemaCoderTest

TheNeuralBit commented on a change in pull request #12521: URL: https://github.com/apache/beam/pull/12521#discussion_r468822570 ## File path: sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/SchemaCoderTest.java ## @@ -290,5 +329,31 @@ public void

[GitHub] [beam] chamikaramj commented on a change in pull request #12485: [BEAM-6064] Improvements to BQ streaming insert performance

chamikaramj commented on a change in pull request #12485: URL: https://github.com/apache/beam/pull/12485#discussion_r468830116 ## File path: sdks/python/apache_beam/io/gcp/bigquery.py ## @@ -304,6 +308,8 @@ def compute_table_name(row): NOTE: This job name template does not

[GitHub] [beam] chamikaramj commented on a change in pull request #12485: [BEAM-6064] Improvements to BQ streaming insert performance

chamikaramj commented on a change in pull request #12485: URL: https://github.com/apache/beam/pull/12485#discussion_r468829856 ## File path: sdks/python/apache_beam/io/gcp/bigquery.py ## @@ -304,6 +308,8 @@ def compute_table_name(row): NOTE: This job name template does not

[GitHub] [beam] chamikaramj commented on a change in pull request #12485: [BEAM-6064] Improvements to BQ streaming insert performance

chamikaramj commented on a change in pull request #12485: URL: https://github.com/apache/beam/pull/12485#discussion_r468830450 ## File path: sdks/python/apache_beam/io/gcp/bigquery.py ## @@ -304,6 +308,8 @@ def compute_table_name(row): NOTE: This job name template does not

[GitHub] [beam] pabloem commented on pull request #12489: [BEAM-6064] Add an option to avoid insert_ids on BQ in exchange for faster insertions

pabloem commented on pull request #12489: URL: https://github.com/apache/beam/pull/12489#issuecomment-672280905 Run Python 3.8 PostCommit This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] robinyqiu opened a new pull request #12536: [BEAM-10611] Simplification: Use new ZetaSQL API to get/create Values

robinyqiu opened a new pull request #12536: URL: https://github.com/apache/beam/pull/12536 Simplify getting/creating ZetaSQL values using the new API. r: @apilloud @ZijieSong946 Post-Commit Tests Status (on master branch)

[GitHub] [beam] KevinGG commented on a change in pull request #12444: Added a whitespace lint as part of python lint precommit

KevinGG commented on a change in pull request #12444: URL: https://github.com/apache/beam/pull/12444#discussion_r468881675 ## File path: sdks/python/scripts/run_whitespacelint.sh ## @@ -0,0 +1,32 @@ +#!/bin/bash +# +#Licensed to the Apache Software Foundation (ASF) under

[GitHub] [beam] mxm commented on a change in pull request #12481: [BEAM-10571] Use schemas in ExternalConfigurationPayload

mxm commented on a change in pull request #12481: URL: https://github.com/apache/beam/pull/12481#discussion_r467556781 ## File path: sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java ## @@ -1452,12 +1446,12 @@ public void

[GitHub] [beam] mxm commented on pull request #12481: [BEAM-10571] Use schemas in ExternalConfigurationPayload

mxm commented on pull request #12481: URL: https://github.com/apache/beam/pull/12481#issuecomment-672083483 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[GitHub] [beam] boyuanzz commented on a change in pull request #12526: [BEAM-10663] Disable python kafka integration tests

boyuanzz commented on a change in pull request #12526: URL: https://github.com/apache/beam/pull/12526#discussion_r468724341 ## File path: sdks/python/apache_beam/io/external/xlang_kafkaio_it_test.py ## @@ -94,6 +94,7 @@ def run_xlang_kafkaio(self, pipeline):

[GitHub] [beam] tysonjh commented on a change in pull request #12435: [BEAM-10616] Added Python Pardo load tests for streaming on Dataflow

tysonjh commented on a change in pull request #12435: URL: https://github.com/apache/beam/pull/12435#discussion_r468749356 ## File path: sdks/python/apache_beam/testing/load_tests/pardo_test.py ## @@ -125,7 +125,9 @@ def process(self, element, state=state_param):

[GitHub] [beam] lukecwik commented on pull request #12519: [BEAM-10670] Make Read execute as a splittable DoFn by default for the Java DirectRunner.

lukecwik commented on pull request #12519: URL: https://github.com/apache/beam/pull/12519#issuecomment-672146059 Run JavaPortabilityApiJava11 PreCommit This is an automated message from the Apache Git Service. To respond to

[GitHub] [beam] ihji opened a new pull request #12533: [BEAM-10679] improving XLang KafkaIO streaming test

ihji opened a new pull request #12533: URL: https://github.com/apache/beam/pull/12533 Using state instead of early triggering and combine per key. This would reduce the flakiness in the streaming test. Thank you for your contribution! Follow this

[GitHub] [beam] ihji commented on pull request #12533: [BEAM-10679] improving XLang KafkaIO streaming test

ihji commented on pull request #12533: URL: https://github.com/apache/beam/pull/12533#issuecomment-672209441 Run Python 3.5 PostCommit This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] TheNeuralBit merged pull request #12526: [BEAM-10663] Disable python kafka integration tests

TheNeuralBit merged pull request #12526: URL: https://github.com/apache/beam/pull/12526 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [beam] Ardagan commented on pull request #12139: [DO NOT REVIEW] Scd py bq test

Ardagan commented on pull request #12139: URL: https://github.com/apache/beam/pull/12139#issuecomment-672240516 run python postcommit This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] pabloem commented on a change in pull request #12485: [BEAM-6064] Improvements to BQ streaming insert performance

pabloem commented on a change in pull request #12485: URL: https://github.com/apache/beam/pull/12485#discussion_r468842890 ## File path: sdks/python/apache_beam/io/gcp/bigquery.py ## @@ -304,6 +308,8 @@ def compute_table_name(row): NOTE: This job name template does not have

[GitHub] [beam] pabloem commented on pull request #12492: [BEAM-6807] Implement an Azure blobstore filesystem for Python SDK

pabloem commented on pull request #12492: URL: https://github.com/apache/beam/pull/12492#issuecomment-672273518 Run Python2_PVR_Flink PreCommit This is an automated message from the Apache Git Service. To respond to the

[GitHub] [beam] KevinGG commented on a change in pull request #12460: [BEAM-10545] HtmlView module

KevinGG commented on a change in pull request #12460: URL: https://github.com/apache/beam/pull/12460#discussion_r468764276 ## File path: sdks/python/apache_beam/runners/interactive/extensions/apache-beam-jupyterlab-sidepanel/src/common/HtmlView.tsx ## @@ -0,0 +1,119 @@ +//

[GitHub] [beam] aaltay commented on pull request #12528: Extending archiveJunit Jenkins post-commit task with stability history

aaltay commented on pull request #12528: URL: https://github.com/apache/beam/pull/12528#issuecomment-672142551 /cc @tysonjh This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [beam] TheNeuralBit commented on pull request #12518: [BEAM-10663] Workaround of AutoValueSchema doesn't work with SchemaFieldName

TheNeuralBit commented on pull request #12518: URL: https://github.com/apache/beam/pull/12518#issuecomment-672181034 Ah ok. Maybe we should just change that suite to use the DirectRunner? I'm not sure why it's using Flink CC: @chamikaramj, @ihji in case they know

[GitHub] [beam] chamikaramj commented on pull request #12518: [BEAM-10663] Workaround of AutoValueSchema doesn't work with SchemaFieldName

chamikaramj commented on pull request #12518: URL: https://github.com/apache/beam/pull/12518#issuecomment-672191125 Is bundle finalization not an issue for DirectRunner ? If so we should be able to run the Kafka test on DirectRunner.

[GitHub] [beam] pabloem commented on a change in pull request #12489: [BEAM-6064] Add an option to avoid insert_ids on BQ in exchange for faster insertions

pabloem commented on a change in pull request #12489: URL: https://github.com/apache/beam/pull/12489#discussion_r468825940 ## File path: sdks/python/apache_beam/io/gcp/bigquery.py ## @@ -304,6 +308,8 @@ def compute_table_name(row): NOTE: This job name template does not have

[GitHub] [beam] pabloem commented on a change in pull request #12489: [BEAM-6064] Add an option to avoid insert_ids on BQ in exchange for faster insertions

pabloem commented on a change in pull request #12489: URL: https://github.com/apache/beam/pull/12489#discussion_r468843122 ## File path: sdks/python/apache_beam/io/gcp/bigquery.py ## @@ -1048,6 +1055,11 @@ def __init__( to be passed when creating a BigQuery table.

[GitHub] [beam] tysonjh commented on a change in pull request #12516: [BEAM-9547] Implement dataframes top, join, merge.

tysonjh commented on a change in pull request #12516: URL: https://github.com/apache/beam/pull/12516#discussion_r468707336 ## File path: sdks/python/apache_beam/dataframe/frames.py ## @@ -54,6 +54,42 @@ def agg(self, *args, **kwargs): 'order-sensitive') diff =

[GitHub] [beam] KevinGG commented on a change in pull request #12460: [BEAM-10545] HtmlView module

KevinGG commented on a change in pull request #12460: URL: https://github.com/apache/beam/pull/12460#discussion_r468740290 ## File path: sdks/python/apache_beam/runners/interactive/extensions/apache-beam-jupyterlab-sidepanel/src/common/HtmlView.tsx ## @@ -0,0 +1,119 @@ +//

[GitHub] [beam] KevinGG commented on a change in pull request #12460: [BEAM-10545] HtmlView module

KevinGG commented on a change in pull request #12460: URL: https://github.com/apache/beam/pull/12460#discussion_r468764276 ## File path: sdks/python/apache_beam/runners/interactive/extensions/apache-beam-jupyterlab-sidepanel/src/common/HtmlView.tsx ## @@ -0,0 +1,119 @@ +//

[GitHub] [beam] KevinGG commented on a change in pull request #12460: [BEAM-10545] HtmlView module

KevinGG commented on a change in pull request #12460: URL: https://github.com/apache/beam/pull/12460#discussion_r468764276 ## File path: sdks/python/apache_beam/runners/interactive/extensions/apache-beam-jupyterlab-sidepanel/src/common/HtmlView.tsx ## @@ -0,0 +1,119 @@ +//

[GitHub] [beam] kennknowles merged pull request #12366: [BEAM-10572] Eliminate nullability errors from :sdks:java:extensions:sql:datacatalog

kennknowles merged pull request #12366: URL: https://github.com/apache/beam/pull/12366 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [beam] pabloem commented on a change in pull request #12485: [BEAM-6064] Improvements to BQ streaming insert performance

pabloem commented on a change in pull request #12485: URL: https://github.com/apache/beam/pull/12485#discussion_r468825099 ## File path: sdks/python/apache_beam/io/gcp/bigquery.py ## @@ -1419,7 +1448,18 @@ def __init__( Default is to retry always. This means that

[GitHub] [beam] pabloem commented on a change in pull request #12485: [BEAM-6064] Improvements to BQ streaming insert performance

pabloem commented on a change in pull request #12485: URL: https://github.com/apache/beam/pull/12485#discussion_r468825677 ## File path: sdks/python/apache_beam/io/gcp/bigquery.py ## @@ -304,6 +308,8 @@ def compute_table_name(row): NOTE: This job name template does not have

[GitHub] [beam] TheNeuralBit commented on pull request #12481: [BEAM-10571] Use schemas in ExternalConfigurationPayload

TheNeuralBit commented on pull request #12481: URL: https://github.com/apache/beam/pull/12481#issuecomment-672256723 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] TheNeuralBit commented on pull request #12522: Fix format string in PipelineValidator

TheNeuralBit commented on pull request #12522: URL: https://github.com/apache/beam/pull/12522#issuecomment-672263017 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] pabloem merged pull request #12485: [BEAM-6064] Improvements to BQ streaming insert performance

pabloem merged pull request #12485: URL: https://github.com/apache/beam/pull/12485 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [beam] TheNeuralBit merged pull request #12521: [BEAM-8125] Add verifyDeterministic test to SchemaCoderTest

TheNeuralBit merged pull request #12521: URL: https://github.com/apache/beam/pull/12521 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [beam] ihji removed a comment on pull request #12533: [BEAM-10679] improving XLang KafkaIO streaming test

ihji removed a comment on pull request #12533: URL: https://github.com/apache/beam/pull/12533#issuecomment-672209441 Run Python 3.5 PostCommit This is an automated message from the Apache Git Service. To respond to the

[GitHub] [beam] TobKed commented on a change in pull request #12452: [BEAM-10623] Add workflow to run Beam python tests on Linux/Windows/Mac platforms

TobKed commented on a change in pull request #12452: URL: https://github.com/apache/beam/pull/12452#discussion_r468876965 ## File path: CI.md ## @@ -75,8 +75,28 @@ run categories. Here is a summary of the run categories with regards of the jobs Those jobs often have matrix

[GitHub] [beam] TobKed commented on a change in pull request #12452: [BEAM-10623] Add workflow to run Beam python tests on Linux/Windows/Mac platforms

TobKed commented on a change in pull request #12452: URL: https://github.com/apache/beam/pull/12452#discussion_r468877500 ## File path: CI.md ## @@ -75,8 +75,28 @@ run categories. Here is a summary of the run categories with regards of the jobs Those jobs often have matrix

[GitHub] [beam] pabloem commented on a change in pull request #12492: [BEAM-6807] Implement an Azure blobstore filesystem for Python SDK

pabloem commented on a change in pull request #12492: URL: https://github.com/apache/beam/pull/12492#discussion_r468877314 ## File path: sdks/python/apache_beam/io/azure/blobstoragefilesystem_test.py ## @@ -0,0 +1,315 @@ +# -*- coding: utf-8 -*- +# +# Licensed to the Apache

[GitHub] [beam] mxm commented on pull request #12385: [BEAM-10527] Migrate Flink and Spark tests to pytest.

mxm commented on pull request #12385: URL: https://github.com/apache/beam/pull/12385#issuecomment-672090815 > > > I have already spent a long time trying to fix quotes, so I can't help but wondering: why do we need flinkCompatibilityMatrixPROCESS in the first place, when it is not being

[GitHub] [beam] mxm edited a comment on pull request #12385: [BEAM-10527] Migrate Flink and Spark tests to pytest.

mxm edited a comment on pull request #12385: URL: https://github.com/apache/beam/pull/12385#issuecomment-672090815 > > Another solution I had in mind was reworking the `--environment_config` option. JSON blobs are unwieldy, and overloading the `--environment_config` option is confusing to

[GitHub] [beam] amaliujia commented on pull request #12532: [Beam-9543] support MATCH_RECOGNIZE with NFA

amaliujia commented on pull request #12532: URL: https://github.com/apache/beam/pull/12532#issuecomment-672126518 cc @aaltay This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [beam] tvalentyn commented on a change in pull request #12452: [BEAM-10623] Add workflow to run Beam python tests on Linux/Windows/Mac platforms

tvalentyn commented on a change in pull request #12452: URL: https://github.com/apache/beam/pull/12452#discussion_r468768265 ## File path: CI.md ## @@ -75,8 +75,28 @@ run categories. Here is a summary of the run categories with regards of the jobs Those jobs often have

[GitHub] [beam] boyuanzz commented on pull request #12518: [BEAM-10663] Workaround of AutoValueSchema doesn't work with SchemaFieldName

boyuanzz commented on pull request #12518: URL: https://github.com/apache/beam/pull/12518#issuecomment-672158339 > Will it be possible to re-enable the Kafka test on direct runner? I don't think we have Kafka test on direct runner.

[GitHub] [beam] danielxjd removed a comment on pull request #12223: [Beam-4379] Make ParquetIO read splittable

danielxjd removed a comment on pull request #12223: URL: https://github.com/apache/beam/pull/12223#issuecomment-672092490 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the

[GitHub] [beam] boyuanzz commented on a change in pull request #12531: [BEAM-10676] Use the fire timestamp as the output timestamp for timers

boyuanzz commented on a change in pull request #12531: URL: https://github.com/apache/beam/pull/12531#discussion_r468822213 ## File path: sdks/python/apache_beam/runners/worker/bundle_processor.py ## @@ -647,7 +647,7 @@ def set(self, ts): windows=(self._window, ),

[GitHub] [beam] robertwb commented on pull request #12534: [BEAM-9547] Implement some methods for deferred Series.

robertwb commented on pull request #12534: URL: https://github.com/apache/beam/pull/12534#issuecomment-672230522 R: @ibzib This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [beam] robinyqiu merged pull request #12515: Upgrade to ZetaSQL 2020.08.1

robinyqiu merged pull request #12515: URL: https://github.com/apache/beam/pull/12515 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [beam] pabloem commented on a change in pull request #12492: [BEAM-6807] Implement an Azure blobstore filesystem for Python SDK

pabloem commented on a change in pull request #12492: URL: https://github.com/apache/beam/pull/12492#discussion_r468878930 ## File path: sdks/python/apache_beam/io/azure/blobstorageio_test.py ## @@ -0,0 +1,86 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one

[GitHub] [beam] saavannanavati commented on pull request #12352: [BEAM-10549] Improve runtime type checking performance for the Python SDK

saavannanavati commented on pull request #12352: URL: https://github.com/apache/beam/pull/12352#issuecomment-672292191 The microbenchmark results! It looks like `performance_runtime_type_check` is ~10% slower than no type check, but significantly faster than `runtime_type_check`

[GitHub] [beam] kkucharc commented on a change in pull request #12527: [BEAM-10672] Added Python Combine load tests for streaming on Dataflow

kkucharc commented on a change in pull request #12527: URL: https://github.com/apache/beam/pull/12527#discussion_r468725264 ## File path: .test-infra/jenkins/job_LoadTests_Combine_Python.groovy ## @@ -93,15 +93,23 @@ def loadTestConfigurations = { datasetName ->

[GitHub] [beam] TheNeuralBit commented on a change in pull request #12498: [BEAM-10654] Implemented ExternalSchemaIOTransformRegistrar for jdbc

TheNeuralBit commented on a change in pull request #12498: URL: https://github.com/apache/beam/pull/12498#discussion_r468722620 ## File path: sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcSchemaIOProvider.java ## @@ -0,0 +1,172 @@ +/* + * Licensed to the

[GitHub] [beam] danielxjd removed a comment on pull request #12223: [Beam-4379] Make ParquetIO read splittable

danielxjd removed a comment on pull request #12223: URL: https://github.com/apache/beam/pull/12223#issuecomment-670715872 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the

[GitHub] [beam] tysonjh commented on pull request #12499: [BEAM-10602] Fix load test metrics in Grafana dashboard

tysonjh commented on pull request #12499: URL: https://github.com/apache/beam/pull/12499#issuecomment-672113428 > @tysonjh You should be able to run this locally with the backup data which is automatically retrieved from the GCS bucket when you run `docker-compose up`. Basically, the

[GitHub] [beam] TheNeuralBit commented on pull request #12481: [BEAM-10571] Use schemas in ExternalConfigurationPayload

TheNeuralBit commented on pull request #12481: URL: https://github.com/apache/beam/pull/12481#issuecomment-672186752 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] emilymye commented on pull request #12505: [WIP][BEAM-8106] Add version to java container image name

emilymye commented on pull request #12505: URL: https://github.com/apache/beam/pull/12505#issuecomment-672204447 Run Python2_PVR_Flink PreCommit This is an automated message from the Apache Git Service. To respond to the

[GitHub] [beam] robertwb opened a new pull request #12534: [BEAM-9547] Implement some methods for deferred Series.

robertwb opened a new pull request #12534: URL: https://github.com/apache/beam/pull/12534 Now less than 50% of the pandas doctests are skipped. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and

[GitHub] [beam] aaltay merged pull request #12529: Moving /tmp directory cleanup of CI workers to Inventory Jenkins job

aaltay merged pull request #12529: URL: https://github.com/apache/beam/pull/12529 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [beam] Ardagan commented on pull request #12139: [DO NOT REVIEW] Scd py bq test

Ardagan commented on pull request #12139: URL: https://github.com/apache/beam/pull/12139#issuecomment-672253789 Run Python 2 PostCommit This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] boyuanzz merged pull request #12518: [BEAM-10663] Workaround of AutoValueSchema doesn't work with SchemaFieldName

boyuanzz merged pull request #12518: URL: https://github.com/apache/beam/pull/12518 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [beam] tvalentyn commented on pull request #12526: [BEAM-10663] Disable python kafka integration tests

tvalentyn commented on pull request #12526: URL: https://github.com/apache/beam/pull/12526#issuecomment-672258012 Thanks everyone for reporting and fixing this error. This is an automated message from the Apache Git Service.

[GitHub] [beam] TheNeuralBit opened a new pull request #12535: [BEAM-10500] Make KeyedTimerDataCoder encode output timestamp

TheNeuralBit opened a new pull request #12535: URL: https://github.com/apache/beam/pull/12535 KeyedTimerDataCoder doesn't actually encode the outputTimestamp separately, instead re-using the timestamp when decoding. This leads to occasional flakes when the two `new Instant()` instances

[GitHub] [beam] amaliujia commented on pull request #12515: Upgrade to ZetaSQL 2020.08.1

amaliujia commented on pull request #12515: URL: https://github.com/apache/beam/pull/12515#issuecomment-672271568 Thank you! This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [beam] damondouglas commented on pull request #12506: [BEAM-9680] Add Filter with ParDo lesson to Go SDK Katas

damondouglas commented on pull request #12506: URL: https://github.com/apache/beam/pull/12506#issuecomment-672365280 @lostluck the [stepik course](https://stepik.org/course/70387) has been updated and the `*-remote.yaml` files committed to this PR. This PR is ready to merge.

[GitHub] [beam] pabloem commented on pull request #12489: [BEAM-6064] Add an option to avoid insert_ids on BQ in exchange for faster insertions

pabloem commented on pull request #12489: URL: https://github.com/apache/beam/pull/12489#issuecomment-672370947 Run Python 3.8 PostCommit This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] robinyqiu merged pull request #12536: [BEAM-10611] Simplification: Use new ZetaSQL API to get/create Values

robinyqiu merged pull request #12536: URL: https://github.com/apache/beam/pull/12536 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [beam] robinyqiu opened a new pull request #12539: Fix some typos

robinyqiu opened a new pull request #12539: URL: https://github.com/apache/beam/pull/12539 Fix some typos found during code import to Google codebase. r: @aaltay @kennknowles Post-Commit Tests Status (on master branch)

[GitHub] [beam] boyuanzz commented on a change in pull request #12519: [BEAM-10670] Make Read execute as a splittable DoFn by default for the Java DirectRunner.

boyuanzz commented on a change in pull request #12519: URL: https://github.com/apache/beam/pull/12519#discussion_r468874809 ## File path: runners/core-java/src/main/java/org/apache/beam/runners/core/OutputAndTimeBoundedSplittableProcessElementInvoker.java ## @@ -211,10 +211,6

[GitHub] [beam] KevinGG commented on pull request #12444: Added a whitespace lint as part of python lint precommit

KevinGG commented on pull request #12444: URL: https://github.com/apache/beam/pull/12444#issuecomment-672309994 > Few things regarding this change > > 1. Isn't the spotless pre-commit right place to extend with such checks? Many developers probably already added that to their

[GitHub] [beam] TheNeuralBit commented on a change in pull request #12297: [BEAM-10137] Add KinesisIO for cross-language usage with python wrapper

TheNeuralBit commented on a change in pull request #12297: URL: https://github.com/apache/beam/pull/12297#discussion_r468907524 ## File path: sdks/python/apache_beam/io/kinesis.py ## @@ -0,0 +1,317 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +#

[GitHub] [beam] TobKed commented on pull request #12452: [BEAM-10623] Add workflow to run Beam python tests on Linux/Windows/Mac platforms

TobKed commented on pull request #12452: URL: https://github.com/apache/beam/pull/12452#issuecomment-672344986 > Thanks, @TobKed , I don't have additional comments. Once all reviewers, whose feedback you expect, give their LGTM, please squash fixup commits, and leave only the commits that

[GitHub] [beam] TheNeuralBit commented on a change in pull request #12297: [BEAM-10137] Add KinesisIO for cross-language usage with python wrapper

TheNeuralBit commented on a change in pull request #12297: URL: https://github.com/apache/beam/pull/12297#discussion_r468925737 ## File path: sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/KinesisTransformRegistrar.java ## @@ -0,0 +1,268 @@ +/* + * Licensed

[GitHub] [beam] chamikaramj commented on a change in pull request #12263: [BEAM-10492] Add missing sideinput handling to DLP transforms

chamikaramj commented on a change in pull request #12263: URL: https://github.com/apache/beam/pull/12263#discussion_r468930529 ## File path: sdks/java/extensions/ml/src/main/java/org/apache/beam/sdk/extensions/ml/DLPDeidentifyText.java ## @@ -177,19 +177,24 @@ public

  1   2   >