[GitHub] [beam] robinyqiu opened a new pull request #12539: Fix some typos

2020-08-11 Thread GitBox
robinyqiu opened a new pull request #12539: URL: https://github.com/apache/beam/pull/12539 Fix some typos found during code import to Google codebase. r: @aaltay @kennknowles Post-Commit Tests Status (on master branch)

[GitHub] [beam] udim commented on a change in pull request #12257: [BEAM-2762] Generate Python coverage reports during pre-commit

2020-08-11 Thread GitBox
udim commented on a change in pull request #12257: URL: https://github.com/apache/beam/pull/12257#discussion_r468946744 ## File path: sdks/python/tox.ini ## @@ -172,6 +172,17 @@ extras = test,gcp,interactive,aws commands = {toxinidir}/scripts/run_pytest.sh {envname}

[GitHub] [beam] udim commented on pull request #12257: [BEAM-2762] Generate Python coverage reports during pre-commit

2020-08-11 Thread GitBox
udim commented on pull request #12257: URL: https://github.com/apache/beam/pull/12257#issuecomment-672414494 We can give this a try by adding this to `CommonJobProperties.groovy`: ``` credentialsBinding { // other values string("CODECOV_TOKEN",

[GitHub] [beam] udim commented on a change in pull request #12257: [BEAM-2762] Generate Python coverage reports during pre-commit

2020-08-11 Thread GitBox
udim commented on a change in pull request #12257: URL: https://github.com/apache/beam/pull/12257#discussion_r468943368 ## File path: sdks/python/tox.ini ## @@ -172,6 +172,17 @@ extras = test,gcp,interactive,aws commands = {toxinidir}/scripts/run_pytest.sh {envname}

[GitHub] [beam] y1chi commented on a change in pull request #12537: [BEAM-10200] Add optional experiment to enable heap dump through the …

2020-08-11 Thread GitBox
y1chi commented on a change in pull request #12537: URL: https://github.com/apache/beam/pull/12537#discussion_r468939630 ## File path: sdks/python/apache_beam/runners/worker/worker_status.py ## @@ -152,7 +170,11 @@ def generate_status_response(self): all_status_sections =

[GitHub] [beam] angoenka commented on a change in pull request #12537: [BEAM-10200] Add optional experiment to enable heap dump through the …

2020-08-11 Thread GitBox
angoenka commented on a change in pull request #12537: URL: https://github.com/apache/beam/pull/12537#discussion_r468938985 ## File path: sdks/python/apache_beam/runners/worker/worker_status.py ## @@ -152,7 +170,11 @@ def generate_status_response(self):

[GitHub] [beam] pabloem commented on pull request #12489: [BEAM-6064] Add an option to avoid insert_ids on BQ in exchange for faster insertions

2020-08-11 Thread GitBox
pabloem commented on pull request #12489: URL: https://github.com/apache/beam/pull/12489#issuecomment-672395931 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the

[GitHub] [beam] iindyk commented on a change in pull request #12420: Extending ApproximateQuantiles functionality to deal with non-uniform weights.

2020-08-11 Thread GitBox
iindyk commented on a change in pull request #12420: URL: https://github.com/apache/beam/pull/12420#discussion_r468937788 ## File path: sdks/python/apache_beam/transforms/stats.py ## @@ -398,8 +424,8 @@ class ApproximateQuantilesCombineFn(CombineFn):

[GitHub] [beam] iindyk commented on a change in pull request #12420: Extending ApproximateQuantiles functionality to deal with non-uniform weights.

2020-08-11 Thread GitBox
iindyk commented on a change in pull request #12420: URL: https://github.com/apache/beam/pull/12420#discussion_r468937924 ## File path: sdks/python/apache_beam/transforms/stats.py ## @@ -576,17 +633,34 @@ def _interpolate(self, i_buffers, count, step, offset):

[GitHub] [beam] iindyk commented on a change in pull request #12420: Extending ApproximateQuantiles functionality to deal with non-uniform weights.

2020-08-11 Thread GitBox
iindyk commented on a change in pull request #12420: URL: https://github.com/apache/beam/pull/12420#discussion_r468937732 ## File path: sdks/python/apache_beam/transforms/stats.py ## @@ -263,30 +265,38 @@ class Globally(PTransform): Args: num_quantiles: number of

[GitHub] [beam] robinyqiu merged pull request #12536: [BEAM-10611] Simplification: Use new ZetaSQL API to get/create Values

2020-08-11 Thread GitBox
robinyqiu merged pull request #12536: URL: https://github.com/apache/beam/pull/12536 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [beam] tvalentyn commented on a change in pull request #12263: [BEAM-10492] Add missing sideinput handling to DLP transforms

2020-08-11 Thread GitBox
tvalentyn commented on a change in pull request #12263: URL: https://github.com/apache/beam/pull/12263#discussion_r468935806 ## File path: sdks/java/extensions/ml/src/main/java/org/apache/beam/sdk/extensions/ml/DLPDeidentifyText.java ## @@ -177,19 +177,24 @@ public

[GitHub] [beam] ibzib commented on pull request #12538: [BEAM-10681] Set metrics supported in Spark portable runner.

2020-08-11 Thread GitBox
ibzib commented on pull request #12538: URL: https://github.com/apache/beam/pull/12538#issuecomment-672390647 Run Java Spark PortableValidatesRunner Batch This is an automated message from the Apache Git Service. To respond

[GitHub] [beam] ibzib opened a new pull request #12538: [BEAM-10681] Set metrics supported in Spark portable runner.

2020-08-11 Thread GitBox
ibzib opened a new pull request #12538: URL: https://github.com/apache/beam/pull/12538 **Please** add a meaningful description for your change here Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly

[GitHub] [beam] chamikaramj commented on a change in pull request #12263: [BEAM-10492] Add missing sideinput handling to DLP transforms

2020-08-11 Thread GitBox
chamikaramj commented on a change in pull request #12263: URL: https://github.com/apache/beam/pull/12263#discussion_r468930529 ## File path: sdks/java/extensions/ml/src/main/java/org/apache/beam/sdk/extensions/ml/DLPDeidentifyText.java ## @@ -177,19 +177,24 @@ public

[GitHub] [beam] TheNeuralBit commented on a change in pull request #12297: [BEAM-10137] Add KinesisIO for cross-language usage with python wrapper

2020-08-11 Thread GitBox
TheNeuralBit commented on a change in pull request #12297: URL: https://github.com/apache/beam/pull/12297#discussion_r468925737 ## File path: sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/KinesisTransformRegistrar.java ## @@ -0,0 +1,268 @@ +/* + * Licensed

[GitHub] [beam] pabloem commented on pull request #12489: [BEAM-6064] Add an option to avoid insert_ids on BQ in exchange for faster insertions

2020-08-11 Thread GitBox
pabloem commented on pull request #12489: URL: https://github.com/apache/beam/pull/12489#issuecomment-672370947 Run Python 3.8 PostCommit This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] tvalentyn merged pull request #12525: [BEAM-10644] Mark Beam 2.24.0 as the last release with Py2 and Py35 support.

2020-08-11 Thread GitBox
tvalentyn merged pull request #12525: URL: https://github.com/apache/beam/pull/12525 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [beam] damondouglas commented on pull request #12506: [BEAM-9680] Add Filter with ParDo lesson to Go SDK Katas

2020-08-11 Thread GitBox
damondouglas commented on pull request #12506: URL: https://github.com/apache/beam/pull/12506#issuecomment-672365280 @lostluck the [stepik course](https://stepik.org/course/70387) has been updated and the `*-remote.yaml` files committed to this PR. This PR is ready to merge.

[GitHub] [beam] TobKed commented on pull request #12452: [BEAM-10623] Add workflow to run Beam python tests on Linux/Windows/Mac platforms

2020-08-11 Thread GitBox
TobKed commented on pull request #12452: URL: https://github.com/apache/beam/pull/12452#issuecomment-672344986 > Thanks, @TobKed , I don't have additional comments. Once all reviewers, whose feedback you expect, give their LGTM, please squash fixup commits, and leave only the commits that

[GitHub] [beam] TheNeuralBit commented on a change in pull request #12297: [BEAM-10137] Add KinesisIO for cross-language usage with python wrapper

2020-08-11 Thread GitBox
TheNeuralBit commented on a change in pull request #12297: URL: https://github.com/apache/beam/pull/12297#discussion_r468907524 ## File path: sdks/python/apache_beam/io/kinesis.py ## @@ -0,0 +1,317 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +#

[GitHub] [beam] y1chi opened a new pull request #12537: [BEAM-10200] Add optional experiment to enable heap dump through the …

2020-08-11 Thread GitBox
y1chi opened a new pull request #12537: URL: https://github.com/apache/beam/pull/12537 …status api for portable runners. **Please** add a meaningful description for your change here Thank you for your contribution! Follow this checklist to help

[GitHub] [beam] y1chi commented on pull request #12537: [BEAM-10200] Add optional experiment to enable heap dump through the …

2020-08-11 Thread GitBox
y1chi commented on pull request #12537: URL: https://github.com/apache/beam/pull/12537#issuecomment-672338978 R: @angoenka This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [beam] lukecwik commented on pull request #12519: [BEAM-10670] Make Read execute as a splittable DoFn by default for the Java DirectRunner.

2020-08-11 Thread GitBox
lukecwik commented on pull request #12519: URL: https://github.com/apache/beam/pull/12519#issuecomment-672327320 Thanks for taking a look, I'm going to break this up into a few PRs to separate it from what was needed for the DirectRunner.

[GitHub] [beam] TobKed commented on a change in pull request #12513: [BEAM-BEAM-10667] Make gh-actions wheels bucket name stored as a secret

2020-08-11 Thread GitBox
TobKed commented on a change in pull request #12513: URL: https://github.com/apache/beam/pull/12513#discussion_r468897533 ## File path: CI.md ## @@ -89,7 +89,9 @@ Those jobs often have matrix run strategy which runs several different variation Some of the jobs require

[GitHub] [beam] boyuanzz commented on a change in pull request #12519: [BEAM-10670] Make Read execute as a splittable DoFn by default for the Java DirectRunner.

2020-08-11 Thread GitBox
boyuanzz commented on a change in pull request #12519: URL: https://github.com/apache/beam/pull/12519#discussion_r468874809 ## File path: runners/core-java/src/main/java/org/apache/beam/runners/core/OutputAndTimeBoundedSplittableProcessElementInvoker.java ## @@ -211,10 +211,6

[GitHub] [beam] KevinGG commented on pull request #12444: Added a whitespace lint as part of python lint precommit

2020-08-11 Thread GitBox
KevinGG commented on pull request #12444: URL: https://github.com/apache/beam/pull/12444#issuecomment-672309994 > Few things regarding this change > > 1. Isn't the spotless pre-commit right place to extend with such checks? Many developers probably already added that to their

[GitHub] [beam] TheNeuralBit merged pull request #12481: [BEAM-10571] Use schemas in ExternalConfigurationPayload

2020-08-11 Thread GitBox
TheNeuralBit merged pull request #12481: URL: https://github.com/apache/beam/pull/12481 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [beam] TheNeuralBit commented on pull request #12481: [BEAM-10571] Use schemas in ExternalConfigurationPayload

2020-08-11 Thread GitBox
TheNeuralBit commented on pull request #12481: URL: https://github.com/apache/beam/pull/12481#issuecomment-672308417 PreCommit seems to have passed, just no update here: https://ci-beam.apache.org/job/beam_PreCommit_Java_Phrase/2603/ I've also validated this with some internal test

[GitHub] [beam] pabloem commented on pull request #12489: [BEAM-6064] Add an option to avoid insert_ids on BQ in exchange for faster insertions

2020-08-11 Thread GitBox
pabloem commented on pull request #12489: URL: https://github.com/apache/beam/pull/12489#issuecomment-672302553 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [beam] KevinGG commented on a change in pull request #12444: Added a whitespace lint as part of python lint precommit

2020-08-11 Thread GitBox
KevinGG commented on a change in pull request #12444: URL: https://github.com/apache/beam/pull/12444#discussion_r468885295 ## File path: .test-infra/jenkins/job_PreCommit_Whitespace.groovy ## @@ -0,0 +1,30 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one +

[GitHub] [beam] tvalentyn commented on a change in pull request #12513: [BEAM-BEAM-10667] Make gh-actions wheels bucket name stored as a secret

2020-08-11 Thread GitBox
tvalentyn commented on a change in pull request #12513: URL: https://github.com/apache/beam/pull/12513#discussion_r468884894 ## File path: CI.md ## @@ -89,7 +89,9 @@ Those jobs often have matrix run strategy which runs several different variation Some of the jobs require

[GitHub] [beam] TobKed commented on a change in pull request #12452: [BEAM-10623] Add workflow to run Beam python tests on Linux/Windows/Mac platforms

2020-08-11 Thread GitBox
TobKed commented on a change in pull request #12452: URL: https://github.com/apache/beam/pull/12452#discussion_r468884891 ## File path: sdks/python/apache_beam/testing/util.py ## @@ -334,3 +335,19 @@ def open_shards(glob_pattern, mode='rt', encoding='utf-8'):

[GitHub] [beam] tvalentyn edited a comment on pull request #12452: [BEAM-10623] Add workflow to run Beam python tests on Linux/Windows/Mac platforms

2020-08-11 Thread GitBox
tvalentyn edited a comment on pull request #12452: URL: https://github.com/apache/beam/pull/12452#issuecomment-672296133 Thanks, @TobKed , I don't have additional comments. Once all reviewers, whose feedback you expect, give their LGTM, please squash fixup commits, and leave only the

[GitHub] [beam] tvalentyn commented on pull request #12452: [BEAM-10623] Add workflow to run Beam python tests on Linux/Windows/Mac platforms

2020-08-11 Thread GitBox
tvalentyn commented on pull request #12452: URL: https://github.com/apache/beam/pull/12452#issuecomment-672296133 Thanks, @TobKed , I don't have additional comments. Once all reviewers, whose feedback you expect, gave their LGTM, please squash fixup commits, and leave only the commits

[GitHub] [beam] KevinGG commented on a change in pull request #12444: Added a whitespace lint as part of python lint precommit

2020-08-11 Thread GitBox
KevinGG commented on a change in pull request #12444: URL: https://github.com/apache/beam/pull/12444#discussion_r468881675 ## File path: sdks/python/scripts/run_whitespacelint.sh ## @@ -0,0 +1,32 @@ +#!/bin/bash +# +#Licensed to the Apache Software Foundation (ASF) under

[GitHub] [beam] saavannanavati commented on pull request #12352: [BEAM-10549] Improve runtime type checking performance for the Python SDK

2020-08-11 Thread GitBox
saavannanavati commented on pull request #12352: URL: https://github.com/apache/beam/pull/12352#issuecomment-672292191 The microbenchmark results! It looks like `performance_runtime_type_check` is ~10% slower than no type check, but significantly faster than `runtime_type_check`

[GitHub] [beam] pabloem commented on a change in pull request #12492: [BEAM-6807] Implement an Azure blobstore filesystem for Python SDK

2020-08-11 Thread GitBox
pabloem commented on a change in pull request #12492: URL: https://github.com/apache/beam/pull/12492#discussion_r468878930 ## File path: sdks/python/apache_beam/io/azure/blobstorageio_test.py ## @@ -0,0 +1,86 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one

[GitHub] [beam] TobKed commented on a change in pull request #12452: [BEAM-10623] Add workflow to run Beam python tests on Linux/Windows/Mac platforms

2020-08-11 Thread GitBox
TobKed commented on a change in pull request #12452: URL: https://github.com/apache/beam/pull/12452#discussion_r468877557 ## File path: CI.md ## @@ -75,8 +75,28 @@ run categories. Here is a summary of the run categories with regards of the jobs Those jobs often have matrix

[GitHub] [beam] pabloem commented on a change in pull request #12492: [BEAM-6807] Implement an Azure blobstore filesystem for Python SDK

2020-08-11 Thread GitBox
pabloem commented on a change in pull request #12492: URL: https://github.com/apache/beam/pull/12492#discussion_r468877314 ## File path: sdks/python/apache_beam/io/azure/blobstoragefilesystem_test.py ## @@ -0,0 +1,315 @@ +# -*- coding: utf-8 -*- +# +# Licensed to the Apache

[GitHub] [beam] TobKed commented on a change in pull request #12452: [BEAM-10623] Add workflow to run Beam python tests on Linux/Windows/Mac platforms

2020-08-11 Thread GitBox
TobKed commented on a change in pull request #12452: URL: https://github.com/apache/beam/pull/12452#discussion_r468877500 ## File path: CI.md ## @@ -75,8 +75,28 @@ run categories. Here is a summary of the run categories with regards of the jobs Those jobs often have matrix

[GitHub] [beam] TobKed commented on a change in pull request #12452: [BEAM-10623] Add workflow to run Beam python tests on Linux/Windows/Mac platforms

2020-08-11 Thread GitBox
TobKed commented on a change in pull request #12452: URL: https://github.com/apache/beam/pull/12452#discussion_r468876965 ## File path: CI.md ## @@ -75,8 +75,28 @@ run categories. Here is a summary of the run categories with regards of the jobs Those jobs often have matrix

[GitHub] [beam] robinyqiu opened a new pull request #12536: [BEAM-10611] Simplification: Use new ZetaSQL API to get/create Values

2020-08-11 Thread GitBox
robinyqiu opened a new pull request #12536: URL: https://github.com/apache/beam/pull/12536 Simplify getting/creating ZetaSQL values using the new API. r: @apilloud @ZijieSong946 Post-Commit Tests Status (on master branch)

[GitHub] [beam] pabloem commented on pull request #12489: [BEAM-6064] Add an option to avoid insert_ids on BQ in exchange for faster insertions

2020-08-11 Thread GitBox
pabloem commented on pull request #12489: URL: https://github.com/apache/beam/pull/12489#issuecomment-672280905 Run Python 3.8 PostCommit This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] robinyqiu merged pull request #12515: Upgrade to ZetaSQL 2020.08.1

2020-08-11 Thread GitBox
robinyqiu merged pull request #12515: URL: https://github.com/apache/beam/pull/12515 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [beam] robertwb commented on pull request #12516: [BEAM-9547] Implement dataframes top, join, merge.

2020-08-11 Thread GitBox
robertwb commented on pull request #12516: URL: https://github.com/apache/beam/pull/12516#issuecomment-672277371 > This is going to take me a while to review since I have very little Python experience and even less Panda experience. If you want a more thorough, faster review, I suggest

[GitHub] [beam] robertwb commented on a change in pull request #12516: [BEAM-9547] Implement dataframes top, join, merge.

2020-08-11 Thread GitBox
robertwb commented on a change in pull request #12516: URL: https://github.com/apache/beam/pull/12516#discussion_r468862256 ## File path: sdks/python/apache_beam/dataframe/frames.py ## @@ -54,6 +54,42 @@ def agg(self, *args, **kwargs): 'order-sensitive') diff =

[GitHub] [beam] pabloem commented on pull request #12492: [BEAM-6807] Implement an Azure blobstore filesystem for Python SDK

2020-08-11 Thread GitBox
pabloem commented on pull request #12492: URL: https://github.com/apache/beam/pull/12492#issuecomment-672273518 Run Python2_PVR_Flink PreCommit This is an automated message from the Apache Git Service. To respond to the

[GitHub] [beam] amaliujia commented on pull request #12515: Upgrade to ZetaSQL 2020.08.1

2020-08-11 Thread GitBox
amaliujia commented on pull request #12515: URL: https://github.com/apache/beam/pull/12515#issuecomment-672271568 Thank you! This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [beam] TheNeuralBit opened a new pull request #12535: [BEAM-10500] Make KeyedTimerDataCoder encode output timestamp

2020-08-11 Thread GitBox
TheNeuralBit opened a new pull request #12535: URL: https://github.com/apache/beam/pull/12535 KeyedTimerDataCoder doesn't actually encode the outputTimestamp separately, instead re-using the timestamp when decoding. This leads to occasional flakes when the two `new Instant()` instances

[GitHub] [beam] ihji removed a comment on pull request #12533: [BEAM-10679] improving XLang KafkaIO streaming test

2020-08-11 Thread GitBox
ihji removed a comment on pull request #12533: URL: https://github.com/apache/beam/pull/12533#issuecomment-672209441 Run Python 3.5 PostCommit This is an automated message from the Apache Git Service. To respond to the

[GitHub] [beam] TheNeuralBit merged pull request #12521: [BEAM-8125] Add verifyDeterministic test to SchemaCoderTest

2020-08-11 Thread GitBox
TheNeuralBit merged pull request #12521: URL: https://github.com/apache/beam/pull/12521 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [beam] pabloem merged pull request #12485: [BEAM-6064] Improvements to BQ streaming insert performance

2020-08-11 Thread GitBox
pabloem merged pull request #12485: URL: https://github.com/apache/beam/pull/12485 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [beam] TheNeuralBit commented on pull request #12522: Fix format string in PipelineValidator

2020-08-11 Thread GitBox
TheNeuralBit commented on pull request #12522: URL: https://github.com/apache/beam/pull/12522#issuecomment-672263017 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] pabloem commented on a change in pull request #12489: [BEAM-6064] Add an option to avoid insert_ids on BQ in exchange for faster insertions

2020-08-11 Thread GitBox
pabloem commented on a change in pull request #12489: URL: https://github.com/apache/beam/pull/12489#discussion_r468843122 ## File path: sdks/python/apache_beam/io/gcp/bigquery.py ## @@ -1048,6 +1055,11 @@ def __init__( to be passed when creating a BigQuery table.

[GitHub] [beam] pabloem commented on a change in pull request #12485: [BEAM-6064] Improvements to BQ streaming insert performance

2020-08-11 Thread GitBox
pabloem commented on a change in pull request #12485: URL: https://github.com/apache/beam/pull/12485#discussion_r468842890 ## File path: sdks/python/apache_beam/io/gcp/bigquery.py ## @@ -304,6 +308,8 @@ def compute_table_name(row): NOTE: This job name template does not have

[GitHub] [beam] tvalentyn commented on pull request #12526: [BEAM-10663] Disable python kafka integration tests

2020-08-11 Thread GitBox
tvalentyn commented on pull request #12526: URL: https://github.com/apache/beam/pull/12526#issuecomment-672258012 Thanks everyone for reporting and fixing this error. This is an automated message from the Apache Git Service.

[GitHub] [beam] TheNeuralBit commented on pull request #12481: [BEAM-10571] Use schemas in ExternalConfigurationPayload

2020-08-11 Thread GitBox
TheNeuralBit commented on pull request #12481: URL: https://github.com/apache/beam/pull/12481#issuecomment-672256723 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] boyuanzz merged pull request #12518: [BEAM-10663] Workaround of AutoValueSchema doesn't work with SchemaFieldName

2020-08-11 Thread GitBox
boyuanzz merged pull request #12518: URL: https://github.com/apache/beam/pull/12518 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [beam] Ardagan commented on pull request #12139: [DO NOT REVIEW] Scd py bq test

2020-08-11 Thread GitBox
Ardagan commented on pull request #12139: URL: https://github.com/apache/beam/pull/12139#issuecomment-672253789 Run Python 2 PostCommit This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] chamikaramj commented on a change in pull request #12485: [BEAM-6064] Improvements to BQ streaming insert performance

2020-08-11 Thread GitBox
chamikaramj commented on a change in pull request #12485: URL: https://github.com/apache/beam/pull/12485#discussion_r468830450 ## File path: sdks/python/apache_beam/io/gcp/bigquery.py ## @@ -304,6 +308,8 @@ def compute_table_name(row): NOTE: This job name template does not

[GitHub] [beam] chamikaramj commented on a change in pull request #12485: [BEAM-6064] Improvements to BQ streaming insert performance

2020-08-11 Thread GitBox
chamikaramj commented on a change in pull request #12485: URL: https://github.com/apache/beam/pull/12485#discussion_r468829856 ## File path: sdks/python/apache_beam/io/gcp/bigquery.py ## @@ -304,6 +308,8 @@ def compute_table_name(row): NOTE: This job name template does not

[GitHub] [beam] chamikaramj commented on a change in pull request #12485: [BEAM-6064] Improvements to BQ streaming insert performance

2020-08-11 Thread GitBox
chamikaramj commented on a change in pull request #12485: URL: https://github.com/apache/beam/pull/12485#discussion_r468830116 ## File path: sdks/python/apache_beam/io/gcp/bigquery.py ## @@ -304,6 +308,8 @@ def compute_table_name(row): NOTE: This job name template does not

[GitHub] [beam] pabloem commented on a change in pull request #12489: [BEAM-6064] Add an option to avoid insert_ids on BQ in exchange for faster insertions

2020-08-11 Thread GitBox
pabloem commented on a change in pull request #12489: URL: https://github.com/apache/beam/pull/12489#discussion_r468825940 ## File path: sdks/python/apache_beam/io/gcp/bigquery.py ## @@ -304,6 +308,8 @@ def compute_table_name(row): NOTE: This job name template does not have

[GitHub] [beam] pabloem commented on a change in pull request #12485: [BEAM-6064] Improvements to BQ streaming insert performance

2020-08-11 Thread GitBox
pabloem commented on a change in pull request #12485: URL: https://github.com/apache/beam/pull/12485#discussion_r468825677 ## File path: sdks/python/apache_beam/io/gcp/bigquery.py ## @@ -304,6 +308,8 @@ def compute_table_name(row): NOTE: This job name template does not have

[GitHub] [beam] pabloem commented on a change in pull request #12485: [BEAM-6064] Improvements to BQ streaming insert performance

2020-08-11 Thread GitBox
pabloem commented on a change in pull request #12485: URL: https://github.com/apache/beam/pull/12485#discussion_r468825099 ## File path: sdks/python/apache_beam/io/gcp/bigquery.py ## @@ -1419,7 +1448,18 @@ def __init__( Default is to retry always. This means that

[GitHub] [beam] Ardagan commented on pull request #12139: [DO NOT REVIEW] Scd py bq test

2020-08-11 Thread GitBox
Ardagan commented on pull request #12139: URL: https://github.com/apache/beam/pull/12139#issuecomment-672240516 run python postcommit This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] TheNeuralBit merged pull request #12526: [BEAM-10663] Disable python kafka integration tests

2020-08-11 Thread GitBox
TheNeuralBit merged pull request #12526: URL: https://github.com/apache/beam/pull/12526 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [beam] TheNeuralBit commented on pull request #12526: [BEAM-10663] Disable python kafka integration tests

2020-08-11 Thread GitBox
TheNeuralBit commented on pull request #12526: URL: https://github.com/apache/beam/pull/12526#issuecomment-672235923 Looks like the pending jobs are actually complete, they just haven't updated the PR. Merging now. This is

[GitHub] [beam] TheNeuralBit commented on a change in pull request #12521: [BEAM-8125] Add verifyDeterministic test to SchemaCoderTest

2020-08-11 Thread GitBox
TheNeuralBit commented on a change in pull request #12521: URL: https://github.com/apache/beam/pull/12521#discussion_r468822570 ## File path: sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/SchemaCoderTest.java ## @@ -290,5 +329,31 @@ public void

[GitHub] [beam] robertwb commented on pull request #12534: [BEAM-9547] Implement some methods for deferred Series.

2020-08-11 Thread GitBox
robertwb commented on pull request #12534: URL: https://github.com/apache/beam/pull/12534#issuecomment-672230522 R: @ibzib This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [beam] boyuanzz commented on a change in pull request #12531: [BEAM-10676] Use the fire timestamp as the output timestamp for timers

2020-08-11 Thread GitBox
boyuanzz commented on a change in pull request #12531: URL: https://github.com/apache/beam/pull/12531#discussion_r468822213 ## File path: sdks/python/apache_beam/runners/worker/bundle_processor.py ## @@ -647,7 +647,7 @@ def set(self, ts): windows=(self._window, ),

[GitHub] [beam] aaltay merged pull request #12529: Moving /tmp directory cleanup of CI workers to Inventory Jenkins job

2020-08-11 Thread GitBox
aaltay merged pull request #12529: URL: https://github.com/apache/beam/pull/12529 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [beam] robertwb opened a new pull request #12534: [BEAM-9547] Implement some methods for deferred Series.

2020-08-11 Thread GitBox
robertwb opened a new pull request #12534: URL: https://github.com/apache/beam/pull/12534 Now less than 50% of the pandas doctests are skipped. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and

[GitHub] [beam] ihji commented on pull request #12533: [BEAM-10679] improving XLang KafkaIO streaming test

2020-08-11 Thread GitBox
ihji commented on pull request #12533: URL: https://github.com/apache/beam/pull/12533#issuecomment-672209854 R: @chamikaramj This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [beam] ihji commented on pull request #12533: [BEAM-10679] improving XLang KafkaIO streaming test

2020-08-11 Thread GitBox
ihji commented on pull request #12533: URL: https://github.com/apache/beam/pull/12533#issuecomment-672209441 Run Python 3.5 PostCommit This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] ihji opened a new pull request #12533: [BEAM-10679] improving XLang KafkaIO streaming test

2020-08-11 Thread GitBox
ihji opened a new pull request #12533: URL: https://github.com/apache/beam/pull/12533 Using state instead of early triggering and combine per key. This would reduce the flakiness in the streaming test. Thank you for your contribution! Follow this

[GitHub] [beam] kkucharc commented on a change in pull request #12435: [BEAM-10616] Added Python Pardo load tests for streaming on Dataflow

2020-08-11 Thread GitBox
kkucharc commented on a change in pull request #12435: URL: https://github.com/apache/beam/pull/12435#discussion_r468807769 ## File path: .test-infra/jenkins/job_LoadTests_ParDo_Python.groovy ## @@ -151,3 +151,35 @@

[GitHub] [beam] emilymye commented on pull request #12505: [WIP][BEAM-8106] Add version to java container image name

2020-08-11 Thread GitBox
emilymye commented on pull request #12505: URL: https://github.com/apache/beam/pull/12505#issuecomment-672207756 retest this please This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [beam] emilymye removed a comment on pull request #12505: [WIP][BEAM-8106] Add version to java container image name

2020-08-11 Thread GitBox
emilymye removed a comment on pull request #12505: URL: https://github.com/apache/beam/pull/12505#issuecomment-671632378 Run Python2_PVR_Flink PreCommit This is an automated message from the Apache Git Service. To

[GitHub] [beam] emilymye commented on pull request #12505: [WIP][BEAM-8106] Add version to java container image name

2020-08-11 Thread GitBox
emilymye commented on pull request #12505: URL: https://github.com/apache/beam/pull/12505#issuecomment-672204447 Run Python2_PVR_Flink PreCommit This is an automated message from the Apache Git Service. To respond to the

[GitHub] [beam] boyuanzz commented on pull request #12518: [BEAM-10663] Workaround of AutoValueSchema doesn't work with SchemaFieldName

2020-08-11 Thread GitBox
boyuanzz commented on pull request #12518: URL: https://github.com/apache/beam/pull/12518#issuecomment-672201149 > Is bundle finalization not an issue for DirectRunner ? If so we should be able to run the Kafka test on DirectRunner. Within https://github.com/apache/beam/pull/12488,

[GitHub] [beam] TheNeuralBit merged pull request #12530: [BEAM-7996] Add Python SqlTransform test that includes a MAP input and output

2020-08-11 Thread GitBox
TheNeuralBit merged pull request #12530: URL: https://github.com/apache/beam/pull/12530 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [beam] TheNeuralBit commented on pull request #12522: Fix format string in PipelineValidator

2020-08-11 Thread GitBox
TheNeuralBit commented on pull request #12522: URL: https://github.com/apache/beam/pull/12522#issuecomment-672195554 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] chamikaramj commented on pull request #12518: [BEAM-10663] Workaround of AutoValueSchema doesn't work with SchemaFieldName

2020-08-11 Thread GitBox
chamikaramj commented on pull request #12518: URL: https://github.com/apache/beam/pull/12518#issuecomment-672191125 Is bundle finalization not an issue for DirectRunner ? If so we should be able to run the Kafka test on DirectRunner.

[GitHub] [beam] TheNeuralBit commented on pull request #12481: [BEAM-10571] Use schemas in ExternalConfigurationPayload

2020-08-11 Thread GitBox
TheNeuralBit commented on pull request #12481: URL: https://github.com/apache/beam/pull/12481#issuecomment-672186752 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] TheNeuralBit commented on a change in pull request #12526: [BEAM-10663] Disable python kafka integration tests

2020-08-11 Thread GitBox
TheNeuralBit commented on a change in pull request #12526: URL: https://github.com/apache/beam/pull/12526#discussion_r468789977 ## File path: sdks/python/apache_beam/io/external/xlang_kafkaio_it_test.py ## @@ -94,6 +94,7 @@ def run_xlang_kafkaio(self, pipeline):

[GitHub] [beam] TheNeuralBit commented on pull request #12518: [BEAM-10663] Workaround of AutoValueSchema doesn't work with SchemaFieldName

2020-08-11 Thread GitBox
TheNeuralBit commented on pull request #12518: URL: https://github.com/apache/beam/pull/12518#issuecomment-672181034 Ah ok. Maybe we should just change that suite to use the DirectRunner? I'm not sure why it's using Flink CC: @chamikaramj, @ihji in case they know

[GitHub] [beam] danielxjd removed a comment on pull request #12223: [Beam-4379] Make ParquetIO read splittable

2020-08-11 Thread GitBox
danielxjd removed a comment on pull request #12223: URL: https://github.com/apache/beam/pull/12223#issuecomment-672092490 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the

[GitHub] [beam] kennknowles commented on a change in pull request #12521: [BEAM-8125] Add verifyDeterministic test to SchemaCoderTest

2020-08-11 Thread GitBox
kennknowles commented on a change in pull request #12521: URL: https://github.com/apache/beam/pull/12521#discussion_r468783212 ## File path: sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/SchemaCoderTest.java ## @@ -290,5 +329,31 @@ public void

[GitHub] [beam] boyuanzz edited a comment on pull request #12518: [BEAM-10663] Workaround of AutoValueSchema doesn't work with SchemaFieldName

2020-08-11 Thread GitBox
boyuanzz edited a comment on pull request #12518: URL: https://github.com/apache/beam/pull/12518#issuecomment-672158339 > Will it be possible to re-enable the Kafka test on direct runner? I don't think we have Kafka test on direct runner. Here is the only place I can find to run

[GitHub] [beam] KevinGG commented on a change in pull request #12460: [BEAM-10545] HtmlView module

2020-08-11 Thread GitBox
KevinGG commented on a change in pull request #12460: URL: https://github.com/apache/beam/pull/12460#discussion_r468778647 ## File path: sdks/python/apache_beam/runners/interactive/extensions/apache-beam-jupyterlab-sidepanel/src/common/HtmlView.tsx ## @@ -0,0 +1,119 @@ +//

[GitHub] [beam] boyuanzz commented on pull request #12518: [BEAM-10663] Workaround of AutoValueSchema doesn't work with SchemaFieldName

2020-08-11 Thread GitBox
boyuanzz commented on pull request #12518: URL: https://github.com/apache/beam/pull/12518#issuecomment-672158339 > Will it be possible to re-enable the Kafka test on direct runner? I don't think we have Kafka test on direct runner.

[GitHub] [beam] tvalentyn commented on a change in pull request #12452: [BEAM-10623] Add workflow to run Beam python tests on Linux/Windows/Mac platforms

2020-08-11 Thread GitBox
tvalentyn commented on a change in pull request #12452: URL: https://github.com/apache/beam/pull/12452#discussion_r468768265 ## File path: CI.md ## @@ -75,8 +75,28 @@ run categories. Here is a summary of the run categories with regards of the jobs Those jobs often have

[GitHub] [beam] TheNeuralBit commented on pull request #12518: [BEAM-10663] Workaround of AutoValueSchema doesn't work with SchemaFieldName

2020-08-11 Thread GitBox
TheNeuralBit commented on pull request #12518: URL: https://github.com/apache/beam/pull/12518#issuecomment-672156135 Will it be possible to re-enable the Kafka test on direct runner? This is an automated message from the

[GitHub] [beam] KevinGG commented on a change in pull request #12460: [BEAM-10545] HtmlView module

2020-08-11 Thread GitBox
KevinGG commented on a change in pull request #12460: URL: https://github.com/apache/beam/pull/12460#discussion_r468740290 ## File path: sdks/python/apache_beam/runners/interactive/extensions/apache-beam-jupyterlab-sidepanel/src/common/HtmlView.tsx ## @@ -0,0 +1,119 @@ +//

[GitHub] [beam] kennknowles merged pull request #12366: [BEAM-10572] Eliminate nullability errors from :sdks:java:extensions:sql:datacatalog

2020-08-11 Thread GitBox
kennknowles merged pull request #12366: URL: https://github.com/apache/beam/pull/12366 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [beam] KevinGG commented on a change in pull request #12460: [BEAM-10545] HtmlView module

2020-08-11 Thread GitBox
KevinGG commented on a change in pull request #12460: URL: https://github.com/apache/beam/pull/12460#discussion_r468764276 ## File path: sdks/python/apache_beam/runners/interactive/extensions/apache-beam-jupyterlab-sidepanel/src/common/HtmlView.tsx ## @@ -0,0 +1,119 @@ +//

[GitHub] [beam] KevinGG commented on a change in pull request #12460: [BEAM-10545] HtmlView module

2020-08-11 Thread GitBox
KevinGG commented on a change in pull request #12460: URL: https://github.com/apache/beam/pull/12460#discussion_r468764276 ## File path: sdks/python/apache_beam/runners/interactive/extensions/apache-beam-jupyterlab-sidepanel/src/common/HtmlView.tsx ## @@ -0,0 +1,119 @@ +//

[GitHub] [beam] lukecwik commented on pull request #12519: [BEAM-10670] Make Read execute as a splittable DoFn by default for the Java DirectRunner.

2020-08-11 Thread GitBox
lukecwik commented on pull request #12519: URL: https://github.com/apache/beam/pull/12519#issuecomment-672146059 Run JavaPortabilityApiJava11 PreCommit This is an automated message from the Apache Git Service. To respond to

  1   2   >