[GitHub] [beam] lukecwik removed a comment on pull request #12488: [BEAM-10656] Enable bundle finalization within the Java direct runner.

2020-08-06 Thread GitBox
lukecwik removed a comment on pull request #12488: URL: https://github.com/apache/beam/pull/12488#issuecomment-670246745 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [beam] AldairCoronel commented on pull request #12492: [BEAM-6807] Implement an Azure blobstore filesystem for Python SDK

2020-08-06 Thread GitBox
AldairCoronel commented on pull request #12492: URL: https://github.com/apache/beam/pull/12492#issuecomment-670282714 R: @pabloem This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [beam] AldairCoronel opened a new pull request #12492: [BEAM-6807] Implement an Azure blobstore filesystem for Python SDK

2020-08-06 Thread GitBox
AldairCoronel opened a new pull request #12492: URL: https://github.com/apache/beam/pull/12492 **Please** add a meaningful description for your change here Thank you for your contribution! Follow this checklist to help us incorporate your contribution

[GitHub] [beam] reuvenlax commented on pull request #12474: [BEAM-10650] OrderedListState API

2020-08-06 Thread GitBox
reuvenlax commented on pull request #12474: URL: https://github.com/apache/beam/pull/12474#issuecomment-670272212 Run Dataflow ValidatesRunner This is an automated message from the Apache Git Service. To respond to the

[GitHub] [beam] reuvenlax commented on pull request #12474: [BEAM-10650] OrderedListState API

2020-08-06 Thread GitBox
reuvenlax commented on pull request #12474: URL: https://github.com/apache/beam/pull/12474#issuecomment-670272121 retest this please This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [beam] robertwb commented on a change in pull request #12440: [BEAM-10619] Report ratio of implemented pandas tests

2020-08-06 Thread GitBox
robertwb commented on a change in pull request #12440: URL: https://github.com/apache/beam/pull/12440#discussion_r466760298 ## File path: sdks/python/apache_beam/dataframe/pandas_doctests_test.py ## @@ -90,6 +113,16 @@ def test_series_tests(self): result =

[GitHub] [beam] TheNeuralBit commented on pull request #11980: [BEAM-9546] DataframeTransform can now consume a schema-aware PCollection

2020-08-06 Thread GitBox
TheNeuralBit commented on pull request #11980: URL: https://github.com/apache/beam/pull/11980#issuecomment-670261701 R: @robertwb This is ready for review This is an automated message from the Apache Git Service. To

[GitHub] [beam] youngoli commented on a change in pull request #12445: [BEAM-9919] Added an External Transform API to Go SDK

2020-08-06 Thread GitBox
youngoli commented on a change in pull request #12445: URL: https://github.com/apache/beam/pull/12445#discussion_r466758649 ## File path: sdks/go/examples/xlang/wordcount/xlang_wordcount.go ## @@ -0,0 +1,107 @@ +// Licensed to the Apache Software Foundation (ASF) under one or

[GitHub] [beam] aaltay commented on pull request #12482: [BEAM-7390] Add top code snippets

2020-08-06 Thread GitBox
aaltay commented on pull request #12482: URL: https://github.com/apache/beam/pull/12482#issuecomment-670258058 > > @aaltay Looking at the PR checks, it looks like the staging never ran. How can we trigger it? > > I do not know. You can ask on dev@ list? Might be a known issue

[GitHub] [beam] aaltay merged pull request #12472: [BEAM-7390] Add sum code snippets

2020-08-06 Thread GitBox
aaltay merged pull request #12472: URL: https://github.com/apache/beam/pull/12472 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [beam] youngoli commented on a change in pull request #12471: [BEAM-9615] Add initial Schema to Go conversions.

2020-08-06 Thread GitBox
youngoli commented on a change in pull request #12471: URL: https://github.com/apache/beam/pull/12471#discussion_r466753786 ## File path: sdks/go/pkg/beam/core/runtime/graphx/schema/schema.go ## @@ -0,0 +1,269 @@ +// Licensed to the Apache Software Foundation (ASF) under one

[GitHub] [beam] youngoli commented on a change in pull request #12471: [BEAM-9615] Add initial Schema to Go conversions.

2020-08-06 Thread GitBox
youngoli commented on a change in pull request #12471: URL: https://github.com/apache/beam/pull/12471#discussion_r466753786 ## File path: sdks/go/pkg/beam/core/runtime/graphx/schema/schema.go ## @@ -0,0 +1,269 @@ +// Licensed to the Apache Software Foundation (ASF) under one

[GitHub] [beam] apilloud commented on pull request #12483: [BEAM-9182] [BEAM-10651] Support NULL query parameters in ZetaSQL

2020-08-06 Thread GitBox
apilloud commented on pull request #12483: URL: https://github.com/apache/beam/pull/12483#issuecomment-670256055 This probably unblocks #11742 as well... This is an automated message from the Apache Git Service. To respond

[GitHub] [beam] apilloud commented on a change in pull request #12483: [BEAM-9182] [BEAM-10651] Support NULL query parameters in ZetaSQL

2020-08-06 Thread GitBox
apilloud commented on a change in pull request #12483: URL: https://github.com/apache/beam/pull/12483#discussion_r466754168 ## File path: sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BeamSqlUnparseContext.java ## @@ -147,4

[GitHub] [beam] robertwb opened a new pull request #12491: Avoid re-encoding row types.

2020-08-06 Thread GitBox
robertwb opened a new pull request #12491: URL: https://github.com/apache/beam/pull/12491 There is no benefit in decoding these only to recode them to pass them back to the SDK. Thank you for your contribution! Follow this checklist to help us

[GitHub] [beam] pabloem commented on a change in pull request #12427: [BEAM-2855] nexmark python suite implement queries 0, 1, 2 and 9

2020-08-06 Thread GitBox
pabloem commented on a change in pull request #12427: URL: https://github.com/apache/beam/pull/12427#discussion_r466750803 ## File path: sdks/python/apache_beam/testing/benchmarks/nexmark/nexmark_util.py ## @@ -103,6 +121,107 @@ def process(self, elem): yield event

[GitHub] [beam] saavannanavati commented on a change in pull request #12352: [BEAM-10549] Improve runtime type checking performance for the Python SDK

2020-08-06 Thread GitBox
saavannanavati commented on a change in pull request #12352: URL: https://github.com/apache/beam/pull/12352#discussion_r466747648 ## File path: sdks/python/apache_beam/runners/worker/opcounters.py ## @@ -224,8 +230,25 @@ def _observable_callback_inner(value, is_encoded=False):

[GitHub] [beam] saavannanavati commented on pull request #12352: [BEAM-10549] Improve runtime type checking performance for the Python SDK

2020-08-06 Thread GitBox
saavannanavati commented on pull request #12352: URL: https://github.com/apache/beam/pull/12352#issuecomment-670247889 > As discussed in person, let's push the construction of a `{source_str: TypeConstraint}` all the way up to the visitor. Just pushed these changes - PTAL I

[GitHub] [beam] ibzib commented on pull request #12484: [BEAM-10653] Modularize BeamSqlDslUdfUdafTest.

2020-08-06 Thread GitBox
ibzib commented on pull request #12484: URL: https://github.com/apache/beam/pull/12484#issuecomment-670247000 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [beam] lukecwik commented on pull request #12488: [BEAM-10656] Enable bundle finalization within the Java direct runner.

2020-08-06 Thread GitBox
lukecwik commented on pull request #12488: URL: https://github.com/apache/beam/pull/12488#issuecomment-670246745 Run Java Precommit This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [beam] amaliujia edited a comment on pull request #12484: [BEAM-10653] Modularize BeamSqlDslUdfUdafTest.

2020-08-06 Thread GitBox
amaliujia edited a comment on pull request #12484: URL: https://github.com/apache/beam/pull/12484#issuecomment-670246038 LGTM. Thanks! This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] amaliujia commented on pull request #12484: [BEAM-10653] Modularize BeamSqlDslUdfUdafTest.

2020-08-06 Thread GitBox
amaliujia commented on pull request #12484: URL: https://github.com/apache/beam/pull/12484#issuecomment-670246038 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [beam] ihji commented on a change in pull request #12223: [Beam-4379] Make ParquetIO read splittable

2020-08-06 Thread GitBox
ihji commented on a change in pull request #12223: URL: https://github.com/apache/beam/pull/12223#discussion_r466739699 ## File path: sdks/java/io/parquet/src/main/java/org/apache/beam/sdk/io/parquet/ParquetIO.java ## @@ -154,10 +187,15 @@ public static ReadFiles

[GitHub] [beam] udim commented on pull request #12242: [BEAM-10427] Benchmark runtime typechecking for the Python SDK

2020-08-06 Thread GitBox
udim commented on pull request #12242: URL: https://github.com/apache/beam/pull/12242#issuecomment-670239967 > Sorry I didn't have a chance to look at this sooner. > > Since this is a purely local change, I would suggest rather than adding all this infrastructure simply creating a

[GitHub] [beam] George-Wu commented on a change in pull request #12473: [BEAM-10601] DICOM API Beam IO connector e2e test

2020-08-06 Thread GitBox
George-Wu commented on a change in pull request #12473: URL: https://github.com/apache/beam/pull/12473#discussion_r466736948 ## File path: sdks/python/apache_beam/io/gcp/dicomio_integration_test.py ## @@ -0,0 +1,215 @@ +# +# Licensed to the Apache Software Foundation (ASF)

[GitHub] [beam] George-Wu commented on a change in pull request #12473: [BEAM-10601] DICOM API Beam IO connector e2e test

2020-08-06 Thread GitBox
George-Wu commented on a change in pull request #12473: URL: https://github.com/apache/beam/pull/12473#discussion_r466736799 ## File path: sdks/python/apache_beam/io/gcp/dicomio_integration_test.py ## @@ -0,0 +1,215 @@ +# +# Licensed to the Apache Software Foundation (ASF)

[GitHub] [beam] George-Wu commented on a change in pull request #12473: [BEAM-10601] DICOM API Beam IO connector e2e test

2020-08-06 Thread GitBox
George-Wu commented on a change in pull request #12473: URL: https://github.com/apache/beam/pull/12473#discussion_r466736883 ## File path: sdks/python/apache_beam/io/gcp/dicomio_integration_test.py ## @@ -0,0 +1,215 @@ +# +# Licensed to the Apache Software Foundation (ASF)

[GitHub] [beam] abhiy13 opened a new pull request #12490: [BEAM-10124] Add ContextualTextIO

2020-08-06 Thread GitBox
abhiy13 opened a new pull request #12490: URL: https://github.com/apache/beam/pull/12490 [[BEAM-10124]](https://issues.apache.org/jira/browse/BEAM-10124) Added ContextualTextIO. Thank you for your contribution! Follow this checklist to help us

[GitHub] [beam] robertwb commented on a change in pull request #12352: [BEAM-10549] Improve runtime type checking performance for the Python SDK

2020-08-06 Thread GitBox
robertwb commented on a change in pull request #12352: URL: https://github.com/apache/beam/pull/12352#discussion_r466729599 ## File path: sdks/python/apache_beam/runners/worker/bundle_processor.py ## @@ -188,7 +189,8 @@ def __init__(self,

[GitHub] [beam] reuvenlax commented on a change in pull request #12474: [BEAM-10650] OrderedListState API

2020-08-06 Thread GitBox
reuvenlax commented on a change in pull request #12474: URL: https://github.com/apache/beam/pull/12474#discussion_r466729615 ## File path: sdks/java/core/src/main/java/org/apache/beam/sdk/state/OrderedListState.java ## @@ -0,0 +1,51 @@ +/* + * Licensed to the Apache Software

[GitHub] [beam] reuvenlax commented on pull request #12474: [BEAM-10650] OrderedListState API

2020-08-06 Thread GitBox
reuvenlax commented on pull request #12474: URL: https://github.com/apache/beam/pull/12474#issuecomment-670231511 run flink validatesrunner This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] reuvenlax commented on pull request #12474: [BEAM-10650] OrderedListState API

2020-08-06 Thread GitBox
reuvenlax commented on pull request #12474: URL: https://github.com/apache/beam/pull/12474#issuecomment-670231434 run dataflow validatesrunner This is an automated message from the Apache Git Service. To respond to the

[GitHub] [beam] robertwb commented on a change in pull request #12352: [BEAM-10549] Improve runtime type checking performance for the Python SDK

2020-08-06 Thread GitBox
robertwb commented on a change in pull request #12352: URL: https://github.com/apache/beam/pull/12352#discussion_r466727740 ## File path: website/www/site/content/en/documentation/sdks/python-type-safety.md ## @@ -210,7 +210,21 @@ However, if you enable runtime type checking,

[GitHub] [beam] robertwb commented on a change in pull request #12352: [BEAM-10549] Improve runtime type checking performance for the Python SDK

2020-08-06 Thread GitBox
robertwb commented on a change in pull request #12352: URL: https://github.com/apache/beam/pull/12352#discussion_r466727543 ## File path: sdks/python/apache_beam/runners/worker/bundle_processor.py ## @@ -188,7 +188,8 @@ def __init__(self,

[GitHub] [beam] TheNeuralBit commented on pull request #12440: [BEAM-10619] Report ratio of implemented pandas tests

2020-08-06 Thread GitBox
TheNeuralBit commented on pull request #12440: URL: https://github.com/apache/beam/pull/12440#issuecomment-670226776 Updated the report to collect information about wont implement, now it looks like this: DataFrame: ``` 572 total test cases:

[GitHub] [beam] TheNeuralBit commented on a change in pull request #12440: [BEAM-10619] Report ratio of implemented pandas tests

2020-08-06 Thread GitBox
TheNeuralBit commented on a change in pull request #12440: URL: https://github.com/apache/beam/pull/12440#discussion_r466723059 ## File path: sdks/python/apache_beam/dataframe/doctests.py ## @@ -290,34 +298,60 @@ def to_callable(cond): super(BeamDataframeDoctestRunner,

[GitHub] [beam] lukecwik commented on pull request #12488: [BEAM-10656] Enable bundle finalization within the Java direct runner.

2020-08-06 Thread GitBox
lukecwik commented on pull request #12488: URL: https://github.com/apache/beam/pull/12488#issuecomment-670224938 retest this please This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [beam] pabloem opened a new pull request #12489: Bq no insert ids

2020-08-06 Thread GitBox
pabloem opened a new pull request #12489: URL: https://github.com/apache/beam/pull/12489 **Please** add a meaningful description for your change here Thank you for your contribution! Follow this checklist to help us incorporate your contribution

[GitHub] [beam] TheNeuralBit commented on a change in pull request #12440: [BEAM-10619] Report ratio of implemented pandas tests

2020-08-06 Thread GitBox
TheNeuralBit commented on a change in pull request #12440: URL: https://github.com/apache/beam/pull/12440#discussion_r466721877 ## File path: sdks/python/apache_beam/dataframe/doctests.py ## @@ -290,34 +298,60 @@ def to_callable(cond): super(BeamDataframeDoctestRunner,

[GitHub] [beam] lukecwik commented on pull request #12488: [BEAM-10656] Enable bundle finalization within the Java direct runner.

2020-08-06 Thread GitBox
lukecwik commented on pull request #12488: URL: https://github.com/apache/beam/pull/12488#issuecomment-670223324 R: @amaliujia @ibzib CC: @boyuanzz This should enable testing unbounded SDFs using the direct runner with bundle finalization callbacks.

[GitHub] [beam] lukecwik opened a new pull request #12488: [BEAM-10656] Enable bundle finalization within the Java direct runner.

2020-08-06 Thread GitBox
lukecwik opened a new pull request #12488: URL: https://github.com/apache/beam/pull/12488 This is towards making all UnboundedSources execute as splittable dofns within the direct runner using the SDF unbounded source wrapper since it relies on bundle finalization to handle checkpoints.

[GitHub] [beam] ettirapp commented on pull request #12487: Fixing Typos in GCS and S3 Filesystem Tests

2020-08-06 Thread GitBox
ettirapp commented on pull request #12487: URL: https://github.com/apache/beam/pull/12487#issuecomment-670219384 R: @pabloem This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [beam] ettirapp opened a new pull request #12487: Fixing Typos in GCS and S3 Filesystem Tests

2020-08-06 Thread GitBox
ettirapp opened a new pull request #12487: URL: https://github.com/apache/beam/pull/12487 Fixing typos in gcs filesystem tests and removing duplicate tests in s3 filesystem. Thank you for your contribution! Follow this checklist to help us

[GitHub] [beam] ibzib opened a new pull request #12486: [BEAM-9558] Remove usage of empty data/timers to signify last.

2020-08-06 Thread GitBox
ibzib opened a new pull request #12486: URL: https://github.com/apache/beam/pull/12486 **Please** add a meaningful description for your change here Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly

[GitHub] [beam] leiyiz commented on a change in pull request #12427: [BEAM-2855] nexmark python suite implement queries 0, 1, 2 and 9

2020-08-06 Thread GitBox
leiyiz commented on a change in pull request #12427: URL: https://github.com/apache/beam/pull/12427#discussion_r466705734 ## File path: sdks/python/apache_beam/testing/benchmarks/nexmark/nexmark_util.py ## @@ -103,6 +121,107 @@ def process(self, elem): yield event

[GitHub] [beam] leiyiz commented on a change in pull request #12427: [BEAM-2855] nexmark python suite implement queries 0, 1, 2 and 9

2020-08-06 Thread GitBox
leiyiz commented on a change in pull request #12427: URL: https://github.com/apache/beam/pull/12427#discussion_r466704695 ## File path: sdks/python/apache_beam/testing/benchmarks/nexmark/queries/query2.py ## @@ -29,17 +29,15 @@ from __future__ import absolute_import import

[GitHub] [beam] leiyiz commented on a change in pull request #12427: [BEAM-2855] nexmark python suite implement queries 0, 1, 2 and 9

2020-08-06 Thread GitBox
leiyiz commented on a change in pull request #12427: URL: https://github.com/apache/beam/pull/12427#discussion_r466704612 ## File path: sdks/python/apache_beam/testing/benchmarks/nexmark/queries/query1.py ## @@ -29,20 +29,16 @@ import apache_beam as beam from

[GitHub] [beam] apilloud commented on pull request #12480: [BEAM-10647] Fixes get_query_location bug in BigQueryWrapper

2020-08-06 Thread GitBox
apilloud commented on pull request #12480: URL: https://github.com/apache/beam/pull/12480#issuecomment-670206489 Jenkins is having issues right now (see https://issues.apache.org/jira/browse/INFRA-20649) but will merge this when it is fixed. Please remind me if I forget!

[GitHub] [beam] apilloud commented on a change in pull request #12480: [BEAM-10647] Fixes get_query_location bug in BigQueryWrapper

2020-08-06 Thread GitBox
apilloud commented on a change in pull request #12480: URL: https://github.com/apache/beam/pull/12480#discussion_r466701216 ## File path: sdks/python/apache_beam/io/gcp/bigquery_tools_test.py ## @@ -50,6 +49,15 @@ from apache_beam.io.gcp.internal.clients import bigquery from

[GitHub] [beam] allenpradeep commented on pull request #12255: [BEAM-9910]Add mutation byte counters to SpannerIO.write

2020-08-06 Thread GitBox
allenpradeep commented on pull request #12255: URL: https://github.com/apache/beam/pull/12255#issuecomment-670204460 > @allenpradeep - any progress on this? @aaltay I had similar thoughts on this as @chamikaramj had with respect to performance. I'll would want to hear feedback

[GitHub] [beam] allenpradeep commented on pull request #12255: [BEAM-9910]Add mutation byte counters to SpannerIO.write

2020-08-06 Thread GitBox
allenpradeep commented on pull request #12255: URL: https://github.com/apache/beam/pull/12255#issuecomment-670202696 > @allenpradeep given that this seems to be a per-element change, have we done a performance analysis to confirm that there's won't be a significant performance degradation

[GitHub] [beam] galuszkak commented on a change in pull request #12480: [BEAM-10647] Fixes get_query_location bug in BigQueryWrapper

2020-08-06 Thread GitBox
galuszkak commented on a change in pull request #12480: URL: https://github.com/apache/beam/pull/12480#discussion_r466696088 ## File path: sdks/python/apache_beam/io/gcp/bigquery_tools.py ## @@ -318,15 +319,20 @@ def get_query_location(self, project_id, query,

[GitHub] [beam] pabloem commented on pull request #12485: [BEAM-6064] Improvements to BQ streaming insert performance

2020-08-06 Thread GitBox
pabloem commented on pull request #12485: URL: https://github.com/apache/beam/pull/12485#issuecomment-670199488 Run Python 3.8 PostCommit This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] robertwb merged pull request #12476: [BEAM-10645] Create context for allowing non-parallel dataframe operations.

2020-08-06 Thread GitBox
robertwb merged pull request #12476: URL: https://github.com/apache/beam/pull/12476 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [beam] robertwb commented on a change in pull request #12440: [BEAM-10619] Report ratio of implemented pandas tests

2020-08-06 Thread GitBox
robertwb commented on a change in pull request #12440: URL: https://github.com/apache/beam/pull/12440#discussion_r466686042 ## File path: sdks/python/apache_beam/dataframe/doctests.py ## @@ -290,34 +298,60 @@ def to_callable(cond): super(BeamDataframeDoctestRunner,

[GitHub] [beam] TheNeuralBit commented on a change in pull request #12440: [BEAM-10619] Report ratio of implemented pandas tests

2020-08-06 Thread GitBox
TheNeuralBit commented on a change in pull request #12440: URL: https://github.com/apache/beam/pull/12440#discussion_r466684332 ## File path: sdks/python/apache_beam/dataframe/doctests.py ## @@ -290,34 +298,60 @@ def to_callable(cond): super(BeamDataframeDoctestRunner,

[GitHub] [beam] pabloem opened a new pull request #12485: [BEAM-6064] Improvements to BQ streaming insert performance

2020-08-06 Thread GitBox
pabloem opened a new pull request #12485: URL: https://github.com/apache/beam/pull/12485 Consider results from this analysis: https://docs.google.com/document/d/1s8VRkN4qKdgGkDOZQiwowmD3GVyVV9UskJQwTdKfRCE/edit# This change helps increase EPS per worker from ~100 to >1000 per

[GitHub] [beam] pabloem commented on pull request #12485: [BEAM-6064] Improvements to BQ streaming insert performance

2020-08-06 Thread GitBox
pabloem commented on pull request #12485: URL: https://github.com/apache/beam/pull/12485#issuecomment-670189538 Run Python 3.8 PostCommit This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] boyuanzz commented on a change in pull request #11749: [BEAM-9977] Implement ReadFromKafkaViaSDF

2020-08-06 Thread GitBox
boyuanzz commented on a change in pull request #11749: URL: https://github.com/apache/beam/pull/11749#discussion_r466683752 ## File path: sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/ReadFromKafkaDoFn.java ## @@ -0,0 +1,403 @@ +/* + * Licensed to the Apache

[GitHub] [beam] boyuanzz commented on a change in pull request #11749: [BEAM-9977] Implement ReadFromKafkaViaSDF

2020-08-06 Thread GitBox
boyuanzz commented on a change in pull request #11749: URL: https://github.com/apache/beam/pull/11749#discussion_r466680059 ## File path: sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java ## @@ -198,6 +209,102 @@ *... * } * + * Read from

[GitHub] [beam] dpmills commented on a change in pull request #12474: [BEAM-10650] OrderedListState API

2020-08-06 Thread GitBox
dpmills commented on a change in pull request #12474: URL: https://github.com/apache/beam/pull/12474#discussion_r466677463 ## File path: sdks/java/core/src/main/java/org/apache/beam/sdk/state/OrderedListState.java ## @@ -0,0 +1,51 @@ +/* + * Licensed to the Apache Software

[GitHub] [beam] aaltay commented on pull request #12482: [BEAM-7390] Add top code snippets

2020-08-06 Thread GitBox
aaltay commented on pull request #12482: URL: https://github.com/apache/beam/pull/12482#issuecomment-670180992 > @aaltay Looking at the PR checks, it looks like the staging never ran. How can we trigger it? I do not know. You can ask on dev@ list?

[GitHub] [beam] boyuanzz commented on a change in pull request #11749: [BEAM-9977] Implement ReadFromKafkaViaSDF

2020-08-06 Thread GitBox
boyuanzz commented on a change in pull request #11749: URL: https://github.com/apache/beam/pull/11749#discussion_r466673488 ## File path: sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java ## @@ -1051,33 +1198,352 @@ public void

[GitHub] [beam] robinyqiu commented on pull request #12483: Support NULL query parameters in ZetaSQL

2020-08-06 Thread GitBox
robinyqiu commented on pull request #12483: URL: https://github.com/apache/beam/pull/12483#issuecomment-670176079 Tested internally. No regression found, except for the known issue tracked in https://issues.apache.org/jira/browse/BEAM-9191 and

[GitHub] [beam] reuvenlax commented on a change in pull request #12474: [BEAM-10650] OrderedListState API

2020-08-06 Thread GitBox
reuvenlax commented on a change in pull request #12474: URL: https://github.com/apache/beam/pull/12474#discussion_r46953 ## File path: sdks/java/core/src/main/java/org/apache/beam/sdk/state/OrderedListState.java ## @@ -0,0 +1,60 @@ +/* + * Licensed to the Apache Software

[GitHub] [beam] boyuanzz commented on a change in pull request #11749: [BEAM-9977] Implement ReadFromKafkaViaSDF

2020-08-06 Thread GitBox
boyuanzz commented on a change in pull request #11749: URL: https://github.com/apache/beam/pull/11749#discussion_r46972 ## File path: sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java ## @@ -1051,33 +1198,352 @@ public void

[GitHub] [beam] ibzib opened a new pull request #12484: [BEAM-10653] Modularize BeamSqlDslUdfUdafTest.

2020-08-06 Thread GitBox
ibzib opened a new pull request #12484: URL: https://github.com/apache/beam/pull/12484 Split tests with multiple, independent branches into separate test cases. The only part removed was in testUdaf. The two cases in testUdaf were different in the past, but converged at some

[GitHub] [beam] davidyan74 commented on a change in pull request #12413: [BEAM-10603] Add max count to utils.to_element_list

2020-08-06 Thread GitBox
davidyan74 commented on a change in pull request #12413: URL: https://github.com/apache/beam/pull/12413#discussion_r466657201 ## File path: sdks/python/apache_beam/runners/interactive/utils.py ## @@ -27,30 +27,46 @@ import pandas as pd from

[GitHub] [beam] tysonjh commented on pull request #12366: [BEAM-10572] Eliminate nullability errors from :sdks:java:extensions:sql:datacatalog

2020-08-06 Thread GitBox
tysonjh commented on pull request #12366: URL: https://github.com/apache/beam/pull/12366#issuecomment-670165578 > > Filed [kelloggm/checkerframework-gradle-plugin#117](https://github.com/kelloggm/checkerframework-gradle-plugin/issues/117). We don't have to wait for a fix, though. We can

[GitHub] [beam] tysonjh commented on a change in pull request #12435: [BEAM-10616] Added Python Pardo load tests for streaming on Dataflow

2020-08-06 Thread GitBox
tysonjh commented on a change in pull request #12435: URL: https://github.com/apache/beam/pull/12435#discussion_r466648416 ## File path: .test-infra/jenkins/job_LoadTests_ParDo_Python.groovy ## @@ -151,3 +151,35 @@

[GitHub] [beam] saavannanavati commented on pull request #12257: [BEAM-2762] Generate Python coverage reports during pre-commit

2020-08-06 Thread GitBox
saavannanavati commented on pull request #12257: URL: https://github.com/apache/beam/pull/12257#issuecomment-670161660 run python precommit This is an automated message from the Apache Git Service. To respond to the

[GitHub] [beam] aaltay commented on pull request #12342: [Beam-7925] Add column projection to ParquetIO

2020-08-06 Thread GitBox
aaltay commented on pull request #12342: URL: https://github.com/apache/beam/pull/12342#issuecomment-670161122 /cc @chamikaramj This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [beam] aaltay commented on pull request #12381: GH-Actions workflow checks are GCP variables set [depends on BEAM-10599]

2020-08-06 Thread GitBox
aaltay commented on pull request #12381: URL: https://github.com/apache/beam/pull/12381#issuecomment-670160482 https://github.com/apache/beam/pull/12405 was merged. Could this be updated? This is an automated message from

[GitHub] [beam] aaltay commented on pull request #12255: [BEAM-9910]Add mutation byte counters to SpannerIO.write

2020-08-06 Thread GitBox
aaltay commented on pull request #12255: URL: https://github.com/apache/beam/pull/12255#issuecomment-670160224 @allenpradeep - any progress on this? This is an automated message from the Apache Git Service. To respond to the

[GitHub] [beam] aaltay merged pull request #12424: [BEAM-8543] Dataflow streaming timers are not strictly time ordered when set earlier mid-bundle

2020-08-06 Thread GitBox
aaltay merged pull request #12424: URL: https://github.com/apache/beam/pull/12424 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [beam] aaltay commented on pull request #12389: [BEAM-10587] Support Maps in BigQuery

2020-08-06 Thread GitBox
aaltay commented on pull request #12389: URL: https://github.com/apache/beam/pull/12389#issuecomment-670159620 Hey @rworley-monster - any progress on this? Do you need help? This is an automated message from the Apache Git

[GitHub] [beam] rohdesamuel commented on pull request #12413: [BEAM-10603] Add max count to utils.to_element_list

2020-08-06 Thread GitBox
rohdesamuel commented on pull request #12413: URL: https://github.com/apache/beam/pull/12413#issuecomment-670158103 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [beam] chamikaramj merged pull request #12423: [BEAM-10135][BEAM-10136] Refactor jdbc external transform registrar

2020-08-06 Thread GitBox
chamikaramj merged pull request #12423: URL: https://github.com/apache/beam/pull/12423 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [beam] chamikaramj commented on pull request #12423: [BEAM-10135][BEAM-10136] Refactor jdbc external transform registrar

2020-08-06 Thread GitBox
chamikaramj commented on pull request #12423: URL: https://github.com/apache/beam/pull/12423#issuecomment-670157940 We can merge this. BTW I'm curious, have you tested the jdbc.py against Dataflow ? If not can you try that ? Instructions for running x-lang wrappers against

[GitHub] [beam] chamikaramj merged pull request #12325: [BEAM-10543] Upgrade Kafka cross-language python tests

2020-08-06 Thread GitBox
chamikaramj merged pull request #12325: URL: https://github.com/apache/beam/pull/12325 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [beam] TheNeuralBit commented on a change in pull request #12452: [BEAM-10623] Add workflow to run Beam python tests on Linux/Windows/Mac platforms

2020-08-06 Thread GitBox
TheNeuralBit commented on a change in pull request #12452: URL: https://github.com/apache/beam/pull/12452#discussion_r466635592 ## File path: sdks/python/apache_beam/testing/datatype_inference_test.py ## @@ -174,6 +175,7 @@ def test_infer_typehints_schema(self, _, data,

[GitHub] [beam] piotr-szuberski commented on pull request #12422: [BEAM-601] Run KinesisIOIT with localstack

2020-08-06 Thread GitBox
piotr-szuberski commented on pull request #12422: URL: https://github.com/apache/beam/pull/12422#issuecomment-670143609 Run Java PostCommit This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] piotr-szuberski commented on pull request #12422: [BEAM-601] Run KinesisIOIT with localstack

2020-08-06 Thread GitBox
piotr-szuberski commented on pull request #12422: URL: https://github.com/apache/beam/pull/12422#issuecomment-670143022 @aromanenko-dev I've done what you suggested without one thing that is left to discuss (the unresolved comment)

[GitHub] [beam] piotr-szuberski commented on a change in pull request #12422: [BEAM-601] Run KinesisIOIT with localstack

2020-08-06 Thread GitBox
piotr-szuberski commented on a change in pull request #12422: URL: https://github.com/apache/beam/pull/12422#discussion_r466629615 ## File path: sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/BasicKinesisProvider.java ## @@ -39,16 +40,27 @@ private final

[GitHub] [beam] lostluck merged pull request #12471: [BEAM-9615] Add initial Schema to Go conversions.

2020-08-06 Thread GitBox
lostluck merged pull request #12471: URL: https://github.com/apache/beam/pull/12471 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [beam] piotr-szuberski commented on a change in pull request #12422: [BEAM-601] Run KinesisIOIT with localstack

2020-08-06 Thread GitBox
piotr-szuberski commented on a change in pull request #12422: URL: https://github.com/apache/beam/pull/12422#discussion_r466621223 ## File path: sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/KinesisIOIT.java ## @@ -95,32 +123,114 @@ private void runRead()

[GitHub] [beam] piotr-szuberski commented on a change in pull request #12422: [BEAM-601] Run KinesisIOIT with localstack

2020-08-06 Thread GitBox
piotr-szuberski commented on a change in pull request #12422: URL: https://github.com/apache/beam/pull/12422#discussion_r466620915 ## File path: sdks/java/io/kinesis/build.gradle ## @@ -50,6 +50,7 @@ dependencies { testCompile library.java.powermock testCompile

[GitHub] [beam] piotr-szuberski commented on a change in pull request #12422: [BEAM-601] Run KinesisIOIT with localstack

2020-08-06 Thread GitBox
piotr-szuberski commented on a change in pull request #12422: URL: https://github.com/apache/beam/pull/12422#discussion_r466621117 ## File path: sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/KinesisIO.java ## @@ -413,6 +413,28 @@ public Read

[GitHub] [beam] amaliujia merged pull request #12461: [BEAM-10633] UdfImpl hits NPE when returns a List

2020-08-06 Thread GitBox
amaliujia merged pull request #12461: URL: https://github.com/apache/beam/pull/12461 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [beam] amaliujia commented on pull request #12461: [BEAM-10633] UdfImpl hits NPE when returns a List

2020-08-06 Thread GitBox
amaliujia commented on pull request #12461: URL: https://github.com/apache/beam/pull/12461#issuecomment-670133998 +1 on refactoring it in another PR. This is an automated message from the Apache Git Service. To respond to

[GitHub] [beam] ibzib commented on a change in pull request #12461: [BEAM-10633] UdfImpl hits NPE when returns a List

2020-08-06 Thread GitBox
ibzib commented on a change in pull request #12461: URL: https://github.com/apache/beam/pull/12461#discussion_r466619308 ## File path: sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslUdfUdafTest.java ## @@ -101,6 +104,21 @@ public void

[GitHub] [beam] robinyqiu opened a new pull request #12483: Support NULL query parameters in ZetaSQL

2020-08-06 Thread GitBox
robinyqiu opened a new pull request #12483: URL: https://github.com/apache/beam/pull/12483 Previously we substitute all parameters with literals. This works in most cases except when the parameter is NULL, because some operations allows NULL parameter but not NULL literal.

[GitHub] [beam] udim commented on pull request #12257: [BEAM-2762] Generate Python coverage reports during pre-commit

2020-08-06 Thread GitBox
udim commented on pull request #12257: URL: https://github.com/apache/beam/pull/12257#issuecomment-670131768 run python precommit This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [beam] ibzib commented on a change in pull request #12461: [BEAM-10633] UdfImpl hits NPE when returns a List

2020-08-06 Thread GitBox
ibzib commented on a change in pull request #12461: URL: https://github.com/apache/beam/pull/12461#discussion_r466615046 ## File path: sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslUdfUdafTest.java ## @@ -101,6 +104,21 @@ public void

[GitHub] [beam] amaliujia commented on a change in pull request #12481: [BEAM-10571] Use schemas in ExternalConfigurationPayload

2020-08-06 Thread GitBox
amaliujia commented on a change in pull request #12481: URL: https://github.com/apache/beam/pull/12481#discussion_r466614630 ## File path: runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/ValidateRunnerXlangTest.java ## @@ -222,14 +228,21

[GitHub] [beam] rohdesamuel commented on pull request #12412: [BEAM-10603] Add ElementLimiters to all Cache Managers.

2020-08-06 Thread GitBox
rohdesamuel commented on pull request #12412: URL: https://github.com/apache/beam/pull/12412#issuecomment-670124406 R: @pabloem This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [beam] saavannanavati commented on a change in pull request #12257: [BEAM-2762] Generate Python coverage reports during pre-commit

2020-08-06 Thread GitBox
saavannanavati commented on a change in pull request #12257: URL: https://github.com/apache/beam/pull/12257#discussion_r466614068 ## File path: sdks/python/test-suites/tox/py38/build.gradle ## @@ -32,4 +32,4 @@ check.dependsOn formatter apply from: "../common.gradle" //

[GitHub] [beam] udim commented on a change in pull request #12257: [BEAM-2762] Generate Python coverage reports during pre-commit

2020-08-06 Thread GitBox
udim commented on a change in pull request #12257: URL: https://github.com/apache/beam/pull/12257#discussion_r466612715 ## File path: sdks/python/test-suites/tox/py38/build.gradle ## @@ -32,4 +32,4 @@ check.dependsOn formatter apply from: "../common.gradle" //

[GitHub] [beam] TheNeuralBit commented on a change in pull request #12481: [BEAM-10571] Use schemas in ExternalConfigurationPayload

2020-08-06 Thread GitBox
TheNeuralBit commented on a change in pull request #12481: URL: https://github.com/apache/beam/pull/12481#discussion_r466611999 ## File path: sdks/java/expansion-service/build.gradle ## @@ -44,6 +44,8 @@ dependencies { compile library.java.slf4j_api runtimeOnly

[GitHub] [beam] ananvay commented on pull request #11720: [BEAM-9729] Ignore bundle registration requests.

2020-08-06 Thread GitBox
ananvay commented on pull request #11720: URL: https://github.com/apache/beam/pull/11720#issuecomment-670116185 /cc: @y1chi This is an automated message from the Apache Git Service. To respond to the message, please log on

  1   2   >