[GitHub] [beam] mwalenia commented on pull request #11534: Push ioit tests metrics to influxdb

2020-05-05 Thread GitBox
mwalenia commented on pull request #11534: URL: https://github.com/apache/beam/pull/11534#issuecomment-624468886 retest this please This is an automated message from the Apache Git Service. To respond to the message, please l

[GitHub] [beam] mwalenia commented on pull request #11534: Push ioit tests metrics to influxdb

2020-05-05 Thread GitBox
mwalenia commented on pull request #11534: URL: https://github.com/apache/beam/pull/11534#issuecomment-624469096 If the tests pass, I'm giving it a LGTM :) Thanks for the contribution! This is an automated message from the Ap

[GitHub] [beam] bashir2 commented on pull request #11617: Update programming-guide.md with some typo fixes.

2020-05-05 Thread GitBox
bashir2 commented on pull request #11617: URL: https://github.com/apache/beam/pull/11617#issuecomment-624460783 Reviewer: @melap per website/OWNERS file. This is an automated message from the Apache Git Service. To respond to

[GitHub] [beam] bashir2 opened a new pull request #11617: Update programming-guide.md with some typo fixes.

2020-05-05 Thread GitBox
bashir2 opened a new pull request #11617: URL: https://github.com/apache/beam/pull/11617 **Please** add a meaningful description for your change here Just fixed some typos like "Oten", "the the". Thank you for your contribution! Follow this check

[GitHub] [beam] robinyqiu commented on pull request #11272: [BEAM-9641] Support ZetaSQL DATE type as a Beam LogicalType

2020-05-05 Thread GitBox
robinyqiu commented on pull request #11272: URL: https://github.com/apache/beam/pull/11272#issuecomment-624452448 Could you help trigger the tests again? For the comment on range: Thanks for pointing it out. I overlooked this problem. I would like to create a separate PR to address i

[GitHub] [beam] robinyqiu commented on a change in pull request #11272: [BEAM-9641] Support ZetaSQL DATE type as a Beam LogicalType

2020-05-05 Thread GitBox
robinyqiu commented on a change in pull request #11272: URL: https://github.com/apache/beam/pull/11272#discussion_r418871819 ## File path: sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamCalcRel.java ## @@ -427,17 +430,12 @@ private stati

[GitHub] [beam] robinyqiu commented on pull request #11272: [BEAM-9641] Support ZetaSQL DATE type as a Beam LogicalType

2020-05-05 Thread GitBox
robinyqiu commented on pull request #11272: URL: https://github.com/apache/beam/pull/11272#issuecomment-624451644 Ah, just realized that the previous comments were not sent out. This is an automated message from the Apache Gi

[GitHub] [beam] pabloem commented on pull request #11086: [BEAM-8910] Make custom BQ source read from Avro

2020-05-05 Thread GitBox
pabloem commented on pull request #11086: URL: https://github.com/apache/beam/pull/11086#issuecomment-624431077 Run PythonDocker PreCommit This is an automated message from the Apache Git Service. To respond to the message, p

[GitHub] [beam] pabloem commented on pull request #11086: [BEAM-8910] Make custom BQ source read from Avro

2020-05-05 Thread GitBox
pabloem commented on pull request #11086: URL: https://github.com/apache/beam/pull/11086#issuecomment-624431173 Run Python2_PVR_Flink PreCommit This is an automated message from the Apache Git Service. To respond to the messa

[GitHub] [beam] pabloem commented on pull request #11086: [BEAM-8910] Make custom BQ source read from Avro

2020-05-05 Thread GitBox
pabloem commented on pull request #11086: URL: https://github.com/apache/beam/pull/11086#issuecomment-624430895 Run Python 2 PostCommit This is an automated message from the Apache Git Service. To respond to the message, plea

[GitHub] [beam] pabloem commented on pull request #11086: [BEAM-8910] Make custom BQ source read from Avro

2020-05-05 Thread GitBox
pabloem commented on pull request #11086: URL: https://github.com/apache/beam/pull/11086#issuecomment-624431013 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [beam] stale[bot] commented on pull request #10791: [BEAM-9250] Update release guide with more instructions.

2020-05-05 Thread GitBox
stale[bot] commented on pull request #10791: URL: https://github.com/apache/beam/pull/10791#issuecomment-624403721 This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull

[GitHub] [beam] stale[bot] commented on pull request #11034: [BEAM-9424] Allow grouping by LogicalType

2020-05-05 Thread GitBox
stale[bot] commented on pull request #11034: URL: https://github.com/apache/beam/pull/11034#issuecomment-624403737 This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull

[GitHub] [beam] rahul8383 commented on a change in pull request #11609: [BEAM-9887] Throw IllegalArgumentException when building Row with logical types with Invalid input

2020-05-05 Thread GitBox
rahul8383 commented on a change in pull request #11609: URL: https://github.com/apache/beam/pull/11609#discussion_r420487951 ## File path: sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/logicaltypes/LogicalTypesTest.java ## @@ -97,4 +99,19 @@ public void testNanosDur

[GitHub] [beam] rahul8383 commented on a change in pull request #11609: [BEAM-9887] Throw IllegalArgumentException when building Row with logical types with Invalid input

2020-05-05 Thread GitBox
rahul8383 commented on a change in pull request #11609: URL: https://github.com/apache/beam/pull/11609#discussion_r420486636 ## File path: sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/logicaltypes/LogicalTypesTest.java ## @@ -97,4 +99,19 @@ public void testNanosDur

[GitHub] [beam] pabloem commented on pull request #11086: [BEAM-8910] Make custom BQ source read from Avro

2020-05-05 Thread GitBox
pabloem commented on pull request #11086: URL: https://github.com/apache/beam/pull/11086#issuecomment-624370689 Run Python 3.7 PostCommit This is an automated message from the Apache Git Service. To respond to the message, pl

[GitHub] [beam] amaliujia commented on pull request #11581: [BEAM-8307] NPE in Calcite dialect when input PCollection has logical…

2020-05-05 Thread GitBox
amaliujia commented on pull request #11581: URL: https://github.com/apache/beam/pull/11581#issuecomment-624366230 retest this please This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [beam] iemejia commented on pull request #11613: [BEAM-9833] Fix yaml issues, sort labels and merge button as only strategy

2020-05-05 Thread GitBox
iemejia commented on pull request #11613: URL: https://github.com/apache/beam/pull/11613#issuecomment-624365589 I am ok with squashing if it creates extra commits, but it does not seem to be the case or does it create the additional extra merge commit? I just want to ensure we follow the r

[GitHub] [beam] pabloem commented on pull request #11613: [BEAM-9833] Fix yaml issues, sort labels and merge button as only strategy

2020-05-05 Thread GitBox
pabloem commented on pull request #11613: URL: https://github.com/apache/beam/pull/11613#issuecomment-624363164 I think squash is used by many, and it facilitates receiving contributions without the extra round trip to the contributors. I think we need to discuss more before removing it. W

[GitHub] [beam] pabloem commented on a change in pull request #11590: [BEAM-8944] Improve UnboundedThreadPoolExecutor performance

2020-05-05 Thread GitBox
pabloem commented on a change in pull request #11590: URL: https://github.com/apache/beam/pull/11590#discussion_r420468171 ## File path: sdks/python/apache_beam/utils/thread_pool_executor.py ## @@ -137,35 +101,33 @@ def submit(self, fn, *args, **kwargs): """ future =

[GitHub] [beam] rahul8383 commented on pull request #11609: [BEAM-9887] Throw IllegalArgumentException when building Row with logical types with Invalid input

2020-05-05 Thread GitBox
rahul8383 commented on pull request #11609: URL: https://github.com/apache/beam/pull/11609#issuecomment-624360661 > We always convert logical types to their base type when serializing with SchemaCoder, and convert back to the input type when deserializing. Other than that I think the only

[GitHub] [beam] aaltay commented on pull request #11616: Use csv reader instead of split to read csv data.

2020-05-05 Thread GitBox
aaltay commented on pull request #11616: URL: https://github.com/apache/beam/pull/11616#issuecomment-624360261 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [beam] KevinGG commented on pull request #11616: Use csv reader instead of split to read csv data.

2020-05-05 Thread GitBox
KevinGG commented on pull request #11616: URL: https://github.com/apache/beam/pull/11616#issuecomment-624359913 R: @aaltay PTAL, thx! This is an automated message from the Apache Git Service. To respond to the message

[GitHub] [beam] TheNeuralBit commented on pull request #11554: [BEAM-9876] Migrate the Beam website from Jekyll to Hugo to enable localization of the site content

2020-05-05 Thread GitBox
TheNeuralBit commented on pull request #11554: URL: https://github.com/apache/beam/pull/11554#issuecomment-624359510 Ok I removed `beam_PreCommit_Website_Commit/src/website/www` and `beam_PreCommit_Website_Stage_GCS_Commit/src/website/www` on `apache-beam-jenkins-{1..15} `. Hopefully that

[GitHub] [beam] pabloem commented on pull request #11086: [BEAM-8910] Make custom BQ source read from Avro

2020-05-05 Thread GitBox
pabloem commented on pull request #11086: URL: https://github.com/apache/beam/pull/11086#issuecomment-624356609 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

[GitHub] [beam] iemejia commented on a change in pull request #11614: Disable two flaky tests (BEAM-8035, BEAM-9164)

2020-05-05 Thread GitBox
iemejia commented on a change in pull request #11614: URL: https://github.com/apache/beam/pull/11614#discussion_r420461169 ## File path: runners/flink/src/test/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/UnboundedSourceWrapperTest.java ## @@ -242,6 +24

[GitHub] [beam] iemejia commented on pull request #11613: [BEAM-9833] Fix yaml issues, sort labels and merge button as only strategy

2020-05-05 Thread GitBox
iemejia commented on pull request #11613: URL: https://github.com/apache/beam/pull/11613#issuecomment-624353261 @robertwb Agree, a consensus that nobody respects :). In the case of this PR I set into the merge approach because we have only [three options](https://help.github.com/en/github/

[GitHub] [beam] KevinGG opened a new pull request #11616: Use csv reader instead of split to read csv data.

2020-05-05 Thread GitBox
KevinGG opened a new pull request #11616: URL: https://github.com/apache/beam/pull/11616 There might be comma in the csv formatted data itself. A naive split will generate malformed data and cause errors. Using `csv.reader` solves the issue. Thank you for

[GitHub] [beam] pabloem commented on pull request #11296: [BEAM-9640] Sketching watermark tracking on FnApiRunner

2020-05-05 Thread GitBox
pabloem commented on pull request #11296: URL: https://github.com/apache/beam/pull/11296#issuecomment-624351936 > Sorry it took so long to get to this. Most of my questions are around watermark advancement. no worries. This is a critical component, and I have other work to do, so I'

[GitHub] [beam] pabloem commented on a change in pull request #11582: [BEAM-9650] Add ReadAllFromBigQuery PTransform

2020-05-05 Thread GitBox
pabloem commented on a change in pull request #11582: URL: https://github.com/apache/beam/pull/11582#discussion_r420456908 ## File path: sdks/python/apache_beam/io/gcp/bigquery.py ## @@ -1641,3 +1644,314 @@ def process(self, unused_element, signal): *self._args

[GitHub] [beam] TheNeuralBit commented on pull request #11554: [BEAM-9876] Migrate the Beam website from Jekyll to Hugo to enable localization of the site content

2020-05-05 Thread GitBox
TheNeuralBit commented on pull request #11554: URL: https://github.com/apache/beam/pull/11554#issuecomment-624346035 We need to stop re-testing this. Its creating a bunch of files owned by root on the workers, so then subsequent runs fail because they don't have permissions to clean it up:

[GitHub] [beam] damondouglas commented on a change in pull request #11564: [Beam-9679] Add Core Transforms section / Map lesson to the Go SDK katas

2020-05-05 Thread GitBox
damondouglas commented on a change in pull request #11564: URL: https://github.com/apache/beam/pull/11564#discussion_r420451192 ## File path: learning/katas/go/Core Transforms/Map/ParDo OneToMany/task.md ## @@ -0,0 +1,32 @@ + + +# ParDo - One to Many + +In the previous kata we

[GitHub] [beam] pabloem commented on pull request #11554: [BEAM-9876] Migrate the Beam website from Jekyll to Hugo to enable localization of the site content

2020-05-05 Thread GitBox
pabloem commented on pull request #11554: URL: https://github.com/apache/beam/pull/11554#issuecomment-624343145 retest this please This is an automated message from the Apache Git Service. To respond to the message, please lo

[GitHub] [beam] pabloem commented on a change in pull request #11582: [BEAM-9650] Add ReadAllFromBigQuery PTransform

2020-05-05 Thread GitBox
pabloem commented on a change in pull request #11582: URL: https://github.com/apache/beam/pull/11582#discussion_r420386613 ## File path: sdks/python/apache_beam/io/gcp/bigquery.py ## @@ -1641,3 +1644,314 @@ def process(self, unused_element, signal): *self._args

[GitHub] [beam] pabloem commented on pull request #11554: [BEAM-9876] Migrate the Beam website from Jekyll to Hugo to enable localization of the site content

2020-05-05 Thread GitBox
pabloem commented on pull request #11554: URL: https://github.com/apache/beam/pull/11554#issuecomment-624337831 retest this please This is an automated message from the Apache Git Service. To respond to the message, please lo

[GitHub] [beam] pabloem commented on pull request #11554: [BEAM-9876] Migrate the Beam website from Jekyll to Hugo to enable localization of the site content

2020-05-05 Thread GitBox
pabloem commented on pull request #11554: URL: https://github.com/apache/beam/pull/11554#issuecomment-624337083 retest this please This is an automated message from the Apache Git Service. To respond to the message, pleas

[GitHub] [beam] lukecwik commented on pull request #11590: [BEAM-8944] Improve UnboundedThreadPoolExecutor performance

2020-05-05 Thread GitBox
lukecwik commented on pull request #11590: URL: https://github.com/apache/beam/pull/11590#issuecomment-624329537 Run Python Load Tests ParDo Flink Streaming This is an automated message from the Apache Git Service. To respond

[GitHub] [beam] reuvenlax commented on a change in pull request #11350: [BEAM-1589] Added @OnWindowExpiration annotation.

2020-05-05 Thread GitBox
reuvenlax commented on a change in pull request #11350: URL: https://github.com/apache/beam/pull/11350#discussion_r420423316 ## File path: runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java ## @@ -857,6 +863,223 @@ public BundleFinalizer bundleF

[GitHub] [beam] thetorpedodog opened a new pull request #11615: passert.Equals: sort output strings for easier reading

2020-05-05 Thread GitBox
thetorpedodog opened a new pull request #11615: URL: https://github.com/apache/beam/pull/11615 R: @lostluck --- Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [![Build Status](https://builds.apache.org/j

[GitHub] [beam] pabloem commented on pull request #11554: [BEAM-9876] Migrate the Beam website from Jekyll to Hugo to enable localization of the site content

2020-05-05 Thread GitBox
pabloem commented on pull request #11554: URL: https://github.com/apache/beam/pull/11554#issuecomment-624321585 retest this please This is an automated message from the Apache Git Service. To respond to the message, please lo

[GitHub] [beam] pabloem commented on pull request #11554: [BEAM-9876] Migrate the Beam website from Jekyll to Hugo to enable localization of the site content

2020-05-05 Thread GitBox
pabloem commented on pull request #11554: URL: https://github.com/apache/beam/pull/11554#issuecomment-624319547 retest this please This is an automated message from the Apache Git Service. To respond to the message, please lo

[GitHub] [beam] robertwb commented on a change in pull request #11610: [BEAM-9825] | Implement Intersect,Union,Except transforms

2020-05-05 Thread GitBox
robertwb commented on a change in pull request #11610: URL: https://github.com/apache/beam/pull/11610#discussion_r420415887 ## File path: sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/SetFns.java ## @@ -0,0 +1,261 @@ +/* + * Licensed to the Apache Software Founda

[GitHub] [beam] je-ik commented on pull request #11614: Disable two flaky tests (BEAM-8035, BEAM-9164)

2020-05-05 Thread GitBox
je-ik commented on pull request #11614: URL: https://github.com/apache/beam/pull/11614#issuecomment-624317271 @robertwb Looks like we should introduce some measures to solve this (not sure which measures these should be), because like we are accumulating these ignored tests: ``` ~/gi

[GitHub] [beam] tweise commented on a change in pull request #11558: [BEAM-8742] Add stateful and timely processing benchmarks

2020-05-05 Thread GitBox
tweise commented on a change in pull request #11558: URL: https://github.com/apache/beam/pull/11558#discussion_r420389628 ## File path: .test-infra/jenkins/job_LoadTests_ParDo_Flink_Python.groovy ## @@ -142,9 +249,22 @@ PhraseTriggeringPostCommitBuilder.postCommitJob( 'Load

[GitHub] [beam] robertwb commented on pull request #11613: [BEAM-9833] Fix yaml issues, sort labels and merge button as only strategy

2020-05-05 Thread GitBox
robertwb commented on pull request #11613: URL: https://github.com/apache/beam/pull/11613#issuecomment-624311669 I thought consensus was that we did want to encourage squash for those PRs with a huge pile of fixup commits (and otherwise no semantically meaningful commits). -

[GitHub] [beam] robertwb commented on pull request #11614: Disable two flaky tests (BEAM-8035, BEAM-9164)

2020-05-05 Thread GitBox
robertwb commented on pull request #11614: URL: https://github.com/apache/beam/pull/11614#issuecomment-624310548 I'm not sure we have more visibility into disabled tests than the jira entry (which shouldn't get closed until the tests are fixed and/or deemed unneeded and deleted. ---

[GitHub] [beam] robertwb commented on a change in pull request #11296: [BEAM-9640] Sketching watermark tracking on FnApiRunner

2020-05-05 Thread GitBox
robertwb commented on a change in pull request #11296: URL: https://github.com/apache/beam/pull/11296#discussion_r420388089 ## File path: sdks/python/apache_beam/runners/portability/fn_api_runner/fn_runner.py ## @@ -363,16 +387,17 @@ def _run_bundle_multiple_times_for_testing(

[GitHub] [beam] udim commented on pull request #11038: [BEAM-7746] More typing fixes

2020-05-05 Thread GitBox
udim commented on pull request #11038: URL: https://github.com/apache/beam/pull/11038#issuecomment-624296105 > Hi everyone, I have some availability to finish this PR off now. I'm going to rebase it soon. @udim do you have the time to help me get this through review? Yeah, I'll make

[GitHub] [beam] je-ik commented on a change in pull request #11614: Disable two flaky tests (BEAM-8035, BEAM-9164)

2020-05-05 Thread GitBox
je-ik commented on a change in pull request #11614: URL: https://github.com/apache/beam/pull/11614#discussion_r420390006 ## File path: runners/flink/src/test/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/UnboundedSourceWrapperTest.java ## @@ -242,6 +243,

[GitHub] [beam] iemejia commented on a change in pull request #11614: Disable two flaky tests (BEAM-8035, BEAM-9164)

2020-05-05 Thread GitBox
iemejia commented on a change in pull request #11614: URL: https://github.com/apache/beam/pull/11614#discussion_r420387278 ## File path: runners/flink/src/test/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/UnboundedSourceWrapperTest.java ## @@ -242,6 +24

[GitHub] [beam] ibzib commented on pull request #11403: [DO NOT MERGE] Run all PostCommit and PreCommit Tests against Release Branch

2020-05-05 Thread GitBox
ibzib commented on pull request #11403: URL: https://github.com/apache/beam/pull/11403#issuecomment-624286892 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[GitHub] [beam] je-ik commented on pull request #11614: Disable two flaky tests (BEAM-8035, BEAM-9164)

2020-05-05 Thread GitBox
je-ik commented on pull request #11614: URL: https://github.com/apache/beam/pull/11614#issuecomment-624285935 @robertwb Thanks for approval, do we have a way of visualizing ignored tests? I'm a little afraid these test might get ignored for ever, which might be unfortunate. -

[GitHub] [beam] je-ik opened a new pull request #11614: Disable two flaky tests (BEAM-8035, BEAM-9164)

2020-05-05 Thread GitBox
je-ik opened a new pull request #11614: URL: https://github.com/apache/beam/pull/11614 Disables two flaky tests until resolution is found. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easil

[GitHub] [beam] Ardagan commented on pull request #11555: [BEAM-8134] Grafana dashboards for Load Tests and IO IT Performance Tests

2020-05-05 Thread GitBox
Ardagan commented on pull request #11555: URL: https://github.com/apache/beam/pull/11555#issuecomment-624265259 Hey Kamil, can we also add a proper landing page for metrics site? People regularly can't navigate to dashboards they need. Adding landing page with intuitive navigation would

[GitHub] [beam] mxm commented on pull request #11558: [BEAM-8742] Add stateful and timely processing benchmarks

2020-05-05 Thread GitBox
mxm commented on pull request #11558: URL: https://github.com/apache/beam/pull/11558#issuecomment-624264625 Run Python Load Tests ParDo Flink Streaming This is an automated message from the Apache Git Service. To respond to t

[GitHub] [beam] mxm removed a comment on pull request #11558: [BEAM-8742] Add stateful and timely processing benchmarks

2020-05-05 Thread GitBox
mxm removed a comment on pull request #11558: URL: https://github.com/apache/beam/pull/11558#issuecomment-624264392 Run Python Load Tests ParDo Flink Streaming This is an automated message from the Apache Git Service. To resp

[GitHub] [beam] mxm commented on pull request #11558: [BEAM-8742] Add stateful and timely processing benchmarks

2020-05-05 Thread GitBox
mxm commented on pull request #11558: URL: https://github.com/apache/beam/pull/11558#issuecomment-624264392 Run Python Load Tests ParDo Flink Streaming This is an automated message from the Apache Git Service. To respond to t

[GitHub] [beam] mxm removed a comment on pull request #11558: [BEAM-8742] Add stateful and timely processing benchmarks

2020-05-05 Thread GitBox
mxm removed a comment on pull request #11558: URL: https://github.com/apache/beam/pull/11558#issuecomment-624229343 Run Python Load Tests ParDo Flink Streaming This is an automated message from the Apache Git Service. To resp

[GitHub] [beam] ihji commented on pull request #11574: [BEAM-9449] Pass PipelineOptions through expansion service

2020-05-05 Thread GitBox
ihji commented on pull request #11574: URL: https://github.com/apache/beam/pull/11574#issuecomment-624260369 > > I think we should also consider adding optional pipeline_options argument to ExternalTransform given that each different expansion service needs different pipeline options. >

[GitHub] [beam] aaltay commented on pull request #11555: [BEAM-8134] Grafana dashboards for Load Tests and IO IT Performance Tests

2020-05-05 Thread GitBox
aaltay commented on pull request #11555: URL: https://github.com/apache/beam/pull/11555#issuecomment-624248353 /cc @chamikaramj @tysonjh @kennknowles -- optional review request, if you would like to take a quick look at new benchmarks at http://metrics.beam.apache.org. (Instructions

[GitHub] [beam] aaltay commented on pull request #11555: [BEAM-8134] Grafana dashboards for Load Tests and IO IT Performance Tests

2020-05-05 Thread GitBox
aaltay commented on pull request #11555: URL: https://github.com/apache/beam/pull/11555#issuecomment-624247136 Some comments: - I do see missing data. (Example: http://metrics.beam.apache.org/d/fK0U4JqWz/cogbk-load-tests?orgId=1 -- all graphs missing recent data, java | coGBK | 100B rec

[GitHub] [beam] aaltay edited a comment on pull request #11555: [BEAM-8134] Grafana dashboards for Load Tests and IO IT Performance Tests

2020-05-05 Thread GitBox
aaltay edited a comment on pull request #11555: URL: https://github.com/apache/beam/pull/11555#issuecomment-624247136 Some comments: - I do see missing data. (Example: http://metrics.beam.apache.org/d/fK0U4JqWz/cogbk-load-tests?orgId=1 -- all graphs missing recent data, java | coGBK | 1

[GitHub] [beam] TheNeuralBit commented on a change in pull request #11609: [BEAM-9887] Throw IllegalArgumentException when building Row with logical types with Invalid input

2020-05-05 Thread GitBox
TheNeuralBit commented on a change in pull request #11609: URL: https://github.com/apache/beam/pull/11609#discussion_r420336807 ## File path: sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/logicaltypes/LogicalTypesTest.java ## @@ -97,4 +99,19 @@ public void testNanos

[GitHub] [beam] TheNeuralBit commented on pull request #11609: [BEAM-9887] Throw IllegalArgumentException when building Row with logical types with Invalid input

2020-05-05 Thread GitBox
TheNeuralBit commented on pull request #11609: URL: https://github.com/apache/beam/pull/11609#issuecomment-624243528 > Will this line ever get hit? We always convert logical types to their base type when serializing with SchemaCoder, and convert back to the input type when deserializ

[GitHub] [beam] aaltay commented on pull request #11555: [BEAM-8134] Grafana dashboards for Load Tests and IO IT Performance Tests

2020-05-05 Thread GitBox
aaltay commented on pull request #11555: URL: https://github.com/apache/beam/pull/11555#issuecomment-624242630 > Done. I pushed modified version to the website (http://metrics.beam.apache.org) I do not see the new dashboard here. How can I find it? I see these three: Code

[GitHub] [beam] mxm commented on pull request #11558: [BEAM-8742] Add stateful and timely processing benchmarks

2020-05-05 Thread GitBox
mxm commented on pull request #11558: URL: https://github.com/apache/beam/pull/11558#issuecomment-624229343 Run Python Load Tests ParDo Flink Streaming This is an automated message from the Apache Git Service. To respond to t

[GitHub] [beam] mxm commented on pull request #11558: [BEAM-8742] Add stateful and timely processing benchmarks

2020-05-05 Thread GitBox
mxm commented on pull request #11558: URL: https://github.com/apache/beam/pull/11558#issuecomment-624227730 Run Seed Job This is an automated message from the Apache Git Service. To respond to the message, please log on to Gi

[GitHub] [beam] je-ik commented on pull request #11612: [BEAM-9888] Drop data based on input watermark in @RequiresTimeSortedInput

2020-05-05 Thread GitBox
je-ik commented on pull request #11612: URL: https://github.com/apache/beam/pull/11612#issuecomment-624225596 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [beam] iemejia opened a new pull request #11613: [BEAM-9833] Fix yaml issues, sort labels and merge button as only strategy

2020-05-05 Thread GitBox
iemejia opened a new pull request #11613: URL: https://github.com/apache/beam/pull/11613 Issues reported by yamllint and some minor fixes. Also set merge button as the only strategy because we don't want to encourage (o even make possible the other two). R: @pabloem CC: @ibzib

[GitHub] [beam] Ardagan commented on pull request #11555: [BEAM-8134] Grafana dashboards for Load Tests and IO IT Performance Tests

2020-05-05 Thread GitBox
Ardagan commented on pull request #11555: URL: https://github.com/apache/beam/pull/11555#issuecomment-624216529 Some dashboards seem to miss data, but that's due to not all data migrated IIUC. LGTM otherwise. @aaltay can you take a look as well please? --

[GitHub] [beam] Ardagan edited a comment on pull request #11555: [BEAM-8134] Grafana dashboards for Load Tests and IO IT Performance Tests

2020-05-05 Thread GitBox
Ardagan edited a comment on pull request #11555: URL: https://github.com/apache/beam/pull/11555#issuecomment-624213555 > @aaltay > > > It would be great if we can make data-points clickable with links to relevant job > > Grafana has a feature called Data links [1] that could b

[GitHub] [beam] Ardagan commented on pull request #11555: [BEAM-8134] Grafana dashboards for Load Tests and IO IT Performance Tests

2020-05-05 Thread GitBox
Ardagan commented on pull request #11555: URL: https://github.com/apache/beam/pull/11555#issuecomment-624213555 > @aaltay > > > It would be great if we can make data-points clickable with links to relevant job > > Grafana has a feature called Data links [1] that could be use h

[GitHub] [beam] aaltay edited a comment on pull request #11210: [BEAM-8949] SpannerIO integration tests

2020-05-05 Thread GitBox
aaltay edited a comment on pull request #11210: URL: https://github.com/apache/beam/pull/11210#issuecomment-624213199 > @aaltay - Wordcount stream tests are failing. They are running successfully on my local machine. Jira ticket is already created for that - https://issues.apache.org/jira/

[GitHub] [beam] aaltay commented on pull request #11210: [BEAM-8949] SpannerIO integration tests

2020-05-05 Thread GitBox
aaltay commented on pull request #11210: URL: https://github.com/apache/beam/pull/11210#issuecomment-624213199 > @aaltay - Wordcount stream tests are failing. They are running successfully on my local machine. Jira ticket is already created for that - https://issues.apache.org/jira/browse/

[GitHub] [beam] aaltay commented on pull request #11210: [BEAM-8949] SpannerIO integration tests

2020-05-05 Thread GitBox
aaltay commented on pull request #11210: URL: https://github.com/apache/beam/pull/11210#issuecomment-624213031 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please l

[GitHub] [beam] Ardagan commented on pull request #11582: [BEAM-9650] Add ReadAllFromBigQuery PTransform

2020-05-05 Thread GitBox
Ardagan commented on pull request #11582: URL: https://github.com/apache/beam/pull/11582#issuecomment-624210475 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [beam] pabloem commented on pull request #11086: [BEAM-8910] Make custom BQ source read from Avro

2020-05-05 Thread GitBox
pabloem commented on pull request #11086: URL: https://github.com/apache/beam/pull/11086#issuecomment-624209882 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

[GitHub] [beam] Ardagan commented on a change in pull request #11582: [BEAM-9650] Add ReadAllFromBigQuery PTransform

2020-05-05 Thread GitBox
Ardagan commented on a change in pull request #11582: URL: https://github.com/apache/beam/pull/11582#discussion_r420286996 ## File path: sdks/python/apache_beam/io/gcp/bigquery.py ## @@ -283,6 +284,8 @@ def compute_table_name(row): 'BigQuerySink', 'WriteToBigQuery',

[GitHub] [beam] amaliujia commented on pull request #11610: [BEAM-9825] | Implement Intersect,Union,Except transforms

2020-05-05 Thread GitBox
amaliujia commented on pull request #11610: URL: https://github.com/apache/beam/pull/11610#issuecomment-624199945 @darshanj You can run `./gradlew spotlessApply` to fix checkstyle issues. You need to run `./gradlew ${module}:check` command to not only run tests but also run styl

[GitHub] [beam] amaliujia commented on pull request #11610: [BEAM-9825] | Implement Intersect,Union,Except transforms

2020-05-05 Thread GitBox
amaliujia commented on pull request #11610: URL: https://github.com/apache/beam/pull/11610#issuecomment-624199252 cc @Mark-Zeng to make sure I tagged the right person. This is an automated message from the Apache Git Service.

[GitHub] [beam] Ardagan commented on pull request #11477: [BEAM-9650] Add PeriodicSequence generator.

2020-05-05 Thread GitBox
Ardagan commented on pull request #11477: URL: https://github.com/apache/beam/pull/11477#issuecomment-624198780 Run Website_Stage_GCS PreCommit This is an automated message from the Apache Git Service. To respond to the messa

[GitHub] [beam] amaliujia edited a comment on pull request #11610: [BEAM-9825] | Implement Intersect,Union,Except transforms

2020-05-05 Thread GitBox
amaliujia edited a comment on pull request #11610: URL: https://github.com/apache/beam/pull/11610#issuecomment-624196073 R: @amaliujia cc: @jhnmora000 @Mark-Zeng (expose more PRs to GSoC students) This is an automated mes

[GitHub] [beam] je-ik commented on pull request #11612: [BEAM-9888] Drop data based on input watermark in @RequiresTimeSortedInput

2020-05-05 Thread GitBox
je-ik commented on pull request #11612: URL: https://github.com/apache/beam/pull/11612#issuecomment-624197391 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [beam] amaliujia commented on pull request #11610: [BEAM-9825] | Implement Intersect,Union,Except transforms

2020-05-05 Thread GitBox
amaliujia commented on pull request #11610: URL: https://github.com/apache/beam/pull/11610#issuecomment-624196073 R: @amaliujia cc: @jhnmora000 @MarkZeng1998 (expose more PRs to GSoC students) This is an automated message

[GitHub] [beam] robertwb commented on pull request #11557: [BEAM-9845] Stage artifacts over expansion service.

2020-05-05 Thread GitBox
robertwb commented on pull request #11557: URL: https://github.com/apache/beam/pull/11557#issuecomment-624188539 Thanks. Those suites passed locally, I'll look into what's going on here. This is an automated message from the

[GitHub] [beam] mxm commented on pull request #11558: [BEAM-8742] Add stateful and timely processing benchmarks

2020-05-05 Thread GitBox
mxm commented on pull request #11558: URL: https://github.com/apache/beam/pull/11558#issuecomment-624184021 Run Python Load Tests ParDo Flink Streaming This is an automated message from the Apache Git Service. To respond to t

[GitHub] [beam] mxm commented on pull request #11558: [BEAM-8742] Add stateful and timely processing benchmarks

2020-05-05 Thread GitBox
mxm commented on pull request #11558: URL: https://github.com/apache/beam/pull/11558#issuecomment-624180353 Run Seed Job This is an automated message from the Apache Git Service. To respond to the message, please log on to Gi

[GitHub] [beam] kamilwu removed a comment on pull request #11274: [BEAM-9633] Add PubsubIO performance test

2020-05-05 Thread GitBox
kamilwu removed a comment on pull request #11274: URL: https://github.com/apache/beam/pull/11274#issuecomment-624175575 Retest this please This is an automated message from the Apache Git Service. To respond to the message, p

[GitHub] [beam] kamilwu commented on pull request #11274: [BEAM-9633] Add PubsubIO performance test

2020-05-05 Thread GitBox
kamilwu commented on pull request #11274: URL: https://github.com/apache/beam/pull/11274#issuecomment-624176044 Retest this please This is an automated message from the Apache Git Service. To respond to the message, please lo

[GitHub] [beam] kamilwu commented on pull request #11274: [BEAM-9633] Add PubsubIO performance test

2020-05-05 Thread GitBox
kamilwu commented on pull request #11274: URL: https://github.com/apache/beam/pull/11274#issuecomment-624175575 Retest this please This is an automated message from the Apache Git Service. To respond to the message, please lo

[GitHub] [beam] kamilwu commented on pull request #11555: [BEAM-8134] Grafana dashboards for Load Tests and IO IT Performance Tests

2020-05-05 Thread GitBox
kamilwu commented on pull request #11555: URL: https://github.com/apache/beam/pull/11555#issuecomment-624173530 @aaltay > It would be great if we can make data-points clickable with links to relevant job Grafana has a feature called Data links [1] that could be use here. But the

[GitHub] [beam] robertwb commented on pull request #11038: [BEAM-7746] More typing fixes

2020-05-05 Thread GitBox
robertwb commented on pull request #11038: URL: https://github.com/apache/beam/pull/11038#issuecomment-624170238 Yes, let's make it happen! I'll help out as well. On Mon, May 4, 2020 at 11:47 AM Chad Dombrova wrote: > > I've rebased onto master. We jumped from 32 errors to 260

[GitHub] [beam] TheNeuralBit commented on pull request #11574: [BEAM-9449] Pass PipelineOptions through expansion service

2020-05-05 Thread GitBox
TheNeuralBit commented on pull request #11574: URL: https://github.com/apache/beam/pull/11574#issuecomment-624169642 > I think we should also consider adding optional pipeline_options argument to ExternalTransform given that each different expansion service needs different pipeline options

[GitHub] [beam] je-ik commented on pull request #11612: [BEAM-9888] Drop data based on input watermark in @RequiresTimeSortedInput

2020-05-05 Thread GitBox
je-ik commented on pull request #11612: URL: https://github.com/apache/beam/pull/11612#issuecomment-624163277 cc @dmvk @JozoVilcek This is an automated message from the Apache Git Service. To respond to the message, please l

[GitHub] [beam] je-ik commented on pull request #11612: [BEAM-9888] Drop data based on input watermark in @RequiresTimeSortedInput

2020-05-05 Thread GitBox
je-ik commented on pull request #11612: URL: https://github.com/apache/beam/pull/11612#issuecomment-624162961 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [beam] reuvenlax edited a comment on pull request #11609: [BEAM-9887] Throw IllegalArgumentException when building Row with logical types with Invalid input

2020-05-05 Thread GitBox
reuvenlax edited a comment on pull request #11609: URL: https://github.com/apache/beam/pull/11609#issuecomment-624156907 @TheNeuralBit withFieldValue should replace addValues for most users. addValues is difficult and error prone and withFieldValues allows building a row based on named fie

[GitHub] [beam] chamikaramj commented on pull request #11607: [BEAM-9430] Makes sure the watermarks returned by estimators are within bounds

2020-05-05 Thread GitBox
chamikaramj commented on pull request #11607: URL: https://github.com/apache/beam/pull/11607#issuecomment-624161195 Closing this temporarily while I look into this further. This is an automated message from the Apache Git Ser

[GitHub] [beam] rahul8383 commented on pull request #11609: [BEAM-9887] Throw IllegalArgumentException when building Row with logical types with Invalid input

2020-05-05 Thread GitBox
rahul8383 commented on pull request #11609: URL: https://github.com/apache/beam/pull/11609#issuecomment-624160514 @reuvenlax @TheNeuralBit Thanks for the review. Could you please clarify the questions that I have asked above? I wanted to understand more about logical types for my P

[GitHub] [beam] reuvenlax commented on pull request #11609: [BEAM-9887] Throw IllegalArgumentException when building Row with logical types with Invalid input

2020-05-05 Thread GitBox
reuvenlax commented on pull request #11609: URL: https://github.com/apache/beam/pull/11609#issuecomment-624156907 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [beam] rahul8383 edited a comment on pull request #11609: [BEAM-9887] Throw IllegalArgumentException when building Row with logical types with Invalid input

2020-05-05 Thread GitBox
rahul8383 edited a comment on pull request #11609: URL: https://github.com/apache/beam/pull/11609#issuecomment-624153984 @TheNeuralBit `schemas.logicaltypes.LogicalTypesTest.testFixedBytesIllegalArgument` Test will fail even if `addValue()` method is used instead of `withFieldValue()` m

  1   2   >