Re: [PR] Skip calling Compute in RemoveSafeDeltaCounterCell if value exists [beam]

2024-04-15 Thread via GitHub
m-trieu commented on code in PR #30913: URL: https://github.com/apache/beam/pull/30913#discussion_r1566653940 ## runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/RemoveSafeDeltaCounterCell.java: ## @@ -62,6 +62,12 @@ public void

Re: [PR] [#30870]: support consumer polling timeout in KafkaIO expansion service [beam]

2024-04-15 Thread via GitHub
xianhualiu commented on PR #30915: URL: https://github.com/apache/beam/pull/30915#issuecomment-2058150496 @damccorm please merge. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [I] [Bug]: GRCPIO 1.62.1 deadlocks in Beam SDK worker with Dataflow [beam]

2024-04-15 Thread via GitHub
tvalentyn commented on issue #30867: URL: https://github.com/apache/beam/issues/30867#issuecomment-2058072150 There is a confirmed issue https://github.com/googleapis/python-bigtable/issues/949 that affects google-api-core and grpcio library, which caused a regression in Apache Beam

Re: [PR] Bump sqlparse from 0.4.4 to 0.5.0 in /sdks/python/container/py311 [beam]

2024-04-15 Thread via GitHub
dependabot[bot] commented on PR #30982: URL: https://github.com/apache/beam/pull/30982#issuecomment-2058065375 OK, I won't notify you again about this release, but will get in touch when a new version is available. If you'd rather skip all updates until the next major or minor version, let

Re: [PR] Bump sqlparse from 0.4.4 to 0.5.0 in /sdks/python/container/py311 [beam]

2024-04-15 Thread via GitHub
tvalentyn closed pull request #30982: Bump sqlparse from 0.4.4 to 0.5.0 in /sdks/python/container/py311 URL: https://github.com/apache/beam/pull/30982 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] Skip generating docs for yaml/integration_tests.py. [beam]

2024-04-15 Thread via GitHub
tvalentyn merged PR #30986: URL: https://github.com/apache/beam/pull/30986 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Skip generating docs for yaml/integration_tests.py. [beam]

2024-04-15 Thread via GitHub
tvalentyn commented on PR #30986: URL: https://github.com/apache/beam/pull/30986#issuecomment-2058062278 LGTM -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] Skip generating docs for yaml/integration_tests.py. [beam]

2024-04-15 Thread via GitHub
github-actions[bot] commented on PR #30986: URL: https://github.com/apache/beam/pull/30986#issuecomment-2058045495 Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`: R: @jrmccluskey for label python. Available commands: -

Re: [PR] [YAML] Add the ability to pre-process yaml files with jinja2. [beam]

2024-04-15 Thread via GitHub
robertwb commented on PR #30976: URL: https://github.com/apache/beam/pull/30976#issuecomment-2058043075 RowCoderTest, YamlIntegration, and Docs issues seem pre-existing. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] Create YAML Join Transform [beam]

2024-04-15 Thread via GitHub
itodotimothy6 commented on code in PR #30734: URL: https://github.com/apache/beam/pull/30734#discussion_r1566240502 ## sdks/python/apache_beam/yaml/yaml_join.py: ## @@ -0,0 +1,273 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license

Re: [PR] Bump sqlparse from 0.4.4 to 0.5.0 in /sdks/python/container/py310 [beam]

2024-04-15 Thread via GitHub
github-actions[bot] commented on PR #30983: URL: https://github.com/apache/beam/pull/30983#issuecomment-2058021968 Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment `assign set of reviewers` -- This is an automated

Re: [PR] Bump sqlparse from 0.4.4 to 0.5.0 in /sdks/python/container/py38 [beam]

2024-04-15 Thread via GitHub
github-actions[bot] commented on PR #30985: URL: https://github.com/apache/beam/pull/30985#issuecomment-2058021878 Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment `assign set of reviewers` -- This is an automated

Re: [PR] Bump sqlparse from 0.4.4 to 0.5.0 in /sdks/python/container/py311 [beam]

2024-04-15 Thread via GitHub
github-actions[bot] commented on PR #30982: URL: https://github.com/apache/beam/pull/30982#issuecomment-2058021994 Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`: R: @tvalentyn for label python. Available commands: -

Re: [PR] Bump sqlparse from 0.4.4 to 0.5.0 in /sdks/python/container/py39 [beam]

2024-04-15 Thread via GitHub
github-actions[bot] commented on PR #30984: URL: https://github.com/apache/beam/pull/30984#issuecomment-2058021920 Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`: R: @shunping for label python. Available commands: - `stop

[PR] Skip generating docs for yaml/integration_tests.py. [beam]

2024-04-15 Thread via GitHub
robertwb opened a new pull request, #30986: URL: https://github.com/apache/beam/pull/30986 There's no value to generating docs for these, and the auto-generated names are causing warnings. Thank you for your contribution! Follow this checklist to help

Re: [PR] Implementing lull reporting at bundle level processing [beam]

2024-04-15 Thread via GitHub
dustin12 commented on code in PR #30693: URL: https://github.com/apache/beam/pull/30693#discussion_r1566540948 ## runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/ExecutionStateTracker.java: ## @@ -335,6 +352,19 @@ protected void takeSampleOnce(long

Re: [PR] [YAML] Add the ability to pre-process yaml files with jinja2. [beam]

2024-04-15 Thread via GitHub
robertwb commented on PR #30976: URL: https://github.com/apache/beam/pull/30976#issuecomment-2057977117 isort wants to put jinja2 after apache_beam, e.g. ``` import apache_beam as beam +import jinja2 ``` Not sure how to resolve that. -- This is an automated message

Re: [PR] Add code change guide contributor-doc [beam]

2024-04-15 Thread via GitHub
robertwb commented on code in PR #30879: URL: https://github.com/apache/beam/pull/30879#discussion_r1566531486 ## contributor-docs/code-change-guide.md: ## @@ -0,0 +1,519 @@ + + +This guide is for Beam users and developers changing and testing Beam code. +Specifically, this

Re: [PR] Implementing lull reporting at bundle level processing [beam]

2024-04-15 Thread via GitHub
arvindram03 commented on code in PR #30693: URL: https://github.com/apache/beam/pull/30693#discussion_r1566528993 ## runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowExecutionContext.java: ## @@ -310,12 +350,64 @@ public

Re: [PR] Implementing lull reporting at bundle level processing [beam]

2024-04-15 Thread via GitHub
arvindram03 commented on code in PR #30693: URL: https://github.com/apache/beam/pull/30693#discussion_r1566527849 ## runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/DataflowExecutionStateTrackerTest.java: ## @@ -123,12 +158,115 @@

Re: [PR] Implementing lull reporting at bundle level processing [beam]

2024-04-15 Thread via GitHub
arvindram03 commented on code in PR #30693: URL: https://github.com/apache/beam/pull/30693#discussion_r1566528219 ## runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowOperationContext.java: ## @@ -296,55 +267,6 @@ public

Re: [PR] Implementing lull reporting at bundle level processing [beam]

2024-04-15 Thread via GitHub
arvindram03 commented on code in PR #30693: URL: https://github.com/apache/beam/pull/30693#discussion_r1566525149 ## runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StackTraceUtil.java: ## @@ -0,0 +1,42 @@ +/* + * Licensed to the

Re: [PR] Implementing lull reporting at bundle level processing [beam]

2024-04-15 Thread via GitHub
arvindram03 commented on code in PR #30693: URL: https://github.com/apache/beam/pull/30693#discussion_r1566524587 ## runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowOperationContext.java: ## @@ -185,41 +181,19 @@ public

[PR] Bump sqlparse from 0.4.4 to 0.5.0 in /sdks/python/container/py38 [beam]

2024-04-15 Thread via GitHub
dependabot[bot] opened a new pull request, #30985: URL: https://github.com/apache/beam/pull/30985 Bumps [sqlparse](https://github.com/andialbrecht/sqlparse) from 0.4.4 to 0.5.0. Changelog Sourced from https://github.com/andialbrecht/sqlparse/blob/master/CHANGELOG;>sqlparse's

[PR] Bump sqlparse from 0.4.4 to 0.5.0 in /sdks/python/container/py39 [beam]

2024-04-15 Thread via GitHub
dependabot[bot] opened a new pull request, #30984: URL: https://github.com/apache/beam/pull/30984 Bumps [sqlparse](https://github.com/andialbrecht/sqlparse) from 0.4.4 to 0.5.0. Changelog Sourced from https://github.com/andialbrecht/sqlparse/blob/master/CHANGELOG;>sqlparse's

[PR] Bump sqlparse from 0.4.4 to 0.5.0 in /sdks/python/container/py310 [beam]

2024-04-15 Thread via GitHub
dependabot[bot] opened a new pull request, #30983: URL: https://github.com/apache/beam/pull/30983 Bumps [sqlparse](https://github.com/andialbrecht/sqlparse) from 0.4.4 to 0.5.0. Changelog Sourced from https://github.com/andialbrecht/sqlparse/blob/master/CHANGELOG;>sqlparse's

[PR] Bump sqlparse from 0.4.4 to 0.5.0 in /sdks/python/container/py311 [beam]

2024-04-15 Thread via GitHub
dependabot[bot] opened a new pull request, #30982: URL: https://github.com/apache/beam/pull/30982 Bumps [sqlparse](https://github.com/andialbrecht/sqlparse) from 0.4.4 to 0.5.0. Changelog Sourced from https://github.com/andialbrecht/sqlparse/blob/master/CHANGELOG;>sqlparse's

Re: [PR] Update the upper bound for google-auth-httplib2 to 'google-auth-httplib2>=0.1.0,<0.3.0' [beam]

2024-04-15 Thread via GitHub
tvalentyn commented on PR #30962: URL: https://github.com/apache/beam/pull/30962#issuecomment-2057939797 waiting for tests to finish -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Implementing lull reporting at bundle level processing [beam]

2024-04-15 Thread via GitHub
arvindram03 commented on code in PR #30693: URL: https://github.com/apache/beam/pull/30693#discussion_r1566512976 ## runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/ExecutionStateTracker.java: ## @@ -139,8 +140,17 @@ public String getDescription() { */

Re: [PR] use strimzi kafka on beam_utility cluster [beam]

2024-04-15 Thread via GitHub
Abacn commented on PR #30967: URL: https://github.com/apache/beam/pull/30967#issuecomment-2057923752 https://github.com/apache/beam/actions/runs/8696844369/job/23850939058 There is error ``` # Warning: 'patchesStrategicMerge' is deprecated. Please use 'patches' instead. Run

Re: [PR] use strimzi kafka on beam_utility cluster [beam]

2024-04-15 Thread via GitHub
Abacn commented on PR #30967: URL: https://github.com/apache/beam/pull/30967#issuecomment-2057904620 thanks. Let me duplicate this branch and run the workflow once -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] Add some metrics for CoGBK profiling. [beam]

2024-04-15 Thread via GitHub
github-actions[bot] commented on PR #30979: URL: https://github.com/apache/beam/pull/30979#issuecomment-2057898733 Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control -- This is an automated message from the Apache Git

Re: [PR] Add some metrics for CoGBK profiling. [beam]

2024-04-15 Thread via GitHub
robertwb commented on PR #30979: URL: https://github.com/apache/beam/pull/30979#issuecomment-2057897221 R: @priyansndesai -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[PR] Add some metrics for CoGBK profiling. [beam]

2024-04-15 Thread via GitHub
robertwb opened a new pull request, #30979: URL: https://github.com/apache/beam/pull/30979 This could be useful in guiding future optimizations. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and

Re: [PR] add terraform for utility cluster. Add name override to gke [beam]

2024-04-15 Thread via GitHub
damondouglas commented on code in PR #30847: URL: https://github.com/apache/beam/pull/30847#discussion_r1566440957 ## .test-infra/terraform/google-cloud-platform/utility-cluster/strimzi-contoller.tf: ## @@ -0,0 +1,36 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

Re: [PR] add terraform for utility cluster. Add name override to gke [beam]

2024-04-15 Thread via GitHub
damondouglas commented on code in PR #30847: URL: https://github.com/apache/beam/pull/30847#discussion_r1566440957 ## .test-infra/terraform/google-cloud-platform/utility-cluster/strimzi-contoller.tf: ## @@ -0,0 +1,36 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

Re: [PR] python sdk: fix several bugs regarding avto <-> beam schema conversion [beam]

2024-04-15 Thread via GitHub
benkonz commented on code in PR #30770: URL: https://github.com/apache/beam/pull/30770#discussion_r1566453505 ## sdks/python/apache_beam/io/avroio.py: ## @@ -649,7 +684,11 @@ def beam_schema_to_avro_schema( def beam_type_to_avro_type(beam_type: schema_pb2.FieldType) ->

Re: [PR] python sdk: fix several bugs regarding avto <-> beam schema conversion [beam]

2024-04-15 Thread via GitHub
benkonz commented on code in PR #30770: URL: https://github.com/apache/beam/pull/30770#discussion_r1566452436 ## sdks/python/apache_beam/io/avroio.py: ## @@ -649,7 +684,11 @@ def beam_schema_to_avro_schema( def beam_type_to_avro_type(beam_type: schema_pb2.FieldType) ->

Re: [PR] Automatically execute unbounded pipelines in streaming mode. [beam]

2024-04-15 Thread via GitHub
robertwb merged PR #30959: URL: https://github.com/apache/beam/pull/30959 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] [YAML] Add the ability to pre-process yaml files with jinja2. [beam]

2024-04-15 Thread via GitHub
robertwb commented on code in PR #30976: URL: https://github.com/apache/beam/pull/30976#discussion_r1566446325 ## sdks/python/apache_beam/yaml/main.py: ## @@ -67,6 +105,12 @@ def _pipeline_spec_from_args(known_args): def run(argv=None): known_args, pipeline_args =

[I] [Feature Request]: Support structured logging in Python [beam]

2024-04-15 Thread via GitHub
tvalentyn opened a new issue, #30978: URL: https://github.com/apache/beam/issues/30978 ### What would you like to happen? Per https://lists.apache.org/thread/9ckf6tg4w3wlowyqnnzlspdlh83zm9lq : - There is interest in having structured logging functionality - We have a design

Re: [PR] add terraform for utility cluster. Add name override to gke [beam]

2024-04-15 Thread via GitHub
damondouglas commented on code in PR #30847: URL: https://github.com/apache/beam/pull/30847#discussion_r1566440957 ## .test-infra/terraform/google-cloud-platform/utility-cluster/strimzi-contoller.tf: ## @@ -0,0 +1,36 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

Re: [PR] [Python] Allow lambda function in bigtable handler to build a custom row key [beam]

2024-04-15 Thread via GitHub
damccorm commented on code in PR #30974: URL: https://github.com/apache/beam/pull/30974#discussion_r1566430927 ## sdks/python/apache_beam/transforms/enrichment_handlers/bigtable.py: ## @@ -82,8 +89,13 @@ def __init__( self._row_filter = row_filter self._app_profile_id

Re: [PR] [YAML] Add the ability to pre-process yaml files with jinja2. [beam]

2024-04-15 Thread via GitHub
damccorm commented on code in PR #30976: URL: https://github.com/apache/beam/pull/30976#discussion_r1566419436 ## sdks/python/apache_beam/yaml/main.py: ## @@ -67,6 +105,12 @@ def _pipeline_spec_from_args(known_args): def run(argv=None): known_args, pipeline_args =

Re: [PR] Switch to width 2 exponential histograms [beam]

2024-04-15 Thread via GitHub
JayajP closed pull request #30977: Switch to width 2 exponential histograms URL: https://github.com/apache/beam/pull/30977 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] Add a link to Grafana statistics for flaky workflows [beam]

2024-04-15 Thread via GitHub
damccorm commented on code in PR #30934: URL: https://github.com/apache/beam/pull/30934#discussion_r1566386234 ## .test-infra/tools/flaky_test_detection.py: ## @@ -24,6 +24,18 @@ ALERT_NAME = "flaky_test" GIT_ORG = "apache" GRAFANA_URL = "http://metrics.beam.apache.org;

Re: [PR] Add a link to Grafana statistics for flaky workflows [beam]

2024-04-15 Thread via GitHub
damccorm commented on code in PR #30934: URL: https://github.com/apache/beam/pull/30934#discussion_r1566386234 ## .test-infra/tools/flaky_test_detection.py: ## @@ -24,6 +24,18 @@ ALERT_NAME = "flaky_test" GIT_ORG = "apache" GRAFANA_URL = "http://metrics.beam.apache.org;

Re: [PR] remove CLOUDSDK env variable , not needed here [beam]

2024-04-15 Thread via GitHub
damccorm merged PR #30970: URL: https://github.com/apache/beam/pull/30970 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[PR] Switch to width 2 exponential histograms [beam]

2024-04-15 Thread via GitHub
JayajP opened a new pull request, #30977: URL: https://github.com/apache/beam/pull/30977 Switch to width 2 exponential histograms Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: -

Re: [PR] [YAML] Add the ability to pre-process yaml files with jinja2. [beam]

2024-04-15 Thread via GitHub
robertwb commented on code in PR #30976: URL: https://github.com/apache/beam/pull/30976#discussion_r1566374434 ## sdks/python/apache_beam/yaml/main.py: ## @@ -67,6 +105,12 @@ def _pipeline_spec_from_args(known_args): def run(argv=None): known_args, pipeline_args =

Re: [PR] python sdk: fix several bugs regarding avto <-> beam schema conversion [beam]

2024-04-15 Thread via GitHub
benkonz commented on code in PR #30770: URL: https://github.com/apache/beam/pull/30770#discussion_r1566373258 ## sdks/python/apache_beam/io/avroio_test.py: ## @@ -149,17 +151,37 @@ def _run_avro_test( def test_schema_read_write(self): with tempfile.TemporaryDirectory()

Re: [PR] [YAML] Add the ability to pre-process yaml files with jinja2. [beam]

2024-04-15 Thread via GitHub
damccorm commented on code in PR #30976: URL: https://github.com/apache/beam/pull/30976#discussion_r1566365704 ## sdks/python/apache_beam/yaml/main.py: ## @@ -45,9 +47,45 @@ def _configure_parser(argv): help='none: do no pipeline validation against the schema; '

Re: [PR] [Flink] finalize checkpoint marks in the new Flink source implementation [beam]

2024-04-15 Thread via GitHub
je-ik commented on PR #30849: URL: https://github.com/apache/beam/pull/30849#issuecomment-2057696518 Ah, I missed this before https://github.com/apache/beam/pull/30971 was merged. I'll try to consolidate the two PRs. -- This is an automated message from the Apache Git Service. To respond

Re: [PR] [YAML] Add the ability to pre-process yaml files with jinja2. [beam]

2024-04-15 Thread via GitHub
robertwb commented on code in PR #30976: URL: https://github.com/apache/beam/pull/30976#discussion_r1566346899 ## sdks/python/apache_beam/yaml/main.py: ## @@ -45,9 +47,45 @@ def _configure_parser(argv): help='none: do no pipeline validation against the schema; '

Re: [PR] [YAML] Add the ability to pre-process yaml files with jinja2. [beam]

2024-04-15 Thread via GitHub
damccorm commented on code in PR #30976: URL: https://github.com/apache/beam/pull/30976#discussion_r1566345277 ## sdks/python/apache_beam/yaml/main.py: ## @@ -67,6 +105,12 @@ def _pipeline_spec_from_args(known_args): def run(argv=None): known_args, pipeline_args =

Re: [PR] [YAML] Add the ability to pre-process yaml files with jinja2. [beam]

2024-04-15 Thread via GitHub
damccorm commented on code in PR #30976: URL: https://github.com/apache/beam/pull/30976#discussion_r1566345277 ## sdks/python/apache_beam/yaml/main.py: ## @@ -67,6 +105,12 @@ def _pipeline_spec_from_args(known_args): def run(argv=None): known_args, pipeline_args =

Re: [I] [Bug]: Kafka consumer offsets are not being committed to Kafka in Beam v2.52.0 on Flink Runner [beam]

2024-04-15 Thread via GitHub
je-ik closed issue #30539: [Bug]: Kafka consumer offsets are not being committed to Kafka in Beam v2.52.0 on Flink Runner URL: https://github.com/apache/beam/issues/30539 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] #29902 finalize checkpoints after checkpoint [beam]

2024-04-15 Thread via GitHub
je-ik merged PR #30971: URL: https://github.com/apache/beam/pull/30971 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] [YAML] Add the ability to pre-process yaml files with jinja2. [beam]

2024-04-15 Thread via GitHub
damccorm commented on code in PR #30976: URL: https://github.com/apache/beam/pull/30976#discussion_r1566344279 ## sdks/python/apache_beam/yaml/main.py: ## @@ -67,6 +105,12 @@ def _pipeline_spec_from_args(known_args): def run(argv=None): known_args, pipeline_args =

Re: [PR] [YAML] Add the ability to pre-process yaml files with jinja2. [beam]

2024-04-15 Thread via GitHub
robertwb commented on code in PR #30976: URL: https://github.com/apache/beam/pull/30976#discussion_r1566344084 ## sdks/python/apache_beam/yaml/main.py: ## @@ -67,6 +105,12 @@ def _pipeline_spec_from_args(known_args): def run(argv=None): known_args, pipeline_args =

Re: [PR] [YAML] Add the ability to pre-process yaml files with jinja2. [beam]

2024-04-15 Thread via GitHub
robertwb commented on code in PR #30976: URL: https://github.com/apache/beam/pull/30976#discussion_r1566343526 ## sdks/python/apache_beam/yaml/main.py: ## @@ -67,6 +105,12 @@ def _pipeline_spec_from_args(known_args): def run(argv=None): known_args, pipeline_args =

Re: [PR] [YAML] Add the ability to pre-process yaml files with jinja2. [beam]

2024-04-15 Thread via GitHub
damccorm commented on code in PR #30976: URL: https://github.com/apache/beam/pull/30976#discussion_r1566338553 ## sdks/python/apache_beam/yaml/main.py: ## @@ -45,9 +47,45 @@ def _configure_parser(argv): help='none: do no pipeline validation against the schema; '

Re: [PR] [Python] Allow lambda function in bigtable handler to build a custom row key [beam]

2024-04-15 Thread via GitHub
github-actions[bot] commented on PR #30974: URL: https://github.com/apache/beam/pull/30974#issuecomment-2057661226 Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`: R: @damccorm for label python. R: @damccorm for label build.

Re: [PR] [YAML] Add the ability to pre-process yaml files with jinja2. [beam]

2024-04-15 Thread via GitHub
Polber commented on code in PR #30976: URL: https://github.com/apache/beam/pull/30976#discussion_r1566336507 ## sdks/python/apache_beam/yaml/main.py: ## @@ -67,6 +105,12 @@ def _pipeline_spec_from_args(known_args): def run(argv=None): known_args, pipeline_args =

Re: [PR] [YAML] Add the ability to pre-process yaml files with jinja2. [beam]

2024-04-15 Thread via GitHub
github-actions[bot] commented on PR #30976: URL: https://github.com/apache/beam/pull/30976#issuecomment-2057623102 Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control -- This is an automated message from the Apache Git

[PR] [YAML] Add the ability to pre-process yaml files with jinja2. [beam]

2024-04-15 Thread via GitHub
robertwb opened a new pull request, #30976: URL: https://github.com/apache/beam/pull/30976 This will allow Beam YAML to be used for various template applications. Thank you for your contribution! Follow this checklist to help us incorporate your

Re: [PR] [YAML] Add the ability to pre-process yaml files with jinja2. [beam]

2024-04-15 Thread via GitHub
robertwb commented on PR #30976: URL: https://github.com/apache/beam/pull/30976#issuecomment-2057621210 R: @Polber @damccorm -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] add terraform for utility cluster. Add name override to gke [beam]

2024-04-15 Thread via GitHub
volatilemolotov commented on code in PR #30847: URL: https://github.com/apache/beam/pull/30847#discussion_r1566302782 ## .test-infra/terraform/google-cloud-platform/utility-cluster/strimzi-contoller.tf: ## @@ -0,0 +1,36 @@ +/* + * Licensed to the Apache Software Foundation

Re: [PR] Update the upper bound for google-auth-httplib2 to 'google-auth-httplib2>=0.1.0,<0.3.0' [beam]

2024-04-15 Thread via GitHub
liferoad commented on code in PR #30962: URL: https://github.com/apache/beam/pull/30962#discussion_r1566286863 ## sdks/python/container/py311/base_image_requirements.txt: ## @@ -73,7 +73,6 @@ google-cloud-vision==3.7.2 google-crc32c==1.5.0 google-resumable-media==2.7.0

Re: [PR] [bug30870]: make consumer polling timeout configurable for KafkaIO.Read [beam]

2024-04-15 Thread via GitHub
Abacn commented on PR #30877: URL: https://github.com/apache/beam/pull/30877#issuecomment-2057569432 Also as followup `:sdks:java:io:kafka:upgrade:test` should be exercised in Kafka IO PreCommit -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [PR] Update the upper bound for google-auth-httplib2 to 'google-auth-httplib2>=0.1.0,<0.3.0' [beam]

2024-04-15 Thread via GitHub
tvalentyn commented on code in PR #30962: URL: https://github.com/apache/beam/pull/30962#discussion_r1566268222 ## sdks/python/container/py311/base_image_requirements.txt: ## @@ -73,7 +73,6 @@ google-cloud-vision==3.7.2 google-crc32c==1.5.0 google-resumable-media==2.7.0

Re: [PR] Update the upper bound for google-auth-httplib2 to 'google-auth-httplib2>=0.1.0,<0.3.0' [beam]

2024-04-15 Thread via GitHub
liferoad commented on code in PR #30962: URL: https://github.com/apache/beam/pull/30962#discussion_r1566264530 ## sdks/python/container/py311/base_image_requirements.txt: ## @@ -73,7 +73,6 @@ google-cloud-vision==3.7.2 google-crc32c==1.5.0 google-resumable-media==2.7.0

Re: [PR] [bug30870]: make consumer polling timeout configurable for KafkaIO.Read [beam]

2024-04-15 Thread via GitHub
Abacn commented on PR #30877: URL: https://github.com/apache/beam/pull/30877#issuecomment-2057540824 It appears this breaks Java IOs PreCommit: #30941 please fix -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [I] [Failing Test]: Java IO Precommit failing due to KafkaIOTranslation being out of sync [beam]

2024-04-15 Thread via GitHub
Abacn commented on issue #30941: URL: https://github.com/apache/beam/issues/30941#issuecomment-2057538972 This affects PreCommit so bump to P1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] Move logic to build a Labeled Metric Name to a util file. [beam]

2024-04-15 Thread via GitHub
Abacn merged PR #30796: URL: https://github.com/apache/beam/pull/30796 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] [Do Not Merge Now] Test 'google-auth-httplib2>=0.1.0,<0.3.0' [beam]

2024-04-15 Thread via GitHub
liferoad commented on code in PR #30962: URL: https://github.com/apache/beam/pull/30962#discussion_r1566256675 ## sdks/python/setup.py: ## @@ -442,7 +442,7 @@ def get_portability_package_data(): # Until this issue is closed #

Re: [PR] Skip calling Compute in RemoveSafeDeltaCounterCell if value exists [beam]

2024-04-15 Thread via GitHub
JayajP commented on PR #30913: URL: https://github.com/apache/beam/pull/30913#issuecomment-2057526487 R: @m-trieu -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] Skip calling Compute in RemoveSafeDeltaCounterCell if value exists [beam]

2024-04-15 Thread via GitHub
github-actions[bot] commented on PR #30913: URL: https://github.com/apache/beam/pull/30913#issuecomment-2057513871 Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control -- This is an automated message from the Apache Git

Re: [PR] Skip calling Compute in RemoveSafeDeltaCounterCell if value exists [beam]

2024-04-15 Thread via GitHub
JayajP commented on PR #30913: URL: https://github.com/apache/beam/pull/30913#issuecomment-2057511721 R: @m-trieu -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [I] [Bug]: GRCPIO 1.62.1 deadlocks in Beam SDK worker with Dataflow [beam]

2024-04-15 Thread via GitHub
tvalentyn commented on issue #30867: URL: https://github.com/apache/beam/issues/30867#issuecomment-2057505210 we can fix the bug we are currently investigating and then come back to your issue and see if you still reproduce it. -- This is an automated message from the Apache Git Service.

[PR] [Python] Allow lambda function in bigtable handler to build a custom row key [beam]

2024-04-15 Thread via GitHub
riteshghorse opened a new pull request, #30974: URL: https://github.com/apache/beam/pull/30974 Allow users to use a custom function to build a composite row key from for the Bigtable row. Thank you for your contribution! Follow this checklist to help

Re: [PR] [Do Not Merge Now] Test 'google-auth-httplib2>=0.1.0,<=0.2.0' [beam]

2024-04-15 Thread via GitHub
tvalentyn commented on code in PR #30962: URL: https://github.com/apache/beam/pull/30962#discussion_r1566227018 ## sdks/python/setup.py: ## @@ -442,7 +442,7 @@ def get_portability_package_data(): # Until this issue is closed #

Re: [PR] [Do Not Merge Now] Test 'google-auth-httplib2>=0.1.0,<=0.2.0' [beam]

2024-04-15 Thread via GitHub
liferoad commented on code in PR #30962: URL: https://github.com/apache/beam/pull/30962#discussion_r156664 ## sdks/python/setup.py: ## @@ -442,7 +442,7 @@ def get_portability_package_data(): # Until this issue is closed #

Re: [PR] #29902 finalize checkpoints after checkpoint [beam]

2024-04-15 Thread via GitHub
je-ik commented on PR #30971: URL: https://github.com/apache/beam/pull/30971#issuecomment-2057465989 Rebased this on top of master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [I] [Bug]: OnTimer("loopingTimer") not triggered on 2.52.0 [beam]

2024-04-15 Thread via GitHub
je-ik closed issue #29816: [Bug]: OnTimer("loopingTimer") not triggered on 2.52.0 URL: https://github.com/apache/beam/issues/29816 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] [runners-flink] Fix watermark emission for empty splits (#29816) [beam]

2024-04-15 Thread via GitHub
je-ik merged PR #30969: URL: https://github.com/apache/beam/pull/30969 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Fix build wheels due to setuptools changed naming [beam]

2024-04-15 Thread via GitHub
tvalentyn merged PR #30972: URL: https://github.com/apache/beam/pull/30972 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Upload PerWorkerMetrics every 30 second instead of every 10 seconds [beam]

2024-04-15 Thread via GitHub
Abacn merged PR #30795: URL: https://github.com/apache/beam/pull/30795 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] add terraform for utility cluster. Add name override to gke [beam]

2024-04-15 Thread via GitHub
damondouglas commented on code in PR #30847: URL: https://github.com/apache/beam/pull/30847#discussion_r1566090575 ## .test-infra/terraform/google-cloud-platform/utility-cluster/strimzi-contoller.tf: ## @@ -0,0 +1,36 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

Re: [PR] [Python] Add feast feature store handler for enrichment transform [beam]

2024-04-15 Thread via GitHub
github-actions[bot] commented on PR #30957: URL: https://github.com/apache/beam/pull/30957#issuecomment-2057275241 Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`: R: @liferoad for label python. R: @Abacn for label build.

Re: [PR] [Python] Add feast feature store handler for enrichment transform [beam]

2024-04-15 Thread via GitHub
riteshghorse commented on PR #30957: URL: https://github.com/apache/beam/pull/30957#issuecomment-2057270121 assign set of reviewers -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] [Python] Add feast feature store handler for enrichment transform [beam]

2024-04-15 Thread via GitHub
riteshghorse commented on PR #30957: URL: https://github.com/apache/beam/pull/30957#issuecomment-2057260872 Feast integration tests passed https://github.com/apache/beam/assets/25881114/f15b17ea-744b-404f-960b-367c0ce55af8;> Unrelated failures in other test suites -- This is an

Re: [PR] Fix build wheels due to setuptools changed naming [beam]

2024-04-15 Thread via GitHub
github-actions[bot] commented on PR #30972: URL: https://github.com/apache/beam/pull/30972#issuecomment-2057219733 Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control -- This is an automated message from the Apache Git

Re: [PR] Fix build wheels due to setuptools changed naming [beam]

2024-04-15 Thread via GitHub
Abacn commented on PR #30972: URL: https://github.com/apache/beam/pull/30972#issuecomment-2057217353 R: @tvalentyn @liferoad -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] [Do Not Merge Now] Test 'google-auth-httplib2>=0.1.0,<=0.2.0' [beam]

2024-04-15 Thread via GitHub
tvalentyn commented on code in PR #30962: URL: https://github.com/apache/beam/pull/30962#discussion_r1566013771 ## sdks/python/setup.py: ## @@ -442,7 +442,7 @@ def get_portability_package_data(): # Until this issue is closed #

Re: [PR] Fix build wheels due to setuptools changed naming [beam]

2024-04-15 Thread via GitHub
Abacn commented on PR #30972: URL: https://github.com/apache/beam/pull/30972#issuecomment-2057182605 triggered https://github.com/apache/beam/actions/runs/8692146995 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[PR] Fix build wheels due to setuptools changed naming [beam]

2024-04-15 Thread via GitHub
Abacn opened a new pull request, #30972: URL: https://github.com/apache/beam/pull/30972 This is due to https://github.com/pypa/setuptools/issues/4300 **Please** add a meaningful description for your change here Thank you for your contribution!

Re: [PR] Restore pip logs also for requirements.txt case [beam]

2024-04-15 Thread via GitHub
tvalentyn merged PR #30936: URL: https://github.com/apache/beam/pull/30936 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Restore pip logs also for requirements.txt case [beam]

2024-04-15 Thread via GitHub
tvalentyn commented on PR #30936: URL: https://github.com/apache/beam/pull/30936#issuecomment-2057175437 Failures are unrelated, merging to make sure this lands in the release in time. -- This is an automated message from the Apache Git Service. To respond to the message, please log on

Re: [PR] change grafana creds to use readonly account for beammetrics cloudsql db [beam]

2024-04-15 Thread via GitHub
Abacn merged PR #30968: URL: https://github.com/apache/beam/pull/30968 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

  1   2   >