Re: [PR] [YAML] Add the ability to pre-process yaml files with jinja2. [beam]

2024-04-16 Thread via GitHub
damccorm commented on PR #30976: URL: https://github.com/apache/beam/pull/30976#issuecomment-2059831558 Precommit failures look real? Its not failing on master, but I also don't see any relevant changes here -

Re: [PR] Use base 2 exponential histograms in BigQuerySinkMetrics [beam]

2024-04-16 Thread via GitHub
github-actions[bot] commented on PR #31000: URL: https://github.com/apache/beam/pull/31000#issuecomment-2059853042 Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control -- This is an automated message from the Apache Git

Re: [PR] Downgrade google-api-core to mitigate #30927 [beam]

2024-04-16 Thread via GitHub
damccorm commented on code in PR #31004: URL: https://github.com/apache/beam/pull/31004#discussion_r1567964520 ## sdks/python/setup.py: ## @@ -436,7 +436,9 @@ def get_portability_package_data(): ], 'gcp': [ 'cachetools>=3.1.0,<6', -

Re: [PR] Revert #30877 and #30915 to fix Java IOs PreCommit - Kafka upgrade test [beam]

2024-04-16 Thread via GitHub
damccorm commented on PR #31001: URL: https://github.com/apache/beam/pull/31001#issuecomment-2059957502 > we need to do the forward fix on this. Yep, agreed (chatted offline about this already as well) -- This is an automated message from the Apache Git Service. To respond to the

Re: [PR] Iceberg SchemaTransforms and Translation [beam]

2024-04-16 Thread via GitHub
chamikaramj commented on code in PR #30910: URL: https://github.com/apache/beam/pull/30910#discussion_r1568031375 ## model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/schema_aware_transforms.proto: ## @@ -0,0 +1,73 @@ +/* + * Licensed to the Apache Software

Re: [PR] Cache parsed metrics in StreamingStepMetricsContainer [beam]

2024-04-16 Thread via GitHub
JayajP commented on PR #31005: URL: https://github.com/apache/beam/pull/31005#issuecomment-2060169215 R: @m-trieu -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] Revert #30877 and #30915 to fix Java IOs PreCommit - Kafka upgrade test [beam]

2024-04-16 Thread via GitHub
Abacn commented on PR #31001: URL: https://github.com/apache/beam/pull/31001#issuecomment-2059772229 CC: @xianhualiu @damccorm In case the two linked Issues not resolved by release cut let's get this in so the release branch is green -- This is an automated message from the Apache Git

Re: [PR] Bump github.com/apple/swift-nio-http2 from 1.27.0 to 1.28.1 [beam-swift]

2024-04-16 Thread via GitHub
brucearctor commented on PR #4: URL: https://github.com/apache/beam-swift/pull/4#issuecomment-2059808312 all good - easy to miss notifcations. I'm donig a larger sweep of security notifications across our codebases, so finding a few things that seem to have been ignored. I'll

Re: [PR] [#30941]fix upgrade test due to missed config ConsumerPollingTimeout [beam]

2024-04-16 Thread via GitHub
chamikaramj commented on code in PR #30998: URL: https://github.com/apache/beam/pull/30998#discussion_r1567865679 ## sdks/java/io/kafka/upgrade/src/main/java/org/apache/beam/sdk/io/kafka/upgrade/KafkaIOTranslation.java: ## @@ -102,6 +102,7 @@ static class

Re: [PR] Bump github.com/apple/swift-nio-http2 from 1.27.0 to 1.28.1 [beam-swift]

2024-04-16 Thread via GitHub
byronellis commented on PR #4: URL: https://github.com/apache/beam-swift/pull/4#issuecomment-2059822864 Thanks! Will merge when I get a chance!On Apr 16, 2024, at 12:44 PM, brucearctor ***@***.***> wrote: @brucearctor approved this pull request. —Reply to this email

Re: [PR] Support BQ clustering with value provider [beam]

2024-04-16 Thread via GitHub
liferoad commented on PR #30460: URL: https://github.com/apache/beam/pull/30460#issuecomment-2059950154 This is requested by one customer. What is blocking this PR? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] Create YAML Join Transform [beam]

2024-04-16 Thread via GitHub
Polber commented on PR #30734: URL: https://github.com/apache/beam/pull/30734#issuecomment-2060172249 It looks like you addressed all my initial comments, and overall looks good. I will probably take another deeper look in the morning, but don't block on me. -- This is an automated

Re: [PR] Cache parsed metrics in StreamingStepMetricsContainer [beam]

2024-04-16 Thread via GitHub
github-actions[bot] commented on PR #31005: URL: https://github.com/apache/beam/pull/31005#issuecomment-2060170072 Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control -- This is an automated message from the Apache Git

Re: [PR] Bump github.com/apple/swift-nio-http2 from 1.27.0 to 1.28.1 [beam-swift]

2024-04-16 Thread via GitHub
byronellis commented on PR #4: URL: https://github.com/apache/beam-swift/pull/4#issuecomment-2059801579 Hi, sorry didn’t see that PR come through. Absolutely no reason we wouldn’t update! Traveling right now so once I’m not on my phone I’ll take a lookOn Apr 16, 2024, at 12:33 PM,

Re: [PR] Add code change guide contributor-doc [beam]

2024-04-16 Thread via GitHub
Abacn commented on code in PR #30879: URL: https://github.com/apache/beam/pull/30879#discussion_r1567918819 ## contributor-docs/code-change-guide.md: ## @@ -0,0 +1,519 @@ + + +This guide is for Beam users and developers changing and testing Beam code. +Specifically, this guide

Re: [PR] Skip calling Compute in RemoveSafeDeltaCounterCell if value exists [beam]

2024-04-16 Thread via GitHub
JayajP commented on code in PR #30913: URL: https://github.com/apache/beam/pull/30913#discussion_r1567932972 ## runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/RemoveSafeDeltaCounterCell.java: ## @@ -62,6 +62,12 @@ public void

Re: [PR] Disable profiling in cythonized code by default. [beam]

2024-04-16 Thread via GitHub
tvalentyn merged PR #30938: URL: https://github.com/apache/beam/pull/30938 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [I] [Feature Request]: Allow passing a custom function for row key in BigTableEnrichmentHandler [beam]

2024-04-16 Thread via GitHub
riteshghorse closed issue #30975: [Feature Request]: Allow passing a custom function for row key in BigTableEnrichmentHandler URL: https://github.com/apache/beam/issues/30975 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] Bump github.com/apple/swift-nio-http2 from 1.27.0 to 1.28.1 [beam-swift]

2024-04-16 Thread via GitHub
tvalentyn merged PR #4: URL: https://github.com/apache/beam-swift/pull/4 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] [#30941]fix upgrade test due to missed config ConsumerPollingTimeout [beam]

2024-04-16 Thread via GitHub
github-actions[bot] commented on PR #30998: URL: https://github.com/apache/beam/pull/30998#issuecomment-2060021687 Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment `assign set of reviewers` -- This is an automated

Re: [I] [Feature Request]: Allow passing a custom function for row key in BigTableEnrichmentHandler [beam]

2024-04-16 Thread via GitHub
riteshghorse commented on issue #30975: URL: https://github.com/apache/beam/issues/30975#issuecomment-2060221303 Done in #30974 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Revert #30877 and #30915 to fix Java IOs PreCommit - Kafka upgrade test [beam]

2024-04-16 Thread via GitHub
liferoad commented on PR #31001: URL: https://github.com/apache/beam/pull/31001#issuecomment-2059947840 we need to do the forward fix on this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] [YAML] Add the ability to pre-process yaml files with jinja2. [beam]

2024-04-16 Thread via GitHub
robertwb commented on PR #30976: URL: https://github.com/apache/beam/pull/30976#issuecomment-2060017241 A merge to master seems to have resolved the row coder issues. Running a full suite of tests again. -- This is an automated message from the Apache Git Service. To respond to the

[PR] Use base 2 exponential histograms in BigQuerySinkMetrics [beam]

2024-04-16 Thread via GitHub
JayajP opened a new pull request, #31000: URL: https://github.com/apache/beam/pull/31000 Use base 2 exponential histograms in BigQuerySinkMetrics. This shows better performance on load tests either because the bucket index computation is faster or because there are fewer buckets to send to

Re: [PR] [#30941]fix upgrade test due to missed config ConsumerPollingTimeout [beam]

2024-04-16 Thread via GitHub
xianhualiu commented on code in PR #30998: URL: https://github.com/apache/beam/pull/30998#discussion_r1567904910 ## sdks/java/io/kafka/upgrade/src/main/java/org/apache/beam/sdk/io/kafka/upgrade/KafkaIOTranslation.java: ## @@ -102,6 +102,7 @@ static class

Re: [PR] Skip calling Compute in RemoveSafeDeltaCounterCell if value exists [beam]

2024-04-16 Thread via GitHub
Abacn merged PR #30913: URL: https://github.com/apache/beam/pull/30913 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] [#30941]fix upgrade test due to missed config ConsumerPollingTimeout [beam]

2024-04-16 Thread via GitHub
xianhualiu commented on code in PR #30998: URL: https://github.com/apache/beam/pull/30998#discussion_r1567950746 ## sdks/java/io/kafka/upgrade/src/main/java/org/apache/beam/sdk/io/kafka/upgrade/KafkaIOTranslation.java: ## @@ -320,6 +324,12 @@ public Row toConfigRow(Read

Re: [PR] [#30941]fix upgrade test due to missed config ConsumerPollingTimeout [beam]

2024-04-16 Thread via GitHub
xianhualiu commented on code in PR #30998: URL: https://github.com/apache/beam/pull/30998#discussion_r1567950918 ## sdks/java/io/kafka/upgrade/src/main/java/org/apache/beam/sdk/io/kafka/upgrade/KafkaIOTranslation.java: ## @@ -102,6 +102,7 @@ static class

Re: [PR] Downgrade google-api-core to mitigate #30927 [beam]

2024-04-16 Thread via GitHub
tvalentyn commented on code in PR #31004: URL: https://github.com/apache/beam/pull/31004#discussion_r1567973507 ## sdks/python/setup.py: ## @@ -436,7 +436,9 @@ def get_portability_package_data(): ], 'gcp': [ 'cachetools>=3.1.0,<6', -

Re: [PR] Downgrade google-api-core to mitigate #30927 [beam]

2024-04-16 Thread via GitHub
tvalentyn commented on code in PR #31004: URL: https://github.com/apache/beam/pull/31004#discussion_r1567974278 ## sdks/python/setup.py: ## @@ -436,7 +436,9 @@ def get_portability_package_data(): ], 'gcp': [ 'cachetools>=3.1.0,<6', -

Re: [PR] Add some metrics for CoGBK profiling. [beam]

2024-04-16 Thread via GitHub
robertwb merged PR #30979: URL: https://github.com/apache/beam/pull/30979 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Add some metrics for CoGBK profiling. [beam]

2024-04-16 Thread via GitHub
robertwb commented on PR #30979: URL: https://github.com/apache/beam/pull/30979#issuecomment-2059977366 We're seeing the same structural value error on master: https://github.com/apache/beam/actions/runs/8709671441 -- This is an automated message from the Apache Git Service. To respond

Re: [PR] Minor updates to StreamingInsertsMetrics [beam]

2024-04-16 Thread via GitHub
github-actions[bot] commented on PR #31003: URL: https://github.com/apache/beam/pull/31003#issuecomment-2060126914 Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control -- This is an automated message from the Apache Git

Re: [PR] Minor updates to StreamingInsertsMetrics [beam]

2024-04-16 Thread via GitHub
JayajP commented on PR #31003: URL: https://github.com/apache/beam/pull/31003#issuecomment-2060124415 R: @m-trieu -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] [#30941]fix upgrade test due to missed config ConsumerPollingTimeout [beam]

2024-04-16 Thread via GitHub
github-actions[bot] commented on PR #30998: URL: https://github.com/apache/beam/pull/30998#issuecomment-2060196426 Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`: R: @damccorm for label python. R: @kennknowles for label java.

Re: [I] [Bug]: GRCPIO 1.62.1 deadlocks in Beam SDK worker with Dataflow [beam]

2024-04-16 Thread via GitHub
tvalentyn commented on issue #30867: URL: https://github.com/apache/beam/issues/30867#issuecomment-2059740318 > @tvalentyn is the fix here basically just disallowing 1.59.0-1.62.1 in We have at least two changes in that depend on new versions of GRPCIO, we'd have to roll them back. I

[PR] Revert #30877 and #30915 to fix Java IOs PreCommit - Kafka upgrade test [beam]

2024-04-16 Thread via GitHub
Abacn opened a new pull request, #31001: URL: https://github.com/apache/beam/pull/31001 Fix #30941; mitigate #30997 - we'll need to redo this change in a way that not breaking transform upgrade **Please** add a meaningful description for your change here

Re: [PR] Change ManagedSchemaTransformProvider to take a Row config [beam]

2024-04-16 Thread via GitHub
ahmedabu98 commented on PR #30937: URL: https://github.com/apache/beam/pull/30937#issuecomment-2059859123 Actually, I don't think this is possible because we don't know the underlying transform's configuration schema ahead of time. Even if we do via some mapping, this still won't work

[PR] Minor updates to StreamingInsertsMetrics [beam]

2024-04-16 Thread via GitHub
JayajP opened a new pull request, #31003: URL: https://github.com/apache/beam/pull/31003 **Please** add a meaningful description for your change here Thank you for your contribution! Follow this checklist to help us incorporate your contribution

Re: [PR] Add code change guide contributor-doc [beam]

2024-04-16 Thread via GitHub
Abacn commented on code in PR #30879: URL: https://github.com/apache/beam/pull/30879#discussion_r1567919805 ## contributor-docs/code-change-guide.md: ## @@ -0,0 +1,519 @@ + + +This guide is for Beam users and developers changing and testing Beam code. +Specifically, this guide

[PR] Downgrade google-api-core to mitigate #30927 [beam]

2024-04-16 Thread via GitHub
tvalentyn opened a new pull request, #31004: URL: https://github.com/apache/beam/pull/31004 A change in google-api-core 2.17.0 triggered a regression introduced in grpcio==1.58.0, which will be fixed in grpcio==1.63.0 and possibly in grpcio==1.62.2. The regression affects Beam 2.55.0

Re: [PR] Downgrade google-api-core to mitigate #30927 [beam]

2024-04-16 Thread via GitHub
tvalentyn commented on PR #31004: URL: https://github.com/apache/beam/pull/31004#issuecomment-2059954063 R: @damccorm -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] Downgrade google-api-core to mitigate #30927 [beam]

2024-04-16 Thread via GitHub
tvalentyn commented on code in PR #31004: URL: https://github.com/apache/beam/pull/31004#discussion_r1567971669 ## sdks/python/setup.py: ## @@ -436,7 +436,9 @@ def get_portability_package_data(): ], 'gcp': [ 'cachetools>=3.1.0,<6', -

Re: [I] [Bug]: Uber jar not work with Dataflow runner v2 due to `UnknownCoderWrapper` for Beam 2.55 [beam]

2024-04-16 Thread via GitHub
chamikaramj commented on issue #30994: URL: https://github.com/apache/beam/issues/30994#issuecomment-2060077026 I don't think this is necessarily a bug but a limitation due the uber jar not properly registering coders. I tried a job with more logs and seems like we are simply not able to

[PR] Cache parsed metrics in StreamingStepMetricsContainer [beam]

2024-04-16 Thread via GitHub
JayajP opened a new pull request, #31005: URL: https://github.com/apache/beam/pull/31005 Cache parsed metrics in StreamingStepMetricsContainer since this is a non-trivial amount of work. We will cache parsed names for both PerWorkerCounters and PerWorkerHistograms. - Parsed

Re: [PR] Follow up of recent GHA breakages [beam]

2024-04-16 Thread via GitHub
Abacn merged PR #30996: URL: https://github.com/apache/beam/pull/30996 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Bump github.com/apple/swift-nio-http2 from 1.27.0 to 1.28.1 [beam-swift]

2024-04-16 Thread via GitHub
brucearctor commented on PR #4: URL: https://github.com/apache/beam-swift/pull/4#issuecomment-2059793600 @byronellis -- seeing this security update. Any reason we don't use it? There are not existing tests, so unclear whether bumping version breaks things. I will assume this repo

Re: [PR] [#30941]fix upgrade test due to missed config ConsumerPollingTimeout [beam]

2024-04-16 Thread via GitHub
chamikaramj commented on code in PR #30998: URL: https://github.com/apache/beam/pull/30998#discussion_r1567857231 ## sdks/java/io/kafka/upgrade/src/main/java/org/apache/beam/sdk/io/kafka/upgrade/KafkaIOTranslation.java: ## @@ -102,6 +102,7 @@ static class

Re: [PR] Remove caching of global window side input param when invoking. [beam]

2024-04-16 Thread via GitHub
scwhittle merged PR #30991: URL: https://github.com/apache/beam/pull/30991 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Revert #30877 and #30915 to fix Java IOs PreCommit - Kafka upgrade test [beam]

2024-04-16 Thread via GitHub
github-actions[bot] commented on PR #31001: URL: https://github.com/apache/beam/pull/31001#issuecomment-2059839186 Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`: R: @liferoad for label python. R: @robertwb for label java. R:

Re: [I] [Bug]: RunInference with automatic refresh using a global window side inputs takes too long to update sometimes [beam]

2024-04-16 Thread via GitHub
scwhittle commented on issue #28776: URL: https://github.com/apache/beam/issues/28776#issuecomment-2059855132 @AnandInguva Can you verify the above fixes your issue? It does for my separate testing pipeline -- This is an automated message from the Apache Git Service. To respond to the

Re: [PR] [#30941]fix upgrade test due to missed config ConsumerPollingTimeout [beam]

2024-04-16 Thread via GitHub
chamikaramj commented on code in PR #30998: URL: https://github.com/apache/beam/pull/30998#discussion_r1567906726 ## sdks/java/io/kafka/upgrade/src/main/java/org/apache/beam/sdk/io/kafka/upgrade/KafkaIOTranslation.java: ## @@ -320,6 +324,12 @@ public Row toConfigRow(Read

Re: [PR] [#30941]fix upgrade test due to missed config ConsumerPollingTimeout [beam]

2024-04-16 Thread via GitHub
chamikaramj commented on code in PR #30998: URL: https://github.com/apache/beam/pull/30998#discussion_r1567908457 ## sdks/java/io/kafka/upgrade/src/main/java/org/apache/beam/sdk/io/kafka/upgrade/KafkaIOTranslation.java: ## @@ -102,6 +102,7 @@ static class

Re: [PR] Add code change guide contributor-doc [beam]

2024-04-16 Thread via GitHub
Abacn commented on code in PR #30879: URL: https://github.com/apache/beam/pull/30879#discussion_r1567920232 ## contributor-docs/code-change-guide.md: ## @@ -0,0 +1,519 @@ + + +This guide is for Beam users and developers changing and testing Beam code. +Specifically, this guide

Re: [PR] [Flink] finalize checkpoint marks in the new Flink source implementation (#30849) [beam]

2024-04-16 Thread via GitHub
Abacn merged PR #30987: URL: https://github.com/apache/beam/pull/30987 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Flink 1.17 [beam]

2024-04-16 Thread via GitHub
Abacn commented on code in PR #30197: URL: https://github.com/apache/beam/pull/30197#discussion_r1567951803 ## runners/flink/1.12/build.gradle: ## @@ -16,18 +16,10 @@ * limitations under the License. */ -def basePath = '..' -/* All properties required for loading the

Re: [PR] Use base 2 exponential histograms in BigQuerySinkMetrics [beam]

2024-04-16 Thread via GitHub
JayajP commented on PR #31000: URL: https://github.com/apache/beam/pull/31000#issuecomment-2059851330 R: @m-trieu -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] Add mechanism for skipping batching if your data is prebatched [beam]

2024-04-16 Thread via GitHub
damccorm merged PR #30961: URL: https://github.com/apache/beam/pull/30961 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Add code change guide contributor-doc [beam]

2024-04-16 Thread via GitHub
Abacn commented on PR #30879: URL: https://github.com/apache/beam/pull/30879#issuecomment-2059898073 > There's a lot of good information here, but it's unclear if it's targeted at the casual (I'm just trying to contribute a single transform) vs. advanced (I just got hired to develop Beam

Re: [PR] Downgrade google-api-core to mitigate #30927 [beam]

2024-04-16 Thread via GitHub
github-actions[bot] commented on PR #31004: URL: https://github.com/apache/beam/pull/31004#issuecomment-2059955612 Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control -- This is an automated message from the Apache Git

Re: [PR] Downgrade google-api-core to mitigate #30927 [beam]

2024-04-16 Thread via GitHub
tvalentyn commented on code in PR #31004: URL: https://github.com/apache/beam/pull/31004#discussion_r1567972540 ## sdks/python/setup.py: ## @@ -436,7 +436,9 @@ def get_portability_package_data(): ], 'gcp': [ 'cachetools>=3.1.0,<6', -

Re: [PR] Revert #30877 and #30915 to fix Java IOs PreCommit - Kafka upgrade test [beam]

2024-04-16 Thread via GitHub
xianhualiu commented on PR #31001: URL: https://github.com/apache/beam/pull/31001#issuecomment-206012 please hold on merging this PR if possible. The https://github.com/apache/beam/pull/30998 will fix the test issue. -- This is an automated message from the Apache Git Service. To

Re: [PR] Cache parsed metrics in StreamingStepMetricsContainer [beam]

2024-04-16 Thread via GitHub
JayajP commented on PR #31005: URL: https://github.com/apache/beam/pull/31005#issuecomment-2060205011 Run Java PreCommit -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [I] FakeJobsService tries to base64 decode a plain QueryString resulting in an error. [beam]

2024-04-16 Thread via GitHub
hanalaydrus commented on issue #26935: URL: https://github.com/apache/beam/issues/26935#issuecomment-2060241169 This happen to me too, have you found the solution? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] Minor updates to StreamingInsertsMetrics [beam]

2024-04-16 Thread via GitHub
JayajP commented on PR #31003: URL: https://github.com/apache/beam/pull/31003#issuecomment-2060249277 Run PreCommit Java GCP IO Direct -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[I] The Python tests job is flaky [beam]

2024-04-16 Thread via GitHub
github-actions[bot] opened a new issue, #31006: URL: https://github.com/apache/beam/issues/31006 The Python tests is failing over 50% of the time. Please visit https://github.com/apache/beam/actions/workflows/python_tests.yml?query=is%3Afailure+branch%3Amaster to see all failed workflow

Re: [PR] Iceberg SchemaTransforms and Translation [beam]

2024-04-16 Thread via GitHub
ahmedabu98 commented on code in PR #30910: URL: https://github.com/apache/beam/pull/30910#discussion_r1568192720 ## sdks/java/io/iceberg/src/main/java/org/apache/beam/io/iceberg/IcebergIO.java: ## @@ -19,73 +19,116 @@ import static

Re: [I] [Feature Request]: SpannerIO: support max commit delay [beam]

2024-04-16 Thread via GitHub
kberezin-nshl commented on issue #31007: URL: https://github.com/apache/beam/issues/31007#issuecomment-2060420112 .take-issue -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[PR] SpannerIO: support max commit delay [beam]

2024-04-16 Thread via GitHub
kberezin-nshl opened a new pull request, #31008: URL: https://github.com/apache/beam/pull/31008 Addresses https://github.com/apache/beam/issues/31007 Thank you for your contribution! Follow this checklist to help us incorporate your contribution

Re: [PR] SpannerIO: support max commit delay [beam]

2024-04-16 Thread via GitHub
github-actions[bot] commented on PR #31008: URL: https://github.com/apache/beam/pull/31008#issuecomment-2060433514 Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control -- This is an automated message from the Apache Git

Re: [PR] SpannerIO: support max commit delay [beam]

2024-04-16 Thread via GitHub
kberezin-nshl commented on PR #31008: URL: https://github.com/apache/beam/pull/31008#issuecomment-2060431846 R: @damccorm -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] Iceberg SchemaTransforms and Translation [beam]

2024-04-16 Thread via GitHub
ahmedabu98 commented on code in PR #30910: URL: https://github.com/apache/beam/pull/30910#discussion_r1568193051 ## sdks/java/io/iceberg/src/main/java/org/apache/beam/io/iceberg/IcebergIOTranslation.java: ## @@ -0,0 +1,241 @@ +/* + * Licensed to the Apache Software Foundation

Re: [PR] Iceberg SchemaTransforms and Translation [beam]

2024-04-16 Thread via GitHub
ahmedabu98 commented on code in PR #30910: URL: https://github.com/apache/beam/pull/30910#discussion_r1568193214 ## sdks/java/io/iceberg/src/main/java/org/apache/beam/io/iceberg/IcebergIOTranslation.java: ## @@ -0,0 +1,241 @@ +/* + * Licensed to the Apache Software Foundation

Re: [PR] Iceberg SchemaTransforms and Translation [beam]

2024-04-16 Thread via GitHub
ahmedabu98 commented on code in PR #30910: URL: https://github.com/apache/beam/pull/30910#discussion_r1568193684 ## sdks/java/managed/src/main/java/org/apache/beam/sdk/managed/Managed.java: ## @@ -108,8 +107,8 @@ public static ManagedTransform read(String source) { }

Re: [PR] Minor updates to StreamingInsertsMetrics [beam]

2024-04-16 Thread via GitHub
JayajP commented on PR #31003: URL: https://github.com/apache/beam/pull/31003#issuecomment-2060263015 Run Java_GCP_IO_Direct PreCommit -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] Use base 2 exponential histograms in BigQuerySinkMetrics [beam]

2024-04-16 Thread via GitHub
JayajP commented on PR #31000: URL: https://github.com/apache/beam/pull/31000#issuecomment-2060296545 Run Java_GCP_IO_Direct PreCommit -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] Iceberg SchemaTransforms and Translation [beam]

2024-04-16 Thread via GitHub
ahmedabu98 commented on code in PR #30910: URL: https://github.com/apache/beam/pull/30910#discussion_r1568192121 ## model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/schema_aware_transforms.proto: ## @@ -0,0 +1,73 @@ +/* + * Licensed to the Apache Software

Re: [PR] Iceberg SchemaTransforms and Translation [beam]

2024-04-16 Thread via GitHub
ahmedabu98 commented on code in PR #30910: URL: https://github.com/apache/beam/pull/30910#discussion_r1568192121 ## model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/schema_aware_transforms.proto: ## @@ -0,0 +1,73 @@ +/* + * Licensed to the Apache Software

Re: [PR] Update python transform catalog [beam]

2024-04-16 Thread via GitHub
damccorm commented on PR #30788: URL: https://github.com/apache/beam/pull/30788#issuecomment-2059270821 No problem - looks like it did fix the lint checks, so I'll merge once the coverage check finishes (assuming nothing broke unexpectedly) -- This is an automated message from the Apache

Re: [PR] [yaml] backtick generated aliases on sql mappings [beam]

2024-04-16 Thread via GitHub
Polber commented on code in PR #30895: URL: https://github.com/apache/beam/pull/30895#discussion_r1567555783 ## sdks/python/apache_beam/yaml/tests/maptofields.yaml: ## @@ -0,0 +1,62 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor

Re: [PR] Add a link to Grafana statistics for flaky workflows [beam]

2024-04-16 Thread via GitHub
damccorm merged PR #30934: URL: https://github.com/apache/beam/pull/30934 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [I] [Bug]: GRCPIO 1.62.1 deadlocks in Beam SDK worker with Dataflow [beam]

2024-04-16 Thread via GitHub
DerRidda commented on issue #30867: URL: https://github.com/apache/beam/issues/30867#issuecomment-2059239874 So, I removed all other dependency pins and just updated grpcio to 1.63.0rc1 (didn't even do grpcio-status in the beginning) on SDK version 2.55.1 with [the patch

Re: [I] [Bug]: GRCPIO 1.62.1 deadlocks in Beam SDK worker with Dataflow [beam]

2024-04-16 Thread via GitHub
damccorm commented on issue #30867: URL: https://github.com/apache/beam/issues/30867#issuecomment-2059261252 @tvalentyn is the fix here basically just disallowing 1.59.0-1.62.1 in https://github.com/apache/beam/blob/e59d313c3b17aaf45176ed5035068f01858809a0/sdks/python/setup.py#L368 ?

Re: [PR] Update python transform catalog [beam]

2024-04-16 Thread via GitHub
hjtran commented on PR #30788: URL: https://github.com/apache/beam/pull/30788#issuecomment-2059265662 SGTM, thanks @damccorm ! Sorry I didn't have time to take another look at this last week. -- This is an automated message from the Apache Git Service. To respond to the message, please

Re: [PR] [YAML] Add the ability to pre-process yaml files with jinja2. [beam]

2024-04-16 Thread via GitHub
damccorm commented on code in PR #30976: URL: https://github.com/apache/beam/pull/30976#discussion_r1567500942 ## sdks/python/apache_beam/yaml/main_test.py: ## @@ -0,0 +1,91 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license

Re: [PR] IcebergIO translation [beam]

2024-04-16 Thread via GitHub
ahmedabu98 closed pull request #30933: IcebergIO translation URL: https://github.com/apache/beam/pull/30933 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

Re: [PR] add terraform for utility cluster. Add name override to gke [beam]

2024-04-16 Thread via GitHub
volatilemolotov commented on PR #30847: URL: https://github.com/apache/beam/pull/30847#issuecomment-2059330063 @damondouglas I have added a number of changes that implement most of what has been discussed. Please take a look when you have time. Thanks -- This is an automated message

[I] [Bug]: Uber jar not work with Dataflow runner v2 due to `UnknownCoderWrapper` for Beam 2.55 [beam]

2024-04-16 Thread via GitHub
Abacn opened a new issue, #30994: URL: https://github.com/apache/beam/issues/30994 ### What happened? Minimum reproduce: A minimum pipeline: ``` PipelineOptions options = PipelineOptionsFactory.fromArgs(argv).create(); Pipeline p = Pipeline.create(options);

[PR] Add ReadChangeStream IO param to adjust backlog estimates for replication delay [beam]

2024-04-16 Thread via GitHub
jackdingilian opened a new pull request, #30995: URL: https://github.com/apache/beam/pull/30995 ReadChangeStreamIO estimates the backlog (via `getSize`) for each partition using `partitionLowWatermark * partitionThroughtput`. The low watermark is held back by replication delay for

Re: [PR] [yaml] backtick generated aliases on sql mappings [beam]

2024-04-16 Thread via GitHub
Polber commented on code in PR #30895: URL: https://github.com/apache/beam/pull/30895#discussion_r1567565392 ## website/www/site/content/en/documentation/sdks/yaml-udf.md: ## @@ -133,11 +184,26 @@ expressions are supported as well another_col: "col2 + col3" ```

Re: [PR] Add ReadChangeStream IO param to adjust backlog estimates for replication delay [beam]

2024-04-16 Thread via GitHub
jackdingilian commented on PR #30995: URL: https://github.com/apache/beam/pull/30995#issuecomment-2059372211 @tonytanger for Bigtable review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] Add a link to Grafana statistics for flaky workflows [beam]

2024-04-16 Thread via GitHub
damccorm commented on code in PR #30934: URL: https://github.com/apache/beam/pull/30934#discussion_r1567429718 ## .test-infra/tools/flaky_test_detection.py: ## @@ -24,6 +24,18 @@ ALERT_NAME = "flaky_test" GIT_ORG = "apache" GRAFANA_URL = "http://metrics.beam.apache.org;

Re: [PR] Fix ConcurrentModification exception possible in DataflowExecutionStateSampler [beam]

2024-04-16 Thread via GitHub
github-actions[bot] commented on PR #30993: URL: https://github.com/apache/beam/pull/30993#issuecomment-2059247521 Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment `assign set of reviewers` -- This is an automated

Re: [PR] replace clock.milliseconds with stopwatch [beam]

2024-04-16 Thread via GitHub
damccorm merged PR #30678: URL: https://github.com/apache/beam/pull/30678 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[PR] [Flink] finalize checkpoint marks in the new Flink source implementation (#30849) [beam]

2024-04-16 Thread via GitHub
je-ik opened a new pull request, #30987: URL: https://github.com/apache/beam/pull/30987 Merged from #30849 Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] Mention the

[PR] Remove flink 1.12 1.13 [beam]

2024-04-16 Thread via GitHub
je-ik opened a new pull request, #30988: URL: https://github.com/apache/beam/pull/30988 Removes Flink runner 1.12 and 1.13 after 1.17 will be merged. Thank you for your contribution! Follow this checklist to help us incorporate your contribution

Re: [PR] Implementing lull reporting at bundle level processing [beam]

2024-04-16 Thread via GitHub
arvindram03 commented on PR #30693: URL: https://github.com/apache/beam/pull/30693#issuecomment-2058398756 > LGTM after the 3 comments are addressed. The comments are addressed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on

Re: [PR] [Flink] finalize checkpoint marks in the new Flink source implementation (#30849) [beam]

2024-04-16 Thread via GitHub
je-ik commented on PR #30987: URL: https://github.com/apache/beam/pull/30987#issuecomment-2058366138 R: @jto @Abacn Merged from earlier implementation of #30849 after #30971 has been merged. -- This is an automated message from the Apache Git Service. To respond to the message, please

Re: [PR] [Flink] finalize checkpoint marks in the new Flink source implementation (#30849) [beam]

2024-04-16 Thread via GitHub
github-actions[bot] commented on PR #30987: URL: https://github.com/apache/beam/pull/30987#issuecomment-2058367835 Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control -- This is an automated message from the Apache Git

Re: [PR] Remove flink 1.12 1.13 [beam]

2024-04-16 Thread via GitHub
je-ik commented on PR #30988: URL: https://github.com/apache/beam/pull/30988#issuecomment-2058439543 R: @Abacn -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

  1   2   >