Re: [PR] Correct per-entry HashMap overhead in WindmillStateCache [beam]

2024-03-19 Thread via GitHub
dmitryor commented on PR #30672: URL: https://github.com/apache/beam/pull/30672#issuecomment-2006005444 R: @scwhittle -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[PR] Correct per-entry HashMap overhead in WindmillStateCache [beam]

2024-03-19 Thread via GitHub
dmitryor opened a new pull request, #30672: URL: https://github.com/apache/beam/pull/30672 Existing code incorrectly assumes a per-entry HashMap overhead is 16 bytes. In reality it's 32 bytes [per this article](https://appsintheopen.com/posts/52-the-memory-overhead-of-java-ojects),

Re: [PR] Correct per-entry HashMap overhead in WindmillStateCache [beam]

2024-03-19 Thread via GitHub
github-actions[bot] commented on PR #30672: URL: https://github.com/apache/beam/pull/30672#issuecomment-2006009059 Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control -- This is an automated message from the Apache Git

Re: [PR] Correct per-entry HashMap overhead in WindmillStateCache [beam]

2024-03-19 Thread via GitHub
scwhittle commented on PR #30672: URL: https://github.com/apache/beam/pull/30672#issuecomment-2006346607 > Task :runners:google-cloud-dataflow-java:worker:test org.apache.beam.runners.dataflow.worker.windmill.state.WindmillStateCacheTest > testBasic FAILED

[PR] add pythonpath to cloudcoverage tox environment [beam]

2024-03-19 Thread via GitHub
volatilemolotov opened a new pull request, #30673: URL: https://github.com/apache/beam/pull/30673 Adds PYTHONPATH environment variable to cloudcoverage tox environment set to toxinidir (sdks/python). This allows pytest to load all the modules and run the correct coverage report

Re: [I] [Failing Test]: PostCommit Java SingleStoreIO IT failing [beam]

2024-03-19 Thread via GitHub
AdalbertMemSQL commented on issue #30564: URL: https://github.com/apache/beam/issues/30564#issuecomment-2006636094 Hey @Abacn Is it possible to somehow retrieve full workload logs? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [I] [Feature Request]: Refresh side input from BigQuery [beam]

2024-03-19 Thread via GitHub
BostjanBozic commented on issue #26196: URL: https://github.com/apache/beam/issues/26196#issuecomment-2006766022 Just a question - if you use global window (since Pub/Sub would be unbounded source), would `PeriodicImpulse` come into play or would you need to use `GenerateSequence` for

Re: [PR] add flag for direct path that reads from system properties [beam]

2024-03-19 Thread via GitHub
scwhittle commented on code in PR #30588: URL: https://github.com/apache/beam/pull/30588#discussion_r1530507529 ## runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/options/DataflowStreamingPipelineOptions.java: ## @@ -211,6 +211,14 @@ public

Re: [PR] Bump github.com/aws/aws-sdk-go-v2/credentials from 1.17.4 to 1.17.8 in /sdks [beam]

2024-03-19 Thread via GitHub
lostluck commented on PR #30669: URL: https://github.com/apache/beam/pull/30669#issuecomment-2007677567 LGTM -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] Bump github.com/aws/aws-sdk-go-v2/credentials from 1.17.4 to 1.17.8 in /sdks [beam]

2024-03-19 Thread via GitHub
lostluck merged PR #30669: URL: https://github.com/apache/beam/pull/30669 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Add BigTableIO Stress test [beam]

2024-03-19 Thread via GitHub
Abacn commented on code in PR #30630: URL: https://github.com/apache/beam/pull/30630#discussion_r1530769586 ## it/google-cloud-platform/src/main/java/org/apache/beam/it/gcp/IOStressTestBase.java: ## @@ -0,0 +1,158 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

Re: [PR] Add config validation to kafka read schema transform [beam]

2024-03-19 Thread via GitHub
ahmedabu98 commented on code in PR #30625: URL: https://github.com/apache/beam/pull/30625#discussion_r1530772951 ## sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaReadSchemaTransformConfiguration.java: ## @@ -59,20 +63,25 @@ public void validate() {

Re: [I] Performance Regression or Improvement: cogbk_python_batch_load_test_2GB_of_100B_records_with_a_multiple_key:runtime [beam]

2024-03-19 Thread via GitHub
liferoad closed issue #30659: Performance Regression or Improvement: cogbk_python_batch_load_test_2GB_of_100B_records_with_a_multiple_key:runtime URL: https://github.com/apache/beam/issues/30659 -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [I] Performance Regression or Improvement: gbk_python_batch_load_test_fanout_4_times_with_2GB_10byte_records_total:runtime [beam]

2024-03-19 Thread via GitHub
liferoad closed issue #30658: Performance Regression or Improvement: gbk_python_batch_load_test_fanout_4_times_with_2GB_10byte_records_total:runtime URL: https://github.com/apache/beam/issues/30658 -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [PR] Use 50 workers for 2GB 10 bytes combine test [beam]

2024-03-19 Thread via GitHub
Abacn commented on PR #30655: URL: https://github.com/apache/beam/pull/30655#issuecomment-2007343641 Can we change the cron string to sth like '50 5 * * *' so the risk of interfering with other workflow is minimum

Re: [PR] Add config validation to kafka read schema transform [beam]

2024-03-19 Thread via GitHub
Polber commented on code in PR #30625: URL: https://github.com/apache/beam/pull/30625#discussion_r1530776385 ## sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaReadSchemaTransformConfiguration.java: ## @@ -59,20 +63,25 @@ public void validate() { final

[PR] Revert "Disable remote gradle cache until it is cleaned (#30584)" [beam]

2024-03-19 Thread via GitHub
Abacn opened a new pull request, #30674: URL: https://github.com/apache/beam/pull/30674 This reverts commit a3ea9ef706cf798fc1f6b026dcdf7171434e74d8. **Please** add a meaningful description for your change here Thank you for your contribution!

Re: [PR] Revert "Disable remote gradle cache until it is cleaned (#30584)" [beam]

2024-03-19 Thread via GitHub
Abacn commented on PR #30674: URL: https://github.com/apache/beam/pull/30674#issuecomment-2007649109 Cache removed : https://issues.apache.org/jira/projects/INFRA/issues/INFRA-25595?filter=allopenissues -- This is an automated message from the Apache Git Service. To respond to the

Re: [PR] add a way for channels to be closed manually [beam]

2024-03-19 Thread via GitHub
scwhittle merged PR #30425: URL: https://github.com/apache/beam/pull/30425 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Add config validation to kafka read schema transform [beam]

2024-03-19 Thread via GitHub
github-actions[bot] commented on PR #30625: URL: https://github.com/apache/beam/pull/30625#issuecomment-2007582202 Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`: R: @damondouglas for label java. R: @ahmedabu98 for label io.

Re: [PR] Bump github.com/google/uuid from 1.5.0 to 1.6.0 in /sdks [beam]

2024-03-19 Thread via GitHub
dependabot[bot] commented on PR #30254: URL: https://github.com/apache/beam/pull/30254#issuecomment-2007685637 Looks like github.com/google/uuid is up-to-date now, so this is no longer needed. -- This is an automated message from the Apache Git Service. To respond to the message, please

Re: [PR] Bump github.com/google/uuid from 1.5.0 to 1.6.0 in /sdks [beam]

2024-03-19 Thread via GitHub
dependabot[bot] closed pull request #30254: Bump github.com/google/uuid from 1.5.0 to 1.6.0 in /sdks URL: https://github.com/apache/beam/pull/30254 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] Bump github.com/testcontainers/testcontainers-go from 0.26.0 to 0.29.1 in /sdks [beam]

2024-03-19 Thread via GitHub
lostluck commented on PR #30557: URL: https://github.com/apache/beam/pull/30557#issuecomment-2007686397 @dependabot rebase -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] [Python] Check feature store existence at construction time [beam]

2024-03-19 Thread via GitHub
damccorm commented on PR #30668: URL: https://github.com/apache/beam/pull/30668#issuecomment-2007237814 3.10 failure looks like an unrelated GameStatsIT failure. So this should be safe to move forward -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [PR] Use 50 workers for 2GB 10 bytes combine test [beam]

2024-03-19 Thread via GitHub
Abacn merged PR #30655: URL: https://github.com/apache/beam/pull/30655 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [I] [Bug]: beam_LoadTests_Python_Combine_Dataflow_Streaming failing [beam]

2024-03-19 Thread via GitHub
Abacn closed issue #23904: [Bug]: beam_LoadTests_Python_Combine_Dataflow_Streaming failing URL: https://github.com/apache/beam/issues/23904 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] Use 50 workers for 2GB 10 bytes combine test [beam]

2024-03-19 Thread via GitHub
Abacn commented on PR #30655: URL: https://github.com/apache/beam/pull/30655#issuecomment-2007640750 merge now and let's see how it goes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] update confluent dependency version to 7.6.0 [beam]

2024-03-19 Thread via GitHub
Abacn merged PR #30638: URL: https://github.com/apache/beam/pull/30638 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [I] [Task]: Update confluent libraries versions from 5.3.2 to more recent in kafka io extension [beam]

2024-03-19 Thread via GitHub
Abacn closed issue #30610: [Task]: Update confluent libraries versions from 5.3.2 to more recent in kafka io extension URL: https://github.com/apache/beam/issues/30610 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] Support ValueProvider for _CustomBigQueryStorageSource [beam]

2024-03-19 Thread via GitHub
Abacn merged PR #30662: URL: https://github.com/apache/beam/pull/30662 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[PR] Use autovalue's @Memoized in ExponentialBuckets [beam]

2024-03-19 Thread via GitHub
JayajP opened a new pull request, #30676: URL: https://github.com/apache/beam/pull/30676 Based on [autovalue's documentation](https://github.com/google/auto/blob/main/value/userguide/howto.md#-memoize-cache-derived-properties). For `ExponentialBuckets`, the `Base`,

[I] Seeking guidance on new runner implementation [beam]

2024-03-19 Thread via GitHub
moradology opened a new issue, #30675: URL: https://github.com/apache/beam/issues/30675 OK, so I'm very interested in making the batch subset of apache beam play nicely with EMR-Serverless. Unfortunately, this is difficult to pull off with the portable runner - perhaps impossible even - as

Re: [I] Performance Regression or Improvement: cogbk_python_batch_load_test_2GB_of_100B_records_with_a_single_key:runtime [beam]

2024-03-19 Thread via GitHub
liferoad closed issue #30652: Performance Regression or Improvement: cogbk_python_batch_load_test_2GB_of_100B_records_with_a_single_key:runtime URL: https://github.com/apache/beam/issues/30652 -- This is an automated message from the Apache Git Service. To respond to the message, please

Re: [I] Performance Regression or Improvement: gbk_python_batch_load_test_2gb_of_100B_records:runtime [beam]

2024-03-19 Thread via GitHub
liferoad closed issue #30651: Performance Regression or Improvement: gbk_python_batch_load_test_2gb_of_100B_records:runtime URL: https://github.com/apache/beam/issues/30651 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [I] Performance Regression or Improvement: gbk_python_batch_load_test_2gb_of_10B_records:runtime [beam]

2024-03-19 Thread via GitHub
liferoad closed issue #30650: Performance Regression or Improvement: gbk_python_batch_load_test_2gb_of_10B_records:runtime URL: https://github.com/apache/beam/issues/30650 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [I] Performance Regression or Improvement: combine_python_batch_2gb_10_byte_records:runtime [beam]

2024-03-19 Thread via GitHub
liferoad closed issue #30649: Performance Regression or Improvement: combine_python_batch_2gb_10_byte_records:runtime URL: https://github.com/apache/beam/issues/30649 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [I] Performance Regression or Improvement: gbk_python_batch_load_test_fanout_8_times_with_2GB_10byte_records_total:runtime [beam]

2024-03-19 Thread via GitHub
liferoad closed issue #30640: Performance Regression or Improvement: gbk_python_batch_load_test_fanout_8_times_with_2GB_10byte_records_total:runtime URL: https://github.com/apache/beam/issues/30640 -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [I] [Task]: Check if feature store exists in VertexAIFeatureStoreEnrichmentHandler [beam]

2024-03-19 Thread via GitHub
riteshghorse closed issue #30541: [Task]: Check if feature store exists in VertexAIFeatureStoreEnrichmentHandler URL: https://github.com/apache/beam/issues/30541 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [I] [Task]: Check if feature store exists in VertexAIFeatureStoreEnrichmentHandler [beam]

2024-03-19 Thread via GitHub
riteshghorse commented on issue #30541: URL: https://github.com/apache/beam/issues/30541#issuecomment-2007266896 Fixed by #30668 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Update Python Dependencies from BRANCH weekly_update_python_dependencies_1710635460 [beam]

2024-03-19 Thread via GitHub
Abacn merged PR #30656: URL: https://github.com/apache/beam/pull/30656 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Bump github.com/google/uuid from 1.5.0 to 1.6.0 in /sdks [beam]

2024-03-19 Thread via GitHub
lostluck commented on PR #30254: URL: https://github.com/apache/beam/pull/30254#issuecomment-2007684387 @dependabot rebase -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] [Python] Check feature store existence at construction time [beam]

2024-03-19 Thread via GitHub
damccorm commented on PR #30668: URL: https://github.com/apache/beam/pull/30668#issuecomment-2007222555 Oh, looks like postcommits are failing though, could you take a look please? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] [Python] Check feature store existence at construction time [beam]

2024-03-19 Thread via GitHub
damccorm commented on PR #30668: URL: https://github.com/apache/beam/pull/30668#issuecomment-2007225474 3.11 failure looks like an unrelated pubsub test -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [Python] Check feature store existence at construction time [beam]

2024-03-19 Thread via GitHub
riteshghorse merged PR #30668: URL: https://github.com/apache/beam/pull/30668 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] [Python] Check feature store existence at construction time [beam]

2024-03-19 Thread via GitHub
riteshghorse commented on PR #30668: URL: https://github.com/apache/beam/pull/30668#issuecomment-2007260795 Yeah look like it, It is passing on master - https://github.com/apache/beam/actions/runs/8338077957/job/22817805661. Thanks! -- This is an automated message from the Apache

Re: [PR] Tune maximum thread count for streaming dataflow worker executor dynamically. [beam]

2024-03-19 Thread via GitHub
scwhittle commented on code in PR #30439: URL: https://github.com/apache/beam/pull/30439#discussion_r1530536071 ## runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/BoundedQueueExecutor.java: ## @@ -59,8 +72,8 @@ public

Re: [PR] Use 50 workers for 2GB 10 bytes combine test [beam]

2024-03-19 Thread via GitHub
liferoad commented on PR #30655: URL: https://github.com/apache/beam/pull/30655#issuecomment-2007621658 > Can we change the cron string to sth like '50 5 * * *' so the risk of interfering with other workflow is minimum > >

Re: [PR] [Python] Upgrade mypy version to 1.7.1 [beam]

2024-03-19 Thread via GitHub
riteshghorse closed pull request #29707: [Python] Upgrade mypy version to 1.7.1 URL: https://github.com/apache/beam/pull/29707 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] Revert "Disable remote gradle cache until it is cleaned (#30584)" [beam]

2024-03-19 Thread via GitHub
github-actions[bot] commented on PR #30674: URL: https://github.com/apache/beam/pull/30674#issuecomment-2007765764 Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment `assign set of reviewers` -- This is an automated

[PR] Minimize scope of expensive lock [beam]

2024-03-19 Thread via GitHub
damccorm opened a new pull request, #30679: URL: https://github.com/apache/beam/pull/30679 We've seen this section hang, eating up the lock when shutdown takes a long time. This should minimize the size of the critical section to a single copy of `self.last_access_times.items()` so that it

Re: [PR] Deduplicate common environments. [beam]

2024-03-19 Thread via GitHub
github-actions[bot] commented on PR #30681: URL: https://github.com/apache/beam/pull/30681#issuecomment-2008172678 Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control -- This is an automated message from the Apache Git

Re: [PR] Deduplicate common environments. [beam]

2024-03-19 Thread via GitHub
robertwb commented on PR #30681: URL: https://github.com/apache/beam/pull/30681#issuecomment-2008171114 R: @chamikaramj -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] Tune maximum thread count for streaming dataflow worker executor dynamically. [beam]

2024-03-19 Thread via GitHub
MelodyShen commented on code in PR #30439: URL: https://github.com/apache/beam/pull/30439#discussion_r1531166753 ## runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/util/BoundedQueueExecutorTest.java: ## @@ -0,0 +1,232 @@ +/* + *

Re: [PR] Add Redistribute transform to model, Java SDK, and most active runners [beam]

2024-03-19 Thread via GitHub
kennknowles commented on PR #30545: URL: https://github.com/apache/beam/pull/30545#issuecomment-2007801662 The problem in the portable runner was that it was not registered as a "native" transform. I don't know the details, but this influences the fuser. -- This is an automated message

Re: [I] The PostCommit Python ValidatesRunner Spark job is flaky [beam]

2024-03-19 Thread via GitHub
Abacn commented on issue #30645: URL: https://github.com/apache/beam/issues/30645#issuecomment-2007928104 This is due to https://github.com/apache/beam/pull/30587#issuecomment-2004812901 CC: @robertwb -- This is an automated message from the Apache Git Service. To respond to the

Re: [PR] replace clock.milliseconds with stopwatch [beam]

2024-03-19 Thread via GitHub
github-actions[bot] commented on PR #30678: URL: https://github.com/apache/beam/pull/30678#issuecomment-2007928098 Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment `assign set of reviewers` -- This is an automated

Re: [I] The PostCommit XVR Samza job is flaky [beam]

2024-03-19 Thread via GitHub
Abacn closed issue #30601: The PostCommit XVR Samza job is flaky URL: https://github.com/apache/beam/issues/30601 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] Tune maximum thread count for streaming dataflow worker executor dynamically. [beam]

2024-03-19 Thread via GitHub
MelodyShen commented on code in PR #30439: URL: https://github.com/apache/beam/pull/30439#discussion_r1531096133 ## runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/util/BoundedQueueExecutorTest.java: ## @@ -0,0 +1,232 @@ +/* + *

[PR] Deduplicate common environments. [beam]

2024-03-19 Thread via GitHub
robertwb opened a new pull request, #30681: URL: https://github.com/apache/beam/pull/30681 This can be especially useful for those pipelines with many cross-language transforms. Thank you for your contribution! Follow this checklist to help us

Re: [PR] Use autovalue's @Memoized in ExponentialBuckets [beam]

2024-03-19 Thread via GitHub
JayajP commented on PR #30676: URL: https://github.com/apache/beam/pull/30676#issuecomment-2007940060 R: @m-trieu -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] Use autovalue's @Memoized in ExponentialBuckets [beam]

2024-03-19 Thread via GitHub
github-actions[bot] commented on PR #30676: URL: https://github.com/apache/beam/pull/30676#issuecomment-2007942325 Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control -- This is an automated message from the Apache Git

Re: [PR] add ExternalTransformProvider example [beam]

2024-03-19 Thread via GitHub
liferoad commented on code in PR #30666: URL: https://github.com/apache/beam/pull/30666#discussion_r1530967292 ## examples/multi-language/README.md: ## @@ -28,6 +28,8 @@ This project provides examples of Apache Beam * **python/javacount** - A Python pipeline that counts words

[PR] Disable unsupported custom window type test on samza and spark. [beam]

2024-03-19 Thread via GitHub
robertwb opened a new pull request, #30680: URL: https://github.com/apache/beam/pull/30680 Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] Mention the appropriate issue in

Re: [PR] Deduplicate common environments. [beam]

2024-03-19 Thread via GitHub
robertwb commented on code in PR #30681: URL: https://github.com/apache/beam/pull/30681#discussion_r1531179623 ## sdks/python/apache_beam/runners/common.py: ## @@ -1941,3 +1945,64 @@ def validate_transform(transform_id): for t in pipeline_proto.root_transform_ids:

Re: [PR] Use autovalue's @Memoized in ExponentialBuckets [beam]

2024-03-19 Thread via GitHub
github-actions[bot] commented on PR #30676: URL: https://github.com/apache/beam/pull/30676#issuecomment-2007873357 Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment `assign set of reviewers` -- This is an automated

Re: [PR] Revert "Disable remote gradle cache until it is cleaned (#30584)" [beam]

2024-03-19 Thread via GitHub
Abacn commented on PR #30674: URL: https://github.com/apache/beam/pull/30674#issuecomment-2007874461 R: @damccorm -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] Add Redistribute transform to model, Java SDK, and most active runners [beam]

2024-03-19 Thread via GitHub
kennknowles commented on PR #30545: URL: https://github.com/apache/beam/pull/30545#issuecomment-2008089051 OK this is now green. The test suites that are still running were green before and have not been touched. Please review! -- This is an automated message from the Apache Git Service.

Re: [PR] Revert "Disable remote gradle cache until it is cleaned (#30584)" [beam]

2024-03-19 Thread via GitHub
github-actions[bot] commented on PR #30674: URL: https://github.com/apache/beam/pull/30674#issuecomment-2007876434 Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control -- This is an automated message from the Apache Git

Re: [PR] Minimize scope of expensive lock [beam]

2024-03-19 Thread via GitHub
damccorm commented on PR #30679: URL: https://github.com/apache/beam/pull/30679#issuecomment-2008095596 R: @tvalentyn -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] Add Redistribute transform to model, Java SDK, and most active runners [beam]

2024-03-19 Thread via GitHub
kennknowles commented on code in PR #30545: URL: https://github.com/apache/beam/pull/30545#discussion_r1531106246 ## runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java: ## @@ -917,6 +919,41 @@ private void

Re: [PR] Deduplicate common environments. [beam]

2024-03-19 Thread via GitHub
chamikaramj commented on code in PR #30681: URL: https://github.com/apache/beam/pull/30681#discussion_r1531155798 ## sdks/python/apache_beam/runners/common.py: ## @@ -1941,3 +1945,64 @@ def validate_transform(transform_id): for t in pipeline_proto.root_transform_ids:

Re: [I] [Bug]: Dataflow worker container resolved to legacy runner label if not explicitly disable/enable runner v2 in 2.54.0+.dev [beam]

2024-03-19 Thread via GitHub
Abacn commented on issue #30634: URL: https://github.com/apache/beam/issues/30634#issuecomment-2007838586 This also affects

Re: [I] The PostCommit Python ValidatesRunner Samza job is flaky [beam]

2024-03-19 Thread via GitHub
Abacn commented on issue #30657: URL: https://github.com/apache/beam/issues/30657#issuecomment-2007926950 This is due to https://github.com/apache/beam/pull/30587#issuecomment-2004812901 CC: @robertwb -- This is an automated message from the Apache Git Service. To respond to the

Re: [PR] Minimize scope of expensive lock [beam]

2024-03-19 Thread via GitHub
github-actions[bot] commented on PR #30679: URL: https://github.com/apache/beam/pull/30679#issuecomment-2008098106 Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control -- This is an automated message from the Apache Git

Re: [PR] Tune maximum thread count for streaming dataflow worker executor dynamically. [beam]

2024-03-19 Thread via GitHub
MelodyShen commented on code in PR #30439: URL: https://github.com/apache/beam/pull/30439#discussion_r1531099531 ## runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/util/BoundedQueueExecutorTest.java: ## @@ -0,0 +1,232 @@ +/* + *

Re: [PR] Tune maximum thread count for streaming dataflow worker executor dynamically. [beam]

2024-03-19 Thread via GitHub
MelodyShen commented on code in PR #30439: URL: https://github.com/apache/beam/pull/30439#discussion_r1531100277 ## runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/BoundedQueueExecutor.java: ## @@ -59,8 +72,8 @@ public

Re: [PR] replace clock.milliseconds with stopwatch [beam]

2024-03-19 Thread via GitHub
clmccart commented on PR #30678: URL: https://github.com/apache/beam/pull/30678#issuecomment-2007781631 cc: @tudorm -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] Add Redistribute transform to model, Java SDK, and most active runners [beam]

2024-03-19 Thread via GitHub
kennknowles commented on code in PR #30545: URL: https://github.com/apache/beam/pull/30545#discussion_r1531103041 ## model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/beam_runner_api.proto: ## @@ -813,6 +813,10 @@ message GroupIntoBatchesPayload { int64

Re: [PR] Update Python Dependencies from BRANCH weekly_update_python_dependencies_1708820977 [beam]

2024-03-19 Thread via GitHub
damondouglas closed pull request #30412: Update Python Dependencies from BRANCH weekly_update_python_dependencies_1708820977 URL: https://github.com/apache/beam/pull/30412 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] Add BigTableIO Stress test [beam]

2024-03-19 Thread via GitHub
akashorabek commented on PR #30630: URL: https://github.com/apache/beam/pull/30630#issuecomment-2008682755 Run Kotlin_Examples PreCommit -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [I] Seeking guidance on new runner implementation [beam]

2024-03-19 Thread via GitHub
cisaacstern commented on issue #30675: URL: https://github.com/apache/beam/issues/30675#issuecomment-2008380280  @moradology! Over in my WIP for DaskRunner SideInputs, I've got this:

Re: [PR] Disable unsupported custom window type test on samza and spark. [beam]

2024-03-19 Thread via GitHub
Abacn merged PR #30680: URL: https://github.com/apache/beam/pull/30680 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [I] The PostCommit Python ValidatesRunner Samza job is flaky [beam]

2024-03-19 Thread via GitHub
Abacn closed issue #30657: The PostCommit Python ValidatesRunner Samza job is flaky URL: https://github.com/apache/beam/issues/30657 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Add BigTableIO Stress test [beam]

2024-03-19 Thread via GitHub
akashorabek commented on code in PR #30630: URL: https://github.com/apache/beam/pull/30630#discussion_r1531474427 ## it/google-cloud-platform/src/main/java/org/apache/beam/it/gcp/IOStressTestBase.java: ## @@ -0,0 +1,158 @@ +/* + * Licensed to the Apache Software Foundation

[PR] Bump github.com/docker/docker from 25.0.3+incompatible to 25.0.5+incompatible in /sdks [beam]

2024-03-19 Thread via GitHub
dependabot[bot] opened a new pull request, #30684: URL: https://github.com/apache/beam/pull/30684 Bumps [github.com/docker/docker](https://github.com/docker/docker) from 25.0.3+incompatible to 25.0.5+incompatible. Release notes Sourced from

[PR] Bump golang.org/x/oauth2 from 0.17.0 to 0.18.0 in /sdks [beam]

2024-03-19 Thread via GitHub
dependabot[bot] opened a new pull request, #30685: URL: https://github.com/apache/beam/pull/30685 Bumps [golang.org/x/oauth2](https://github.com/golang/oauth2) from 0.17.0 to 0.18.0. Commits

Re: [PR] Bump golang.org/x/oauth2 from 0.17.0 to 0.18.0 in /sdks [beam]

2024-03-19 Thread via GitHub
github-actions[bot] commented on PR #30685: URL: https://github.com/apache/beam/pull/30685#issuecomment-2008666471 Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`: R: @riteshghorse for label go. Available commands: - `stop

Re: [PR] Bump github.com/docker/docker from 25.0.3+incompatible to 25.0.5+incompatible in /sdks [beam]

2024-03-19 Thread via GitHub
github-actions[bot] commented on PR #30684: URL: https://github.com/apache/beam/pull/30684#issuecomment-2008666508 Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`: R: @jrmccluskey for label go. Available commands: - `stop

Re: [PR] Implement ordered list state for FnApi. [beam]

2024-03-19 Thread via GitHub
acrites commented on code in PR #30317: URL: https://github.com/apache/beam/pull/30317#discussion_r1531296538 ## model/fn-execution/src/main/proto/org/apache/beam/model/fn_execution/v1/beam_fn_api.proto: ## Review Comment: Or are we going to piggy-back on multimap for

Re: [I] The PostCommit Python ValidatesRunner Samza job is flaky [beam]

2024-03-19 Thread via GitHub
github-actions[bot] commented on issue #30657: URL: https://github.com/apache/beam/issues/30657#issuecomment-2008452865 Reopening since the workflow is still flaky -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[I] The PreCommit Java job is flaky [beam]

2024-03-19 Thread via GitHub
github-actions[bot] opened a new issue, #30683: URL: https://github.com/apache/beam/issues/30683 The PreCommit Java is failing over 50% of the time Please visit https://github.com/apache/beam/actions/workflows/beam_PreCommit_Java.yml?query=is%3Afailure+branch%3Amaster to see the logs.

Re: [PR] Bump github.com/docker/docker from 25.0.3+incompatible to 25.0.5+incompatible in /sdks [beam]

2024-03-19 Thread via GitHub
codecov[bot] commented on PR #30684: URL: https://github.com/apache/beam/pull/30684#issuecomment-2008651036 ## [Codecov](https://app.codecov.io/gh/apache/beam/pull/30684?dropdown=coverage=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) Report All

Re: [PR] Bump golang.org/x/oauth2 from 0.17.0 to 0.18.0 in /sdks [beam]

2024-03-19 Thread via GitHub
codecov[bot] commented on PR #30685: URL: https://github.com/apache/beam/pull/30685#issuecomment-2008651162 ## [Codecov](https://app.codecov.io/gh/apache/beam/pull/30685?dropdown=coverage=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) Report All

Re: [PR] Add BigTableIO Stress test [beam]

2024-03-19 Thread via GitHub
akashorabek commented on PR #30630: URL: https://github.com/apache/beam/pull/30630#issuecomment-2008651373 Run Java_Examples_Dataflow PreCommit -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] Implement ordered list state for FnApi. [beam]

2024-03-19 Thread via GitHub
acrites commented on code in PR #30317: URL: https://github.com/apache/beam/pull/30317#discussion_r1531294554 ## model/fn-execution/src/main/proto/org/apache/beam/model/fn_execution/v1/beam_fn_api.proto: ## Review Comment: Do we also need to add something here:

[PR] [Python] Add redis client to python dependencies [beam]

2024-03-19 Thread via GitHub
riteshghorse opened a new pull request, #30677: URL: https://github.com/apache/beam/pull/30677 With the addition of redis cache support to the Enrichment transform, it will be good to package redis client dependency with apache-beam package. Otherwise user will have to install it

Re: [PR] add ExternalTransformProvider example [beam]

2024-03-19 Thread via GitHub
ahmedabu98 commented on code in PR #30666: URL: https://github.com/apache/beam/pull/30666#discussion_r1530961429 ## examples/multi-language/python/wordcount_external.py: ## @@ -0,0 +1,102 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor

Re: [PR] Disable unsupported custom window type test on samza and spark. [beam]

2024-03-19 Thread via GitHub
robertwb commented on PR #30680: URL: https://github.com/apache/beam/pull/30680#issuecomment-2007996651 R: @Abacn -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] Disable unsupported custom window type test on samza and spark. [beam]

2024-03-19 Thread via GitHub
github-actions[bot] commented on PR #30680: URL: https://github.com/apache/beam/pull/30680#issuecomment-2007999326 Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control -- This is an automated message from the Apache Git

Re: [PR] Revert "Disable remote gradle cache until it is cleaned (#30584)" [beam]

2024-03-19 Thread via GitHub
damccorm merged PR #30674: URL: https://github.com/apache/beam/pull/30674 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

  1   2   >