(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new addd47d086e Updating config from bot
addd47d086e is described below
commit addd47d086e13075424c242f5be60588226e0dc8
Author: github-actions
AuthorDate: Fri May 10 05:05:41 2024 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-31243.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-31243.json
b/scripts/ci/pr-bot/state/pr-state/pr-31243.json
new file mode 100644
index 000..242a48d7d3b
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-31243.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": true,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 73162f52348 Updating config from bot
73162f52348 is described below
commit 73162f52348f2848b74066acabf8d7b4ee549013
Author: github-actions
AuthorDate: Fri May 10 05:05:39 2024 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-31244.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-31244.json
b/scripts/ci/pr-bot/state/pr-state/pr-31244.json
new file mode 100644
index 000..242a48d7d3b
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-31244.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": true,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
(beam) branch dependabot/go_modules/sdks/github.com/testcontainers/testcontainers-go-0.30.0 deleted (was 54cd70688e9)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/go_modules/sdks/github.com/testcontainers/testcontainers-go-0.30.0 in repository https://gitbox.apache.org/repos/asf/beam.git was 54cd70688e9 Bump github.com/testcontainers/testcontainers-go in /sdks The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
(beam) branch dependabot/go_modules/sdks/github.com/testcontainers/testcontainers-go-0.31.0 created (now f9be67a89a8)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/go_modules/sdks/github.com/testcontainers/testcontainers-go-0.31.0 in repository https://gitbox.apache.org/repos/asf/beam.git at f9be67a89a8 Bump github.com/testcontainers/testcontainers-go in /sdks No new revisions were added by this update.
(beam) branch dependabot/go_modules/sdks/github.com/docker/docker-26.1.1incompatible deleted (was 440444008d0)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/go_modules/sdks/github.com/docker/docker-26.1.1incompatible in repository https://gitbox.apache.org/repos/asf/beam.git was 440444008d0 Bump github.com/docker/docker in /sdks The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
(beam) branch dependabot/go_modules/sdks/github.com/docker/docker-26.1.2incompatible created (now be595b0f3da)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/go_modules/sdks/github.com/docker/docker-26.1.2incompatible in repository https://gitbox.apache.org/repos/asf/beam.git at be595b0f3da Bump github.com/docker/docker in /sdks No new revisions were added by this update.
(beam) branch nightly-refs/heads/master updated (287ed388a68 -> 365c2d92965)
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a change to branch nightly-refs/heads/master
in repository https://gitbox.apache.org/repos/asf/beam.git
from 287ed388a68 Fix trivial inference tests for Python 3.12 support
(#31170)
add 7ed94383d2e Don't hardcode python version in tox.ini for lint, mypy
and docs. (#31222)
add 057eb06514a Add a container for Python 3.12. (#31216)
add 736f2b963dd Turn off abandoned node enforcement in some Reshuffle unit
tests where it is not needed
add 13708eaedeb Merge pull request #31229: Turn off abandoned node
enforcement in some Reshuffle unit tests where it is not needed
add e0bc8e770a7 Add IcebergIO integration tests (#31220)
add 365c2d92965 Support Kafka Managed IO (#31172)
No new revisions were added by this update.
Summary of changes:
...n => beam_PostCommit_Java_Hadoop_Versions.json} | 2 +-
.../{IO_Iceberg.yml => IO_Iceberg_Unit_Tests.yml} | 16 +-
build.gradle.kts | 6 +-
.../apache/beam/sdk/schemas/utils/YamlUtils.java | 8 +
.../apache/beam/sdk/transforms/ReshuffleTest.java | 4 +
sdks/java/io/iceberg/build.gradle | 26 +-
.../sdk/io/iceberg/SchemaAndRowConversions.java| 29 ++-
.../apache/beam/sdk/io/iceberg/IcebergIOIT.java| 289 +
.../io/iceberg/SchemaAndRowConversionsTest.java| 42 +++
sdks/java/io/kafka/build.gradle| 1 +
.../io/kafka/KafkaReadSchemaTransformProvider.java | 7 +-
.../KafkaReadSchemaTransformProviderTest.java | 53 +++-
.../KafkaWriteSchemaTransformProviderTest.java | 59 +
sdks/java/managed/build.gradle | 1 +
.../java/org/apache/beam/sdk/managed/Managed.java | 14 +-
.../managed/ManagedSchemaTransformProvider.java| 30 ++-
.../sdk/managed/ManagedTransformConstants.java | 52 +++-
sdks/python/container/build.gradle | 23 +-
.../{py311 => py312}/base_image_requirements.txt | 62 +++--
.../python/container/{py310 => py312}/build.gradle | 2 +-
sdks/python/test-suites/tox/py38/build.gradle | 14 -
sdks/python/test-suites/tox/pycommon/build.gradle | 18 +-
sdks/python/tox.ini| 8 +-
settings.gradle.kts| 1 +
24 files changed, 667 insertions(+), 100 deletions(-)
copy .github/trigger_files/{beam_PostCommit_Java_ValidatesRunner_Dataflow.json
=> beam_PostCommit_Java_Hadoop_Versions.json} (97%)
rename .github/workflows/{IO_Iceberg.yml => IO_Iceberg_Unit_Tests.yml} (89%)
create mode 100644
sdks/java/io/iceberg/src/test/java/org/apache/beam/sdk/io/iceberg/IcebergIOIT.java
copy sdks/python/container/{py311 => py312}/base_image_requirements.txt (82%)
copy sdks/python/container/{py310 => py312}/build.gradle (97%)
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new a82491aadd9 Updating config from bot a82491aadd9 is described below commit a82491aadd9c29af9c8fc3014714ad77a8a1cce4 Author: github-actions AuthorDate: Fri May 10 03:05:57 2024 + Updating config from bot --- scripts/ci/pr-bot/state/pr-state/pr-31237.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/pr-state/pr-31237.json b/scripts/ci/pr-bot/state/pr-state/pr-31237.json index b0c5ca106a2..c49d861dcce 100644 --- a/scripts/ci/pr-bot/state/pr-state/pr-31237.json +++ b/scripts/ci/pr-bot/state/pr-state/pr-31237.json @@ -7,5 +7,5 @@ "nextAction": "Reviewers", "stopReviewerNotifications": false, "remindAfterTestsPass": [], - "committerAssigned": false + "committerAssigned": true } \ No newline at end of file
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 5cd0831101e Updating config from bot
5cd0831101e is described below
commit 5cd0831101ef75b24f6c431da9683d52dd88f061
Author: github-actions
AuthorDate: Fri May 10 02:39:43 2024 +
Updating config from bot
---
scripts/ci/pr-bot/state/reviewers-for-label-build.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-build.json
b/scripts/ci/pr-bot/state/reviewers-for-label-build.json
index f7486cbd242..e89ba3fe0eb 100644
--- a/scripts/ci/pr-bot/state/reviewers-for-label-build.json
+++ b/scripts/ci/pr-bot/state/reviewers-for-label-build.json
@@ -2,6 +2,6 @@
"label": "build",
"dateOfLastReviewAssignment": {
"damccorm": 1715128207068,
-"Abacn": 1714527097428
+"Abacn": 1715308778682
}
}
\ No newline at end of file
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new e1cae52935a Updating config from bot
e1cae52935a is described below
commit e1cae52935a459d3db76d20847bd5824eeda2bea
Author: github-actions
AuthorDate: Fri May 10 02:39:40 2024 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-31237.json | 11 +++
1 file changed, 11 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-31237.json
b/scripts/ci/pr-bot/state/pr-state/pr-31237.json
new file mode 100644
index 000..b0c5ca106a2
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-31237.json
@@ -0,0 +1,11 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {
+"python": "shunping",
+"build": "Abacn"
+ },
+ "nextAction": "Reviewers",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new c5e16731074 Updating config from bot c5e16731074 is described below commit c5e16731074256ea460af53275db46b8072d2323 Author: github-actions AuthorDate: Fri May 10 02:39:36 2024 + Updating config from bot --- scripts/ci/pr-bot/state/pr-state/pr-31240.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/pr-state/pr-31240.json b/scripts/ci/pr-bot/state/pr-state/pr-31240.json index e1b3308f259..0046f79aae7 100644 --- a/scripts/ci/pr-bot/state/pr-state/pr-31240.json +++ b/scripts/ci/pr-bot/state/pr-state/pr-31240.json @@ -6,5 +6,5 @@ "nextAction": "Reviewers", "stopReviewerNotifications": false, "remindAfterTestsPass": [], - "committerAssigned": false + "committerAssigned": true } \ No newline at end of file
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new c55947a4180 Updating config from bot c55947a4180 is described below commit c55947a4180e2cf2e7516bb2077397d10df88496 Author: github-actions AuthorDate: Fri May 10 02:39:41 2024 + Updating config from bot --- scripts/ci/pr-bot/state/reviewers-for-label-python.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-python.json b/scripts/ci/pr-bot/state/reviewers-for-label-python.json index 9557856d9d0..7d3d14f863f 100644 --- a/scripts/ci/pr-bot/state/reviewers-for-label-python.json +++ b/scripts/ci/pr-bot/state/reviewers-for-label-python.json @@ -12,6 +12,6 @@ "jrmccluskey": 1715128207068, "riteshghorse": 1715029565371, "liferoad": 1715304582820, -"shunping": 1714777567812 +"shunping": 1715308778682 } } \ No newline at end of file
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new 5175d24c6a1 Updating config from bot 5175d24c6a1 is described below commit 5175d24c6a13a9b1010658bfa39e2421204c0d65 Author: github-actions AuthorDate: Fri May 10 01:29:41 2024 + Updating config from bot --- scripts/ci/pr-bot/state/reviewers-for-label-no-matching-label.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-no-matching-label.json b/scripts/ci/pr-bot/state/reviewers-for-label-no-matching-label.json index 2380139a967..a919405d6e9 100644 --- a/scripts/ci/pr-bot/state/reviewers-for-label-no-matching-label.json +++ b/scripts/ci/pr-bot/state/reviewers-for-label-no-matching-label.json @@ -8,7 +8,7 @@ "johnjcasey": 1714642910919, "jrmccluskey": 1715016955167, "kennknowles": 1714989950311, -"lostluck": 1713531961611, +"lostluck": 1715304578802, "riteshghorse": 1715063769548, "robertwb": 1714025210513, "tvalentyn": 1714408536058,
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new 8df2e6df77d Updating config from bot 8df2e6df77d is described below commit 8df2e6df77d25709b898a523463666425772a38c Author: github-actions AuthorDate: Fri May 10 01:29:45 2024 + Updating config from bot --- scripts/ci/pr-bot/state/reviewers-for-label-python.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-python.json b/scripts/ci/pr-bot/state/reviewers-for-label-python.json index a30f5941ab0..9557856d9d0 100644 --- a/scripts/ci/pr-bot/state/reviewers-for-label-python.json +++ b/scripts/ci/pr-bot/state/reviewers-for-label-python.json @@ -11,7 +11,7 @@ "damccorm": 1715302155876, "jrmccluskey": 1715128207068, "riteshghorse": 1715029565371, -"liferoad": 1714777564212, +"liferoad": 1715304582820, "shunping": 1714777567812 } } \ No newline at end of file
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 822fbf50617 Updating config from bot
822fbf50617 is described below
commit 822fbf50617de19bd2a592fc6d88310ad7aa29e4
Author: github-actions
AuthorDate: Fri May 10 01:29:43 2024 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-31240.json | 10 ++
1 file changed, 10 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-31240.json
b/scripts/ci/pr-bot/state/pr-state/pr-31240.json
new file mode 100644
index 000..e1b3308f259
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-31240.json
@@ -0,0 +1,10 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {
+"python": "liferoad"
+ },
+ "nextAction": "Reviewers",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 6afd16c1659 Updating config from bot
6afd16c1659 is described below
commit 6afd16c165940ff8a20443a47606ec77af4e6cc1
Author: github-actions
AuthorDate: Fri May 10 01:29:40 2024 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-31241.json | 10 ++
1 file changed, 10 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-31241.json
b/scripts/ci/pr-bot/state/pr-state/pr-31241.json
new file mode 100644
index 000..ec82f373f0a
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-31241.json
@@ -0,0 +1,10 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {
+"no-matching-label": "lostluck"
+ },
+ "nextAction": "Reviewers",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
(beam) branch tvalentyn-gha updated (2536cc276db -> 5809f23fc72)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch tvalentyn-gha in repository https://gitbox.apache.org/repos/asf/beam.git from 2536cc276db Add a license for a new dep. add 5809f23fc72 fixup No new revisions were added by this update. Summary of changes: sdks/python/container/license_scripts/dep_urls_py.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new c8dbb5e2bfc Updating config from bot
c8dbb5e2bfc is described below
commit c8dbb5e2bfc2d3a739d4356214f9f5b699512865
Author: github-actions
AuthorDate: Fri May 10 00:49:17 2024 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-31236.json | 10 ++
1 file changed, 10 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-31236.json
b/scripts/ci/pr-bot/state/pr-state/pr-31236.json
new file mode 100644
index 000..88d1920a8bb
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-31236.json
@@ -0,0 +1,10 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {
+"python": "damccorm"
+ },
+ "nextAction": "Reviewers",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new 79d67eeb02c Updating config from bot 79d67eeb02c is described below commit 79d67eeb02c25b3445f285709fbb411efd292afb Author: github-actions AuthorDate: Fri May 10 00:49:18 2024 + Updating config from bot --- scripts/ci/pr-bot/state/reviewers-for-label-python.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-python.json b/scripts/ci/pr-bot/state/reviewers-for-label-python.json index af0cba31000..a30f5941ab0 100644 --- a/scripts/ci/pr-bot/state/reviewers-for-label-python.json +++ b/scripts/ci/pr-bot/state/reviewers-for-label-python.json @@ -8,7 +8,7 @@ "tvalentyn": 1715180748178, "pabloem": 1681281324703, "y1chi": 1667002607045, -"damccorm": 1714777558411, +"damccorm": 1715302155876, "jrmccluskey": 1715128207068, "riteshghorse": 1715029565371, "liferoad": 1714777564212,
(beam) branch tvalentyn-gha updated (6688a500924 -> 2536cc276db)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch tvalentyn-gha in repository https://gitbox.apache.org/repos/asf/beam.git from 6688a500924 Publish snapshots for Py3.12 container. add 2536cc276db Add a license for a new dep. No new revisions were added by this update. Summary of changes: sdks/python/container/license_scripts/dep_urls_py.yaml | 2 ++ 1 file changed, 2 insertions(+)
(beam) branch asf-site updated: Publishing website 2024/05/09 23:38:51 at commit 365c2d9
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new 6f0ea945566 Publishing website 2024/05/09 23:38:51 at commit 365c2d9 6f0ea945566 is described below commit 6f0ea945566f638089b89e70ef443ca9eef714e1 Author: runner AuthorDate: Thu May 9 23:38:51 2024 + Publishing website 2024/05/09 23:38:51 at commit 365c2d9 --- website/generated-content/sitemap.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/generated-content/sitemap.xml b/website/generated-content/sitemap.xml index 350ff8347a1..f0f52e727f4 100644 --- a/website/generated-content/sitemap.xml +++ b/website/generated-content/sitemap.xml @@ -1 +1 @@ -http://www.sitemaps.org/schemas/sitemap/0.9; xmlns:xhtml="http://www.w3.org/1999/xhtml;>/blog/beam-2.56.0/2024-05-09T09:32:10-07:00/categories/blog/2024-05-09T09:32:10-07:00/blog/2024-05-09T09:32:10-07:00/categories/2024-05-09T09:32:10-07:00/catego [...] \ No newline at end of file +http://www.sitemaps.org/schemas/sitemap/0.9; xmlns:xhtml="http://www.w3.org/1999/xhtml;>/blog/beam-2.56.0/2024-05-09T17:14:17-04:00/categories/blog/2024-05-09T17:14:17-04:00/blog/2024-05-09T17:14:17-04:00/categories/2024-05-09T17:14:17-04:00/catego [...] \ No newline at end of file
(beam) branch tvalentyn-gha updated (2f295969f16 -> 6688a500924)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch tvalentyn-gha in repository https://gitbox.apache.org/repos/asf/beam.git discard 2f295969f16 Publish snapshots for Py3.12 container. omit d71dd0ba60d Remove a deprecated test helper. add 6688a500924 Publish snapshots for Py3.12 container. This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (2f295969f16) \ N -- N -- N refs/heads/tvalentyn-gha (6688a500924) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/options/pipeline_options_test.py | 4 ++-- sdks/python/apache_beam/testing/test_pipeline_test.py| 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-)
(beam) branch tvalentyn-gha updated: Publish snapshots for Py3.12 container.
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a commit to branch tvalentyn-gha in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/tvalentyn-gha by this push: new 2f295969f16 Publish snapshots for Py3.12 container. 2f295969f16 is described below commit 2f295969f16191bb50c39d8d833368890fc55c71 Author: Valentyn Tymofieiev AuthorDate: Thu May 9 16:09:55 2024 -0700 Publish snapshots for Py3.12 container. --- .github/workflows/beam_Publish_Beam_SDK_Snapshots.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/.github/workflows/beam_Publish_Beam_SDK_Snapshots.yml b/.github/workflows/beam_Publish_Beam_SDK_Snapshots.yml index 0f9d22c3ff5..23b78c36d7b 100644 --- a/.github/workflows/beam_Publish_Beam_SDK_Snapshots.yml +++ b/.github/workflows/beam_Publish_Beam_SDK_Snapshots.yml @@ -70,6 +70,7 @@ jobs: - "python:container:py39" - "python:container:py310" - "python:container:py311" + - "python:container:py312" steps: - uses: actions/checkout@v4 - name: Setup repository
(beam) branch tvalentyn-gha created (now d71dd0ba60d)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch tvalentyn-gha in repository https://gitbox.apache.org/repos/asf/beam.git at d71dd0ba60d Remove a deprecated test helper. No new revisions were added by this update.
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 5a0906e8a4d Updating config from bot
5a0906e8a4d is described below
commit 5a0906e8a4d45235fac747e5596729dea38800b7
Author: github-actions
AuthorDate: Thu May 9 23:06:03 2024 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-31235.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-31235.json
b/scripts/ci/pr-bot/state/pr-state/pr-31235.json
new file mode 100644
index 000..242a48d7d3b
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-31235.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": true,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
(beam) branch master updated: Support Kafka Managed IO (#31172)
This is an automated email from the ASF dual-hosted git repository.
ahmedabualsaud pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 365c2d92965 Support Kafka Managed IO (#31172)
365c2d92965 is described below
commit 365c2d92965c5e23c23d6e1f3c7a1cd048c872d8
Author: Ahmed Abualsaud <65791736+ahmedab...@users.noreply.github.com>
AuthorDate: Thu May 9 17:14:17 2024 -0400
Support Kafka Managed IO (#31172)
* managed kafka read
* managed kafka write
---
.../apache/beam/sdk/schemas/utils/YamlUtils.java | 8 +++
sdks/java/io/kafka/build.gradle| 1 +
.../io/kafka/KafkaReadSchemaTransformProvider.java | 7 ++-
.../KafkaReadSchemaTransformProviderTest.java | 53 ++-
.../KafkaWriteSchemaTransformProviderTest.java | 59 ++
sdks/java/managed/build.gradle | 1 +
.../java/org/apache/beam/sdk/managed/Managed.java | 14 +++--
.../managed/ManagedSchemaTransformProvider.java| 30 ++-
.../sdk/managed/ManagedTransformConstants.java | 52 ++-
9 files changed, 213 insertions(+), 12 deletions(-)
diff --git
a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/YamlUtils.java
b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/YamlUtils.java
index 122f2d1963b..e631e166e8b 100644
---
a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/YamlUtils.java
+++
b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/YamlUtils.java
@@ -20,6 +20,7 @@ package org.apache.beam.sdk.schemas.utils;
import static org.apache.beam.sdk.values.Row.toRow;
import java.math.BigDecimal;
+import java.util.Collections;
import java.util.List;
import java.util.Map;
import java.util.function.Function;
@@ -181,4 +182,11 @@ public class YamlUtils {
}
return new Yaml().dumpAsMap(map);
}
+
+ public static Map yamlStringToMap(@Nullable String yaml) {
+if (yaml == null || yaml.isEmpty()) {
+ return Collections.emptyMap();
+}
+return new Yaml().load(yaml);
+ }
}
diff --git a/sdks/java/io/kafka/build.gradle b/sdks/java/io/kafka/build.gradle
index 269ddb3f5eb..3e095a2bacc 100644
--- a/sdks/java/io/kafka/build.gradle
+++ b/sdks/java/io/kafka/build.gradle
@@ -90,6 +90,7 @@ dependencies {
provided library.java.everit_json_schema
testImplementation project(path: ":sdks:java:core", configuration:
"shadowTest")
testImplementation project(":sdks:java:io:synthetic")
+ testImplementation project(":sdks:java:managed")
testImplementation project(path: ":sdks:java:extensions:avro",
configuration: "testRuntimeMigration")
testImplementation project(path: ":sdks:java:extensions:protobuf",
configuration: "testRuntimeMigration")
testImplementation project(path: ":sdks:java:io:common", configuration:
"testRuntimeMigration")
diff --git
a/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaReadSchemaTransformProvider.java
b/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaReadSchemaTransformProvider.java
index 2776c388f7c..13240ea9dc4 100644
---
a/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaReadSchemaTransformProvider.java
+++
b/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaReadSchemaTransformProvider.java
@@ -151,11 +151,10 @@ public class KafkaReadSchemaTransformProvider
}
};
}
-
-if (format.equals("RAW")) {
+if ("RAW".equals(format)) {
beamSchema = Schema.builder().addField("payload",
Schema.FieldType.BYTES).build();
valueMapper = getRawBytesToRowFunction(beamSchema);
-} else if (format.equals("PROTO")) {
+} else if ("PROTO".equals(format)) {
String fileDescriptorPath = configuration.getFileDescriptorPath();
String messageName = configuration.getMessageName();
if (fileDescriptorPath != null) {
@@ -165,7 +164,7 @@ public class KafkaReadSchemaTransformProvider
beamSchema = ProtoByteUtils.getBeamSchemaFromProtoSchema(inputSchema,
messageName);
valueMapper =
ProtoByteUtils.getProtoBytesToRowFromSchemaFunction(inputSchema, messageName);
}
-} else if (format.equals("JSON")) {
+} else if ("JSON".equals(format)) {
beamSchema = JsonUtils.beamSchemaFromJsonSchema(inputSchema);
valueMapper = JsonUtils.getJsonBytesToRowFunction(beamSchema);
} else {
diff --git
a/sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaReadSchemaTransformProviderTest.java
b/sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaReadSchemaTransformProviderTest.java
index f6e231c758a..d5962a737ba 100644
---
a/sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaReadSchemaTransformProviderTest.java
+++
(beam) branch master updated (13708eaedeb -> e0bc8e770a7)
This is an automated email from the ASF dual-hosted git repository.
ahmedabualsaud pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
from 13708eaedeb Merge pull request #31229: Turn off abandoned node
enforcement in some Reshuffle unit tests where it is not needed
add e0bc8e770a7 Add IcebergIO integration tests (#31220)
No new revisions were added by this update.
Summary of changes:
...n => beam_PostCommit_Java_Hadoop_Versions.json} | 2 +-
.../{IO_Iceberg.yml => IO_Iceberg_Unit_Tests.yml} | 16 +-
build.gradle.kts | 1 +
sdks/java/io/iceberg/build.gradle | 26 +-
.../sdk/io/iceberg/SchemaAndRowConversions.java| 29 ++-
.../apache/beam/sdk/io/iceberg/IcebergIOIT.java| 289 +
.../io/iceberg/SchemaAndRowConversionsTest.java| 42 +++
7 files changed, 386 insertions(+), 19 deletions(-)
copy .github/trigger_files/{beam_PostCommit_Java_ValidatesRunner_Dataflow.json
=> beam_PostCommit_Java_Hadoop_Versions.json} (97%)
rename .github/workflows/{IO_Iceberg.yml => IO_Iceberg_Unit_Tests.yml} (89%)
create mode 100644
sdks/java/io/iceberg/src/test/java/org/apache/beam/sdk/io/iceberg/IcebergIOIT.java
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 1462162de1a Updating config from bot
1462162de1a is described below
commit 1462162de1af015c6e7e07314d8cac5666a8ac11
Author: github-actions
AuthorDate: Thu May 9 19:17:59 2024 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-31233.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-31233.json
b/scripts/ci/pr-bot/state/pr-state/pr-31233.json
new file mode 100644
index 000..9c2aa5aa212
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-31233.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": true,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
(beam) branch master updated (057eb06514a -> 13708eaedeb)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 057eb06514a Add a container for Python 3.12. (#31216) add 736f2b963dd Turn off abandoned node enforcement in some Reshuffle unit tests where it is not needed add 13708eaedeb Merge pull request #31229: Turn off abandoned node enforcement in some Reshuffle unit tests where it is not needed No new revisions were added by this update. Summary of changes: .../src/test/java/org/apache/beam/sdk/transforms/ReshuffleTest.java | 4 1 file changed, 4 insertions(+)
(beam) branch users/damccorm/batchElementsOvercount updated (8bbc456b431 -> bb40a22aaef)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch users/damccorm/batchElementsOvercount in repository https://gitbox.apache.org/repos/asf/beam.git from 8bbc456b431 Avoid oversizing batch sizes with size estimation function add bb40a22aaef lint No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/transforms/util_test.py | 7 +-- 1 file changed, 5 insertions(+), 2 deletions(-)
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 5bc05f52fd4 Updating config from bot
5bc05f52fd4 is described below
commit 5bc05f52fd461c48721b34f61ee54d186ec28749
Author: github-actions
AuthorDate: Thu May 9 17:17:07 2024 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-31232.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-31232.json
b/scripts/ci/pr-bot/state/pr-state/pr-31232.json
new file mode 100644
index 000..9c2aa5aa212
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-31232.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": true,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
(beam) branch tvalentyn-gha-py39 deleted (was 550b5d7a172)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch tvalentyn-gha-py39 in repository https://gitbox.apache.org/repos/asf/beam.git was 550b5d7a172 Move lint, yapf and dependency compat tests to from py38 to py39 The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
(beam) branch master updated: Add a container for Python 3.12. (#31216)
This is an automated email from the ASF dual-hosted git repository.
tvalentyn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 057eb06514a Add a container for Python 3.12. (#31216)
057eb06514a is described below
commit 057eb06514ad47383ffe33ee1faf0469833dc37d
Author: tvalentyn
AuthorDate: Thu May 9 09:32:10 2024 -0700
Add a container for Python 3.12. (#31216)
* Add a container for py312.
* fixup: bump the upper version.
---
sdks/python/container/build.gradle | 23 ++--
.../container/py312/base_image_requirements.txt| 152 +
sdks/python/container/py312/build.gradle | 28
settings.gradle.kts| 1 +
4 files changed, 191 insertions(+), 13 deletions(-)
diff --git a/sdks/python/container/build.gradle
b/sdks/python/container/build.gradle
index 161d343b303..f07b6f743fa 100644
--- a/sdks/python/container/build.gradle
+++ b/sdks/python/container/build.gradle
@@ -21,7 +21,7 @@ applyGoNature()
description = "Apache Beam :: SDKs :: Python :: Container"
int min_python_version=8
-int max_python_version=11
+int max_python_version=12
configurations {
sdkSourceTarball
@@ -38,10 +38,9 @@ goBuild {
}
tasks.register("buildAll") {
- dependsOn ':sdks:python:container:py38:docker'
- dependsOn ':sdks:python:container:py39:docker'
- dependsOn ':sdks:python:container:py310:docker'
- dependsOn ':sdks:python:container:py311:docker'
+ for(int ver=min_python_version; ver<=max_python_version; ++ver) {
+ dependsOn ':sdks:python:container:py3' + ver + ':docker'
+ }
}
for(int i=min_python_version; i<=max_python_version; ++i) {
@@ -68,17 +67,15 @@ for(int i=min_python_version; i<=max_python_version; ++i) {
}
tasks.register("pushAll") {
- dependsOn ':sdks:python:container:push38'
- dependsOn ':sdks:python:container:push39'
- dependsOn ':sdks:python:container:push310'
- dependsOn ':sdks:python:container:push311'
+ for(int ver=min_python_version; ver<=max_python_version; ++ver) {
+ dependsOn ':sdks:python:container:push3' + ver
+ }
}
tasks.register("generatePythonRequirementsAll") {
- dependsOn ':sdks:python:container:py38:generatePythonRequirements'
- dependsOn ':sdks:python:container:py39:generatePythonRequirements'
- dependsOn ':sdks:python:container:py310:generatePythonRequirements'
- dependsOn ':sdks:python:container:py311:generatePythonRequirements'
+ for(int ver=min_python_version; ver<=max_python_version; ++ver) {
+ dependsOn ':sdks:python:container:py3' + ver +
':generatePythonRequirements'
+ }
}
artifacts {
diff --git a/sdks/python/container/py312/base_image_requirements.txt
b/sdks/python/container/py312/base_image_requirements.txt
new file mode 100644
index 000..4a6147b573b
--- /dev/null
+++ b/sdks/python/container/py312/base_image_requirements.txt
@@ -0,0 +1,152 @@
+#Licensed to the Apache Software Foundation (ASF) under one or more
+#contributor license agreements. See the NOTICE file distributed with
+#this work for additional information regarding copyright ownership.
+#The ASF licenses this file to You under the Apache License, Version 2.0
+#(the "License"); you may not use this file except in compliance with
+#the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+#Unless required by applicable law or agreed to in writing, software
+#distributed under the License is distributed on an "AS IS" BASIS,
+#WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#See the License for the specific language governing permissions and
+#limitations under the License.
+
+# Autogenerated requirements file for Apache Beam py312 container image.
+# Run ./gradlew :sdks:python:container:generatePythonRequirementsAll to update.
+# Do not edit manually, adjust ../base_image_requirements_manual.txt or
+# Apache Beam's setup.py instead, and regenerate the list.
+# You will need Python interpreters for all versions supported by Beam, see:
+# https://s.apache.org/beam-python-dev-wiki
+# Reach out to a committer if you need help.
+
+annotated-types==0.6.0
+attrs==23.2.0
+beautifulsoup4==4.12.3
+bs4==0.0.2
+build==1.2.1
+cachetools==5.3.3
+certifi==2024.2.2
+cffi==1.16.0
+charset-normalizer==3.3.2
+click==8.1.7
+cloudpickle==2.2.1
+cramjam==2.8.3
+crcmod==1.7
+cryptography==42.0.7
+Cython==0.29.37
+deprecation==2.1.0
+dill==0.3.1.1
+dnspython==2.6.1
+docker==7.0.0
+docopt==0.6.2
+docstring_parser==0.16
+execnet==2.1.1
+fastavro==1.9.4
+fasteners==0.19
+freezegun==1.5.0
+future==1.0.0
+google-api-core==2.19.0
+google-api-python-client==2.128.0
+google-apitools==0.5.31
+google-auth==2.29.0
+google-auth-httplib2==0.2.0
+google-cloud-aiplatform==1.50.0
+google-cloud-bigquery==3.22.0
+google-cloud-bigquery-storage==2.25.0
(beam) branch master updated (287ed388a68 -> 7ed94383d2e)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 287ed388a68 Fix trivial inference tests for Python 3.12 support (#31170) add 7ed94383d2e Don't hardcode python version in tox.ini for lint, mypy and docs. (#31222) No new revisions were added by this update. Summary of changes: build.gradle.kts | 5 ++--- sdks/python/test-suites/tox/py38/build.gradle | 14 -- sdks/python/test-suites/tox/pycommon/build.gradle | 18 -- sdks/python/tox.ini | 8 4 files changed, 22 insertions(+), 23 deletions(-)
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new b1ac47283c5 Updating config from bot
b1ac47283c5 is described below
commit b1ac47283c5964a4fde962873b1e758bfcbba057
Author: github-actions
AuthorDate: Thu May 9 15:14:00 2024 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-31230.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-31230.json
b/scripts/ci/pr-bot/state/pr-state/pr-31230.json
new file mode 100644
index 000..9c2aa5aa212
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-31230.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": true,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 93a04c6de35 Updating config from bot
93a04c6de35 is described below
commit 93a04c6de35441e47e7bcb653dc964b10ce44c93
Author: github-actions
AuthorDate: Thu May 9 14:39:29 2024 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-31229.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-31229.json
b/scripts/ci/pr-bot/state/pr-state/pr-31229.json
new file mode 100644
index 000..9c2aa5aa212
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-31229.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": true,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 2e065cc6329 Updating config from bot
2e065cc6329 is described below
commit 2e065cc632932a055ce3fe1dc20b5f4f86ca073f
Author: github-actions
AuthorDate: Thu May 9 13:52:32 2024 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-31228.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-31228.json
b/scripts/ci/pr-bot/state/pr-state/pr-31228.json
new file mode 100644
index 000..9c2aa5aa212
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-31228.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": true,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
(beam) 01/01: Avoid oversizing batch sizes with size estimation function
This is an automated email from the ASF dual-hosted git repository.
damccorm pushed a commit to branch users/damccorm/batchElementsOvercount
in repository https://gitbox.apache.org/repos/asf/beam.git
commit 8bbc456b43194336698b9337be94345afe1b
Author: Danny McCormick
AuthorDate: Thu May 9 09:49:17 2024 -0400
Avoid oversizing batch sizes with size estimation function
---
sdks/python/apache_beam/transforms/util.py | 18 +--
sdks/python/apache_beam/transforms/util_test.py | 30 +
2 files changed, 37 insertions(+), 11 deletions(-)
diff --git a/sdks/python/apache_beam/transforms/util.py
b/sdks/python/apache_beam/transforms/util.py
index edf79b7c798..750d98f0789 100644
--- a/sdks/python/apache_beam/transforms/util.py
+++ b/sdks/python/apache_beam/transforms/util.py
@@ -579,14 +579,15 @@ class _GlobalWindowsBatchingDoFn(DoFn):
self._batch_size_estimator.ignore_next_timing()
def process(self, element):
-self._batch.append(element)
-self._running_batch_size += self._element_size_fn(element)
-if self._running_batch_size >= self._target_batch_size:
+element_size = self._element_size_fn(element)
+if self._running_batch_size + element_size > self._target_batch_size:
with self._batch_size_estimator.record_time(self._running_batch_size):
yield window.GlobalWindows.windowed_value_at_end_of_window(self._batch)
self._batch = []
self._running_batch_size = 0
self._target_batch_size = self._batch_size_estimator.next_batch_size()
+self._batch.append(element)
+self._running_batch_size += element_size
def finish_bundle(self):
if self._batch:
@@ -621,15 +622,18 @@ class _WindowAwareBatchingDoFn(DoFn):
def process(self, element, window=DoFn.WindowParam):
batch = self._batches[window]
-batch.elements.append(element)
-batch.size += self._element_size_fn(element)
-if batch.size >= self._target_batch_size:
+element_size = self._element_size_fn(element)
+if batch.size + element_size > self._target_batch_size:
with self._batch_size_estimator.record_time(batch.size):
yield windowed_value.WindowedValue(
batch.elements, window.max_timestamp(), (window, ))
del self._batches[window]
self._target_batch_size = self._batch_size_estimator.next_batch_size()
-elif len(self._batches) > self._MAX_LIVE_WINDOWS:
+
+self._batches[window].elements.append(element)
+self._batches[window].size += element_size
+
+if len(self._batches) > self._MAX_LIVE_WINDOWS:
window, batch = max(
self._batches.items(),
key=lambda window_batch: window_batch[1].size)
diff --git a/sdks/python/apache_beam/transforms/util_test.py
b/sdks/python/apache_beam/transforms/util_test.py
index 53898d57998..0d2af44ff47 100644
--- a/sdks/python/apache_beam/transforms/util_test.py
+++ b/sdks/python/apache_beam/transforms/util_test.py
@@ -299,15 +299,37 @@ class BatchElementsTest(unittest.TestCase):
res = (
p
| beam.Create([
- 'a', 'a', 'aa', # First batch.
- 'aa', 'a', # Second batch.
- 'a', 'aaa', 'a', 'a' # Third batch.
+ 'a', 'a',# First batch.
+ 'aa',# Second batch.
+ 'a', 'a',# Third batch.
+ 'a', 'aaa', 'a', 'a' # Fourth batch.
], reshuffle=False)
| util.BatchElements(
min_batch_size=10, max_batch_size=10, element_size_fn=len)
| beam.Map(lambda batch: ''.join(batch))
| beam.Map(len))
- assert_that(res, equal_to([12, 11, 10]))
+ assert_that(res, equal_to([2, 10, 10, 10]))
+
+ def test_sized_windowed_batches(self):
+# Assumes a single bundle, in order...
+with TestPipeline() as p:
+ res = (
+ p
+ | beam.Create(range(1, 8), reshuffle=False)
+ | beam.Map(lambda t: window.TimestampedValue('a'*t, t))
+ | beam.WindowInto(window.FixedWindows(3))
+ | util.BatchElements(
+ min_batch_size=11, max_batch_size=11, element_size_fn=len,
clock=FakeClock())
+ | beam.Map(lambda batch: ''.join(batch)))
+ assert_that(
+ res,
+ equal_to([
+ 'a' * (1+2), # Elements in [1, 3)
+ 'a' * (3+4), # Elements in [3, 6)
+ 'a' * 5,
+ 'a' * 6, # Elements in [6, 9)
+ 'a' * 7,
+ ]))
def test_target_duration(self):
clock = FakeClock()
(beam) branch users/damccorm/batchElementsOvercount created (now 8bbc456b431)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch users/damccorm/batchElementsOvercount in repository https://gitbox.apache.org/repos/asf/beam.git at 8bbc456b431 Avoid oversizing batch sizes with size estimation function This branch includes the following new commits: new 8bbc456b431 Avoid oversizing batch sizes with size estimation function The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
