(beam) branch nightly-refs/heads/master updated (66da760357c -> c88a863870c)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch nightly-refs/heads/master in repository https://gitbox.apache.org/repos/asf/beam.git from 66da760357c Update yaml to non-beta link (#31210) add a0c08272d42 Touch trigger files for all relevant ValidatesRunner and PVR suites add bf347fe898b Sickbay extremely flaky Flink ParDoLifecycleTest add ef431ec90f2 Reapply "Add Redistribute transform to Java SDK" add 737be230347 Merge pull request #31156: Reapply "Add Redistribute transform to Java SDK" add 466516b016c [Python] Use BufferedLogger instead of logger to log in boot.go (#31265) add 749c128bef1 Disclaim direct runner result can be different with remote runners (#31250) add c230655a1e5 Implement the hash_words TFT operation (#31249) add 296c13a291d Avoid side input issue where Latest.globally produces 2 elements in a window (#31263) add c88a863870c Add a subsuite of unit tests for ML package and remove them from 'coverage' GHA workflow. (#31255) No new revisions were added by this update. Summary of changes: .../beam_PostCommit_Java_PVR_Flink_Batch.json | 4 + .../beam_PostCommit_Java_PVR_Flink_Docker.json | 4 + .../beam_PostCommit_Java_PVR_Flink_Streaming.json | 4 + .../beam_PostCommit_Java_PVR_Samza.json| 4 + .../beam_PostCommit_Java_PVR_Spark3_Batch.json | 4 + .../beam_PostCommit_Java_PVR_Spark3_Streaming.json | 3 +- ...m_PostCommit_Java_ValidatesRunner_Dataflow.json | 3 +- ...Java_ValidatesRunner_Dataflow_JavaVersions.json | 4 + ...it_Java_ValidatesRunner_Dataflow_Streaming.json | 3 +- ...ostCommit_Java_ValidatesRunner_Dataflow_V2.json | 3 +- ...Java_ValidatesRunner_Dataflow_V2_Streaming.json | 3 +- ...eam_PostCommit_Java_ValidatesRunner_Direct.json | 4 + ...t_Java_ValidatesRunner_Direct_JavaVersions.json | 4 + ...beam_PostCommit_Java_ValidatesRunner_Flink.json | 3 +- ...stCommit_Java_ValidatesRunner_Flink_Java11.json | 3 +- ...beam_PostCommit_Java_ValidatesRunner_Samza.json | 3 +- ...beam_PostCommit_Java_ValidatesRunner_Spark.json | 3 +- ...a_ValidatesRunner_SparkStructuredStreaming.json | 3 +- ...stCommit_Java_ValidatesRunner_Spark_Java11.json | 4 + ...m_PostCommit_Java_ValidatesRunner_Twister2.json | 3 +- .../beam_PostCommit_Java_ValidatesRunner_ULR.json | 3 +- .github/workflows/README.md| 1 + .github/workflows/beam_PreCommit_Python.yml| 3 +- .../workflows/beam_PreCommit_Python_Dataframes.yml | 3 +- .../workflows/beam_PreCommit_Python_Examples.yml | 3 +- ...on_Runners.yml => beam_PreCommit_Python_ML.yml} | 19 +- .../workflows/beam_PreCommit_Python_Runners.yml| 3 +- .../workflows/beam_PreCommit_Python_Transforms.yml | 3 +- .../apache/beam/examples/snippets/Snippets.java| 14 +- runners/flink/flink_runner.gradle | 3 + runners/flink/job-server/flink_job_server.gradle | 2 + runners/portability/java/build.gradle | 2 + runners/samza/job-server/build.gradle | 2 + runners/spark/job-server/spark_job_server.gradle | 8 + runners/spark/spark_runner.gradle | 7 + .../apache/beam/sdk/transforms/Redistribute.java | 300 +++ .../util/construction/PTransformTranslation.java | 8 + .../beam/sdk/transforms/RedistributeTest.java | 410 + .../ml/inference/huggingface_inference.py | 2 +- .../ml/inference/huggingface_inference_test.py | 2 +- .../ml/inference/onnx_inference_test.py| 4 + sdks/python/apache_beam/ml/transforms/tft.py | 44 +++ sdks/python/apache_beam/ml/transforms/tft_test.py | 77 sdks/python/container/boot.go | 4 +- sdks/python/setup.py | 19 + sdks/python/test-suites/tox/common.gradle | 3 + sdks/python/test-suites/tox/py38/build.gradle | 18 +- sdks/python/test-suites/tox/pycommon/build.gradle | 4 +- sdks/python/tox.ini| 8 + .../content/en/documentation/runners/direct.md | 5 + 50 files changed, 1002 insertions(+), 51 deletions(-) create mode 100644 .github/trigger_files/beam_PostCommit_Java_PVR_Flink_Batch.json create mode 100644 .github/trigger_files/beam_PostCommit_Java_PVR_Flink_Docker.json create mode 100644 .github/trigger_files/beam_PostCommit_Java_PVR_Flink_Streaming.json create mode 100644 .github/trigger_files/beam_PostCommit_Java_PVR_Samza.json create mode 100644 .github/trigger_files/beam_PostCommit_Java_PVR_Spark3_Batch.json create mode 100644 .github/trigger_files/beam_PostCommit_Java_ValidatesRunner_Dataflow_JavaVersions.json create mode 100644 .github/trigger_files/beam_PostCommit_Java_ValidatesRunner_Direct.json create mode 100644 .github/trigger_files/beam_PostCommit_Java_ValidatesRunner_Direct_JavaVersions.json create mode
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 7b81b69192b Updating config from bot
7b81b69192b is described below
commit 7b81b69192b055bab9937cfb16122fdfdabe
Author: github-actions
AuthorDate: Tue May 14 01:29:36 2024 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-31284.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-31284.json
b/scripts/ci/pr-bot/state/pr-state/pr-31284.json
new file mode 100644
index 000..242a48d7d3b
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-31284.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": true,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new b5f677877e1 Updating config from bot
b5f677877e1 is described below
commit b5f677877e17848bfdc6d7b5dbc910ce1bef5f11
Author: github-actions
AuthorDate: Tue May 14 00:49:29 2024 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-31283.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-31283.json
b/scripts/ci/pr-bot/state/pr-state/pr-31283.json
new file mode 100644
index 000..242a48d7d3b
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-31283.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": true,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
(beam) branch asf-site updated: Publishing website 2024/05/13 23:37:40 at commit c88a863
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new b11d99491b8 Publishing website 2024/05/13 23:37:40 at commit c88a863
b11d99491b8 is described below
commit b11d99491b8b3e12930b453caaf8fced84e9e855
Author: runner
AuthorDate: Mon May 13 23:37:40 2024 +
Publishing website 2024/05/13 23:37:40 at commit c88a863
---
.../documentation/patterns/side-inputs/index.html | 14 +-
website/generated-content/sitemap.xml | 2 +-
2 files changed, 10 insertions(+), 6 deletions(-)
diff --git
a/website/generated-content/documentation/patterns/side-inputs/index.html
b/website/generated-content/documentation/patterns/side-inputs/index.html
index e75b748e94a..9c5002d5466 100644
--- a/website/generated-content/documentation/patterns/side-inputs/index.html
+++ b/website/generated-content/documentation/patterns/side-inputs/index.html
@@ -40,8 +40,10 @@
// Run in debug mode to see the output.
Pipeline p =
Pipeline.create();
-// Create a
side input that updates each second.
-
PCollectionViewMapString, String map =
+// Create a
side input that updates every 5 seconds.
+
// View as an iterable, not singleton, so that if we happen to
trigger more
+
// than once before Latest.globally is computed we can handle
both elements.
+
PCollectionViewIterableMapString, String mapIterable =
p.apply(GenerateSequence.from(0).withRate(1, Duration.standardSeconds.apply(
ParDo.of(
@@ -62,7 +64,7 @@
.triggering(Repeatedly.forever(AfterProcessingTime.pastFirstElementInPane()))
.discardingFiredPanes())
.apply(Latest.globally())
-.apply(View.asSingleton());
+.apply(View.asIterable());
// Consume
side input. GenerateSequence generates test data.
// Use a real source (like PubSubIO or KafkaIO) in production.
@@ -75,7 +77,9 @@
@ProcessElement
public void process(ProcessContext c,
@Timestamp Instant timestamp) {
-MapString, String keyMap = c.sideInput(map);
+IterableMapString, String si
= c.sideInput(mapIterable);
+// Take an element from the side input iterable (likely length 1)
+
MapString, String keyMap = si.iterator().next();
c.outputWithTimestamp(KV.of(1L, c.element()), Instant.now
LOG.info(
@@ -85,7 +89,7 @@
keyMap.get(Key_A));
}
})
-.withSideInputs(map));
+.withSideInputs(mapIterable));
p.run();
}
diff --git a/website/generated-content/sitemap.xml
b/website/generated-content/sitemap.xml
index 9e2b27e0e5f..81cd04a2387 100644
--- a/website/generated-content/sitemap.xml
+++ b/website/generated-content/sitemap.xml
@@ -1 +1 @@
-http://www.sitemaps.org/schemas/sitemap/0.9;
xmlns:xhtml="http://www.w3.org/1999/xhtml;>/blog/beam-2.56.0/2024-05-13T12:42:28-04:00/categories/blog/2024-05-13T12:42:28-04:00/blog/2024-05-13T12:42:28-04:00/categories/2024-05-13T12:42:28-04:00/catego
[...]
\ No newline at end of file
+http://www.sitemaps.org/schemas/sitemap/0.9;
xmlns:xhtml="http://www.w3.org/1999/xhtml;>/blog/beam-2.56.0/2024-05-13T15:36:46-07:00/categories/blog/2024-05-13T15:36:46-07:00/blog/2024-05-13T15:36:46-07:00/categories/2024-05-13T15:36:46-07:00/catego
[...]
\ No newline at end of file
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 825c591e681 Updating config from bot
825c591e681 is described below
commit 825c591e681cf03bb894f88eabb18f711b38775b
Author: github-actions
AuthorDate: Mon May 13 23:35:07 2024 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-31281.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-31281.json
b/scripts/ci/pr-bot/state/pr-state/pr-31281.json
new file mode 100644
index 000..242a48d7d3b
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-31281.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": true,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
(beam) branch master updated (296c13a291d -> c88a863870c)
This is an automated email from the ASF dual-hosted git repository.
tvalentyn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
from 296c13a291d Avoid side input issue where Latest.globally produces 2
elements in a window (#31263)
add c88a863870c Add a subsuite of unit tests for ML package and remove
them from 'coverage' GHA workflow. (#31255)
No new revisions were added by this update.
Summary of changes:
.github/workflows/README.md | 1 +
.github/workflows/beam_PreCommit_Python.yml | 3 +--
.../workflows/beam_PreCommit_Python_Dataframes.yml| 3 +--
.github/workflows/beam_PreCommit_Python_Examples.yml | 3 +--
...ython_Runners.yml => beam_PreCommit_Python_ML.yml} | 19 +--
.github/workflows/beam_PreCommit_Python_Runners.yml | 3 +--
.../workflows/beam_PreCommit_Python_Transforms.yml| 3 +--
.../apache_beam/ml/inference/huggingface_inference.py | 2 +-
.../ml/inference/huggingface_inference_test.py| 2 +-
.../apache_beam/ml/inference/onnx_inference_test.py | 4
sdks/python/setup.py | 19 +++
sdks/python/test-suites/tox/common.gradle | 3 +++
sdks/python/test-suites/tox/py38/build.gradle | 18 ++
sdks/python/test-suites/tox/pycommon/build.gradle | 4 ++--
sdks/python/tox.ini | 8
15 files changed, 63 insertions(+), 32 deletions(-)
copy .github/workflows/{beam_PreCommit_Python_Runners.yml =>
beam_PreCommit_Python_ML.yml} (88%)
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new 223888e9601 Updating config from bot 223888e9601 is described below commit 223888e9601b78d704b5cb9f460f2c99dcb988a5 Author: github-actions AuthorDate: Mon May 13 22:34:35 2024 + Updating config from bot --- scripts/ci/pr-bot/state/reviewers-for-label-python.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-python.json b/scripts/ci/pr-bot/state/reviewers-for-label-python.json index 963f1e1309b..7ff268976aa 100644 --- a/scripts/ci/pr-bot/state/reviewers-for-label-python.json +++ b/scripts/ci/pr-bot/state/reviewers-for-label-python.json @@ -8,7 +8,7 @@ "tvalentyn": 1715367945090, "pabloem": 1681281324703, "y1chi": 1667002607045, -"damccorm": 1715302155876, +"damccorm": 1715639673063, "jrmccluskey": 1715362487046, "riteshghorse": 1715353551812, "liferoad": 1715304582820,
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 1aa803fcc6c Updating config from bot
1aa803fcc6c is described below
commit 1aa803fcc6c675ea6b5929534afebe3b0507e373
Author: github-actions
AuthorDate: Mon May 13 22:34:34 2024 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-31255.json | 9 ++---
1 file changed, 6 insertions(+), 3 deletions(-)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-31255.json
b/scripts/ci/pr-bot/state/pr-state/pr-31255.json
index 242a48d7d3b..9d3b2ef7b45 100644
--- a/scripts/ci/pr-bot/state/pr-state/pr-31255.json
+++ b/scripts/ci/pr-bot/state/pr-state/pr-31255.json
@@ -1,7 +1,10 @@
{
- "commentedAboutFailingChecks": true,
- "reviewersAssignedForLabels": {},
- "nextAction": "Author",
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {
+"python": "damccorm",
+"build": "damccorm"
+ },
+ "nextAction": "Reviewers",
"stopReviewerNotifications": false,
"remindAfterTestsPass": [],
"committerAssigned": false
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new bd5ed02b1ab Updating config from bot
bd5ed02b1ab is described below
commit bd5ed02b1ab9195c1bdf1dd9d18112a028bbb42f
Author: github-actions
AuthorDate: Mon May 13 22:34:37 2024 +
Updating config from bot
---
scripts/ci/pr-bot/state/reviewers-for-label-build.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-build.json
b/scripts/ci/pr-bot/state/reviewers-for-label-build.json
index e89ba3fe0eb..5a9822d9cc0 100644
--- a/scripts/ci/pr-bot/state/reviewers-for-label-build.json
+++ b/scripts/ci/pr-bot/state/reviewers-for-label-build.json
@@ -1,7 +1,7 @@
{
"label": "build",
"dateOfLastReviewAssignment": {
-"damccorm": 1715128207068,
+"damccorm": 1715639673063,
"Abacn": 1715308778682
}
}
\ No newline at end of file
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 92898ce44f2 Updating config from bot
92898ce44f2 is described below
commit 92898ce44f26396f6ac3dc359fbf9632d9e34456
Author: github-actions
AuthorDate: Mon May 13 21:21:29 2024 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-31279.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-31279.json
b/scripts/ci/pr-bot/state/pr-state/pr-31279.json
new file mode 100644
index 000..9c2aa5aa212
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-31279.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": true,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new c5e9d830379 Updating config from bot
c5e9d830379 is described below
commit c5e9d830379ad19b407367311ea2ae5020aacd05
Author: github-actions
AuthorDate: Mon May 13 21:05:34 2024 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-31274.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-31274.json
b/scripts/ci/pr-bot/state/pr-state/pr-31274.json
new file mode 100644
index 000..242a48d7d3b
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-31274.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": true,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
(beam) branch debugpostrelease updated (119368875d2 -> 0542c83fc1a)
This is an automated email from the ASF dual-hosted git repository. yhu pushed a change to branch debugpostrelease in repository https://gitbox.apache.org/repos/asf/beam.git omit 119368875d2 Un-revert typo fix add 0542c83fc1a Un-revert doc fixes This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (119368875d2) \ N -- N -- N refs/heads/debugpostrelease (0542c83fc1a) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: settings.gradle.kts | 12 1 file changed, 12 insertions(+)
(beam) branch debugpostrelease updated (c38857fe612 -> 119368875d2)
This is an automated email from the ASF dual-hosted git repository. yhu pushed a change to branch debugpostrelease in repository https://gitbox.apache.org/repos/asf/beam.git discard c38857fe612 Revert "[#30789] Add support for Flink 1.18 (#31062)" discard 22d40b0c7ad Revert "[flink] Add documentation for Flink runner versions updates" discard c8d13c03b73 Run validation on Flink 1.17 add 459c87af706 Revert "[flink] Add documentation for Flink runner versions updates" add 557b5bad1b0 Revert "[#30789] Add support for Flink 1.18 (#31062)" add 119368875d2 Un-revert typo fix This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (c38857fe612) \ N -- N -- N refs/heads/debugpostrelease (119368875d2) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: release/build.gradle.kts | 9 - website/www/site/content/en/documentation/runners/flink.md | 4 ++-- 2 files changed, 10 insertions(+), 3 deletions(-)
(beam) branch debugpostrelease updated (c8d13c03b73 -> c38857fe612)
This is an automated email from the ASF dual-hosted git repository. yhu pushed a change to branch debugpostrelease in repository https://gitbox.apache.org/repos/asf/beam.git from c8d13c03b73 Run validation on Flink 1.17 add 22d40b0c7ad Revert "[flink] Add documentation for Flink runner versions updates" add c38857fe612 Revert "[#30789] Add support for Flink 1.18 (#31062)" No new revisions were added by this update. Summary of changes: .../test-properties.json | 2 +- .../workflows/beam_PostCommit_Java_Tpcds_Flink.yml | 2 +- .../beam_PostCommit_Java_ValidatesRunner_Flink.yml | 2 +- CHANGES.md | 1 - gradle.properties | 2 +- runners/flink/1.18/build.gradle| 25 - .../flink/1.18/job-server-container/build.gradle | 26 -- runners/flink/1.18/job-server/build.gradle | 31 -- runners/flink/flink_runner.gradle | 2 +- sdks/go/examples/stringsplit/stringsplit.go| 2 +- sdks/go/examples/wasm/README.md| 2 +- .../python/apache_beam/options/pipeline_options.py | 2 +- sdks/typescript/src/apache_beam/runners/flink.ts | 2 +- settings.gradle.kts| 12 - .../site/content/en/documentation/runners/flink.md | 8 +- 15 files changed, 10 insertions(+), 111 deletions(-) delete mode 100644 runners/flink/1.18/build.gradle delete mode 100644 runners/flink/1.18/job-server-container/build.gradle delete mode 100644 runners/flink/1.18/job-server/build.gradle
(beam) branch master updated: Avoid side input issue where Latest.globally produces 2 elements in a window (#31263)
This is an automated email from the ASF dual-hosted git repository.
damccorm pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 296c13a291d Avoid side input issue where Latest.globally produces 2
elements in a window (#31263)
296c13a291d is described below
commit 296c13a291dd9ce9638c69d0e7d4874968c438fd
Author: Danny McCormick
AuthorDate: Mon May 13 13:51:28 2024 -0500
Avoid side input issue where Latest.globally produces 2 elements in a
window (#31263)
* Avoid side input issue where Latest.globally produces 2 elements in a
window
* Correct var
* Fix compile
---
.../java/org/apache/beam/examples/snippets/Snippets.java | 14 +-
1 file changed, 9 insertions(+), 5 deletions(-)
diff --git
a/examples/java/src/main/java/org/apache/beam/examples/snippets/Snippets.java
b/examples/java/src/main/java/org/apache/beam/examples/snippets/Snippets.java
index 3ca431455a0..4f24c69f74b 100644
---
a/examples/java/src/main/java/org/apache/beam/examples/snippets/Snippets.java
+++
b/examples/java/src/main/java/org/apache/beam/examples/snippets/Snippets.java
@@ -606,8 +606,10 @@ public class Snippets {
// Run in debug mode to see the output.
Pipeline p = Pipeline.create();
-// Create a side input that updates each second.
-PCollectionView> map =
+// Create a side input that updates every 5 seconds.
+// View as an iterable, not singleton, so that if we happen to trigger more
+// than once before Latest.globally is computed we can handle both
elements.
+PCollectionView>> mapIterable =
p.apply(GenerateSequence.from(0).withRate(1,
Duration.standardSeconds(5L)))
.apply(
ParDo.of(
@@ -628,7 +630,7 @@ public class Snippets {
.triggering(Repeatedly.forever(AfterProcessingTime.pastFirstElementInPane()))
.discardingFiredPanes())
.apply(Latest.globally())
-.apply(View.asSingleton());
+.apply(View.asIterable());
// Consume side input. GenerateSequence generates test data.
// Use a real source (like PubSubIO or KafkaIO) in production.
@@ -641,7 +643,9 @@ public class Snippets {
@ProcessElement
public void process(ProcessContext c, @Timestamp Instant
timestamp) {
-Map keyMap = c.sideInput(map);
+Iterable> si =
c.sideInput(mapIterable);
+// Take an element from the side input iterable
(likely length 1)
+Map keyMap = si.iterator().next();
c.outputWithTimestamp(KV.of(1L, c.element()),
Instant.now());
LOG.info(
@@ -651,7 +655,7 @@ public class Snippets {
keyMap.get("Key_A"));
}
})
-.withSideInputs(map));
+.withSideInputs(mapIterable));
p.run();
}
(beam) 01/01: Run validation on Flink 1.17
This is an automated email from the ASF dual-hosted git repository.
yhu pushed a commit to branch debugpostrelease
in repository https://gitbox.apache.org/repos/asf/beam.git
commit c8d13c03b7378723c4951ad6901b672dbb74bde3
Author: Yi Hu
AuthorDate: Mon May 13 14:32:09 2024 -0400
Run validation on Flink 1.17
---
release/build.gradle.kts | 9 +
1 file changed, 1 insertion(+), 8 deletions(-)
diff --git a/release/build.gradle.kts b/release/build.gradle.kts
index ca9982806b9..a14a79689b6 100644
--- a/release/build.gradle.kts
+++ b/release/build.gradle.kts
@@ -36,12 +36,5 @@ dependencies {
task("runJavaExamplesValidationTask") {
group = "Verification"
description = "Run the Beam quickstart across all Java runners"
- dependsOn(":runners:direct-java:runQuickstartJavaDirect")
- dependsOn(":runners:google-cloud-dataflow-java:runQuickstartJavaDataflow")
- dependsOn(":runners:spark:3:runQuickstartJavaSpark")
- dependsOn(":runners:flink:1.18:runQuickstartJavaFlinkLocal")
- dependsOn(":runners:direct-java:runMobileGamingJavaDirect")
- dependsOn(":runners:google-cloud-dataflow-java:runMobileGamingJavaDataflow")
- dependsOn(":runners:twister2:runQuickstartJavaTwister2")
-
dependsOn(":runners:google-cloud-dataflow-java:runMobileGamingJavaDataflowBom")
+ dependsOn(":runners:flink:1.17:runQuickstartJavaFlinkLocal")
}
(beam) branch debugpostrelease created (now c8d13c03b73)
This is an automated email from the ASF dual-hosted git repository. yhu pushed a change to branch debugpostrelease in repository https://gitbox.apache.org/repos/asf/beam.git at c8d13c03b73 Run validation on Flink 1.17 This branch includes the following new commits: new c8d13c03b73 Run validation on Flink 1.17 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
(beam) branch master updated: Implement the hash_words TFT operation (#31249)
This is an automated email from the ASF dual-hosted git repository.
jrmccluskey pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new c230655a1e5 Implement the hash_words TFT operation (#31249)
c230655a1e5 is described below
commit c230655a1e599e25a69b77d8a85fad45e5fc7587
Author: Jack McCluskey <34928439+jrmcclus...@users.noreply.github.com>
AuthorDate: Mon May 13 14:22:25 2024 -0400
Implement the hash_words TFT operation (#31249)
* Implement the hash-words TFT operation
* linting
* Update sdks/python/apache_beam/ml/transforms/tft.py
Co-authored-by: tvalentyn
* tighten key type hint
* formatting
-
Co-authored-by: tvalentyn
---
sdks/python/apache_beam/ml/transforms/tft.py | 44 +
sdks/python/apache_beam/ml/transforms/tft_test.py | 77 +++
2 files changed, 121 insertions(+)
diff --git a/sdks/python/apache_beam/ml/transforms/tft.py
b/sdks/python/apache_beam/ml/transforms/tft.py
index 9b02cf8b75c..550dbedbc7b 100644
--- a/sdks/python/apache_beam/ml/transforms/tft.py
+++ b/sdks/python/apache_beam/ml/transforms/tft.py
@@ -637,3 +637,47 @@ class BagOfWords(TFTOperation):
def count_unique_words(
data: tf.SparseTensor, output_vocab_name: Optional[str]) -> None:
tft.count_per_key(data, key_vocabulary_filename=output_vocab_name)
+
+
+@register_input_dtype(str)
+class HashStrings(TFTOperation):
+ def __init__(
+ self,
+ columns: List[str],
+ hash_buckets: int,
+ key: Optional[Tuple[int, int]] = None,
+ name: Optional[str] = None):
+'''Hashes strings into the provided number of buckets.
+
+Args:
+ columns: A list of the column names to apply the transformation on.
+ hash_buckets: the number of buckets to hash the strings into.
+ key: optional. An array of two Python `uint64`. If passed, output will be
+a deterministic function of `strings` and `key`. Note that hashing will
+be slower if this value is specified.
+ name: optional. A name for this operation.
+
+Raises:
+ ValueError if `hash_buckets` is not a positive and non-zero integer.
+'''
+self.hash_buckets = hash_buckets
+self.key = key
+self.name = name
+
+if hash_buckets < 1:
+ raise ValueError(
+ 'number of hash buckets must be positive, got ', hash_buckets)
+
+super().__init__(columns)
+
+ def apply_transform(
+ self, data: common_types.TensorType,
+ output_col_name: str) -> Dict[str, common_types.TensorType]:
+output_dict = {
+output_col_name: tft.hash_strings(
+strings=data,
+hash_buckets=self.hash_buckets,
+key=self.key,
+name=self.name)
+}
+return output_dict
diff --git a/sdks/python/apache_beam/ml/transforms/tft_test.py
b/sdks/python/apache_beam/ml/transforms/tft_test.py
index ed7c301f3b8..6763032a8eb 100644
--- a/sdks/python/apache_beam/ml/transforms/tft_test.py
+++ b/sdks/python/apache_beam/ml/transforms/tft_test.py
@@ -932,5 +932,82 @@ class BagOfWordsTest(unittest.TestCase):
self.assertEqual(expected_data, actual_data)
+class HashWordsTest(unittest.TestCase):
+ def setUp(self) -> None:
+self.artifact_location = tempfile.mkdtemp()
+
+ def tearDown(self):
+shutil.rmtree(self.artifact_location)
+
+ def test_single_bucket(self):
+list_data = [{'x': 'this is a test string'}]
+expected_values = [np.array([0])]
+with beam.Pipeline() as p:
+ list_result = (
+ p
+ | "listCreate" >> beam.Create(list_data)
+ | "listMLTransform" >> base.MLTransform(
+ write_artifact_location=self.artifact_location).with_transform(
+ tft.HashStrings(columns=['x'], hash_buckets=1)))
+ result = (list_result | beam.Map(lambda x: x.x))
+ assert_that(result, equal_to(expected_values, equals_fn=np.array_equal))
+
+ def test_multi_bucket_one_string(self):
+list_data = [{'x': 'this is a test string'}]
+expected_values = [np.array([1])]
+with beam.Pipeline() as p:
+ list_result = (
+ p
+ | "listCreate" >> beam.Create(list_data)
+ | "listMLTransform" >> base.MLTransform(
+ write_artifact_location=self.artifact_location).with_transform(
+ tft.HashStrings(columns=['x'], hash_buckets=2)))
+ result = (list_result | beam.Map(lambda x: x.x))
+ assert_that(result, equal_to(expected_values, equals_fn=np.array_equal))
+
+ def test_one_bucket_multi_string(self):
+list_data = [{'x': ['these', 'are', 'test', 'strings']}]
+expected_values = [np.array([0, 0, 0, 0])]
+with beam.Pipeline() as p:
+ list_result = (
+ p
+ | "listCreate" >> beam.Create(list_data)
+ | "listMLTransform" >> base.MLTransform(
+
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new b92a96b3c89 Updating config from bot
b92a96b3c89 is described below
commit b92a96b3c895325105d0301f215b69ba06da561f
Author: github-actions
AuthorDate: Mon May 13 18:19:33 2024 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-31273.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-31273.json
b/scripts/ci/pr-bot/state/pr-state/pr-31273.json
new file mode 100644
index 000..9c2aa5aa212
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-31273.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": true,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
(beam) branch users/damccorm/siFix updated (c35f4710bc7 -> 1ed9acbc68e)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch users/damccorm/siFix in repository https://gitbox.apache.org/repos/asf/beam.git from c35f4710bc7 Correct var add 1ed9acbc68e Fix compile No new revisions were added by this update. Summary of changes: .../src/main/java/org/apache/beam/examples/snippets/Snippets.java | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 321876a847d Updating config from bot
321876a847d is described below
commit 321876a847d84f51d19d2eec1a4f447a39d55abe
Author: github-actions
AuthorDate: Mon May 13 16:45:07 2024 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-31268.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-31268.json
b/scripts/ci/pr-bot/state/pr-state/pr-31268.json
new file mode 100644
index 000..9c2aa5aa212
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-31268.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": true,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
(beam) branch master updated (466516b016c -> 749c128bef1)
This is an automated email from the ASF dual-hosted git repository. yhu pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 466516b016c [Python] Use BufferedLogger instead of logger to log in boot.go (#31265) add 749c128bef1 Disclaim direct runner result can be different with remote runners (#31250) No new revisions were added by this update. Summary of changes: website/www/site/content/en/documentation/runners/direct.md | 5 + 1 file changed, 5 insertions(+)
(beam) branch master updated (737be230347 -> 466516b016c)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 737be230347 Merge pull request #31156: Reapply "Add Redistribute transform to Java SDK" add 466516b016c [Python] Use BufferedLogger instead of logger to log in boot.go (#31265) No new revisions were added by this update. Summary of changes: sdks/python/container/boot.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new b37a465a4c2 Updating config from bot
b37a465a4c2 is described below
commit b37a465a4c2fa5cf0869886d51924e8d0c84e0e9
Author: github-actions
AuthorDate: Mon May 13 16:06:51 2024 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-31255.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-31255.json
b/scripts/ci/pr-bot/state/pr-state/pr-31255.json
new file mode 100644
index 000..242a48d7d3b
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-31255.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": true,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new b89db08728e Updating config from bot
b89db08728e is described below
commit b89db08728e11671a6797d36f210b3185d2ce4de
Author: github-actions
AuthorDate: Mon May 13 15:47:06 2024 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-31267.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-31267.json
b/scripts/ci/pr-bot/state/pr-state/pr-31267.json
new file mode 100644
index 000..9c2aa5aa212
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-31267.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": true,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
(beam) branch tvalentyn-patch-6 created (now 9d655d9583c)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch tvalentyn-patch-6 in repository https://gitbox.apache.org/repos/asf/beam.git at 9d655d9583c Fix a typo in skip condition. This branch includes the following new commits: new 9d655d9583c Fix a typo in skip condition. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
(beam) 01/01: Fix a typo in skip condition.
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a commit to branch tvalentyn-patch-6 in repository https://gitbox.apache.org/repos/asf/beam.git commit 9d655d9583ccb948a219164e9760c74691425055 Author: tvalentyn AuthorDate: Mon May 13 08:44:59 2024 -0700 Fix a typo in skip condition. --- sdks/python/apache_beam/dataframe/frames_test.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sdks/python/apache_beam/dataframe/frames_test.py b/sdks/python/apache_beam/dataframe/frames_test.py index b3ddfee95fc..076ab504add 100644 --- a/sdks/python/apache_beam/dataframe/frames_test.py +++ b/sdks/python/apache_beam/dataframe/frames_test.py @@ -1449,7 +1449,7 @@ class DeferredFrameTest(_AbstractFrameTest): self._run_test(lambda s: s.unstack(level=0), s) @unittest.skipIf( - sys.version_info >= (2, 12) and PD_VERSION < (2, 3), + sys.version_info >= (3, 12) and PD_VERSION < (2, 3), 'https://github.com/pandas-dev/pandas/issues/58604') def test_unstack_pandas_example3(self): index = self._unstack_get_categorical_index()
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new d13710c0d9f Updating config from bot
d13710c0d9f is described below
commit d13710c0d9f59ed302a9328e113ce0c6400f8985
Author: github-actions
AuthorDate: Mon May 13 15:43:30 2024 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-31266.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-31266.json
b/scripts/ci/pr-bot/state/pr-state/pr-31266.json
new file mode 100644
index 000..9c2aa5aa212
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-31266.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": true,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new 2f90aadc11c Updating config from bot 2f90aadc11c is described below commit 2f90aadc11c5e8e7ee42a5f4082bab4fe2538fde Author: github-actions AuthorDate: Mon May 13 15:34:42 2024 + Updating config from bot --- scripts/ci/pr-bot/state/reviewers-for-label-io.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-io.json b/scripts/ci/pr-bot/state/reviewers-for-label-io.json index 4b949c6afd7..6581676f552 100644 --- a/scripts/ci/pr-bot/state/reviewers-for-label-io.json +++ b/scripts/ci/pr-bot/state/reviewers-for-label-io.json @@ -9,6 +9,6 @@ "bvolpato": 1712595969392, "manavgarg": 1690826779210, "damondouglas": 1715362487046, -"shunping": 1714599265290 +"shunping": 1715614479475 } } \ No newline at end of file
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new d09e79458c7 Updating config from bot d09e79458c7 is described below commit d09e79458c74376c202254e9ba57d75c5d65ed96 Author: github-actions AuthorDate: Mon May 13 15:34:41 2024 + Updating config from bot --- scripts/ci/pr-bot/state/reviewers-for-label-java.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-java.json b/scripts/ci/pr-bot/state/reviewers-for-label-java.json index e7eaf929b4b..341b561c206 100644 --- a/scripts/ci/pr-bot/state/reviewers-for-label-java.json +++ b/scripts/ci/pr-bot/state/reviewers-for-label-java.json @@ -9,6 +9,6 @@ "Abacn": 1715516043156, "bvolpato": 1712595969392, "m-trieu": 1715158075817, -"damondouglas": 1715084034225 +"damondouglas": 1715614479475 } } \ No newline at end of file
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 1a36e77309a Updating config from bot
1a36e77309a is described below
commit 1a36e77309a5065dc3b25513c0ea938640897a9b
Author: github-actions
AuthorDate: Mon May 13 15:34:40 2024 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-31264.json | 11 +++
1 file changed, 11 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-31264.json
b/scripts/ci/pr-bot/state/pr-state/pr-31264.json
new file mode 100644
index 000..1df7da2a8c7
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-31264.json
@@ -0,0 +1,11 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {
+"java": "damondouglas",
+"io": "shunping"
+ },
+ "nextAction": "Reviewers",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 2c4cc991573 Updating config from bot
2c4cc991573 is described below
commit 2c4cc991573893a2cbd09a87963c6f5066a69ad9
Author: github-actions
AuthorDate: Mon May 13 15:06:15 2024 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-31265.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-31265.json
b/scripts/ci/pr-bot/state/pr-state/pr-31265.json
new file mode 100644
index 000..242a48d7d3b
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-31265.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": true,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
(beam) branch master updated (66da760357c -> 737be230347)
This is an automated email from the ASF dual-hosted git repository.
kenn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
from 66da760357c Update yaml to non-beta link (#31210)
add a0c08272d42 Touch trigger files for all relevant ValidatesRunner and
PVR suites
add bf347fe898b Sickbay extremely flaky Flink ParDoLifecycleTest
add ef431ec90f2 Reapply "Add Redistribute transform to Java SDK"
add 737be230347 Merge pull request #31156: Reapply "Add Redistribute
transform to Java SDK"
No new revisions were added by this update.
Summary of changes:
.../beam_PostCommit_Java_PVR_Flink_Batch.json | 4 +
.../beam_PostCommit_Java_PVR_Flink_Docker.json | 4 +
.../beam_PostCommit_Java_PVR_Flink_Streaming.json | 4 +
.../beam_PostCommit_Java_PVR_Samza.json| 4 +
.../beam_PostCommit_Java_PVR_Spark3_Batch.json | 4 +
.../beam_PostCommit_Java_PVR_Spark3_Streaming.json | 3 +-
...m_PostCommit_Java_ValidatesRunner_Dataflow.json | 3 +-
...Java_ValidatesRunner_Dataflow_JavaVersions.json | 4 +
...it_Java_ValidatesRunner_Dataflow_Streaming.json | 3 +-
...ostCommit_Java_ValidatesRunner_Dataflow_V2.json | 3 +-
...Java_ValidatesRunner_Dataflow_V2_Streaming.json | 3 +-
...eam_PostCommit_Java_ValidatesRunner_Direct.json | 4 +
...t_Java_ValidatesRunner_Direct_JavaVersions.json | 4 +
...beam_PostCommit_Java_ValidatesRunner_Flink.json | 3 +-
...stCommit_Java_ValidatesRunner_Flink_Java11.json | 3 +-
...beam_PostCommit_Java_ValidatesRunner_Samza.json | 3 +-
...beam_PostCommit_Java_ValidatesRunner_Spark.json | 3 +-
...a_ValidatesRunner_SparkStructuredStreaming.json | 3 +-
...stCommit_Java_ValidatesRunner_Spark_Java11.json | 4 +
...m_PostCommit_Java_ValidatesRunner_Twister2.json | 3 +-
.../beam_PostCommit_Java_ValidatesRunner_ULR.json | 3 +-
runners/flink/flink_runner.gradle | 3 +
runners/flink/job-server/flink_job_server.gradle | 2 +
runners/portability/java/build.gradle | 2 +
runners/samza/job-server/build.gradle | 2 +
runners/spark/job-server/spark_job_server.gradle | 8 +
runners/spark/spark_runner.gradle | 7 +
.../apache/beam/sdk/transforms/Redistribute.java | 300 +
.../util/construction/PTransformTranslation.java | 8 +
.../{ReshuffleTest.java => RedistributeTest.java} | 159 +--
30 files changed, 453 insertions(+), 110 deletions(-)
create mode 100644
.github/trigger_files/beam_PostCommit_Java_PVR_Flink_Batch.json
create mode 100644
.github/trigger_files/beam_PostCommit_Java_PVR_Flink_Docker.json
create mode 100644
.github/trigger_files/beam_PostCommit_Java_PVR_Flink_Streaming.json
create mode 100644 .github/trigger_files/beam_PostCommit_Java_PVR_Samza.json
create mode 100644
.github/trigger_files/beam_PostCommit_Java_PVR_Spark3_Batch.json
create mode 100644
.github/trigger_files/beam_PostCommit_Java_ValidatesRunner_Dataflow_JavaVersions.json
create mode 100644
.github/trigger_files/beam_PostCommit_Java_ValidatesRunner_Direct.json
create mode 100644
.github/trigger_files/beam_PostCommit_Java_ValidatesRunner_Direct_JavaVersions.json
create mode 100644
.github/trigger_files/beam_PostCommit_Java_ValidatesRunner_Spark_Java11.json
create mode 100644
sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Redistribute.java
copy
sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/{ReshuffleTest.java
=> RedistributeTest.java} (73%)
(beam) branch users/damccorm/siFix updated (a811d3c5207 -> c35f4710bc7)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch users/damccorm/siFix in repository https://gitbox.apache.org/repos/asf/beam.git from a811d3c5207 Avoid side input issue where Latest.globally produces 2 elements in a window add c35f4710bc7 Correct var No new revisions were added by this update. Summary of changes: .../java/src/main/java/org/apache/beam/examples/snippets/Snippets.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new b4b6d408335 Updating config from bot
b4b6d408335 is described below
commit b4b6d4083354fef974043f296a62a231f065ca29
Author: github-actions
AuthorDate: Mon May 13 13:34:57 2024 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-31263.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-31263.json
b/scripts/ci/pr-bot/state/pr-state/pr-31263.json
new file mode 100644
index 000..242a48d7d3b
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-31263.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": true,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
(beam) 01/01: Avoid side input issue where Latest.globally produces 2 elements in a window
This is an automated email from the ASF dual-hosted git repository.
damccorm pushed a commit to branch users/damccorm/siFix
in repository https://gitbox.apache.org/repos/asf/beam.git
commit a811d3c5207a33d906ae482d13667c9e76779177
Author: Danny McCormick
AuthorDate: Mon May 13 09:12:59 2024 -0400
Avoid side input issue where Latest.globally produces 2 elements in a window
---
.../main/java/org/apache/beam/examples/snippets/Snippets.java | 10 ++
1 file changed, 6 insertions(+), 4 deletions(-)
diff --git
a/examples/java/src/main/java/org/apache/beam/examples/snippets/Snippets.java
b/examples/java/src/main/java/org/apache/beam/examples/snippets/Snippets.java
index 3ca431455a0..cf9bb1b23f4 100644
---
a/examples/java/src/main/java/org/apache/beam/examples/snippets/Snippets.java
+++
b/examples/java/src/main/java/org/apache/beam/examples/snippets/Snippets.java
@@ -606,8 +606,10 @@ public class Snippets {
// Run in debug mode to see the output.
Pipeline p = Pipeline.create();
-// Create a side input that updates each second.
-PCollectionView> map =
+// Create a side input that updates every 5 seconds.
+// View as an iterable, not singleton, so that if we happen to trigger more
+// than once before Latest.globally is computed we can handle both
elements.
+PCollectionView>> mapIterable =
p.apply(GenerateSequence.from(0).withRate(1,
Duration.standardSeconds(5L)))
.apply(
ParDo.of(
@@ -628,7 +630,7 @@ public class Snippets {
.triggering(Repeatedly.forever(AfterProcessingTime.pastFirstElementInPane()))
.discardingFiredPanes())
.apply(Latest.globally())
-.apply(View.asSingleton());
+.apply(View.asIterable());
// Consume side input. GenerateSequence generates test data.
// Use a real source (like PubSubIO or KafkaIO) in production.
@@ -641,7 +643,7 @@ public class Snippets {
@ProcessElement
public void process(ProcessContext c, @Timestamp Instant
timestamp) {
-Map keyMap = c.sideInput(map);
+Map keyMap =
c.sideInput(mapIterable[0]);
c.outputWithTimestamp(KV.of(1L, c.element()),
Instant.now());
LOG.info(
(beam) branch users/damccorm/siFix created (now a811d3c5207)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch users/damccorm/siFix in repository https://gitbox.apache.org/repos/asf/beam.git at a811d3c5207 Avoid side input issue where Latest.globally produces 2 elements in a window This branch includes the following new commits: new a811d3c5207 Avoid side input issue where Latest.globally produces 2 elements in a window The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
(beam) branch dependabot/go_modules/playground/backend/github.com/cloudevents/sdk-go/v2-2.15.2 deleted (was 92a3c014328)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/go_modules/playground/backend/github.com/cloudevents/sdk-go/v2-2.15.2 in repository https://gitbox.apache.org/repos/asf/beam.git was 92a3c014328 Bump github.com/cloudevents/sdk-go/v2 in /playground/backend The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new f25cfadf20f Updating config from bot
f25cfadf20f is described below
commit f25cfadf20fb2fcd7b593f60a89f1e06241965a1
Author: github-actions
AuthorDate: Mon May 13 07:34:34 2024 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-31262.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-31262.json
b/scripts/ci/pr-bot/state/pr-state/pr-31262.json
new file mode 100644
index 000..242a48d7d3b
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-31262.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": true,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
