[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new ae0f64bfcdf Updating config from bot
ae0f64bfcdf is described below
commit ae0f64bfcdf8b494bc80404a37589471adfe4559
Author: github-actions
AuthorDate: Thu Oct 12 05:35:14 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/reviewers-for-label-build.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-build.json
b/scripts/ci/pr-bot/state/reviewers-for-label-build.json
index 6d7bb22369b..635e98e77a7 100644
--- a/scripts/ci/pr-bot/state/reviewers-for-label-build.json
+++ b/scripts/ci/pr-bot/state/reviewers-for-label-build.json
@@ -2,6 +2,6 @@
"label": "build",
"dateOfLastReviewAssignment": {
"damccorm": 1697071765738,
-"Abacn": 1697042173792
+"Abacn": 1697088910723
}
}
\ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new c603eb8293e Updating config from bot
c603eb8293e is described below
commit c603eb8293e54e7d82f288b3cd9ce440bf593c52
Author: github-actions
AuthorDate: Thu Oct 12 05:35:12 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-28894.json | 10 ++
1 file changed, 10 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-28894.json
b/scripts/ci/pr-bot/state/pr-state/pr-28894.json
new file mode 100644
index 000..9c51466b9ee
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-28894.json
@@ -0,0 +1,10 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {
+"build": "Abacn"
+ },
+ "nextAction": "Reviewers",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch nightly-refs/heads/master updated (65eaf45026e -> 9c75db4760c)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch nightly-refs/heads/master in repository https://gitbox.apache.org/repos/asf/beam.git from 65eaf45026e fix lint (#28917) add 99c87a2c68b Updates the Transform Service to accept Python extra packages through the Java API (#28783) add 88139bdc9a0 Retrieve workflow runs for GA Post-Commits Status dashboard (#28844) add e3c1591bbb8 Bump google.golang.org/grpc from 1.58.2 to 1.58.3 in /sdks (#28938) add 62efb967884 Bump golang.org/x/net from 0.16.0 to 0.17.0 in /sdks (#28939) add d437afab961 Run self-assign and pr-updates only in create event (#28928) add a54afde4a05 Fix missing backslash in arguments of job (#28942) add e56d1a1157b Enable missing-timeout pylint check (#28920) add 9cdcb036cc9 move nested windmill state classes to top level files and organize di… (#28506) add 6a456f04b68 Add user counters and msec metrics to prism UI. (#28929) add 2bd23b1 Add counter for MLTransform and data processing transforms (#28927) add c62a81b1c57 Reapply "Populate top-level display data in yaml main. (#28512)" add de48383bdca Fix typo in docstring. add b82dd4849e4 Merge pull request #28931 from robertwb/display-data add 7a1245d4011 Bump github.com/google/go-cmp from 0.5.9 to 0.6.0 in /sdks (#28940) add 2bfcb9fda37 Remove unnecessary a space character that got through Javadoc CI ;) (#28863) add aedfa461fb3 [#20970] Fix gRPC leak by closing ResidualSource at BoundedToUnboundedSourceAdapter.Reader#init() in Dataflow worker (#28548) add bd46c76dd08 Bump snappy-java version to 1.1.10.4 (#28655) add ebad3deb5a0 Add GitHub Workflow Replacements for Jenkins job_LoadTests_Python_CoGBK subgroup (#28764) add dfdbcb9006b Add GitHub Workflow Replacements for Jenkins job_LoadTests_Java_CoGBK subgroup (#28742) add 09f30537015 Add GitHub Workflow Replacements for Jenkins job_LoadTests_Python_SideInput_Dataflow_Batch (#28826) add 6bab4b57992 Add Community Metrics Prober workflow (#28842) add 104c10b3ee5 [RRIO] Create Caller and SetupTeardown interfaces (#28905) add e8e38145d96 Setup Java 21 container (#28833) add 9c75db4760c [YAML] Avro format for PubSub. (#28899) No new revisions were added by this update. Summary of changes: .../beam_LoadTests_Java_CoGBK_Dataflow_Batch.yml | 114 + ...s_Java_CoGBK_Dataflow_V2_Batch_JavaVersions.yml | 132 + ...va_CoGBK_Dataflow_V2_Streaming_JavaVersions.yml | 132 + ...s_Java_CoGBK_SparkStructuredStreaming_Batch.yml | 114 + ...LoadTests_Java_GBK_Dataflow_V2_Batch_Java11.yml | 14 +- ...LoadTests_Java_GBK_Dataflow_V2_Batch_Java17.yml | 14 +- ...Tests_Java_GBK_Dataflow_V2_Streaming_Java11.yml | 14 +- ...Tests_Java_GBK_Dataflow_V2_Streaming_Java17.yml | 14 +- .../beam_LoadTests_Python_CoGBK_Dataflow_Batch.yml | 127 + ...m_LoadTests_Python_CoGBK_Dataflow_Streaming.yml | 126 + .../beam_LoadTests_Python_CoGBK_Flink_Batch.yml| 135 + ...m_LoadTests_Python_SideInput_Dataflow_Batch.yml | 186 ++ .../beam_PostCommit_Java_Examples_Dataflow_ARM.yml |2 +- ...m_PostCommit_Java_Examples_Dataflow_V2_Java.yml |2 +- .../beam_PostCommit_Java_Jpms_Dataflow_Java17.yml |2 +- .../beam_PostCommit_Java_Jpms_Direct_Java17.yml|2 +- ...am_PostCommit_Java_Nexmark_Dataflow_V2_Java.yml |2 +- ..._Python_ValidatesContainer_Dataflow_With_RC.yml |4 +- .../beam_PostCommit_TransformService_Direct.yml|2 +- ...eam_PreCommit_Java_Examples_Dataflow_Java11.yml |2 +- ...eam_PreCommit_Java_Examples_Dataflow_Java17.yml |2 +- .github/workflows/beam_PreCommit_SQL_Java11.yml|2 +- .github/workflows/beam_PreCommit_SQL_Java17.yml|2 +- ...Java17.yml => beam_Prober_CommunityMetrics.yml} | 54 +- ...ava_CoGBK_Dataflow_Batch_100b_Multiple_Keys.txt | 27 + .../java_CoGBK_Dataflow_Batch_100b_Single_Key.txt | 27 + .../java_CoGBK_Dataflow_Batch_10kB.txt | 27 + .../java_CoGBK_Dataflow_Batch_2MB.txt | 27 + ...taflow_V2_Streaming_Java_100b_Multiple_Keys.txt | 29 + ..._Dataflow_V2_Streaming_Java_100b_Single_Key.txt | 29 + .../java_CoGBK_Dataflow_V2_Streaming_Java_10kB.txt | 29 + .../java_CoGBK_Dataflow_V2_Streaming_Java_2MB.txt | 29 + ...tructuredStreaming_Batch_100b_Multiple_Keys.txt | 25 + ...rkStructuredStreaming_Batch_100b_Single_Key.txt | 25 + ...a_CoGBK_SparkStructuredStreaming_Batch_10kB.txt | 25 + ...va_CoGBK_SparkStructuredStreaming_Batch_2MB.txt | 25 + ...hon_CoGBK_Dataflow_Batch_100b_Multiple_Keys.txt | 28 + ...python_CoGBK_Dataflow_Batch_100b_Single_Key.txt | 28 + .../python_CoGBK_Dataflow_Batch_10kB.txt | 28 + .../python_CoGBK_Dataflow_Batch_2MB.txt| 28 + ...GBK_Dataflow_Flink_Batch_100b_Multiple_Keys.txt | 28 + ..._CoGBK_Dataflow_Flink_Batch_100b_Single_Key.txt | 28 +
[beam] branch master updated: Add new resource hint to all sdks for number of cpus per worker machine (#28848)
This is an automated email from the ASF dual-hosted git repository.
yhu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 4a7c4842ace Add new resource hint to all sdks for number of cpus per
worker machine (#28848)
4a7c4842ace is described below
commit 4a7c4842aceb6786040d5cd699b7916c4509bb7b
Author: Kerry Donny-Clark
AuthorDate: Wed Oct 11 23:35:25 2023 -0400
Add new resource hint to all sdks for number of cpus per worker machine
(#28848)
* Adds new resource hint for number of cpus per worker.
* Fixes minor bugs.`
* Go fmt and removes unwanted .python-version file.
* Adds tests.
* Fixes typo.
* Fixes Java tests, adds URN to parsers.
* Addresses FindBugs issue with int parsing.
* Applies Java formatting corrections
* Adds generated go protobufs
---
.../beam/model/pipeline/v1/beam_runner_api.proto | 4 ++
.../beam/model/fnexecution_v1/beam_fn_api.pb.go| 67 --
.../model/fnexecution_v1/beam_fn_api_grpc.pb.go| 2 +-
.../model/fnexecution_v1/beam_provision_api.pb.go | 2 +-
.../fnexecution_v1/beam_provision_api_grpc.pb.go | 2 +-
.../model/jobmanagement_v1/beam_artifact_api.pb.go | 2 +-
.../jobmanagement_v1/beam_artifact_api_grpc.pb.go | 2 +-
.../jobmanagement_v1/beam_expansion_api.pb.go | 2 +-
.../jobmanagement_v1/beam_expansion_api_grpc.pb.go | 2 +-
.../beam/model/jobmanagement_v1/beam_job_api.pb.go | 2 +-
.../model/jobmanagement_v1/beam_job_api_grpc.pb.go | 2 +-
.../beam/model/pipeline_v1/beam_runner_api.pb.go | 81 --
.../model/pipeline_v1/beam_runner_api_grpc.pb.go | 2 +-
sdks/go/pkg/beam/model/pipeline_v1/endpoints.pb.go | 2 +-
.../model/pipeline_v1/external_transforms.pb.go| 2 +-
sdks/go/pkg/beam/model/pipeline_v1/metrics.pb.go | 2 +-
sdks/go/pkg/beam/model/pipeline_v1/schema.pb.go| 2 +-
.../model/pipeline_v1/standard_window_fns.pb.go| 2 +-
sdks/go/pkg/beam/options/resource/hint.go | 37 ++
sdks/go/pkg/beam/options/resource/hint_test.go | 45 ++--
.../transforms/resourcehints/ResourceHints.java| 63 +
.../resourcehints/ResourceHintsTest.java | 11 +--
sdks/python/apache_beam/transforms/resources.py| 16 +
.../apache_beam/transforms/resources_test.py | 6 ++
24 files changed, 281 insertions(+), 79 deletions(-)
diff --git
a/model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/beam_runner_api.proto
b/model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/beam_runner_api.proto
index 2483103b579..db958f183c4 100644
---
a/model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/beam_runner_api.proto
+++
b/model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/beam_runner_api.proto
@@ -1982,5 +1982,9 @@ message StandardResourceHints {
// SDKs should convert the size to bytes, but can allow users to specify
human-friendly units (e.g. GiB).
// Payload: ASCII encoded string of the base 10 representation of an
integer number of bytes.
MIN_RAM_BYTES = 1 [(beam_urn) = "beam:resources:min_ram_bytes:v1"];
+// Describes desired number of CPUs available in transform's execution
environment.
+// SDKs should accept and validate a positive integer count.
+// Payload: ASCII encoded string of the base 10 representation of an
integer number of CPUs.
+CPU_COUNT = 2 [(beam_urn) = "beam:resources:cpu_count:v1"];
}
}
diff --git a/sdks/go/pkg/beam/model/fnexecution_v1/beam_fn_api.pb.go
b/sdks/go/pkg/beam/model/fnexecution_v1/beam_fn_api.pb.go
index 1d547470ea1..9d14cff3c7d 100644
--- a/sdks/go/pkg/beam/model/fnexecution_v1/beam_fn_api.pb.go
+++ b/sdks/go/pkg/beam/model/fnexecution_v1/beam_fn_api.pb.go
@@ -27,7 +27,7 @@
// Code generated by protoc-gen-go. DO NOT EDIT.
// versions:
// protoc-gen-go v1.27.1
-// protocv4.24.0--rc1
+// protocv4.24.4
// source: org/apache/beam/model/fn_execution/v1/beam_fn_api.proto
// TODO: Consider consolidating common components in another package
@@ -1883,30 +1883,57 @@ func (x *ProcessBundleSplitRequest) GetDesiredSplits()
map[string]*ProcessBundle
// first_residual_element.
// - The current bundle, if no further splits happen, will have done exactly
// the work under primary_roots and all elements up to and including the
-// channel splits last_primary_element.
+// channel split's last_primary_element.
//
// This allows the SDK to relinquish ownership of and commit to not process
some
// of the elements that it may have been sent (the residual) while retaining
// ownership and commitment to finish the other portion (the primary).
//
-// For example, lets say the SDK is processing elements A B C D E and a split
-// request comes in. The SDK could return a response with a
[beam] 01/01: Merge pull request #28800: [Website] Announce Beam 2.51.0
This is an automated email from the ASF dual-hosted git repository. kenn pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 0ff6795e268e7deb5611000b068221b424d2b92e Merge: 6b3b3eb603f fb99630d1e0 Author: Kenneth Knowles AuthorDate: Wed Oct 11 22:46:01 2023 -0400 Merge pull request #28800: [Website] Announce Beam 2.51.0 CHANGES.md | 18 +- website/www/site/config.toml | 2 +- website/www/site/content/en/blog/beam-2.51.0.md| 210 + .../www/site/content/en/get-started/downloads.md | 14 +- 4 files changed, 226 insertions(+), 18 deletions(-)
[beam] branch master updated (6b3b3eb603f -> 0ff6795e268)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 6b3b3eb603f Update go_tests.yml to cache (#28954) add fb99630d1e0 [Website] Announce Beam 2.51.0 new 0ff6795e268 Merge pull request #28800: [Website] Announce Beam 2.51.0 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: CHANGES.md | 18 +- website/www/site/config.toml | 2 +- website/www/site/content/en/blog/beam-2.51.0.md| 210 + .../www/site/content/en/get-started/downloads.md | 14 +- 4 files changed, 226 insertions(+), 18 deletions(-) create mode 100644 website/www/site/content/en/blog/beam-2.51.0.md
[beam] branch master updated (9c75db4760c -> 6b3b3eb603f)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 9c75db4760c [YAML] Avro format for PubSub. (#28899) add 6b3b3eb603f Update go_tests.yml to cache (#28954) No new revisions were added by this update. Summary of changes: .github/workflows/go_tests.yml | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
[beam] branch lostluck-cache-go deleted (was 7b6117785ab)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a change to branch lostluck-cache-go in repository https://gitbox.apache.org/repos/asf/beam.git was 7b6117785ab Update go_tests.yml to cache The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
[beam-starter-java] branch dependabot/maven/org.apache.beam-beam-runners-direct-java-2.50.0 deleted (was 373d7fc)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/org.apache.beam-beam-runners-direct-java-2.50.0 in repository https://gitbox.apache.org/repos/asf/beam-starter-java.git was 373d7fc Bump org.apache.beam:beam-runners-direct-java from 2.46.0 to 2.50.0 The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
[beam-starter-java] branch dependabot/maven/org.apache.beam-beam-sdks-java-core-2.50.0 deleted (was 7f1cc06)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/org.apache.beam-beam-sdks-java-core-2.50.0 in repository https://gitbox.apache.org/repos/asf/beam-starter-java.git was 7f1cc06 Bump org.apache.beam:beam-sdks-java-core from 2.46.0 to 2.50.0 The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
[beam-starter-java] branch dependabot/maven/org.apache.beam-beam-sdks-java-core-2.51.0 created (now bab00ef)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/org.apache.beam-beam-sdks-java-core-2.51.0 in repository https://gitbox.apache.org/repos/asf/beam-starter-java.git at bab00ef Bump org.apache.beam:beam-sdks-java-core from 2.46.0 to 2.51.0 No new revisions were added by this update.
[beam-starter-java] branch dependabot/maven/org.apache.beam-beam-runners-direct-java-2.51.0 created (now 6b12641)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/org.apache.beam-beam-runners-direct-java-2.51.0 in repository https://gitbox.apache.org/repos/asf/beam-starter-java.git at 6b12641 Bump org.apache.beam:beam-runners-direct-java from 2.46.0 to 2.51.0 No new revisions were added by this update.
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new 999d0b56433 Updating config from bot 999d0b56433 is described below commit 999d0b564332df8b1b982c6817bc3965d2b116c9 Author: github-actions AuthorDate: Thu Oct 12 01:25:42 2023 + Updating config from bot --- scripts/ci/pr-bot/state/reviewers-for-label-no-matching-label.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-no-matching-label.json b/scripts/ci/pr-bot/state/reviewers-for-label-no-matching-label.json index 3b53d8d6555..85616d1d9eb 100644 --- a/scripts/ci/pr-bot/state/reviewers-for-label-no-matching-label.json +++ b/scripts/ci/pr-bot/state/reviewers-for-label-no-matching-label.json @@ -7,7 +7,7 @@ "damccorm": 1696969088581, "johnjcasey": 1697069812851, "jrmccluskey": 1696620905228, -"kennknowles": 1696512947417, +"kennknowles": 1697073938941, "lostluck": 1696622766402, "riteshghorse": 1696620910329, "robertwb": 1696881991283,
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 7377dd145f6 Updating config from bot
7377dd145f6 is described below
commit 7377dd145f64e1b4caefb066a84ae2e9a698ee9e
Author: github-actions
AuthorDate: Thu Oct 12 01:25:40 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-28956.json | 10 ++
1 file changed, 10 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-28956.json
b/scripts/ci/pr-bot/state/pr-state/pr-28956.json
new file mode 100644
index 000..8ad90f2eea6
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-28956.json
@@ -0,0 +1,10 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {
+"no-matching-label": "kennknowles"
+ },
+ "nextAction": "Reviewers",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch master updated: [YAML] Avro format for PubSub. (#28899)
This is an automated email from the ASF dual-hosted git repository.
robertwb pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 9c75db4760c [YAML] Avro format for PubSub. (#28899)
9c75db4760c is described below
commit 9c75db4760ca3440a0a2ed12f031d84bc011aa96
Author: Robert Bradshaw
AuthorDate: Wed Oct 11 18:19:29 2023 -0700
[YAML] Avro format for PubSub. (#28899)
---
sdks/python/apache_beam/yaml/yaml_io.py | 21 +
sdks/python/apache_beam/yaml/yaml_io_test.py | 68
2 files changed, 89 insertions(+)
diff --git a/sdks/python/apache_beam/yaml/yaml_io.py
b/sdks/python/apache_beam/yaml/yaml_io.py
index 3321644ded5..b2bf150fa55 100644
--- a/sdks/python/apache_beam/yaml/yaml_io.py
+++ b/sdks/python/apache_beam/yaml/yaml_io.py
@@ -23,6 +23,7 @@ Note that in the case that they overlap with other (likely
Java)
implementations of the same transforms, the configs must be kept in sync.
"""
+import io
import os
from typing import Any
from typing import Callable
@@ -32,12 +33,14 @@ from typing import Mapping
from typing import Optional
from typing import Tuple
+import fastavro
import yaml
import apache_beam as beam
import apache_beam.io as beam_io
from apache_beam.io import ReadFromBigQuery
from apache_beam.io import WriteToBigQuery
+from apache_beam.io import avroio
from apache_beam.io.gcp.bigquery import BigQueryDisposition
from apache_beam.portability.api import schema_pb2
from apache_beam.typehints import schemas
@@ -146,6 +149,13 @@ def _create_parser(
elif format == 'json':
beam_schema = json_utils.json_schema_to_beam_schema(schema)
return beam_schema, json_utils.json_parser(beam_schema)
+ elif format == 'avro':
+beam_schema = avroio.avro_schema_to_beam_schema(schema)
+covert_to_row = avroio.avro_dict_to_beam_row(schema, beam_schema)
+return (
+beam_schema,
+lambda record: covert_to_row(
+fastavro.schemaless_reader(io.BytesIO(record), schema)))
else:
raise ValueError(f'Unknown format: {format}')
@@ -162,6 +172,17 @@ def _create_formatter(
return lambda row: getattr(row, field_names[0])
elif format == 'json':
return json_utils.json_formater(beam_schema)
+ elif format == 'avro':
+avro_schema = schema or avroio.beam_schema_to_avro_schema(beam_schema)
+from_row = avroio.beam_row_to_avro_dict(avro_schema, beam_schema)
+
+def formatter(row):
+ buffer = io.BytesIO()
+ fastavro.schemaless_writer(buffer, avro_schema, from_row(row))
+ buffer.seek(0)
+ return buffer.read()
+
+return formatter
else:
raise ValueError(f'Unknown format: {format}')
diff --git a/sdks/python/apache_beam/yaml/yaml_io_test.py
b/sdks/python/apache_beam/yaml/yaml_io_test.py
index 72675da278b..7071860a7bf 100644
--- a/sdks/python/apache_beam/yaml/yaml_io_test.py
+++ b/sdks/python/apache_beam/yaml/yaml_io_test.py
@@ -15,9 +15,12 @@
# limitations under the License.
#
+import io
+import json
import logging
import unittest
+import fastavro
import mock
import apache_beam as beam
@@ -167,6 +170,48 @@ class YamlPubSubTest(unittest.TestCase):
result,
equal_to([beam.Row(payload=b'msg1'), beam.Row(payload=b'msg2')]))
+ _avro_schema = {
+ 'type': 'record',
+ 'name': 'ec',
+ 'fields': [{
+ 'name': 'label', 'type': 'string'
+ }, {
+ 'name': 'rank', 'type': 'int'
+ }]
+ }
+
+ def _encode_avro(self, data):
+buffer = io.BytesIO()
+fastavro.schemaless_writer(buffer, self._avro_schema, data)
+buffer.seek(0)
+return buffer.read()
+
+ def test_read_avro(self):
+
+with beam.Pipeline(options=beam.options.pipeline_options.PipelineOptions(
+pickle_library='cloudpickle')) as p:
+ with mock.patch(
+ 'apache_beam.io.ReadFromPubSub',
+ FakeReadFromPubSub(
+ topic='my_topic',
+ messages=[PubsubMessage(self._encode_avro({'label': '37a',
+ 'rank': 1}), {}),
+PubsubMessage(self._encode_avro({'label': '389a',
+ 'rank': 2}), {})])):
+result = p | YamlTransform(
+'''
+type: ReadFromPubSub
+config:
+ topic: my_topic
+ format: avro
+ schema: %s
+''' % json.dumps(self._avro_schema))
+assert_that(
+result,
+equal_to(
+[beam.Row(label='37a', rank=1), # linebreak
+ beam.Row(label='389a', rank=2)]))
+
def test_read_json(self):
with beam.Pipeline(options=beam.options.pipeline_options.PipelineOptions(
pickle_library='cloudpickle')) as p:
@@ -346,6 +391,29 @@ class YamlPubSubTest(unittest.TestCase):
[beam-starter-java] branch dependabot/gradle/org.apache.beam-beam-runners-direct-java-2.50.0 deleted (was de0213c)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/gradle/org.apache.beam-beam-runners-direct-java-2.50.0 in repository https://gitbox.apache.org/repos/asf/beam-starter-java.git was de0213c Bump org.apache.beam:beam-runners-direct-java from 2.46.0 to 2.50.0 The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
[beam-starter-java] branch dependabot/gradle/org.apache.beam-beam-sdks-java-core-2.51.0 created (now 6b970e9)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/gradle/org.apache.beam-beam-sdks-java-core-2.51.0 in repository https://gitbox.apache.org/repos/asf/beam-starter-java.git at 6b970e9 Bump org.apache.beam:beam-sdks-java-core from 2.46.0 to 2.51.0 No new revisions were added by this update.
[beam-starter-java] branch dependabot/gradle/org.apache.beam-beam-sdks-java-core-2.50.0 deleted (was 21efa29)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/gradle/org.apache.beam-beam-sdks-java-core-2.50.0 in repository https://gitbox.apache.org/repos/asf/beam-starter-java.git was 21efa29 Bump org.apache.beam:beam-sdks-java-core from 2.46.0 to 2.50.0 The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
[beam-starter-java] branch dependabot/gradle/org.apache.beam-beam-runners-direct-java-2.51.0 created (now 6383030)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/gradle/org.apache.beam-beam-runners-direct-java-2.51.0 in repository https://gitbox.apache.org/repos/asf/beam-starter-java.git at 6383030 Bump org.apache.beam:beam-runners-direct-java from 2.46.0 to 2.51.0 No new revisions were added by this update.
[beam] branch master updated (104c10b3ee5 -> e8e38145d96)
This is an automated email from the ASF dual-hosted git repository.
yhu pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
from 104c10b3ee5 [RRIO] Create Caller and SetupTeardown interfaces (#28905)
add e8e38145d96 Setup Java 21 container (#28833)
No new revisions were added by this update.
Summary of changes:
...s_Java_CoGBK_Dataflow_V2_Batch_JavaVersions.yml | 8 ++---
...va_CoGBK_Dataflow_V2_Streaming_JavaVersions.yml | 8 ++---
...LoadTests_Java_GBK_Dataflow_V2_Batch_Java11.yml | 14
...LoadTests_Java_GBK_Dataflow_V2_Batch_Java17.yml | 14
...Tests_Java_GBK_Dataflow_V2_Streaming_Java11.yml | 14
...Tests_Java_GBK_Dataflow_V2_Streaming_Java17.yml | 14
.../beam_PostCommit_Java_Examples_Dataflow_ARM.yml | 2 +-
...m_PostCommit_Java_Examples_Dataflow_V2_Java.yml | 2 +-
.../beam_PostCommit_Java_Jpms_Dataflow_Java17.yml | 2 +-
.../beam_PostCommit_Java_Jpms_Direct_Java17.yml| 2 +-
...am_PostCommit_Java_Nexmark_Dataflow_V2_Java.yml | 2 +-
.../beam_PostCommit_TransformService_Direct.yml| 2 +-
...eam_PreCommit_Java_Examples_Dataflow_Java11.yml | 2 +-
...eam_PreCommit_Java_Examples_Dataflow_Java17.yml | 2 +-
.github/workflows/beam_PreCommit_SQL_Java11.yml| 2 +-
.github/workflows/beam_PreCommit_SQL_Java17.yml| 2 +-
.test-infra/jenkins/JavaTestProperties.groovy | 7 +++-
.test-infra/jenkins/NexmarkBuilder.groovy | 4 +--
.../job_LoadTests_CoGBK_Dataflow_V2_Java11.groovy | 2 +-
.../job_LoadTests_CoGBK_Dataflow_V2_Java17.groovy | 2 +-
.../job_LoadTests_GBK_Dataflow_V2_Java11.groovy| 2 +-
.../job_LoadTests_GBK_Dataflow_V2_Java17.groovy| 2 +-
.../job_LoadTests_ParDo_Dataflow_V2_Java11.groovy | 2 +-
.../job_LoadTests_ParDo_Dataflow_V2_Java17.groovy | 2 +-
...tCommit_Java_Examples_Dataflow_V2_Java11.groovy | 2 +-
...tCommit_Java_Examples_Dataflow_V2_Java17.groovy | 2 +-
...job_PostCommit_Java_Jpms_Dataflow_Java17.groovy | 2 +-
.../job_PostCommit_Java_Jpms_Direct_Java17.groovy | 2 +-
.../job_PostCommit_TransformService_Direct.groovy | 2 +-
..._PreCommit_Java_Examples_Dataflow_Java11.groovy | 2 +-
.../jenkins/job_PreCommit_SQL_Java11.groovy| 2 +-
.../jenkins/job_PreCommit_SQL_Java17.groovy| 2 +-
..._Precommit_Java_Examples_Dataflow_Java17.groovy | 2 +-
build.gradle.kts | 14
.../org/apache/beam/gradle/BeamModulePlugin.groovy | 37 --
.../runners/core/construction/Environments.java| 6 ++--
.../core/construction/EnvironmentsTest.java| 6 +++-
.../google-cloud-dataflow-java/arm/build.gradle| 6 ++--
runners/google-cloud-dataflow-java/build.gradle| 6 ++--
.../beam/runners/dataflow/DataflowRunner.java | 7 ++--
runners/spark/spark_runner.gradle | 5 +--
sdks/java/container/Dockerfile | 3 +-
sdks/java/container/common.gradle | 14
.../java/container/{java11 => java21}/build.gradle | 4 ++-
sdks/java/container/java21/option-jamm.json| 12 +++
sdks/java/testing/jpms-tests/build.gradle | 21 ++--
sdks/java/testing/test-utils/build.gradle | 29 ++---
.../testutils/jvmverification/JvmVerification.java | 16 +-
settings.gradle.kts| 1 +
website/www/site/content/en/roadmap/java-sdk.md| 6 ++--
50 files changed, 193 insertions(+), 133 deletions(-)
copy sdks/java/container/{java11 => java21}/build.gradle (86%)
create mode 100644 sdks/java/container/java21/option-jamm.json
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 9499b05e28e Updating config from bot
9499b05e28e is described below
commit 9499b05e28e35ff38f885c1ceacef6ad5b92f092
Author: github-actions
AuthorDate: Thu Oct 12 00:49:27 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-28954.json | 10 ++
1 file changed, 10 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-28954.json
b/scripts/ci/pr-bot/state/pr-state/pr-28954.json
new file mode 100644
index 000..8a92900947e
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-28954.json
@@ -0,0 +1,10 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {
+"build": "damccorm"
+ },
+ "nextAction": "Reviewers",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 887cf03a50c Updating config from bot
887cf03a50c is described below
commit 887cf03a50c7ed3f872a342286028e076a168e24
Author: github-actions
AuthorDate: Thu Oct 12 00:49:29 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/reviewers-for-label-build.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-build.json
b/scripts/ci/pr-bot/state/reviewers-for-label-build.json
index 81c671b12f3..6d7bb22369b 100644
--- a/scripts/ci/pr-bot/state/reviewers-for-label-build.json
+++ b/scripts/ci/pr-bot/state/reviewers-for-label-build.json
@@ -1,7 +1,7 @@
{
"label": "build",
"dateOfLastReviewAssignment": {
-"damccorm": 1697042167607,
+"damccorm": 1697071765738,
"Abacn": 1697042173792
}
}
\ No newline at end of file
[beam] branch asf-site updated: Publishing website 2023/10/12 00:30:25 at commit 104c10b
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new 63245dbfd5d Publishing website 2023/10/12 00:30:25 at commit 104c10b 63245dbfd5d is described below commit 63245dbfd5d8068f43124fed2061561666ce8883 Author: runner AuthorDate: Thu Oct 12 00:30:25 2023 + Publishing website 2023/10/12 00:30:25 at commit 104c10b --- website/generated-content/sitemap.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/generated-content/sitemap.xml b/website/generated-content/sitemap.xml index 1459471b31e..fb2ab6a064d 100644 --- a/website/generated-content/sitemap.xml +++ b/website/generated-content/sitemap.xml @@ -1 +1 @@ -http://www.sitemaps.org/schemas/sitemap/0.9; xmlns:xhtml="http://www.w3.org/1999/xhtml;>/categories/blog/2023-10-11T10:57:24-07:00/blog/2023-10-11T10:57:24-07:00/categories/2023-10-11T10:57:24-07:00/blog/dyi-content-discovery-platform-genai-beam/2023-10-11T10:57:24-07:00http://www.sitemaps.org/schemas/sitemap/0.9; xmlns:xhtml="http://www.w3.org/1999/xhtml;>/categories/blog/2023-10-11T12:39:38-07:00/blog/2023-10-11T12:39:38-07:00/categories/2023-10-11T12:39:38-07:00/blog/dyi-content-discovery-platform-genai-beam/2023-10-11T12:39:38-07:00
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 9ba95dca9a1 Updating config from bot
9ba95dca9a1 is described below
commit 9ba95dca9a1a586cd6457740a4b63d63c0f2dcc3
Author: github-actions
AuthorDate: Thu Oct 12 00:16:54 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-28953.json | 10 ++
1 file changed, 10 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-28953.json
b/scripts/ci/pr-bot/state/pr-state/pr-28953.json
new file mode 100644
index 000..b29414341ff
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-28953.json
@@ -0,0 +1,10 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {
+"no-matching-label": "johnjcasey"
+ },
+ "nextAction": "Reviewers",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new 74178a3fc1e Updating config from bot 74178a3fc1e is described below commit 74178a3fc1e1e8c1ee43206253b62e206c1468cb Author: github-actions AuthorDate: Thu Oct 12 00:16:56 2023 + Updating config from bot --- scripts/ci/pr-bot/state/reviewers-for-label-no-matching-label.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-no-matching-label.json b/scripts/ci/pr-bot/state/reviewers-for-label-no-matching-label.json index a851b75732a..3b53d8d6555 100644 --- a/scripts/ci/pr-bot/state/reviewers-for-label-no-matching-label.json +++ b/scripts/ci/pr-bot/state/reviewers-for-label-no-matching-label.json @@ -5,7 +5,7 @@ "AnandInguva": 1696897114300, "chamikaramj": 1696966485571, "damccorm": 1696969088581, -"johnjcasey": 1696482359433, +"johnjcasey": 1697069812851, "jrmccluskey": 1696620905228, "kennknowles": 1696512947417, "lostluck": 1696622766402,
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new 8fa77269bd8 Updating config from bot 8fa77269bd8 is described below commit 8fa77269bd8e766bb99bdf8cad7fe7465c14fec9 Author: github-actions AuthorDate: Thu Oct 12 00:07:14 2023 + Updating config from bot --- scripts/ci/pr-bot/state/reviewers-for-label-java.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-java.json b/scripts/ci/pr-bot/state/reviewers-for-label-java.json index eca74696889..7c819cecc6d 100644 --- a/scripts/ci/pr-bot/state/reviewers-for-label-java.json +++ b/scripts/ci/pr-bot/state/reviewers-for-label-java.json @@ -6,7 +6,7 @@ "robertwb": 1696988999226, "kileys": 1674428678843, "apilloud": 1678822446183, -"Abacn": 1696915675163, +"Abacn": 1697069230457, "bvolpato": 1697040392209 } } \ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new d8bb9c2c258 Updating config from bot d8bb9c2c258 is described below commit d8bb9c2c25815b0a8eff246b5b3e1e5d8de3332d Author: github-actions AuthorDate: Thu Oct 12 00:07:16 2023 + Updating config from bot --- scripts/ci/pr-bot/state/reviewers-for-label-io.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-io.json b/scripts/ci/pr-bot/state/reviewers-for-label-io.json index eb7d1b76964..05aa6ffbe5c 100644 --- a/scripts/ci/pr-bot/state/reviewers-for-label-io.json +++ b/scripts/ci/pr-bot/state/reviewers-for-label-io.json @@ -5,7 +5,7 @@ "johnjcasey": 1697040392209, "pabloem": 1691787951165, "Abacn": 1697067901583, -"ahmedabu98": 1696637915510, +"ahmedabu98": 1697069230457, "bvolpato": 1696988999226, "manavgarg": 1690826779210 }
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new f1d9069e8ef Updating config from bot
f1d9069e8ef is described below
commit f1d9069e8ef875d90a782f6cf16ae19762d15f44
Author: github-actions
AuthorDate: Thu Oct 12 00:07:12 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-28952.json | 11 +++
1 file changed, 11 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-28952.json
b/scripts/ci/pr-bot/state/pr-state/pr-28952.json
new file mode 100644
index 000..770d8747020
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-28952.json
@@ -0,0 +1,11 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {
+"java": "Abacn",
+"io": "ahmedabu98"
+ },
+ "nextAction": "Reviewers",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] 01/01: Update go_tests.yml to cache
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a commit to branch lostluck-cache-go in repository https://gitbox.apache.org/repos/asf/beam.git commit 7b6117785abac2c8a4b52db9886ab04fdb93fb5a Author: Robert Burke AuthorDate: Wed Oct 11 16:55:20 2023 -0700 Update go_tests.yml to cache Configure caching go deps properly in Go Tests action. --- .github/workflows/go_tests.yml | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/.github/workflows/go_tests.yml b/.github/workflows/go_tests.yml index 6884dd69252..66c57d219a8 100644 --- a/.github/workflows/go_tests.yml +++ b/.github/workflows/go_tests.yml @@ -46,6 +46,8 @@ jobs: - uses: actions/setup-go@v4 with: go-version: '1.21' + cache-dependency-path: | + sdks/go.sum - name: Delete old coverage run: "cd sdks && rm -rf .coverage.txt || :" - name: Run coverage @@ -70,4 +72,4 @@ jobs: with: work-dir: ./sdks go-package: ./... - go-version-input: 1.21 \ No newline at end of file + go-version-input: 1.21
[beam] branch lostluck-cache-go created (now 7b6117785ab)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a change to branch lostluck-cache-go in repository https://gitbox.apache.org/repos/asf/beam.git at 7b6117785ab Update go_tests.yml to cache This branch includes the following new commits: new 7b6117785ab Update go_tests.yml to cache The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new a55d7e20690 Updating config from bot a55d7e20690 is described below commit a55d7e20690f5ea0435669ed58bc7fcd9c97d0bd Author: github-actions AuthorDate: Wed Oct 11 23:45:06 2023 + Updating config from bot --- scripts/ci/pr-bot/state/reviewers-for-label-io.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-io.json b/scripts/ci/pr-bot/state/reviewers-for-label-io.json index 1a2ae7d072d..eb7d1b76964 100644 --- a/scripts/ci/pr-bot/state/reviewers-for-label-io.json +++ b/scripts/ci/pr-bot/state/reviewers-for-label-io.json @@ -4,7 +4,7 @@ "chamikaramj": 1696867584636, "johnjcasey": 1697040392209, "pabloem": 1691787951165, -"Abacn": 1696594070059, +"Abacn": 1697067901583, "ahmedabu98": 1696637915510, "bvolpato": 1696988999226, "manavgarg": 1690826779210
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new e5d6c5d099a Updating config from bot
e5d6c5d099a is described below
commit e5d6c5d099a9138f2be07a5f4be77f8760975568
Author: github-actions
AuthorDate: Wed Oct 11 23:45:05 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/reviewers-for-label-java.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-java.json
b/scripts/ci/pr-bot/state/reviewers-for-label-java.json
index f3c8d7d11b6..eca74696889 100644
--- a/scripts/ci/pr-bot/state/reviewers-for-label-java.json
+++ b/scripts/ci/pr-bot/state/reviewers-for-label-java.json
@@ -2,7 +2,7 @@
"label": "java",
"dateOfLastReviewAssignment": {
"lukecwik": 1680701852535,
-"kennknowles": 1696887969299,
+"kennknowles": 1697067901583,
"robertwb": 1696988999226,
"kileys": 1674428678843,
"apilloud": 1678822446183,
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 4047d668b7f Updating config from bot
4047d668b7f is described below
commit 4047d668b7f4baed74f0820322f91bb5efdbc9ad
Author: github-actions
AuthorDate: Wed Oct 11 23:45:03 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-28950.json | 7 +--
1 file changed, 5 insertions(+), 2 deletions(-)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-28950.json
b/scripts/ci/pr-bot/state/pr-state/pr-28950.json
index 242a48d7d3b..60655251338 100644
--- a/scripts/ci/pr-bot/state/pr-state/pr-28950.json
+++ b/scripts/ci/pr-bot/state/pr-state/pr-28950.json
@@ -1,7 +1,10 @@
{
"commentedAboutFailingChecks": true,
- "reviewersAssignedForLabels": {},
- "nextAction": "Author",
+ "reviewersAssignedForLabels": {
+"java": "kennknowles",
+"io": "Abacn"
+ },
+ "nextAction": "Reviewers",
"stopReviewerNotifications": false,
"remindAfterTestsPass": [],
"committerAssigned": false
[beam] branch dependabot/go_modules/playground/backend/golang.org/x/net-0.17.0 created (now 608dfbfcbc2)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/go_modules/playground/backend/golang.org/x/net-0.17.0 in repository https://gitbox.apache.org/repos/asf/beam.git at 608dfbfcbc2 Bump golang.org/x/net from 0.8.0 to 0.17.0 in /playground/backend No new revisions were added by this update.
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 93dba060270 Updating config from bot
93dba060270 is described below
commit 93dba06027020695b8b04dd1840bb4ac658b6222
Author: github-actions
AuthorDate: Wed Oct 11 23:34:41 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-28950.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-28950.json
b/scripts/ci/pr-bot/state/pr-state/pr-28950.json
new file mode 100644
index 000..242a48d7d3b
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-28950.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": true,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam-starter-go] branch dependabot/go_modules/golang.org/x/net-0.17.0 created (now b4f14b7)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/go_modules/golang.org/x/net-0.17.0 in repository https://gitbox.apache.org/repos/asf/beam-starter-go.git at b4f14b7 Bump golang.org/x/net from 0.14.0 to 0.17.0 No new revisions were added by this update.
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new 25c60861d20 Updating config from bot 25c60861d20 is described below commit 25c60861d20a015247587e079637043c815e342b Author: github-actions AuthorDate: Wed Oct 11 23:06:05 2023 + Updating config from bot --- scripts/ci/pr-bot/state/pr-state/pr-28899.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/pr-state/pr-28899.json b/scripts/ci/pr-bot/state/pr-state/pr-28899.json index c9972b0d659..7e5492c676d 100644 --- a/scripts/ci/pr-bot/state/pr-state/pr-28899.json +++ b/scripts/ci/pr-bot/state/pr-state/pr-28899.json @@ -6,5 +6,5 @@ "nextAction": "Reviewers", "stopReviewerNotifications": false, "remindAfterTestsPass": [], - "committerAssigned": false + "committerAssigned": true } \ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new ce9e37cb850 Updating config from bot
ce9e37cb850 is described below
commit ce9e37cb85087c01ed0742df91fca3dd6fb3809b
Author: github-actions
AuthorDate: Wed Oct 11 22:54:19 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-28949.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-28949.json
b/scripts/ci/pr-bot/state/pr-state/pr-28949.json
new file mode 100644
index 000..9c2aa5aa212
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-28949.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": true,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 21bb7283119 Updating config from bot
21bb7283119 is described below
commit 21bb72831196cdefe74f0994c63d230277710102
Author: github-actions
AuthorDate: Wed Oct 11 22:09:03 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-28948.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-28948.json
b/scripts/ci/pr-bot/state/pr-state/pr-28948.json
new file mode 100644
index 000..9c2aa5aa212
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-28948.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": true,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch rrio/main/stub deleted (was 104c10b3ee5)
This is an automated email from the ASF dual-hosted git repository. damondouglas pushed a change to branch rrio/main/stub in repository https://gitbox.apache.org/repos/asf/beam.git was 104c10b3ee5 [RRIO] Create Caller and SetupTeardown interfaces (#28905) The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
[beam] branch rrio/main/stub created (now 104c10b3ee5)
This is an automated email from the ASF dual-hosted git repository. damondouglas pushed a change to branch rrio/main/stub in repository https://gitbox.apache.org/repos/asf/beam.git at 104c10b3ee5 [RRIO] Create Caller and SetupTeardown interfaces (#28905) No new revisions were added by this update.
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new a51d2f2e5d0 Updating config from bot
a51d2f2e5d0 is described below
commit a51d2f2e5d07b3d2150497d202f00cee7bb1ddfc
Author: github-actions
AuthorDate: Wed Oct 11 21:35:00 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/reviewers-for-label-go.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-go.json
b/scripts/ci/pr-bot/state/reviewers-for-label-go.json
index b1ddd22a26e..7e5cbcb4a2a 100644
--- a/scripts/ci/pr-bot/state/reviewers-for-label-go.json
+++ b/scripts/ci/pr-bot/state/reviewers-for-label-go.json
@@ -1,7 +1,7 @@
{
"label": "go",
"dateOfLastReviewAssignment": {
-"lostluck": 1697000752516,
+"lostluck": 1697060096355,
"jrmccluskey": 1697000756906,
"youngoli": 1657688896155,
"damccorm": 1680501930289,
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 1b6fe33beb9 Updating config from bot
1b6fe33beb9 is described below
commit 1b6fe33beb9a31940bb055f4eb7e516ced9650ae
Author: github-actions
AuthorDate: Wed Oct 11 21:34:58 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-28947.json | 10 ++
1 file changed, 10 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-28947.json
b/scripts/ci/pr-bot/state/pr-state/pr-28947.json
new file mode 100644
index 000..8ea94810368
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-28947.json
@@ -0,0 +1,10 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {
+"go": "lostluck"
+ },
+ "nextAction": "Reviewers",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch dependabot/go_modules/learning/katas/go/golang.org/x/net-0.17.0 created (now abef4fe1226)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/go_modules/learning/katas/go/golang.org/x/net-0.17.0 in repository https://gitbox.apache.org/repos/asf/beam.git at abef4fe1226 Bump golang.org/x/net from 0.7.0 to 0.17.0 in /learning/katas/go No new revisions were added by this update.
[beam] branch master updated: [RRIO] Create Caller and SetupTeardown interfaces (#28905)
This is an automated email from the ASF dual-hosted git repository.
damondouglas pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 104c10b3ee5 [RRIO] Create Caller and SetupTeardown interfaces (#28905)
104c10b3ee5 is described below
commit 104c10b3ee536a9a3ea52b4dbf62d86b669da5d9
Author: Damon
AuthorDate: Wed Oct 11 12:39:38 2023 -0700
[RRIO] Create Caller and SetupTeardown interfaces (#28905)
* Create test Caller and SetupTeardown interfaces
* Update Javadoc
* Defer Call transform to future PR
* Rename package to requestresponseio
* Add username to TODO
---
sdks/java/io/rrio/build.gradle | 7 +-
.../apache/beam/io/requestresponseio/Caller.java | 27 +
.../beam/io/requestresponseio/SetupTeardown.java | 34 ++
.../UserCodeExecutionException.java| 38 ++
.../requestresponseio/UserCodeQuotaException.java | 42 +++
.../UserCodeTimeoutException.java | 39 ++
.../beam/io/requestresponseio/package-info.java| 20
.../java/org/apache/beam/io/rrio/CallerTest.java | 126
.../org/apache/beam/io/rrio/SetupTeardownTest.java | 132 +
9 files changed, 462 insertions(+), 3 deletions(-)
diff --git a/sdks/java/io/rrio/build.gradle b/sdks/java/io/rrio/build.gradle
index d65df370e0c..32fbd9d22e3 100644
--- a/sdks/java/io/rrio/build.gradle
+++ b/sdks/java/io/rrio/build.gradle
@@ -25,9 +25,10 @@ description = "Apache Beam :: SDKS :: Java :: IO ::
RequestResponseIO (RRIO)"
ext.summary = "Support to read from and write to Web APIs"
dependencies {
-implementation project(path: ":sdks:java:core", configuration: "shadow")
-implementation library.java.joda_time
-implementation library.java.vendored_guava_32_1_2_jre
+// TODO(damondouglas): revert to implementation after project is more
fully developed
+permitUnusedDeclared project(path: ":sdks:java:core", configuration:
"shadow")
+permitUnusedDeclared library.java.joda_time
+permitUnusedDeclared library.java.vendored_guava_32_1_2_jre
testImplementation project(path: ":sdks:java:core", configuration:
"shadowTest")
testImplementation library.java.junit
diff --git
a/sdks/java/io/rrio/src/main/java/org/apache/beam/io/requestresponseio/Caller.java
b/sdks/java/io/rrio/src/main/java/org/apache/beam/io/requestresponseio/Caller.java
new file mode 100644
index 000..32b514c43a1
--- /dev/null
+++
b/sdks/java/io/rrio/src/main/java/org/apache/beam/io/requestresponseio/Caller.java
@@ -0,0 +1,27 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.io.requestresponseio;
+
+import java.io.Serializable;
+
+/** {@link Caller} interfaces user custom code intended for API calls. */
+public interface Caller extends Serializable {
+
+ /** Calls a Web API with the {@link RequestT} and returns a {@link
ResponseT}. */
+ ResponseT call(RequestT request) throws UserCodeExecutionException;
+}
diff --git
a/sdks/java/io/rrio/src/main/java/org/apache/beam/io/requestresponseio/SetupTeardown.java
b/sdks/java/io/rrio/src/main/java/org/apache/beam/io/requestresponseio/SetupTeardown.java
new file mode 100644
index 000..2bdc8113d98
--- /dev/null
+++
b/sdks/java/io/rrio/src/main/java/org/apache/beam/io/requestresponseio/SetupTeardown.java
@@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *
[beam] branch dependabot/github_actions/actions/checkout-4 updated (1e1f4f0849d -> eb1bef466ee)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/github_actions/actions/checkout-4 in repository https://gitbox.apache.org/repos/asf/beam.git omit 1e1f4f0849d Bump actions/checkout from 3 to 4 add 09f30537015 Add GitHub Workflow Replacements for Jenkins job_LoadTests_Python_SideInput_Dataflow_Batch (#28826) add 6bab4b57992 Add Community Metrics Prober workflow (#28842) add eb1bef466ee Bump actions/checkout from 3 to 4 This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (1e1f4f0849d) \ N -- N -- N refs/heads/dependabot/github_actions/actions/checkout-4 (eb1bef466ee) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: ...m_LoadTests_Python_SideInput_Dataflow_Batch.yml | 186 + .github/workflows/beam_Prober_CommunityMetrics.yml | 77 + ...taflow_Batch_10gb_1000window_first_iterable.txt | 30 ...put_Dataflow_Batch_10gb_1000window_iterable.txt | 29 ..._Dataflow_Batch_10gb_1window_first_iterable.txt | 29 ...eInput_Dataflow_Batch_10gb_1window_iterable.txt | 28 ...flow_Batch_1gb_1000window_1key_percent_dict.txt | 30 ...low_Batch_1gb_1000window_99key_percent_dict.txt | 30 ...ataflow_Batch_1gb_1window_1key_percent_dict.txt | 29 ...taflow_Batch_1gb_1window_99key_percent_dict.txt | 29 ...Input_Dataflow_Batch_1gb_1window_first_list.txt | 29 ...n_SideInput_Dataflow_Batch_1gb_1window_list.txt | 28 12 files changed, 554 insertions(+) create mode 100644 .github/workflows/beam_LoadTests_Python_SideInput_Dataflow_Batch.yml create mode 100644 .github/workflows/beam_Prober_CommunityMetrics.yml create mode 100644 .github/workflows/load-tests-job-configs/python_SideInput_Dataflow_Batch_10gb_1000window_first_iterable.txt create mode 100644 .github/workflows/load-tests-job-configs/python_SideInput_Dataflow_Batch_10gb_1000window_iterable.txt create mode 100644 .github/workflows/load-tests-job-configs/python_SideInput_Dataflow_Batch_10gb_1window_first_iterable.txt create mode 100644 .github/workflows/load-tests-job-configs/python_SideInput_Dataflow_Batch_10gb_1window_iterable.txt create mode 100644 .github/workflows/load-tests-job-configs/python_SideInput_Dataflow_Batch_1gb_1000window_1key_percent_dict.txt create mode 100644 .github/workflows/load-tests-job-configs/python_SideInput_Dataflow_Batch_1gb_1000window_99key_percent_dict.txt create mode 100644 .github/workflows/load-tests-job-configs/python_SideInput_Dataflow_Batch_1gb_1window_1key_percent_dict.txt create mode 100644 .github/workflows/load-tests-job-configs/python_SideInput_Dataflow_Batch_1gb_1window_99key_percent_dict.txt create mode 100644 .github/workflows/load-tests-job-configs/python_SideInput_Dataflow_Batch_1gb_1window_first_list.txt create mode 100644 .github/workflows/load-tests-job-configs/python_SideInput_Dataflow_Batch_1gb_1window_list.txt
[beam] branch dependabot/github_actions/actions/checkout-4 updated (e941089a715 -> 1e1f4f0849d)
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a change to branch
dependabot/github_actions/actions/checkout-4
in repository https://gitbox.apache.org/repos/asf/beam.git
discard e941089a715 Bump actions/checkout from 3 to 4
add d26a78297f2 [YAML] Add JSON parsing to PubSub IO. (#28754)
add 6722ff197e8 Schema-aware Avro reading and writing.
add 714d3a9aa04 Add yaml providers for avro.
add 07e26fd014d test,lint
add ce217d36b13 Merge pull request #28855 [YAML] Schemify avroio and add a
yaml provider.
add 57821c191d3 Add python wheel staging to GitHub Actions
build_release_candidate
add 795d366c395 Merge pull request #28822: Add python wheel staging to
GitHub Actions build_release_candidate
add e665c1c73bd [Fix] Broken Healthcare Notebook Link (#28876)
add ee7a2aea975 Fix perf alert tool call (#28872)
add fba098b69db Bump postcss (#28866)
add cb802661f2a [Python]Update state cache size to 100 MB (#28877)
add fc93a36d869 Make parquetio schema aware.
add e94aeb71fff Add parquet yaml providers.
add 9602af91dcd Update error message to account for more detail.
add d7db27efccc Merge branch 'master' into yaml-parquet
add a6947eb3b79 Merge pull request #28850 [BEAM-28684] [YAML] Add Parquet
reading and writing.
add 7531501ff27 Fix interactive tests for Beam Notebooks release (#28880)
add b117ac8b083 Update executionResult (#28884)
add d5b8fb82958 Add edge segment size to filter out change points that are
observed on the data edge (#28780)
add a3bf007cef1 Append UUID when test_name is None for perf tool (#28882)
add 92d5bf87796 Bump github.com/aws/aws-sdk-go-v2 from 1.21.0 to 1.21.1 in
/sdks (#28891)
add 76fbb8e9686 Bump org.nosphere.apache.rat from 0.8.0 to 0.8.1 (#28675)
add 604629798a3 Add dataclasses to perf alert tool and refactor code.
(#28889)
add d17063412ea Update BigQueryIO Documentation (#28591)
add 66bae3fa5ba Bump github.com/aws/aws-sdk-go-v2/feature/s3/manager in
/sdks (#28890)
add ad5c2f55bfe Bump golang.org/x/sync from 0.3.0 to 0.4.0 in /sdks
(#28856)
add 5491b5cdabc Update container version in BeamModulePlugin.groovy
(#28879)
add be3909a62d7 Bump cloud.google.com/go/bigquery from 1.55.0 to 1.56.0 in
/sdks (#28892)
add 9eac03027bc Bump github.com/testcontainers/testcontainers-go in /sdks
(#28761)
add 4ead788840c Reenable TestFhirIO.* on GHA (#28435)
add 21f822f6580 Catch EOFError along with StopIteration (#28896)
add f63e9ed28e4 Additional context for decoding errors. (#27024)
add dafe928a15b Add Java variants of WriteTo{Csv,Json}. (#28380)
add 2bbb3485c78 Populate top-level display data in yaml main. (#28512)
add 3844972d681 Revert "Populate top-level display data in yaml main.
(#28512)"
add ab5992940c1 Merge pull request #28906 from
apache/revert-28512-display-data
add 3d574b485ab Fix sourceJar task dependencies (#28900)
add 152f9c52b57 Bump golang.org/x/oauth2 from 0.12.0 to 0.13.0 in /sdks
(#28912)
add e01de664594 remove unused os import (#28914)
add e130352d7c7 Remove issue_comment trigger for load test, performance
test and other jobs (#28915)
add a7abe61288c Remove issue_comment trigger in most postcommit (#28916)
add 2bccee14cf9 Bump google.golang.org/api from 0.144.0 to 0.146.0 in
/sdks (#28911)
add a859863d392 update dataflow containers (#28904)
add d2420ad6ee5 Fix buildDirectory (#28925)
add 65eaf45026e fix lint (#28917)
add 99c87a2c68b Updates the Transform Service to accept Python extra
packages through the Java API (#28783)
add 88139bdc9a0 Retrieve workflow runs for GA Post-Commits Status
dashboard (#28844)
add e3c1591bbb8 Bump google.golang.org/grpc from 1.58.2 to 1.58.3 in /sdks
(#28938)
add 62efb967884 Bump golang.org/x/net from 0.16.0 to 0.17.0 in /sdks
(#28939)
add d437afab961 Run self-assign and pr-updates only in create event
(#28928)
add a54afde4a05 Fix missing backslash in arguments of job (#28942)
add e56d1a1157b Enable missing-timeout pylint check (#28920)
add 9cdcb036cc9 move nested windmill state classes to top level files and
organize di… (#28506)
add 6a456f04b68 Add user counters and msec metrics to prism UI. (#28929)
add 2bd23b1 Add counter for MLTransform and data processing transforms
(#28927)
add c62a81b1c57 Reapply "Populate top-level display data in yaml main.
(#28512)"
add de48383bdca Fix typo in docstring.
add b82dd4849e4 Merge pull request #28931 from robertwb/display-data
add 7a1245d4011 Bump github.com/google/go-cmp from 0.5.9 to 0.6.0 in /sdks
(#28940)
add 2bfcb9fda37 Remove unnecessary a space character that got through
Javadoc CI ;) (#28863)
add aedfa461fb3 [#20970] Fix gRPC leak by closing ResidualSource at
BoundedToUnboundedSourceAdapter.Reader#init() in Dataflow worker (#28548)
[beam] branch master updated (09f30537015 -> 6bab4b57992)
This is an automated email from the ASF dual-hosted git repository.
yhu pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
from 09f30537015 Add GitHub Workflow Replacements for Jenkins
job_LoadTests_Python_SideInput_Dataflow_Batch (#28826)
add 6bab4b57992 Add Community Metrics Prober workflow (#28842)
No new revisions were added by this update.
Summary of changes:
..._Flink.yml => beam_Prober_CommunityMetrics.yml} | 22 --
1 file changed, 12 insertions(+), 10 deletions(-)
copy .github/workflows/{beam_PostCommit_Go_VR_Flink.yml =>
beam_Prober_CommunityMetrics.yml} (83%)
[beam] branch master updated (dfdbcb9006b -> 09f30537015)
This is an automated email from the ASF dual-hosted git repository.
yhu pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
from dfdbcb9006b Add GitHub Workflow Replacements for Jenkins
job_LoadTests_Java_CoGBK subgroup (#28742)
add 09f30537015 Add GitHub Workflow Replacements for Jenkins
job_LoadTests_Python_SideInput_Dataflow_Batch (#28826)
No new revisions were added by this update.
Summary of changes:
...m_LoadTests_Python_SideInput_Dataflow_Batch.yml | 186 +
...aflow_Batch_10gb_1000window_first_iterable.txt} | 14 +-
...ut_Dataflow_Batch_10gb_1000window_iterable.txt} | 15 +-
...Dataflow_Batch_10gb_1window_first_iterable.txt} | 13 +-
...Input_Dataflow_Batch_10gb_1window_iterable.txt} | 14 +-
...low_Batch_1gb_1000window_1key_percent_dict.txt} | 14 +-
...ow_Batch_1gb_1000window_99key_percent_dict.txt} | 14 +-
...taflow_Batch_1gb_1window_1key_percent_dict.txt} | 13 +-
...aflow_Batch_1gb_1window_99key_percent_dict.txt} | 13 +-
...nput_Dataflow_Batch_1gb_1window_first_list.txt} | 13 +-
..._SideInput_Dataflow_Batch_1gb_1window_list.txt} | 12 +-
11 files changed, 249 insertions(+), 72 deletions(-)
create mode 100644
.github/workflows/beam_LoadTests_Python_SideInput_Dataflow_Batch.yml
copy
.github/workflows/load-tests-job-configs/{python_GBK_Dataflow_Streaming_2GB_of_100B_records.txt
=> python_SideInput_Dataflow_Batch_10gb_1000window_first_iterable.txt} (75%)
copy
.github/workflows/load-tests-job-configs/{python_GBK_Dataflow_Streaming_2GB_of_100B_records.txt
=> python_SideInput_Dataflow_Batch_10gb_1000window_iterable.txt} (74%)
copy
.github/workflows/load-tests-job-configs/{python_GBK_Dataflow_Streaming_2GB_of_100B_records.txt
=> python_SideInput_Dataflow_Batch_10gb_1window_first_iterable.txt} (77%)
copy
.github/workflows/load-tests-job-configs/{python_GBK_Dataflow_Streaming_2GB_of_10B_records.txt
=> python_SideInput_Dataflow_Batch_10gb_1window_iterable.txt} (75%)
copy
.github/workflows/load-tests-job-configs/{python_GBK_Dataflow_Streaming_2GB_of_100B_records.txt
=> python_SideInput_Dataflow_Batch_1gb_1000window_1key_percent_dict.txt} (75%)
copy
.github/workflows/load-tests-job-configs/{python_GBK_Dataflow_Streaming_2GB_of_100B_records.txt
=> python_SideInput_Dataflow_Batch_1gb_1000window_99key_percent_dict.txt} (75%)
copy
.github/workflows/load-tests-job-configs/{python_GBK_Dataflow_Streaming_2GB_of_100B_records.txt
=> python_SideInput_Dataflow_Batch_1gb_1window_1key_percent_dict.txt} (77%)
copy
.github/workflows/load-tests-job-configs/{python_GBK_Dataflow_Streaming_2GB_of_100B_records.txt
=> python_SideInput_Dataflow_Batch_1gb_1window_99key_percent_dict.txt} (77%)
copy
.github/workflows/load-tests-job-configs/{python_GBK_Dataflow_Streaming_2GB_of_100B_records.txt
=> python_SideInput_Dataflow_Batch_1gb_1window_first_list.txt} (77%)
copy
.github/workflows/load-tests-job-configs/{python_GBK_Dataflow_Streaming_2GB_of_10B_records.txt
=> python_SideInput_Dataflow_Batch_1gb_1window_list.txt} (77%)
[beam] branch master updated (ebad3deb5a0 -> dfdbcb9006b)
This is an automated email from the ASF dual-hosted git repository. yhu pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from ebad3deb5a0 Add GitHub Workflow Replacements for Jenkins job_LoadTests_Python_CoGBK subgroup (#28764) add dfdbcb9006b Add GitHub Workflow Replacements for Jenkins job_LoadTests_Java_CoGBK subgroup (#28742) No new revisions were added by this update. Summary of changes: .../beam_LoadTests_Java_CoGBK_Dataflow_Batch.yml | 114 ++ ...s_Java_CoGBK_Dataflow_V2_Batch_JavaVersions.yml | 132 + ...va_CoGBK_Dataflow_V2_Streaming_JavaVersions.yml | 132 + ...s_Java_CoGBK_SparkStructuredStreaming_Batch.yml | 114 ++ ...ava_CoGBK_Dataflow_Batch_100b_Multiple_Keys.txt | 27 + .../java_CoGBK_Dataflow_Batch_100b_Single_Key.txt | 27 + .../java_CoGBK_Dataflow_Batch_10kB.txt | 27 + .../java_CoGBK_Dataflow_Batch_2MB.txt | 27 + ...taflow_V2_Streaming_Java_100b_Multiple_Keys.txt | 29 + ..._Dataflow_V2_Streaming_Java_100b_Single_Key.txt | 29 + .../java_CoGBK_Dataflow_V2_Streaming_Java_10kB.txt | 29 + .../java_CoGBK_Dataflow_V2_Streaming_Java_2MB.txt | 29 + ...tructuredStreaming_Batch_100b_Multiple_Keys.txt | 25 ...rkStructuredStreaming_Batch_100b_Single_Key.txt | 25 ...a_CoGBK_SparkStructuredStreaming_Batch_10kB.txt | 25 ...va_CoGBK_SparkStructuredStreaming_Batch_2MB.txt | 25 16 files changed, 816 insertions(+) create mode 100644 .github/workflows/beam_LoadTests_Java_CoGBK_Dataflow_Batch.yml create mode 100644 .github/workflows/beam_LoadTests_Java_CoGBK_Dataflow_V2_Batch_JavaVersions.yml create mode 100644 .github/workflows/beam_LoadTests_Java_CoGBK_Dataflow_V2_Streaming_JavaVersions.yml create mode 100644 .github/workflows/beam_LoadTests_Java_CoGBK_SparkStructuredStreaming_Batch.yml create mode 100644 .github/workflows/load-tests-job-configs/java_CoGBK_Dataflow_Batch_100b_Multiple_Keys.txt create mode 100644 .github/workflows/load-tests-job-configs/java_CoGBK_Dataflow_Batch_100b_Single_Key.txt create mode 100644 .github/workflows/load-tests-job-configs/java_CoGBK_Dataflow_Batch_10kB.txt create mode 100644 .github/workflows/load-tests-job-configs/java_CoGBK_Dataflow_Batch_2MB.txt create mode 100644 .github/workflows/load-tests-job-configs/java_CoGBK_Dataflow_V2_Streaming_Java_100b_Multiple_Keys.txt create mode 100644 .github/workflows/load-tests-job-configs/java_CoGBK_Dataflow_V2_Streaming_Java_100b_Single_Key.txt create mode 100644 .github/workflows/load-tests-job-configs/java_CoGBK_Dataflow_V2_Streaming_Java_10kB.txt create mode 100644 .github/workflows/load-tests-job-configs/java_CoGBK_Dataflow_V2_Streaming_Java_2MB.txt create mode 100644 .github/workflows/load-tests-job-configs/java_CoGBK_SparkStructuredStreaming_Batch_100b_Multiple_Keys.txt create mode 100644 .github/workflows/load-tests-job-configs/java_CoGBK_SparkStructuredStreaming_Batch_100b_Single_Key.txt create mode 100644 .github/workflows/load-tests-job-configs/java_CoGBK_SparkStructuredStreaming_Batch_10kB.txt create mode 100644 .github/workflows/load-tests-job-configs/java_CoGBK_SparkStructuredStreaming_Batch_2MB.txt
[beam] branch master updated (bd46c76dd08 -> ebad3deb5a0)
This is an automated email from the ASF dual-hosted git repository. yhu pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from bd46c76dd08 Bump snappy-java version to 1.1.10.4 (#28655) add ebad3deb5a0 Add GitHub Workflow Replacements for Jenkins job_LoadTests_Python_CoGBK subgroup (#28764) No new revisions were added by this update. Summary of changes: .../beam_LoadTests_Python_CoGBK_Dataflow_Batch.yml | 127 +++ ...m_LoadTests_Python_CoGBK_Dataflow_Streaming.yml | 126 +++ .../beam_LoadTests_Python_CoGBK_Flink_Batch.yml| 135 + ...hon_CoGBK_Dataflow_Batch_100b_Multiple_Keys.txt | 28 + ...python_CoGBK_Dataflow_Batch_100b_Single_Key.txt | 28 + .../python_CoGBK_Dataflow_Batch_10kB.txt | 28 + .../python_CoGBK_Dataflow_Batch_2MB.txt| 28 + ...GBK_Dataflow_Flink_Batch_100b_Multiple_Keys.txt | 28 + ..._CoGBK_Dataflow_Flink_Batch_100b_Single_Key.txt | 28 + .../python_CoGBK_Dataflow_Flink_Batch_10kB.txt | 28 + ...CoGBK_Dataflow_Streaming_100b_Multiple_Keys.txt | 30 + ...on_CoGBK_Dataflow_Streaming_100b_Single_Key.txt | 30 + .../python_CoGBK_Dataflow_Streaming_10kB.txt | 30 + .../python_CoGBK_Dataflow_Streaming_2MB.txt| 30 + 14 files changed, 704 insertions(+) create mode 100644 .github/workflows/beam_LoadTests_Python_CoGBK_Dataflow_Batch.yml create mode 100644 .github/workflows/beam_LoadTests_Python_CoGBK_Dataflow_Streaming.yml create mode 100644 .github/workflows/beam_LoadTests_Python_CoGBK_Flink_Batch.yml create mode 100644 .github/workflows/load-tests-job-configs/python-cogbk/python_CoGBK_Dataflow_Batch_100b_Multiple_Keys.txt create mode 100644 .github/workflows/load-tests-job-configs/python-cogbk/python_CoGBK_Dataflow_Batch_100b_Single_Key.txt create mode 100644 .github/workflows/load-tests-job-configs/python-cogbk/python_CoGBK_Dataflow_Batch_10kB.txt create mode 100644 .github/workflows/load-tests-job-configs/python-cogbk/python_CoGBK_Dataflow_Batch_2MB.txt create mode 100644 .github/workflows/load-tests-job-configs/python-cogbk/python_CoGBK_Dataflow_Flink_Batch_100b_Multiple_Keys.txt create mode 100644 .github/workflows/load-tests-job-configs/python-cogbk/python_CoGBK_Dataflow_Flink_Batch_100b_Single_Key.txt create mode 100644 .github/workflows/load-tests-job-configs/python-cogbk/python_CoGBK_Dataflow_Flink_Batch_10kB.txt create mode 100644 .github/workflows/load-tests-job-configs/python-cogbk/python_CoGBK_Dataflow_Streaming_100b_Multiple_Keys.txt create mode 100644 .github/workflows/load-tests-job-configs/python-cogbk/python_CoGBK_Dataflow_Streaming_100b_Single_Key.txt create mode 100644 .github/workflows/load-tests-job-configs/python-cogbk/python_CoGBK_Dataflow_Streaming_10kB.txt create mode 100644 .github/workflows/load-tests-job-configs/python-cogbk/python_CoGBK_Dataflow_Streaming_2MB.txt
[beam] branch master updated: Bump snappy-java version to 1.1.10.4 (#28655)
This is an automated email from the ASF dual-hosted git repository.
yhu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new bd46c76dd08 Bump snappy-java version to 1.1.10.4 (#28655)
bd46c76dd08 is described below
commit bd46c76dd086f2546006460d280cbbe7ae19d3ee
Author: Bruno Volpato
AuthorDate: Wed Oct 11 15:23:46 2023 -0400
Bump snappy-java version to 1.1.10.4 (#28655)
---
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
index 705bcb978e1..19ed6aaa427 100644
--- a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
+++ b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
@@ -860,7 +860,7 @@ class BeamModulePlugin implements Plugin {
slf4j_jul_to_slf4j :
"org.slf4j:jul-to-slf4j:$slf4j_version",
slf4j_log4j12 :
"org.slf4j:slf4j-log4j12:$slf4j_version",
slf4j_jcl :
"org.slf4j:slf4j-jcl:$slf4j_version",
-snappy_java :
"org.xerial.snappy:snappy-java:1.1.10.3",
+snappy_java :
"org.xerial.snappy:snappy-java:1.1.10.4",
spark_core :
"org.apache.spark:spark-core_2.11:$spark2_version",
spark_streaming :
"org.apache.spark:spark-streaming_2.11:$spark2_version",
spark3_core :
"org.apache.spark:spark-core_2.12:$spark3_version",
[beam] branch asf-site updated: Publishing website 2023/10/11 18:23:16 at commit aedfa46
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new b555006c3b7 Publishing website 2023/10/11 18:23:16 at commit aedfa46 b555006c3b7 is described below commit b555006c3b7cce9433a664b80f4db832ea73feeb Author: runner AuthorDate: Wed Oct 11 18:23:16 2023 + Publishing website 2023/10/11 18:23:16 at commit aedfa46 --- website/generated-content/sitemap.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/generated-content/sitemap.xml b/website/generated-content/sitemap.xml index 754db1fb62f..1459471b31e 100644 --- a/website/generated-content/sitemap.xml +++ b/website/generated-content/sitemap.xml @@ -1 +1 @@ -http://www.sitemaps.org/schemas/sitemap/0.9; xmlns:xhtml="http://www.w3.org/1999/xhtml;>/categories/blog/2023-10-11T14:05:35+02:00/blog/2023-10-11T14:05:35+02:00/categories/2023-10-11T14:05:35+02:00/blog/dyi-content-discovery-platform-genai-beam/2023-10-11T14:05:35+02:00http://www.sitemaps.org/schemas/sitemap/0.9; xmlns:xhtml="http://www.w3.org/1999/xhtml;>/categories/blog/2023-10-11T10:57:24-07:00/blog/2023-10-11T10:57:24-07:00/categories/2023-10-11T10:57:24-07:00/blog/dyi-content-discovery-platform-genai-beam/2023-10-11T10:57:24-07:00
[beam] branch master updated: [#20970] Fix gRPC leak by closing ResidualSource at BoundedToUnboundedSourceAdapter.Reader#init() in Dataflow worker (#28548)
This is an automated email from the ASF dual-hosted git repository.
kenn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new aedfa461fb3 [#20970] Fix gRPC leak by closing ResidualSource at
BoundedToUnboundedSourceAdapter.Reader#init() in Dataflow worker (#28548)
aedfa461fb3 is described below
commit aedfa461fb354e0e97d75694d2a117d9ad505602
Author: Minbo Bae <49642083+baemi...@users.noreply.github.com>
AuthorDate: Wed Oct 11 10:57:24 2023 -0700
[#20970] Fix gRPC leak by closing ResidualSource at
BoundedToUnboundedSourceAdapter.Reader#init() in Dataflow worker (#28548)
---
.../UnboundedReadFromBoundedSource.java| 14 +-
.../UnboundedReadFromBoundedSourceTest.java| 180 +
.../dataflow/worker/WorkerCustomSources.java | 7 +-
3 files changed, 198 insertions(+), 3 deletions(-)
diff --git
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/UnboundedReadFromBoundedSource.java
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/UnboundedReadFromBoundedSource.java
index 67697636a36..53fad782da9 100644
---
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/UnboundedReadFromBoundedSource.java
+++
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/UnboundedReadFromBoundedSource.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.runners.core.construction;
-import static
org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions.checkArgument;
import static
org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions.checkNotNull;
+import static
org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions.checkState;
import java.io.IOException;
import java.io.InputStream;
@@ -288,6 +288,15 @@ public class UnboundedReadFromBoundedSource extends
PTransform extends
PTransform extends
PTransform getCheckpointMark() {
+checkState(!closed, "getCheckpointMark() call on closed %s",
getClass().getName());
if (reader == null) {
// Reader hasn't started, checkpoint the residualSource.
return new Checkpoint<>(null /* residualElements */, residualSource);
diff --git
a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/UnboundedReadFromBoundedSourceTest.java
b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/UnboundedReadFromBoundedSourceTest.java
index cd4b49262fc..31f6842a42b 100644
---
a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/UnboundedReadFromBoundedSourceTest.java
+++
b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/UnboundedReadFromBoundedSourceTest.java
@@ -26,9 +26,15 @@ import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.OutputStream;
+import java.lang.ref.Reference;
+import java.lang.ref.ReferenceQueue;
+import java.lang.ref.WeakReference;
import java.nio.ByteBuffer;
import java.nio.channels.ReadableByteChannel;
+import java.util.Collections;
+import java.util.HashMap;
import java.util.List;
+import java.util.Map;
import java.util.NoSuchElementException;
import java.util.Random;
import
org.apache.beam.runners.core.construction.UnboundedReadFromBoundedSource.BoundedToUnboundedSourceAdapter;
@@ -69,10 +75,14 @@ import org.junit.rules.ExpectedException;
import org.junit.rules.TemporaryFolder;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
/** Unit tests for {@link UnboundedReadFromBoundedSource}. */
@RunWith(JUnit4.class)
public class UnboundedReadFromBoundedSourceTest {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(UnboundedReadFromBoundedSourceTest.class);
@Rule public TemporaryFolder tmpFolder = new TemporaryFolder();
@@ -280,6 +290,38 @@ public class UnboundedReadFromBoundedSourceTest {
unboundedSource.createReader(options, checkpoint).getCurrent();
}
+ @Test
+ public void testReadersClosedProperly() throws IOException {
+ManagedReaderBoundedSource boundedSource = new
ManagedReaderBoundedSource(0, 10);
+BoundedToUnboundedSourceAdapter unboundedSource =
+new BoundedToUnboundedSourceAdapter<>(boundedSource);
+PipelineOptions options = PipelineOptionsFactory.create();
+
+BoundedToUnboundedSourceAdapter.Reader reader =
+unboundedSource.createReader(options, new Checkpoint(null,
boundedSource));
+
+for (int i = 0; i < 3; ++i) {
+ if (i == 0) {
+assertTrue(reader.start());
+ } else {
+assertTrue(reader.advance());
+ }
+ assertEquals(i, (int) reader.getCurrent());
+}
+
[beam] branch master updated: Remove unnecessary a space character that got through Javadoc CI ;) (#28863)
This is an automated email from the ASF dual-hosted git repository.
johncasey pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 2bfcb9fda37 Remove unnecessary a space character that got through
Javadoc CI ;) (#28863)
2bfcb9fda37 is described below
commit 2bfcb9fda37be8d5b9550a1b118ed87d03fdf12c
Author: Joar Wandborg
AuthorDate: Wed Oct 11 19:00:04 2023 +0200
Remove unnecessary a space character that got through Javadoc CI ;) (#28863)
(Tongue in cheek) Spotless caught me when I did not end the first sentence
of the Javadoc with a a dot, but it did not tell me I had an extra space before
the `` tag at the end of the sentence.
---
.../test/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIOTest.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIOTest.java
b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIOTest.java
index 971b91d89b7..f6224cc24b2 100644
---
a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIOTest.java
+++
b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIOTest.java
@@ -768,7 +768,7 @@ public class BigtableIOTest {
/**
* Regression test for https://github.com/apache/beam/issues/28793;>[Bug]: BigtableSource
- * "Desired bundle size 0 bytes must be greater than 0" #28793 .
+ * "Desired bundle size 0 bytes must be greater than 0" #28793.
*/
@Test
public void testSplittingWithDesiredBundleSizeZero() throws Exception {
[beam] annotated tag v2.51.0 updated (cd653e33b34 -> 1ab140a97a2)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to annotated tag v2.51.0 in repository https://gitbox.apache.org/repos/asf/beam.git *** WARNING: tag v2.51.0 was modified! *** from cd653e33b34 (commit) to 1ab140a97a2 (tag) tagging cd653e33b342bd09c76c2bbaca12597fec5b4a2c (commit) replaces jupyterlab-sidepanel-v3.0.0 by Kenneth Knowles on Wed Oct 11 12:37:30 2023 -0400 - Log - Release 2.51.0 -BEGIN PGP SIGNATURE- iQIzBAABCAAdFiEEA9uj5qvdBL/RVY3BbtVRqK4CRhwFAmUmz08ACgkQbtVRqK4C Rhx78xAAigWUs83oGVebIIz+eQV8SCSj8kuocCFb6PkQW3O97eZK0XqiKoAqA4jb pud6mNQmsycfidA/XPkiycWGIUZQFEfAIOsNkFtBq77FPT4UZvd1hAEYppKGwNlR KQ4tLV4TTmF1PfO3QYlJucmVH21NbHxCPbCuTCYqV/EE8D9tJqIgRjvZpGABs1Mv 8DjH8+7RhGyzy5di/vz1CE4oCZUKZ9Tv0VML7VjrndTid9gOxN7lw5dPLsv8A8kG xfxd+32sucOtu4oxg/a+AIqZsiZKEQkpgncKYl5UUV0nf9jVfNZ5qFZTRvKi+INl suFh4ETYXZJI/dF+YVbQ3IBDIBPEaSMffTRNudQD/H9fRLKpylX4WA592iHi5S2M QNdt+ydoZooNExMAbNaK0/1woDJb8eLIvU2uYdZg4oZ7171R4NIBrfsfACMjKv4q T0GdJQRvnBCPwl0heKSkGlDMLcpzoeebhM2o50waDOoMgXqUNE68S5q7a+98iGyU RxmBlMM4dGofA2lmYNS+40PHPLJadProhZgAG4yN9SZzhsifxSPPCDYIo3VMIDGh VaFktcpTmlmYpueU4WF3bOn3a4fCG3/2EB1gZQ2XBapfeE0Qw1jsqnIKQ2U/VIFy H1Hi8YGH1c/CkXiojjKh09LkkjI5m/45l2hrG5EtwVr94tY2oaI= =UQPu -END PGP SIGNATURE- --- No new revisions were added by this update. Summary of changes:
[beam] annotated tag sdks/v2.51.0 updated (cd653e33b34 -> adf666acc0a)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to annotated tag sdks/v2.51.0 in repository https://gitbox.apache.org/repos/asf/beam.git *** WARNING: tag sdks/v2.51.0 was modified! *** from cd653e33b34 (commit) to adf666acc0a (tag) tagging cd653e33b342bd09c76c2bbaca12597fec5b4a2c (commit) replaces jupyterlab-sidepanel-v3.0.0 by Kenneth Knowles on Wed Oct 11 12:37:01 2023 -0400 - Log - Release 2.51.0 -BEGIN PGP SIGNATURE- iQIzBAABCAAdFiEEA9uj5qvdBL/RVY3BbtVRqK4CRhwFAmUmzzQACgkQbtVRqK4C Rhy6+w/8ChFv+chOqD6LXqAQQSKM0CYe838xLO8oFtkwuqugh09XwNmuy1uM+SRe LUj2GTefQkjB3fff+XeC+iDh20TZyiy4+TK/q99a6i6o4l3ss7JFPZlKeOiuuHh0 P1MRdn3Oki4u+6D2j5ZtobBadLWcfIROHQ+fenjuhxQaRPRzIpU9PqRJwv3whM5u GJUR+tj84Pk7Sv7sEFB8gPf+FZgCQ+Gdxs0zLDHzEwLln8oiE0wrHqT2HGIwLukR 7Uske1AooC8OxRHoVbNeIQldMRbR8x009CzjIzwQzZBYK+vv5y/8UMWKrIy7VGjT 9QUWgDGj0kSdocGV5Xj/PO0jC5bQ2GDzv0wZ2AJNbtfxE53eazA9aoDZ7iTjy28e kHgZsTSKI6TAtI4k14uowrk/FMANBFVVim3fpJ4kemNpvyyLcF7iUQJSuJVP1bq2 mvwnPSSma7zxU8XJNJdMgYq7tOZIagIUNyf3b3yzLV/MtMB8mvnHApP61uIaZdU7 vXZ1Rq6vE4uF4OHH+sX1NhkD3xSC5ozbvjiw75KKF3hoU8M8hwCXftmQDM6qcDt4 MDE9TmP5NSMaAufqHBcBeZUJzygz5zDNCSRucfJmAr+mawRQTyUZehRKy9R2YrG+ 9933sbKTNHJV/bsHQ6Ukfdj4b3JPa6b//RQgdGnSBNwRkJfXsOs= =5Yez -END PGP SIGNATURE- --- No new revisions were added by this update. Summary of changes:
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new bf5b5df082d Updating config from bot
bf5b5df082d is described below
commit bf5b5df082de72a69f8c5c4db88691d8f06ced54
Author: github-actions
AuthorDate: Wed Oct 11 16:36:12 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/reviewers-for-label-build.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-build.json
b/scripts/ci/pr-bot/state/reviewers-for-label-build.json
index ecf3c0c5d4d..b73c484f395 100644
--- a/scripts/ci/pr-bot/state/reviewers-for-label-build.json
+++ b/scripts/ci/pr-bot/state/reviewers-for-label-build.json
@@ -1,7 +1,7 @@
{
"label": "build",
"dateOfLastReviewAssignment": {
-"damccorm": 1696887969299,
+"damccorm": 1697042167607,
"Abacn": 1696897127404
}
}
\ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 6838a6300d5 Updating config from bot
6838a6300d5 is described below
commit 6838a6300d5fe1e3b429239ec957c6ffb4de7c6b
Author: github-actions
AuthorDate: Wed Oct 11 16:36:15 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-28941.json | 10 ++
1 file changed, 10 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-28941.json
b/scripts/ci/pr-bot/state/pr-state/pr-28941.json
new file mode 100644
index 000..9c51466b9ee
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-28941.json
@@ -0,0 +1,10 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {
+"build": "Abacn"
+ },
+ "nextAction": "Reviewers",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 99ff19b5dce Updating config from bot
99ff19b5dce is described below
commit 99ff19b5dce9aa5d0827082675c13e9be8c9ff90
Author: github-actions
AuthorDate: Wed Oct 11 16:36:17 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/reviewers-for-label-build.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-build.json
b/scripts/ci/pr-bot/state/reviewers-for-label-build.json
index b73c484f395..81c671b12f3 100644
--- a/scripts/ci/pr-bot/state/reviewers-for-label-build.json
+++ b/scripts/ci/pr-bot/state/reviewers-for-label-build.json
@@ -2,6 +2,6 @@
"label": "build",
"dateOfLastReviewAssignment": {
"damccorm": 1697042167607,
-"Abacn": 1696897127404
+"Abacn": 1697042173792
}
}
\ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 38712cdb7e9 Updating config from bot
38712cdb7e9 is described below
commit 38712cdb7e910a545b9b78ad287ad1724614ee9d
Author: github-actions
AuthorDate: Wed Oct 11 16:36:09 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-28943.json | 10 ++
1 file changed, 10 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-28943.json
b/scripts/ci/pr-bot/state/pr-state/pr-28943.json
new file mode 100644
index 000..8a92900947e
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-28943.json
@@ -0,0 +1,10 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {
+"build": "damccorm"
+ },
+ "nextAction": "Reviewers",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch dependabot/go_modules/sdks/github.com/google/go-cmp-0.6.0 deleted (was 73958ad1f0c)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/go_modules/sdks/github.com/google/go-cmp-0.6.0 in repository https://gitbox.apache.org/repos/asf/beam.git was 73958ad1f0c Bump github.com/google/go-cmp from 0.5.9 to 0.6.0 in /sdks The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
[beam] branch master updated (2bd23b16666 -> b82dd4849e4)
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 2bd23b1 Add counter for MLTransform and data processing transforms (#28927) new c62a81b1c57 Reapply "Populate top-level display data in yaml main. (#28512)" new de48383bdca Fix typo in docstring. new b82dd4849e4 Merge pull request #28931 from robertwb/display-data The 40662 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/python/apache_beam/pipeline.py | 28 +-- sdks/python/apache_beam/transforms/display.py | 23 ++ sdks/python/apache_beam/yaml/main.py | 17 +--- 3 files changed, 47 insertions(+), 21 deletions(-)
[beam] branch master updated (b82dd4849e4 -> 7a1245d4011)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from b82dd4849e4 Merge pull request #28931 from robertwb/display-data add 7a1245d4011 Bump github.com/google/go-cmp from 0.5.9 to 0.6.0 in /sdks (#28940) No new revisions were added by this update. Summary of changes: sdks/go.mod | 2 +- sdks/go.sum | 3 ++- 2 files changed, 3 insertions(+), 2 deletions(-)
[beam] branch master updated: Add counter for MLTransform and data processing transforms (#28927)
This is an automated email from the ASF dual-hosted git repository.
anandinguva pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 2bd23b1 Add counter for MLTransform and data processing transforms
(#28927)
2bd23b1 is described below
commit 2bd23b10a5a6fdf78f0615d3cd6a101122f2
Author: Anand Inguva <34158215+ananding...@users.noreply.github.com>
AuthorDate: Wed Oct 11 16:20:31 2023 +
Add counter for MLTransform and data processing transforms (#28927)
---
sdks/python/apache_beam/ml/transforms/base.py | 39 +-
sdks/python/apache_beam/ml/transforms/base_test.py | 25 ++
2 files changed, 63 insertions(+), 1 deletion(-)
diff --git a/sdks/python/apache_beam/ml/transforms/base.py
b/sdks/python/apache_beam/ml/transforms/base.py
index a45928f5c8b..49ce6e9ec1e 100644
--- a/sdks/python/apache_beam/ml/transforms/base.py
+++ b/sdks/python/apache_beam/ml/transforms/base.py
@@ -25,6 +25,7 @@ from typing import Sequence
from typing import TypeVar
import apache_beam as beam
+from apache_beam.metrics.metric import Metrics
__all__ = ['MLTransform', 'ProcessHandler', 'BaseOperation']
@@ -88,6 +89,13 @@ class BaseOperation(Generic[OperationInputT,
OperationOutputT], abc.ABC):
transformed_data = {**transformed_data, **artifacts}
return transformed_data
+ def get_counter(self):
+"""
+Returns the counter name for the operation.
+"""
+counter_name = self.__class__.__name__
+return Metrics.counter(MLTransform, f'BeamML_{counter_name}')
+
class ProcessHandler(Generic[ExampleT, MLTransformOutputT], abc.ABC):
"""
@@ -194,6 +202,9 @@ class
MLTransform(beam.PTransform[beam.PCollection[ExampleT],
transforms=transforms) # type: ignore[arg-type]
self._process_handler = process_handler
+self.transforms = transforms
+self._counter = Metrics.counter(
+MLTransform, f'BeamML_{self.__class__.__name__}')
def expand(
self, pcoll: beam.PCollection[ExampleT]
@@ -209,8 +220,11 @@ class
MLTransform(beam.PTransform[beam.PCollection[ExampleT],
Args:
pcoll: A PCollection of ExampleT type.
Returns:
- A PCollection of MLTransformOutputT type.
+ A PCollection of MLTransformOutputT type
"""
+_ = (
+pcoll.pipeline
+| "MLTransformMetricsUsage" >> MLTransformMetricsUsage(self))
return self._process_handler.process_data(pcoll)
def with_transform(self, transform: BaseOperation):
@@ -230,3 +244,26 @@ class
MLTransform(beam.PTransform[beam.PCollection[ExampleT],
raise TypeError(
'transform must be a subclass of BaseOperation. '
'Got: %s instead.' % type(transform))
+
+
+class MLTransformMetricsUsage(beam.PTransform):
+ def __init__(self, ml_transform: MLTransform):
+self._ml_transform = ml_transform
+self._ml_transform._counter.inc()
+
+ def expand(self, pipeline):
+def _increment_counters():
+ # increment for MLTransform.
+ self._ml_transform._counter.inc()
+ # increment if data processing transforms are passed.
+ transforms = (
+ self._ml_transform.transforms or
+ self._ml_transform._process_handler.transforms)
+ if transforms:
+for transform in transforms:
+ transform.get_counter().inc()
+
+_ = (
+pipeline
+| beam.Create([None])
+| beam.Map(lambda _: _increment_counters()))
diff --git a/sdks/python/apache_beam/ml/transforms/base_test.py
b/sdks/python/apache_beam/ml/transforms/base_test.py
index df7a6d26b47..2e447964541 100644
--- a/sdks/python/apache_beam/ml/transforms/base_test.py
+++ b/sdks/python/apache_beam/ml/transforms/base_test.py
@@ -27,6 +27,7 @@ from parameterized import param
from parameterized import parameterized
import apache_beam as beam
+from apache_beam.metrics.metric import MetricsFilter
from apache_beam.testing.util import assert_that
from apache_beam.testing.util import equal_to
@@ -244,6 +245,30 @@ class BaseMLTransformTest(unittest.TestCase):
equal_to(expected_output_y, equals_fn=np.array_equal),
label='actual_output_y')
+ def test_mltransform_with_counter(self):
+transforms = [
+tft.ComputeAndApplyVocabulary(columns=['y']),
+tft.ScaleTo01(columns=['x'])
+]
+data = [{'x': [1, 2, 3], 'y': ['a', 'b', 'c']}]
+with beam.Pipeline() as p:
+ _ = (
+ p | beam.Create(data)
+ | base.MLTransform(
+ transforms=transforms,
+ write_artifact_location=self.artifact_location))
+scale_to_01_counter = MetricsFilter().with_name('BeamML_ScaleTo01')
+vocab_counter = MetricsFilter().with_name(
+'BeamML_ComputeAndApplyVocabulary')
+mltransform_counter = MetricsFilter().with_name('BeamML_MLTransform')
+result = p.result
+self.assertEqual(
+
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new ebfad352502 Updating config from bot ebfad352502 is described below commit ebfad352502c53d82a782e328e582c4cfa69bc19 Author: github-actions AuthorDate: Wed Oct 11 16:06:36 2023 + Updating config from bot --- scripts/ci/pr-bot/state/reviewers-for-label-java.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-java.json b/scripts/ci/pr-bot/state/reviewers-for-label-java.json index f583b228778..f3c8d7d11b6 100644 --- a/scripts/ci/pr-bot/state/reviewers-for-label-java.json +++ b/scripts/ci/pr-bot/state/reviewers-for-label-java.json @@ -7,6 +7,6 @@ "kileys": 1674428678843, "apilloud": 1678822446183, "Abacn": 1696915675163, -"bvolpato": 1696867593588 +"bvolpato": 1697040392209 } } \ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 0f5954ec7ca Updating config from bot
0f5954ec7ca is described below
commit 0f5954ec7ca82bee88587fb693da0c5e513ec11f
Author: github-actions
AuthorDate: Wed Oct 11 16:06:38 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/reviewers-for-label-io.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-io.json
b/scripts/ci/pr-bot/state/reviewers-for-label-io.json
index fe4cbda84f3..1a2ae7d072d 100644
--- a/scripts/ci/pr-bot/state/reviewers-for-label-io.json
+++ b/scripts/ci/pr-bot/state/reviewers-for-label-io.json
@@ -2,7 +2,7 @@
"label": "io",
"dateOfLastReviewAssignment": {
"chamikaramj": 1696867584636,
-"johnjcasey": 1696592081707,
+"johnjcasey": 1697040392209,
"pabloem": 1691787951165,
"Abacn": 1696594070059,
"ahmedabu98": 1696637915510,
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 9391a569002 Updating config from bot
9391a569002 is described below
commit 9391a56900211712c577c1f61762270e688a2279
Author: github-actions
AuthorDate: Wed Oct 11 16:06:34 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-28863.json | 11 +++
1 file changed, 11 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-28863.json
b/scripts/ci/pr-bot/state/pr-state/pr-28863.json
new file mode 100644
index 000..4628b4b5ad0
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-28863.json
@@ -0,0 +1,11 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {
+"java": "bvolpato",
+"io": "johnjcasey"
+ },
+ "nextAction": "Reviewers",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch master updated (9cdcb036cc9 -> 6a456f04b68)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 9cdcb036cc9 move nested windmill state classes to top level files and organize di… (#28506) add 6a456f04b68 Add user counters and msec metrics to prism UI. (#28929) No new revisions were added by this update. Summary of changes: sdks/go/pkg/beam/runners/prism/internal/web/web.go | 40 +- 1 file changed, 39 insertions(+), 1 deletion(-)
[beam] branch master updated (e56d1a1157b -> 9cdcb036cc9)
This is an automated email from the ASF dual-hosted git repository.
tvalentyn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
from e56d1a1157b Enable missing-timeout pylint check (#28920)
add 9cdcb036cc9 move nested windmill state classes to top level files and
organize di… (#28506)
No new revisions were added by this update.
Summary of changes:
.../dataflow/worker/StreamingDataflowWorker.java |2 +
.../worker/StreamingModeExecutionContext.java | 483 ++--
.../beam/runners/dataflow/worker/Weighers.java |2 +-
.../dataflow/worker/WindmillStateInternals.java| 2830
.../dataflow/worker/streaming/ActiveWorkState.java |2 +-
.../worker/streaming/ComputationState.java |2 +-
.../worker/windmill/state/CachingStateTable.java | 275 ++
.../worker/windmill/state/ConcatIterables.java | 55 +
.../dataflow/worker/windmill/state/IdTracker.java | 253 ++
.../worker/windmill/state/PagingIterable.java | 129 +
.../dataflow/worker/windmill/state/RangeCoder.java | 76 +
.../worker/windmill/state/RangeSetCoder.java | 46 +
.../state/SimpleWindmillState.java}| 33 +-
.../dataflow/worker/windmill/state/StateTag.java | 109 +
.../windmill/state/TimestampedValueWithId.java | 42 +
.../worker/windmill/state/ToIterableFunction.java | 79 +
.../windmill/state/ValuesAndContPosition.java | 46 +
.../state/WeightedList.java} | 45 +-
.../worker/windmill/state/WindmillBag.java | 209 ++
.../windmill/state/WindmillCombiningState.java | 170 ++
.../worker/windmill/state/WindmillMap.java | 449
.../worker/windmill/state/WindmillMultimap.java| 732 +
.../worker/windmill/state/WindmillOrderedList.java | 305 +++
.../worker/windmill/state/WindmillSet.java | 146 +
.../worker/windmill/state/WindmillState.java | 68 +
.../{ => windmill/state}/WindmillStateCache.java | 338 +--
.../windmill/state/WindmillStateInternals.java | 171 ++
.../{ => windmill/state}/WindmillStateReader.java | 530 +---
.../worker/windmill/state/WindmillStateUtil.java | 51 +
.../worker/windmill/state/WindmillValue.java | 160 ++
.../windmill/state/WindmillWatermarkHold.java | 259 ++
.../worker/windmill/state/WrappedFuture.java | 64 +
.../worker/StreamingModeExecutionContextTest.java |2 +
.../worker/streaming/ActiveWorkStateTest.java |2 +-
.../state}/WindmillStateCacheTest.java | 135 +-
.../state}/WindmillStateInternalsTest.java | 282 +-
.../state}/WindmillStateReaderTest.java|6 +-
37 files changed, 4677 insertions(+), 3911 deletions(-)
delete mode 100644
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WindmillStateInternals.java
create mode 100644
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/state/CachingStateTable.java
create mode 100644
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/state/ConcatIterables.java
create mode 100644
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/state/IdTracker.java
create mode 100644
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/state/PagingIterable.java
create mode 100644
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/state/RangeCoder.java
create mode 100644
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/state/RangeSetCoder.java
copy
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/{Weighers.java
=> windmill/state/SimpleWindmillState.java} (50%)
create mode 100644
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/state/StateTag.java
create mode 100644
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/state/TimestampedValueWithId.java
create mode 100644
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/state/ToIterableFunction.java
create mode 100644
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/state/ValuesAndContPosition.java
copy
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/{Weighers.java
=> windmill/state/WeightedList.java} (53%)
create mode 100644
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/state/WindmillBag.java
create mode 100644
[beam] branch master updated (a54afde4a05 -> e56d1a1157b)
This is an automated email from the ASF dual-hosted git repository. jrmccluskey pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from a54afde4a05 Fix missing backslash in arguments of job (#28942) add e56d1a1157b Enable missing-timeout pylint check (#28920) No new revisions were added by this update. Summary of changes: sdks/python/.pylintrc| 1 - .../runners/portability/flink_uber_jar_job_server.py | 4 ++-- .../testing/analyzers/github_issues_utils.py | 20 .../testing/benchmarks/nexmark/nexmark_launcher.py | 3 ++- .../testing/load_tests/load_test_metrics_utils.py| 3 ++- 5 files changed, 22 insertions(+), 9 deletions(-)
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new dbfd33ffc41 Updating config from bot dbfd33ffc41 is described below commit dbfd33ffc41f648f217f6c750510b76845432d1d Author: github-actions AuthorDate: Wed Oct 11 14:43:35 2023 + Updating config from bot --- scripts/ci/pr-bot/state/pr-state/pr-28920.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/pr-state/pr-28920.json b/scripts/ci/pr-bot/state/pr-state/pr-28920.json index 26e4be5dc8a..24ff5131212 100644 --- a/scripts/ci/pr-bot/state/pr-state/pr-28920.json +++ b/scripts/ci/pr-bot/state/pr-state/pr-28920.json @@ -4,7 +4,7 @@ "no-matching-label": "chamikaramj" }, "nextAction": "Reviewers", - "stopReviewerNotifications": false, + "stopReviewerNotifications": true, "remindAfterTestsPass": [], "committerAssigned": false } \ No newline at end of file
[beam] branch master updated (d437afab961 -> a54afde4a05)
This is an automated email from the ASF dual-hosted git repository. yhu pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from d437afab961 Run self-assign and pr-updates only in create event (#28928) add a54afde4a05 Fix missing backslash in arguments of job (#28942) No new revisions were added by this update. Summary of changes: .../beam_PostCommit_Python_ValidatesContainer_Dataflow_With_RC.yml| 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
[beam] branch master updated (62efb967884 -> d437afab961)
This is an automated email from the ASF dual-hosted git repository. yhu pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 62efb967884 Bump golang.org/x/net from 0.16.0 to 0.17.0 in /sdks (#28939) add d437afab961 Run self-assign and pr-updates only in create event (#28928) No new revisions were added by this update. Summary of changes: .github/workflows/pr-bot-pr-updates.yml | 1 + .github/workflows/self-assign.yml | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-)
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new 6c735364c1b Updating config from bot 6c735364c1b is described below commit 6c735364c1b26d7d33f052aa736f2f604e381aea Author: github-actions AuthorDate: Wed Oct 11 14:06:12 2023 + Updating config from bot --- scripts/ci/pr-bot/state/pr-state/pr-28931.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/pr-state/pr-28931.json b/scripts/ci/pr-bot/state/pr-state/pr-28931.json index 959c289c017..8a078ee492b 100644 --- a/scripts/ci/pr-bot/state/pr-state/pr-28931.json +++ b/scripts/ci/pr-bot/state/pr-state/pr-28931.json @@ -6,5 +6,5 @@ "nextAction": "Reviewers", "stopReviewerNotifications": false, "remindAfterTestsPass": [], - "committerAssigned": false + "committerAssigned": true } \ No newline at end of file
[beam] branch dependabot/go_modules/sdks/golang.org/x/net-0.17.0 deleted (was 3f1445fea8c)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/go_modules/sdks/golang.org/x/net-0.17.0 in repository https://gitbox.apache.org/repos/asf/beam.git was 3f1445fea8c Bump golang.org/x/net from 0.16.0 to 0.17.0 in /sdks The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
[beam] branch master updated: Bump golang.org/x/net from 0.16.0 to 0.17.0 in /sdks (#28939)
This is an automated email from the ASF dual-hosted git repository. jrmccluskey pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 62efb967884 Bump golang.org/x/net from 0.16.0 to 0.17.0 in /sdks (#28939) 62efb967884 is described below commit 62efb96788455d8b8cddbe9e60335821bf735f65 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> AuthorDate: Wed Oct 11 09:55:42 2023 -0400 Bump golang.org/x/net from 0.16.0 to 0.17.0 in /sdks (#28939) Bumps [golang.org/x/net](https://github.com/golang/net) from 0.16.0 to 0.17.0. - [Commits](https://github.com/golang/net/compare/v0.16.0...v0.17.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- sdks/go.mod | 2 +- sdks/go.sum | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/sdks/go.mod b/sdks/go.mod index 761798d57c9..e5baa554fe6 100644 --- a/sdks/go.mod +++ b/sdks/go.mod @@ -52,7 +52,7 @@ require ( github.com/xitongsys/parquet-go v1.6.2 github.com/xitongsys/parquet-go-source v0.0.0-20220315005136-aec0fe3e777c go.mongodb.org/mongo-driver v1.12.1 - golang.org/x/net v0.16.0 + golang.org/x/net v0.17.0 golang.org/x/oauth2 v0.13.0 golang.org/x/sync v0.4.0 golang.org/x/sys v0.13.0 diff --git a/sdks/go.sum b/sdks/go.sum index ac862972b2a..f1d0c53d968 100644 --- a/sdks/go.sum +++ b/sdks/go.sum @@ -541,8 +541,8 @@ golang.org/x/net v0.0.0-20201224014010-6772e930b67b/go.mod h1:m0MpNAwzfU5UDzcl9v golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg= golang.org/x/net v0.0.0-2022202133-69e39bad7dc2/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c= -golang.org/x/net v0.16.0 h1:7eBu7KsSvFDtSXUIDbh3aqlK4DPsZ1rByC8PFfBThos= -golang.org/x/net v0.16.0/go.mod h1:NxSsAGuq816PNPmqtQdLE42eU2Fs7NoRIZrHJAlaCOE= +golang.org/x/net v0.17.0 h1:pVaXccu2ozPjCXewfr1S7xza/zcXTity9cCdXQYSjIM= +golang.org/x/net v0.17.0/go.mod h1:NxSsAGuq816PNPmqtQdLE42eU2Fs7NoRIZrHJAlaCOE= golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U= golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new fee6e111c52 Updating config from bot fee6e111c52 is described below commit fee6e111c5296d9d9d3d6202648cee142344d881 Author: github-actions AuthorDate: Wed Oct 11 13:34:50 2023 + Updating config from bot --- scripts/ci/pr-bot/state/pr-state/pr-28929.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/pr-state/pr-28929.json b/scripts/ci/pr-bot/state/pr-state/pr-28929.json index aa3638a74ec..1e9dcd816d5 100644 --- a/scripts/ci/pr-bot/state/pr-state/pr-28929.json +++ b/scripts/ci/pr-bot/state/pr-state/pr-28929.json @@ -6,5 +6,5 @@ "nextAction": "Reviewers", "stopReviewerNotifications": false, "remindAfterTestsPass": [], - "committerAssigned": false + "committerAssigned": true } \ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new 2064cf7cd72 Updating config from bot 2064cf7cd72 is described below commit 2064cf7cd72cb3af0f609ec17247a56059c2cbb1 Author: github-actions AuthorDate: Wed Oct 11 13:34:52 2023 + Updating config from bot --- scripts/ci/pr-bot/state/pr-state/pr-28927.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/pr-state/pr-28927.json b/scripts/ci/pr-bot/state/pr-state/pr-28927.json index c43704aa814..76d5fbcc8d6 100644 --- a/scripts/ci/pr-bot/state/pr-state/pr-28927.json +++ b/scripts/ci/pr-bot/state/pr-state/pr-28927.json @@ -6,5 +6,5 @@ "nextAction": "Reviewers", "stopReviewerNotifications": false, "remindAfterTestsPass": [], - "committerAssigned": false + "committerAssigned": true } \ No newline at end of file
[beam] branch dependabot/go_modules/sdks/google.golang.org/grpc-1.58.3 deleted (was 728e3e1979a)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/go_modules/sdks/google.golang.org/grpc-1.58.3 in repository https://gitbox.apache.org/repos/asf/beam.git was 728e3e1979a Bump google.golang.org/grpc from 1.58.2 to 1.58.3 in /sdks The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
[beam] branch master updated: Bump google.golang.org/grpc from 1.58.2 to 1.58.3 in /sdks (#28938)
This is an automated email from the ASF dual-hosted git repository. riteshghorse pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new e3c1591bbb8 Bump google.golang.org/grpc from 1.58.2 to 1.58.3 in /sdks (#28938) e3c1591bbb8 is described below commit e3c1591bbb8c2b91313267fcd39d77720c768b75 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> AuthorDate: Wed Oct 11 09:27:52 2023 -0400 Bump google.golang.org/grpc from 1.58.2 to 1.58.3 in /sdks (#28938) Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.58.2 to 1.58.3. - [Release notes](https://github.com/grpc/grpc-go/releases) - [Commits](https://github.com/grpc/grpc-go/compare/v1.58.2...v1.58.3) --- updated-dependencies: - dependency-name: google.golang.org/grpc dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- sdks/go.mod | 2 +- sdks/go.sum | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/sdks/go.mod b/sdks/go.mod index 0bc60b630bf..761798d57c9 100644 --- a/sdks/go.mod +++ b/sdks/go.mod @@ -59,7 +59,7 @@ require ( golang.org/x/text v0.13.0 google.golang.org/api v0.146.0 google.golang.org/genproto v0.0.0-20230913181813-007df8e322eb - google.golang.org/grpc v1.58.2 + google.golang.org/grpc v1.58.3 google.golang.org/protobuf v1.31.0 gopkg.in/retry.v1 v1.0.3 gopkg.in/yaml.v2 v2.4.0 diff --git a/sdks/go.sum b/sdks/go.sum index 7743b2bc324..ac862972b2a 100644 --- a/sdks/go.sum +++ b/sdks/go.sum @@ -712,8 +712,8 @@ google.golang.org/grpc v1.26.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8 google.golang.org/grpc v1.27.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk= google.golang.org/grpc v1.27.1/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk= google.golang.org/grpc v1.33.2/go.mod h1:JMHMWHQWaTccqQQlmk3MJZS+GWXOdAesneDmEnv2fbc= -google.golang.org/grpc v1.58.2 h1:SXUpjxeVF3FKrTYQI4f4KvbGD5u2xccdYdurwowix5I= -google.golang.org/grpc v1.58.2/go.mod h1:tgX3ZQDlNJGU96V6yHh1T/JeoBQ2TXdr43YbYSsCJk0= +google.golang.org/grpc v1.58.3 h1:BjnpXut1btbtgN/6sp+brB2Kbm2LjNXnidYujAVbSoQ= +google.golang.org/grpc v1.58.3/go.mod h1:tgX3ZQDlNJGU96V6yHh1T/JeoBQ2TXdr43YbYSsCJk0= google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8= google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ01Woi6D6+Kah6886xMZcty6N08ah7+eCXa0= google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQJ+fmap5saPgwCLgHXTUD7jkjRqWcaiX5VyM=
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 9bc38c78e2c Updating config from bot
9bc38c78e2c is described below
commit 9bc38c78e2c5cb5a2441332c3d66fb8289d1ddc6
Author: github-actions
AuthorDate: Wed Oct 11 12:05:34 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-28942.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-28942.json
b/scripts/ci/pr-bot/state/pr-state/pr-28942.json
new file mode 100644
index 000..9c2aa5aa212
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-28942.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": true,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch master updated (99c87a2c68b -> 88139bdc9a0)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 99c87a2c68b Updates the Transform Service to accept Python extra packages through the Java API (#28783) add 88139bdc9a0 Retrieve workflow runs for GA Post-Commits Status dashboard (#28844) No new revisions were added by this update. Summary of changes: .test-infra/metrics/docker-compose.yml| 5 +- .test-infra/metrics/sync/github/requirements.txt | 6 +- .test-infra/metrics/sync/github/sync_workflows.py | 400 +- 3 files changed, 250 insertions(+), 161 deletions(-)
[beam] branch users/damccorm/slash created (now e5023461153)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch users/damccorm/slash in repository https://gitbox.apache.org/repos/asf/beam.git at e5023461153 Fix missing backslash in arguments of job This branch includes the following new commits: new e5023461153 Fix missing backslash in arguments of job The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[beam] 01/01: Fix missing backslash in arguments of job
This is an automated email from the ASF dual-hosted git repository.
damccorm pushed a commit to branch users/damccorm/slash
in repository https://gitbox.apache.org/repos/asf/beam.git
commit e5023461153f4452333a24f0a6737ffad6ab3fe0
Author: Danny McCormick
AuthorDate: Wed Oct 11 08:03:38 2023 -0400
Fix missing backslash in arguments of job
This is causing repeated failures (e.g.
https://github.com/apache/beam/actions/runs/6478627017/job/17590709062)
---
.../beam_PostCommit_Python_ValidatesContainer_Dataflow_With_RC.yml| 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git
a/.github/workflows/beam_PostCommit_Python_ValidatesContainer_Dataflow_With_RC.yml
b/.github/workflows/beam_PostCommit_Python_ValidatesContainer_Dataflow_With_RC.yml
index 18b57aad805..44df23f0296 100644
---
a/.github/workflows/beam_PostCommit_Python_ValidatesContainer_Dataflow_With_RC.yml
+++
b/.github/workflows/beam_PostCommit_Python_ValidatesContainer_Dataflow_With_RC.yml
@@ -87,10 +87,10 @@ jobs:
with:
gradle-command:
:sdks:python:test-suites:dataflow:py${{steps.set_py_ver_clean.outputs.py_ver_clean}}:validatesContainer
arguments: |
--PtestRCDependencies=true
+-PtestRCDependencies=true \
-PpythonVersion=${{ matrix.python_version }} \
- name: Archive code coverage results
uses: actions/upload-artifact@v3
with:
name: python-code-coverage-report
- path: "**/pytest*.xml"
\ No newline at end of file
+ path: "**/pytest*.xml"
[beam] branch asf-site updated: Publishing website 2023/10/11 06:23:20 at commit 99c87a2
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new dcb2c5a5a4f Publishing website 2023/10/11 06:23:20 at commit 99c87a2 dcb2c5a5a4f is described below commit dcb2c5a5a4f6fb2f2e380591167309f31de67358 Author: runner AuthorDate: Wed Oct 11 06:23:21 2023 + Publishing website 2023/10/11 06:23:20 at commit 99c87a2 --- website/generated-content/sitemap.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/generated-content/sitemap.xml b/website/generated-content/sitemap.xml index b6b57409ead..3635e7a0498 100644 --- a/website/generated-content/sitemap.xml +++ b/website/generated-content/sitemap.xml @@ -1 +1 @@ -http://www.sitemaps.org/schemas/sitemap/0.9; xmlns:xhtml="http://www.w3.org/1999/xhtml;>/categories/blog/2023-10-10T16:15:11-04:00/blog/2023-10-10T16:15:11-04:00/categories/2023-10-10T16:15:11-04:00/blog/dyi-content-discovery-platform-genai-beam/2023-10-10T16:15:11-04:00http://www.sitemaps.org/schemas/sitemap/0.9; xmlns:xhtml="http://www.w3.org/1999/xhtml;>/categories/blog/2023-10-10T21:09:58-07:00/blog/2023-10-10T21:09:58-07:00/categories/2023-10-10T21:09:58-07:00/blog/dyi-content-discovery-platform-genai-beam/2023-10-10T21:09:58-07:00
