This is an automated email from the ASF dual-hosted git repository.
Amar3tto pushed a change to branch python-arm-timeout
in repository https://gitbox.apache.org/repos/asf/beam.git
discard 402ae22d211 Increase timeout
add de688c0a199 Close harness data/state channels and don't Recv on Send
EOF (#40262)
add ce221883440 fix performance plot (#40366)
add 985c8fa0a01 Migrate 7 pilot CI workflows from pull_request_target to
pull_request (#40360)
add 8e1e7b3b5fd Temporarily set Bigtable client version upper bound
(#40364)
add e8c94d72a35 Remove outdated pull_request_target permission comments
from workflows (#40371)
add 4555b245462 Remove redundant google-github-actions/auth steps from
self-hosted PR workflows (#40378)
add 7c680c00350 Merge pull request #40367 from
reuvenlax/remove_javadoc_from_java_precommit
add 355e8bd48b5 Expose desiredNumUnboundedSourceSplits for Java and Python
portable pipeline (#40345)
add adc62a984fd Migrate PreCommit workflows from pull_request_target to
pull_request (#40372)
add 9a464d59c3a Set ALLOYDB_PASSWORD directly in test workflows for
sandboxed test DB (#40381)
add 929359adba9 Enforce TLS 1.2+ on ARC webhook GCE Ingress (#40382)
add 086ab9bc601 Merge pull request #40324 from
reuvenlax/improve_test_presubmit
add 5659bb7f826 Migrate PostCommit workflows from pull_request_target to
pull_request (#40373)
add f612c6a13cd Port Wait.On to the Go SDK (#39937)
add b372a883cc5 Add check_run fork guard to beam_PreCommit_Python_Dill.yml
add ea74d0783ca Merge pull request #40383 from
damccorm/fix-python-dill-check-run
add 122d4129ee1 Fix JaCoCo coverage for multiple test tasks
add 2e27eb0c8e0 Merge pull request #40164 from singhmrinal02/master
add faf253a519f [Java] Reduce logging severity for state requests
cancelled by the runner (#40302)
add 4a8296292d1 Migrate secret-using workflows from pull_request_target to
pull_request (#40375)
add 4babf6b7af8 normalize BigQuery CDC
add d945a0f92b9 Merge pull request #40177 from
aIbrahiim/normalize-bigquery-cdc-write
add 24d48c984ae [Python] Fix stateful ParDo taking the timer key coder
from a side input (#40380)
add 4203bab5c92 Remove comment-based workflow run and rerun triggers
(#40388)
add 15a7ce17e1f IcebergIO and AddFiles multi bucket catalog validation
(#39907)
add 69bdf5f2917 Update Jetty to 9.4.58.v20250814 and exclude shaded Jetty
pom metadata in Dataflow worker (#40389)
add 464db0171e9 [Python] Round float seconds to the nearest microsecond in
Duration (#40236)
add 5a218f6c241 assign reviewers immediately when PR is marked ready for
review (#40400)
add dedbf9c8522 Prevent CommitTooLargeException in OrderedEventProcessor
on duplicate floods (#40242)
add 976f0522030 Deprecate ParquetIO.withProjection (#40043)
add 041bd00e4f6 Blog post: Beam modernization at project 10-year mark
(#40368)
add 7711c5345ad bump code coverage to v7 (#40416)
add f6c9c882e88 Bump github.com/moby/moby/api from 1.56.0 to 1.56.1 in
/sdks (#40407)
add 9f78aaf6fee Bump github.com/aws/aws-sdk-go-v2/feature/s3/manager in
/sdks (#40411)
add 7083a3ff0ce Bump github.com/moby/moby/client from 0.6.0 to 0.6.1 in
/sdks (#40408)
add d22c3c32fdb Bump google.golang.org/api from 0.299.0 to 0.300.0 in
/sdks (#40410)
add e2c9256725c [KafkaIO] Add `with_gcp_adc` and `num_partitions` options
to Kafka SchemaTransforms and YAML (#40181)
add 662b91c3d62 Fix exception masking in FileSystems.matchSingleFileSpec
add 782b97c7cdc Replace Mockito FileSystem mock with native FakeFileSystem
in tests
add c7a9bb77f17 Merge pull request #40109: Fix exception masking in
FileSystems.matchSingleFileSpec
add 261f05f142b [Python] Fix LogElements crash on global window timestamps
(#40332)
add c74609d71ee [Python] Support Lakehouse runtime catalog tables in
ReadFromBigQuery… (#40225)
add 0b6d68278b8 [KafkaIO] Use consumer position and lag to estimate end
offsets in KafkaUnboundedReader (#39830)
add ac516686b9f Bump github.com/proullon/ramsql from 0.1.4 to 0.1.5 in
/sdks (#40425)
add c5c59f1da69 Bump cloud.google.com/go/storage from 1.68.0 to 1.69.0 in
/sdks (#40426)
add 63a3479229b Update self-hosted runner versions to meet GitHub's
minimum runner version (#40403)
add d803b846618 [Python] Remove leftover "### Labels" debug log in the
BigQuery source (#40430)
add b456c68217d [Go SDK] Use deterministic protobuf marshaling in protox
helpers (#40385)
add b1b914b8bd0 Update Grafana version to 12.4.12 (#40434)
add 6fe144b3768 [BEAM-40264] Fix Spark MapState and SetState isEmpty after
removals (#40398)
add 32d54e821a6 [Spark] Run ValidatesRunner tests serially, runner metrics
are JVM wide (#40428)
add dfc4d667cd9 [Spark 4] Support stateful ParDo in Structured Streaming
via transformWithState (#40281)
add c164b38e924 Add a security model for Beam (#40421)
add 2485e5dcfe9 Fix Grafana alerting
add bf63b3b3aad Merge pull request #40445 from apache/update-grafana
add 81e55c1a997 Adding release-2.77.0-postrelease to protected branches in
.asf.yaml
add 540f68e3de9 fix bigtable schema issue (#40025)
add 6e305537dd1 Update Beam website to release 2.77.0
add 8cae0843559 Update release date
add c0dc5668e75 Merge pull request #40234 from apache/website-2-77
add 5aa894f3190 [Gemini] Prototype OpenAI RemoteInferenceModelHandler
Implementation (#40239)
add bde760f1995 Update managed-io.md for release 2.77.0-RC4.
add a2760a201c5 Remove whitespace
add 697a4da9667 Merge pull request #40369 from
apache/updates_managed_io_docs_2.77.0_rc4
add be4211d1d24 [Go SDK] Recreate a failed data/state channel without
holding the lock (#40415)
add 274905845dd Add examples for cloudsql (#40027)
add e26d6bacb93 Parallelize per-file rewrites in GcsUtilV2 copy and move
(#40376)
add 0291da13e0f use binding (#40455)
add 5cb33da4830 [Java] Remove CT_CONSTRUCTOR_THROW from the global
SpotBugs filter (#40008)
add 5394822d017 Remove usage of a (leaking) shared handles, which can also
accidentally release the last reference hodling a model in
_SharedMap._keepalive when all DoFns have been gc'ed. (#40441)
add 09fbdf9acdf fix leader board trigger
add d413595e717 trigger workflow
add da910f3bb09 fix leader board triggers
add 05725753096 shorten leader board team trigger delays
add c275c97b2e0 trigger workflow
add 76f6e6d7c0c fire leader board user scores every 10 seconds
add b7cfa6c95a2 Merge pull request #40370 from
aIbrahiim/fix-leaderboard-processing-time-triggers
add fd2bb97d76a Simplify pull request template (#40339)
add 483f14b6dd4 [Python] Isolate async work by key and window (#39996)
add 0f106b32122 [Python] Don't skip a delimiter right after an escaped one
in ReadFromText (#40460)
add 93a84b6a007 Remove kafka reviewers label (#40464)
add c5d866c50fe Let SortAndBatchElements batches reach max_batch_weight
(#40462)
add 7a5f89199d5 [Claude] Plumb through pipeline-level resource hints to
cross-lanugage transforms (#40379)
add af5d596ddbc Remove unused net.researchgate.release Gradle plugin
(#40465)
add 22abffca3a4 [Go SDK] Encode INT16 row fields as big endian to match
Java and Python (#40152)
add debfa0e6088 [#40250] Remove support for Python 3.10 (#40413)
add dabcf50ffbf [YAML] Fix flaky ErrorHandlingTest: don't pickle Scope
into user DoFn closures (#40467)
add 627d7fbb3fd Reduce Gradle build and test log spam in Java CI workflows
(#40458)
add 69bfcacf0be Make AsyncWrapperTest deterministic (#40448) (#40449)
add ddcd6544642 Add FlareDB to Beam runner docs (#40435)
add 40852a71e00 Update JmsIO to ActiveMQ 6.2.5 and jakarta.jms (#38729)
add 0dd464f9801 remove redundant Thank you (#40471)
add a2df8610996 Deflake MqttIOTest read tests (#40393)
add c692c225d97 add redistribute kafka io parameters (#40396)
add fadfc789376 Removes usages of the sentence-transformers package...
(#40439)
new 920f0d4a3e8 Increase timeout
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (402ae22d211)
\
N -- N -- N refs/heads/python-arm-timeout (920f0d4a3e8)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.asf.yaml | 1 +
.github/ACTIONS.md | 14 +-
.github/PULL_REQUEST_TEMPLATE.md | 22 +-
.github/REVIEWERS.yml | 6 +-
.github/actions/rerun-job-action/action.yml | 125 --
.github/actions/setup-action/action.yml | 46 -
.../test-properties.json | 16 +-
.../actions/setup-environment-action/action.yml | 4 +-
.github/actions/setup-k8s-access/action.yml | 4 -
.github/build.gradle | 6 +-
...arc_certificate.tpl => arc_frontend_config.tpl} | 9 +-
.github/gh-actions-self-hosted-runners/arc/gke.tf | 7 +
.../arc/kubernetes.tf | 6 +
.../gh-actions-self-hosted-runners/arc/locals.tf | 1 +
.../arc/variables.tf | 4 +-
.../self-hosted-linux/docker/Dockerfile | 2 +-
.../IO_Iceberg_Integration_Tests.json | 2 +-
.../beam_PostCommit_Java_PVR_Spark3_Streaming.json | 2 +-
.../beam_PostCommit_Java_PVR_Spark4_Batch.json | 3 +-
.../beam_PostCommit_Java_PVR_Spark4_Streaming.json | 3 +-
...Commit_Java_PVR_Spark4_StructuredStreaming.json | 2 +-
.../beam_PostCommit_Java_PVR_Spark_Batch.json | 2 +-
...beam_PostCommit_Java_ValidatesRunner_Spark.json | 2 +-
...eam_PostCommit_Java_ValidatesRunner_Spark4.json | 2 +-
...a_ValidatesRunner_SparkStructuredStreaming.json | 4 +-
.github/trigger_files/beam_PostCommit_Python.json | 2 +-
.../beam_PostCommit_Yaml_Xlang_Direct.json | 2 +-
.github/workflows/IO_Iceberg_Integration_Tests.yml | 18 +-
.../IO_Iceberg_Integration_Tests_Dataflow.yml | 18 +-
..._Iceberg_Managed_Integration_Tests_Dataflow.yml | 18 +-
.github/workflows/IO_Iceberg_Performance_Tests.yml | 18 +-
.github/workflows/IO_Iceberg_Unit_Tests.yml | 27 +-
.github/workflows/README.md | 241 ++--
.github/workflows/beam_CancelStaleDataflowJobs.yml | 14 +-
.../workflows/beam_CleanUpDataprocResources.yml | 4 +-
.github/workflows/beam_CleanUpGCPResources.yml | 14 +-
.../workflows/beam_CleanUpPrebuiltSDKImages.yml | 14 +-
.../workflows/beam_CloudML_Benchmarks_Dataflow.yml | 17 +-
.../beam_IODatastoresCredentialsRotation.yml | 9 +-
.../beam_Inference_Python_Benchmarks_Dataflow.yml | 72 +-
.../beam_Infrastructure_PolicyEnforcer.yml | 4 +-
.../beam_Infrastructure_SecurityLogging.yml | 2 +-
.../beam_Infrastructure_ServiceAccountKeys.yml | 2 +-
.../beam_Infrastructure_UsersPermissions.yml | 5 +-
.github/workflows/beam_Java_JMH.yml | 6 +-
.../beam_Java_LoadTests_Combine_Smoke.yml | 14 +-
.../beam_LoadTests_Go_CoGBK_Dataflow_Batch.yml | 14 +-
.../beam_LoadTests_Go_CoGBK_Flink_batch.yml | 14 +-
.../beam_LoadTests_Go_Combine_Dataflow_Batch.yml | 14 +-
.../beam_LoadTests_Go_Combine_Flink_Batch.yml | 14 +-
.../beam_LoadTests_Go_GBK_Dataflow_Batch.yml | 14 +-
.../beam_LoadTests_Go_GBK_Flink_Batch.yml | 14 +-
.../beam_LoadTests_Go_ParDo_Dataflow_Batch.yml | 14 +-
.../beam_LoadTests_Go_ParDo_Flink_Batch.yml | 14 +-
.../beam_LoadTests_Go_SideInput_Dataflow_Batch.yml | 14 +-
.../beam_LoadTests_Go_SideInput_Flink_Batch.yml | 14 +-
.../beam_LoadTests_Java_CoGBK_Dataflow_Batch.yml | 14 +-
...eam_LoadTests_Java_CoGBK_Dataflow_Streaming.yml | 14 +-
...s_Java_CoGBK_Dataflow_V2_Batch_JavaVersions.yml | 16 +-
...va_CoGBK_Dataflow_V2_Streaming_JavaVersions.yml | 16 +-
...s_Java_CoGBK_SparkStructuredStreaming_Batch.yml | 14 +-
.../beam_LoadTests_Java_Combine_Dataflow_Batch.yml | 14 +-
...m_LoadTests_Java_Combine_Dataflow_Streaming.yml | 14 +-
...Java_Combine_SparkStructuredStreaming_Batch.yml | 14 +-
.../beam_LoadTests_Java_GBK_Dataflow_Batch.yml | 14 +-
.../beam_LoadTests_Java_GBK_Dataflow_Streaming.yml | 14 +-
.../beam_LoadTests_Java_GBK_Dataflow_V2_Batch.yml | 14 +-
...LoadTests_Java_GBK_Dataflow_V2_Batch_Java17.yml | 14 +-
...am_LoadTests_Java_GBK_Dataflow_V2_Streaming.yml | 14 +-
...Tests_Java_GBK_Dataflow_V2_Streaming_Java17.yml | 14 +-
.../workflows/beam_LoadTests_Java_GBK_Smoke.yml | 14 +-
...sts_Java_GBK_SparkStructuredStreaming_Batch.yml | 14 +-
.../beam_LoadTests_Java_ParDo_Dataflow_Batch.yml | 14 +-
...eam_LoadTests_Java_ParDo_Dataflow_Streaming.yml | 14 +-
...s_Java_ParDo_Dataflow_V2_Batch_JavaVersions.yml | 16 +-
...va_ParDo_Dataflow_V2_Streaming_JavaVersions.yml | 16 +-
...s_Java_ParDo_SparkStructuredStreaming_Batch.yml | 14 +-
.github/workflows/beam_LoadTests_Java_PubsubIO.yml | 14 +-
.../beam_LoadTests_Python_CoGBK_Dataflow_Batch.yml | 22 +-
...m_LoadTests_Python_CoGBK_Dataflow_Streaming.yml | 22 +-
.../beam_LoadTests_Python_CoGBK_Flink_Batch.yml | 20 +-
...eam_LoadTests_Python_Combine_Dataflow_Batch.yml | 20 +-
...LoadTests_Python_Combine_Dataflow_Streaming.yml | 20 +-
.../beam_LoadTests_Python_Combine_Flink_Batch.yml | 20 +-
...am_LoadTests_Python_Combine_Flink_Streaming.yml | 18 +-
...LoadTests_Python_FnApiRunner_Microbenchmark.yml | 16 +-
.../beam_LoadTests_Python_GBK_Dataflow_Batch.yml | 24 +-
...eam_LoadTests_Python_GBK_Dataflow_Streaming.yml | 16 +-
.../beam_LoadTests_Python_GBK_Flink_Batch.yml | 14 +-
...adTests_Python_GBK_reiterate_Dataflow_Batch.yml | 18 +-
...sts_Python_GBK_reiterate_Dataflow_Streaming.yml | 18 +-
.../beam_LoadTests_Python_ParDo_Dataflow_Batch.yml | 22 +-
...m_LoadTests_Python_ParDo_Dataflow_Streaming.yml | 22 +-
.../beam_LoadTests_Python_ParDo_Flink_Batch.yml | 20 +-
...beam_LoadTests_Python_ParDo_Flink_Streaming.yml | 24 +-
...m_LoadTests_Python_SideInput_Dataflow_Batch.yml | 34 +-
.github/workflows/beam_LoadTests_Python_Smoke.yml | 18 +-
.../workflows/beam_MetricsCredentialsRotation.yml | 9 +-
.github/workflows/beam_Metrics_Report.yml | 4 +-
.../workflows/beam_PerformanceTests_AvroIOIT.yml | 14 +-
.../beam_PerformanceTests_AvroIOIT_HDFS.yml | 14 +-
...PerformanceTests_BigQueryIO_Batch_Java_Avro.yml | 18 +-
...PerformanceTests_BigQueryIO_Batch_Java_Json.yml | 18 +-
..._PerformanceTests_BigQueryIO_Streaming_Java.yml | 18 +-
...eam_PerformanceTests_BiqQueryIO_Read_Python.yml | 16 +-
...formanceTests_BiqQueryIO_Write_Python_Batch.yml | 16 +-
.github/workflows/beam_PerformanceTests_Cdap.yml | 14 +-
.../beam_PerformanceTests_Compressed_TextIOIT.yml | 14 +-
...m_PerformanceTests_Compressed_TextIOIT_HDFS.yml | 14 +-
.../beam_PerformanceTests_HadoopFormat.yml | 14 +-
.github/workflows/beam_PerformanceTests_JDBC.yml | 14 +-
.../workflows/beam_PerformanceTests_Kafka_IO.yml | 14 +-
.../beam_PerformanceTests_ManyFiles_TextIOIT.yml | 14 +-
...am_PerformanceTests_ManyFiles_TextIOIT_HDFS.yml | 14 +-
.../beam_PerformanceTests_MongoDBIO_IT.yml | 14 +-
.../beam_PerformanceTests_ParquetIOIT.yml | 14 +-
.../beam_PerformanceTests_ParquetIOIT_HDFS.yml | 14 +-
...erformanceTests_PubsubIOIT_Python_Streaming.yml | 16 +-
...m_PerformanceTests_SQLBigQueryIO_Batch_Java.yml | 18 +-
.../beam_PerformanceTests_SingleStoreIO.yml | 14 +-
..._PerformanceTests_SpannerIO_Read_2GB_Python.yml | 16 +-
...manceTests_SpannerIO_Write_2GB_Python_Batch.yml | 16 +-
.../beam_PerformanceTests_SparkReceiver_IO.yml | 14 +-
.../beam_PerformanceTests_TFRecordIOIT.yml | 14 +-
.../beam_PerformanceTests_TFRecordIOIT_HDFS.yml | 14 +-
.../workflows/beam_PerformanceTests_TextIOIT.yml | 14 +-
.../beam_PerformanceTests_TextIOIT_HDFS.yml | 14 +-
.../beam_PerformanceTests_TextIOIT_Python.yml | 16 +-
...PerformanceTests_WordCountIT_PythonVersions.yml | 22 +-
.../workflows/beam_PerformanceTests_XmlIOIT.yml | 14 +-
.../beam_PerformanceTests_XmlIOIT_HDFS.yml | 14 +-
.../beam_PerformanceTests_xlang_KafkaIO_Python.yml | 16 +-
.github/workflows/beam_Playground_CI_Nightly.yml | 8 +-
.github/workflows/beam_Playground_Precommit.yml | 16 +-
.github/workflows/beam_PostCommit_Go.yml | 18 +-
.../workflows/beam_PostCommit_Go_Dataflow_ARM.yml | 20 +-
.github/workflows/beam_PostCommit_Go_VR_Flink.yml | 18 +-
.github/workflows/beam_PostCommit_Go_VR_Spark.yml | 18 +-
.github/workflows/beam_PostCommit_Java.yml | 23 +-
.../beam_PostCommit_Java_Avro_Versions.yml | 23 +-
.../beam_PostCommit_Java_BigQueryEarlyRollout.yml | 23 +-
.../workflows/beam_PostCommit_Java_DataflowV1.yml | 23 +-
.../workflows/beam_PostCommit_Java_DataflowV2.yml | 23 +-
.../beam_PostCommit_Java_Delta_IO_Dataflow.yml | 18 +-
.../beam_PostCommit_Java_Examples_Dataflow.yml | 20 +-
.../beam_PostCommit_Java_Examples_Dataflow_ARM.yml | 23 +-
...beam_PostCommit_Java_Examples_Dataflow_Java.yml | 23 +-
.../beam_PostCommit_Java_Examples_Dataflow_V2.yml | 23 +-
...m_PostCommit_Java_Examples_Dataflow_V2_Java.yml | 25 +-
.../beam_PostCommit_Java_Examples_Direct.yml | 23 +-
.../beam_PostCommit_Java_Examples_Flink.yml | 23 +-
.../beam_PostCommit_Java_Examples_Spark.yml | 23 +-
.../beam_PostCommit_Java_Hadoop_Versions.yml | 23 +-
.../beam_PostCommit_Java_IO_Performance_Tests.yml | 25 +-
.../beam_PostCommit_Java_InfluxDbIO_IT.yml | 16 +-
.../beam_PostCommit_Java_Jpms_Dataflow.yml | 23 +-
...beam_PostCommit_Java_Jpms_Dataflow_Versions.yml | 23 +-
.../workflows/beam_PostCommit_Java_Jpms_Direct.yml | 23 +-
.../beam_PostCommit_Java_Jpms_Direct_Versions.yml | 23 +-
.../beam_PostCommit_Java_Jpms_Flink_Java11.yml | 23 +-
.../beam_PostCommit_Java_Jpms_Spark_Java11.yml | 23 +-
.../beam_PostCommit_Java_Nexmark_Dataflow.yml | 18 +-
.../beam_PostCommit_Java_Nexmark_Dataflow_V2.yml | 18 +-
...am_PostCommit_Java_Nexmark_Dataflow_V2_Java.yml | 20 +-
.../beam_PostCommit_Java_Nexmark_Direct.yml | 18 +-
.../beam_PostCommit_Java_Nexmark_Flink.yml | 18 +-
.../beam_PostCommit_Java_Nexmark_Spark.yml | 18 +-
.../beam_PostCommit_Java_PVR_Flink_Batch.yml | 16 +-
.../beam_PostCommit_Java_PVR_Flink_Streaming.yml | 21 +-
.../beam_PostCommit_Java_PVR_Spark3_Streaming.yml | 23 +-
.../beam_PostCommit_Java_PVR_Spark4_Batch.yml | 20 +-
.../beam_PostCommit_Java_PVR_Spark4_Streaming.yml | 23 +-
...tCommit_Java_PVR_Spark4_StructuredStreaming.yml | 23 +-
.../beam_PostCommit_Java_PVR_Spark_Batch.yml | 20 +-
.../beam_PostCommit_Java_SingleStoreIO_IT.yml | 16 +-
.../beam_PostCommit_Java_Tpcds_Dataflow.yml | 18 +-
.../workflows/beam_PostCommit_Java_Tpcds_Flink.yml | 18 +-
.../workflows/beam_PostCommit_Java_Tpcds_Spark.yml | 18 +-
...am_PostCommit_Java_ValidatesRunner_Dataflow.yml | 23 +-
..._Java_ValidatesRunner_Dataflow_JavaVersions.yml | 23 +-
...mit_Java_ValidatesRunner_Dataflow_Streaming.yml | 23 +-
...a_ValidatesRunner_Dataflow_Streaming_Engine.yml | 23 +-
...atesRunner_Dataflow_Streaming_TagEncodingV2.yml | 23 +-
...PostCommit_Java_ValidatesRunner_Dataflow_V2.yml | 23 +-
..._Java_ValidatesRunner_Dataflow_V2_Streaming.yml | 28 +-
...beam_PostCommit_Java_ValidatesRunner_Direct.yml | 23 +-
...it_Java_ValidatesRunner_Direct_JavaVersions.yml | 23 +-
.../beam_PostCommit_Java_ValidatesRunner_Flink.yml | 21 +-
.../beam_PostCommit_Java_ValidatesRunner_Spark.yml | 23 +-
...beam_PostCommit_Java_ValidatesRunner_Spark4.yml | 23 +-
...va_ValidatesRunner_SparkStructuredStreaming.yml | 23 +-
...am_PostCommit_Java_ValidatesRunner_Twister2.yml | 23 +-
.../beam_PostCommit_Java_ValidatesRunner_ULR.yml | 23 +-
.github/workflows/beam_PostCommit_Javadoc.yml | 18 +-
.../beam_PostCommit_PortableJar_Flink.yml | 25 +-
.../beam_PostCommit_PortableJar_Spark.yml | 25 +-
.github/workflows/beam_PostCommit_Python.yml | 28 +-
.github/workflows/beam_PostCommit_Python_Arm.yml | 23 +-
.../beam_PostCommit_Python_Dependency.yml | 25 +-
.../beam_PostCommit_Python_Examples_Dataflow.yml | 21 +-
.../beam_PostCommit_Python_Examples_Direct.yml | 23 +-
.../beam_PostCommit_Python_Examples_Flink.yml | 23 +-
.../beam_PostCommit_Python_Examples_Spark.yml | 23 +-
.../beam_PostCommit_Python_MongoDBIO_IT.yml | 21 +-
.../beam_PostCommit_Python_Nexmark_Direct.yml | 20 +-
.../beam_PostCommit_Python_Portable_Flink.yml | 26 +-
...stCommit_Python_ValidatesContainer_Dataflow.yml | 30 +-
..._Python_ValidatesContainer_Dataflow_With_RC.yml | 28 +-
..._PostCommit_Python_ValidatesRunner_Dataflow.yml | 23 +-
...eam_PostCommit_Python_ValidatesRunner_Flink.yml | 23 +-
...eam_PostCommit_Python_ValidatesRunner_Spark.yml | 23 +-
.../workflows/beam_PostCommit_Python_Versions.yml | 20 +-
.../beam_PostCommit_Python_Xlang_Gcp_Dataflow.yml | 28 +-
.../beam_PostCommit_Python_Xlang_Gcp_Direct.yml | 28 +-
.../beam_PostCommit_Python_Xlang_IO_Dataflow.yml | 25 +-
.../beam_PostCommit_Python_Xlang_IO_Direct.yml | 25 +-
...am_PostCommit_Python_Xlang_Messaging_Direct.yml | 25 +-
.github/workflows/beam_PostCommit_SQL.yml | 23 +-
.../beam_PostCommit_TransformService_Direct.yml | 27 +-
.github/workflows/beam_PostCommit_Website_Test.yml | 18 +-
.github/workflows/beam_PostCommit_XVR_Direct.yml | 27 +-
.github/workflows/beam_PostCommit_XVR_Flink.yml | 27 +-
.../beam_PostCommit_XVR_GoUsingJava_Dataflow.yml | 23 +-
...eam_PostCommit_XVR_JavaUsingPython_Dataflow.yml | 27 +-
..._PostCommit_XVR_PythonUsingJavaSQL_Dataflow.yml | 23 +-
...eam_PostCommit_XVR_PythonUsingJava_Dataflow.yml | 27 +-
.github/workflows/beam_PostCommit_XVR_Spark3.yml | 27 +-
.../beam_PostCommit_Yaml_Xlang_Direct.yml | 20 +-
.../workflows/beam_PostRelease_NightlySnapshot.yml | 2 +-
.../workflows/beam_PreCommit_CommunityMetrics.yml | 20 +-
.../workflows/beam_PreCommit_Flink_Container.yml | 26 +-
.github/workflows/beam_PreCommit_GHA.yml | 20 +-
.github/workflows/beam_PreCommit_Go.yml | 20 +-
.github/workflows/beam_PreCommit_GoPortable.yml | 20 +-
.github/workflows/beam_PreCommit_GoPrism.yml | 20 +-
.github/workflows/beam_PreCommit_ItFramework.yml | 25 +-
.github/workflows/beam_PreCommit_Java.yml | 29 +-
...eCommit_Java_Amazon-Web-Services2_IO_Direct.yml | 27 +-
.../beam_PreCommit_Java_Amqp_IO_Direct.yml | 27 +-
.../beam_PreCommit_Java_Azure_IO_Direct.yml | 27 +-
.../beam_PreCommit_Java_Cassandra_IO_Direct.yml | 27 +-
.../beam_PreCommit_Java_Cdap_IO_Direct.yml | 27 +-
.../beam_PreCommit_Java_Clickhouse_IO_Direct.yml | 27 +-
.../beam_PreCommit_Java_Csv_IO_Direct.yml | 27 +-
.../beam_PreCommit_Java_Datadog_IO_Direct.yml | 27 +-
.github/workflows/beam_PreCommit_Java_Dataflow.yml | 27 +-
.../beam_PreCommit_Java_Debezium_IO_Direct.yml | 27 +-
.../beam_PreCommit_Java_Delta_IO_Direct.yml | 27 +-
...beam_PreCommit_Java_ElasticSearch_IO_Direct.yml | 27 +-
.../beam_PreCommit_Java_Examples_Dataflow.yml | 25 +-
...eam_PreCommit_Java_Examples_Dataflow_Java11.yml | 25 +-
...Commit_Java_File-schema-transform_IO_Direct.yml | 27 +-
.../beam_PreCommit_Java_Flink_Versions.yml | 19 +-
.../beam_PreCommit_Java_GCP_IO_Direct.yml | 27 +-
.../beam_PreCommit_Java_Google-ads_IO_Direct.yml | 27 +-
.../beam_PreCommit_Java_HBase_IO_Direct.yml | 27 +-
.../beam_PreCommit_Java_HCatalog_IO_Direct.yml | 27 +-
.../beam_PreCommit_Java_Hadoop_IO_Direct.yml | 27 +-
.../workflows/beam_PreCommit_Java_IOs_Direct.yml | 27 +-
.../beam_PreCommit_Java_InfluxDb_IO_Direct.yml | 27 +-
.../beam_PreCommit_Java_JDBC_IO_Direct.yml | 27 +-
.../beam_PreCommit_Java_Jms_IO_Direct.yml | 27 +-
.../beam_PreCommit_Java_Kafka_IO_Direct.yml | 27 +-
.../beam_PreCommit_Java_Kafka_Streams_Runner.yml | 27 +-
.../beam_PreCommit_Java_Kudu_IO_Direct.yml | 27 +-
.../beam_PreCommit_Java_MongoDb_IO_Direct.yml | 27 +-
.../beam_PreCommit_Java_Mqtt_IO_Direct.yml | 27 +-
.../beam_PreCommit_Java_Neo4j_IO_Direct.yml | 27 +-
.../beam_PreCommit_Java_PVR_Flink_Batch.yml | 20 +-
.../beam_PreCommit_Java_PVR_Flink_Docker.yml | 25 +-
.../beam_PreCommit_Java_PVR_Prism_Loopback.yml | 20 +-
.../beam_PreCommit_Java_Parquet_IO_Direct.yml | 27 +-
.../beam_PreCommit_Java_Pulsar_IO_Direct.yml | 27 +-
.../beam_PreCommit_Java_RabbitMq_IO_Direct.yml | 27 +-
.../beam_PreCommit_Java_Redis_IO_Direct.yml | 27 +-
...am_PreCommit_Java_RequestResponse_IO_Direct.yml | 27 +-
.../beam_PreCommit_Java_SingleStore_IO_Direct.yml | 27 +-
.../beam_PreCommit_Java_Snowflake_IO_Direct.yml | 27 +-
.../beam_PreCommit_Java_Solace_IO_Direct.yml | 27 +-
.../beam_PreCommit_Java_Solr_IO_Direct.yml | 27 +-
.../beam_PreCommit_Java_Spark_Versions.yml | 25 +-
.../beam_PreCommit_Java_Splunk_IO_Direct.yml | 27 +-
.../beam_PreCommit_Java_Thrift_IO_Direct.yml | 27 +-
.../beam_PreCommit_Java_Tika_IO_Direct.yml | 27 +-
.../workflows/beam_PreCommit_Kotlin_Examples.yml | 20 +-
.../workflows/beam_PreCommit_Portable_Python.yml | 24 +-
.github/workflows/beam_PreCommit_Prism_Python.yml | 24 +-
.github/workflows/beam_PreCommit_Python.yml | 27 +-
.github/workflows/beam_PreCommit_PythonDocker.yml | 17 +-
.github/workflows/beam_PreCommit_PythonDocs.yml | 20 +-
.../workflows/beam_PreCommit_PythonFormatter.yml | 20 +-
.github/workflows/beam_PreCommit_PythonLint.yml | 20 +-
.../workflows/beam_PreCommit_Python_Coverage.yml | 31 +-
.../workflows/beam_PreCommit_Python_Dataframes.yml | 27 +-
.github/workflows/beam_PreCommit_Python_Dill.yml | 25 +-
.../workflows/beam_PreCommit_Python_Examples.yml | 30 +-
.../beam_PreCommit_Python_Integration.yml | 27 +-
.github/workflows/beam_PreCommit_Python_ML.yml | 32 +-
.../workflows/beam_PreCommit_Python_PVR_Flink.yml | 25 +-
.../workflows/beam_PreCommit_Python_Runners.yml | 27 +-
.../workflows/beam_PreCommit_Python_Transforms.yml | 30 +-
.github/workflows/beam_PreCommit_RAT.yml | 20 +-
.github/workflows/beam_PreCommit_SQL.yml | 29 +-
.github/workflows/beam_PreCommit_SQL_Java17.yml | 27 +-
.github/workflows/beam_PreCommit_Spotless.yml | 20 +-
.github/workflows/beam_PreCommit_Typescript.yml | 20 +-
.github/workflows/beam_PreCommit_Website.yml | 20 +-
.../workflows/beam_PreCommit_Website_Stage_GCS.yml | 28 +-
.github/workflows/beam_PreCommit_Whitespace.yml | 18 +-
.../beam_PreCommit_Xlang_Generated_Transforms.yml | 22 +-
.../workflows/beam_PreCommit_Yaml_Xlang_Direct.yml | 25 +-
.github/workflows/beam_Prober_CommunityMetrics.yml | 16 +-
.github/workflows/beam_Publish_BeamMetrics.yml | 14 +-
.../workflows/beam_Publish_Beam_SDK_Snapshots.yml | 10 +-
.../workflows/beam_Publish_Docker_Snapshots.yml | 14 +-
.../workflows/beam_Publish_Python_VLLM_Image.yml | 2 +-
.github/workflows/beam_Publish_Website.yml | 4 +-
.../beam_Python_CostBenchmarks_Dataflow.yml | 17 +-
...beam_Python_ValidatesContainer_Dataflow_ARM.yml | 16 +-
.github/workflows/beam_Release_NightlySnapshot.yml | 9 +-
.../beam_Release_Python_NightlySnapshot.yml | 9 +-
.../workflows/beam_StressTests_Java_BigQueryIO.yml | 14 +-
.../workflows/beam_StressTests_Java_BigTableIO.yml | 14 +-
.../workflows/beam_StressTests_Java_KafkaIO.yml | 14 +-
.../workflows/beam_StressTests_Java_PubSubIO.yml | 14 +-
.../workflows/beam_StressTests_Java_SpannerIO.yml | 14 +-
.github/workflows/build_release_candidate.yml | 12 +-
.github/workflows/build_wheels.yml | 6 +-
.github/workflows/dask_runner_tests.yml | 6 +-
.github/workflows/go_tests.yml | 4 +-
.github/workflows/java_tests.yml | 2 +-
...python_CoGBK_Flink_Batch_100b_Multiple_Keys.txt | 2 +-
.../python_CoGBK_Flink_Batch_100b_Single_Key.txt | 2 +-
.../python_CoGBK_Flink_Batch_10kB.txt | 2 +-
...hon_Combine_Flink_Batch_2GB_10_byte_records.txt | 2 +-
.../python_Combine_Flink_Batch_2GB_Fanout_4.txt | 2 +-
.../python_Combine_Flink_Batch_2GB_Fanout_8.txt | 2 +-
...Combine_Flink_Streaming_2GB_10_byte_records.txt | 2 +-
...python_Combine_Flink_Streaming_2GB_Fanout_4.txt | 2 +-
...python_Combine_Flink_Streaming_2GB_Fanout_8.txt | 2 +-
...thon_Combine_Flink_Streaming_small_Fanout_1.txt | 2 +-
...thon_Combine_Flink_Streaming_small_Fanout_2.txt | 2 +-
.../python_GBK_Flink_Batch_2GB_of_100B_records.txt | 2 +-
.../python_GBK_Flink_Batch_2GB_of_10B_records.txt | 2 +-
...nout_4_times_with_2GB_10-byte_records_total.txt | 2 +-
...nout_8_times_with_2GB_10-byte_records_total.txt | 2 +-
...K_Flink_Batch_reiterate_4_times_10kB_values.txt | 2 +-
.../python_ParDo_Flink_Batch_10_Counters.txt | 2 +-
.../python_ParDo_Flink_Batch_10_Iterations.txt | 2 +-
.../python_ParDo_Flink_Batch_200_Iterations.txt | 2 +-
.../python_ParDo_Flink_Streaming_100_Counters.txt | 2 +-
.../python_ParDo_Flink_Streaming_10_Counters.txt | 2 +-
.../python_ParDo_Flink_Streaming_10_Iterations.txt | 2 +-
...python_ParDo_Flink_Streaming_200_Iterations.txt | 2 +-
.../python_ParDo_Flink_Streaming_5_Iterations.txt | 2 +-
.github/workflows/local_env_tests.yml | 2 +-
.github/workflows/playground_frontend_test.yml | 2 +-
.github/workflows/python_dependency_tests.yml | 3 +-
.github/workflows/python_tests.yml | 5 +-
.../republish_released_docker_containers.yml | 6 +-
.github/workflows/run_perf_alert_tool.yml | 2 +-
.../run_rc_validation_python_mobile_gaming.yml | 2 +-
.github/workflows/tour_of_beam_backend.yml | 2 +-
.../workflows/tour_of_beam_backend_integration.yml | 2 +-
.github/workflows/tour_of_beam_frontend_test.yml | 2 +-
.github/workflows/typescript_tests.yml | 6 +-
.github/workflows/update_python_dependencies.yml | 1 -
.test-infra/metrics/grafana/Dockerfile | 2 +-
.../metrics/kubernetes/beamgrafana-deploy.yaml | 4 +
.test-infra/mock-apis/pyproject.toml | 2 +-
.test-infra/tools/python_installer.sh | 2 +-
CHANGES.md | 18 +-
CI.md | 2 +-
build.gradle.kts | 48 +-
.../org/apache/beam/gradle/BeamModulePlugin.groovy | 36 +-
.../org/apache/beam/gradle/Repositories.groovy | 2 +-
contributor-docs/release-guide.md | 4 +-
.../datatokenization/utils/SchemasUtils.java | 7 +
.../subprocess/utils/CallingSubProcessUtils.java | 7 +
.../examples/subprocess/utils/ExecutableFile.java | 7 +
.../apache/beam/examples/WindowedWordCountIT.java | 4 +-
gradle.properties | 2 +-
.../it/clickhouse/ClickHouseResourceManager.java | 7 +
.../it/gcp/bigtable/BigtableResourceManager.java | 7 +
.../beam/it/splunk/SplunkResourceManager.java | 7 +
local-env-setup.sh | 4 +-
playground/backend/containers/python/Dockerfile | 2 +-
.../cloudbuild/playground_cd_examples.sh | 8 +-
.../cloudbuild/playground_ci_examples.sh | 16 +-
release/src/main/Dockerfile | 5 +-
.../python-release/python_release_automation.sh | 2 +-
.../beam/runners/core/StatefulDoFnRunner.java | 7 +
.../construction/SerializablePipelineOptions.java | 7 +
.../beam/runners/core/metrics/BoundedTrieData.java | 8 +-
.../beam/runners/core/StateInternalsTest.java | 10 +
runners/direct-java/build.gradle | 10 +-
.../apache/beam/runners/direct/ParDoEvaluator.java | 7 +
.../wrappers/streaming/DoFnOperator.java | 2 +-
.../FlinkStreamingPortablePipelineTranslator.java | 7 +
.../wrappers/streaming/DoFnOperator.java | 2 +-
.../streaming/io/UnboundedSourceWrapper.java | 7 +
.../streaming/state/FlinkStateInternals.java | 9 +-
.../wrappers/streaming/DoFnOperator.java | 2 +-
.../FlinkStreamingPortablePipelineTranslator.java | 7 +
.../translation/functions/FlinkAssignContext.java | 2 +-
.../wrappers/streaming/DoFnOperator.java | 2 +-
.../streaming/KeyedPushedBackElementsHandler.java | 2 +-
.../streaming/io/UnboundedSourceWrapper.java | 7 +
.../io/source/FlinkSourceSplitEnumerator.java | 7 +
.../io/source/LazyFlinkSourceSplitEnumerator.java | 7 +
.../streaming/stableinput/BufferingDoFnRunner.java | 7 +
.../stableinput/KeyedBufferingElementsHandler.java | 7 +
.../streaming/state/FlinkStateInternals.java | 9 +-
.../beam/runners/dataflow/DataflowRunnerTest.java | 12 +-
.../runners/dataflow/TestDataflowRunnerTest.java | 4 +-
.../google-cloud-dataflow-java/worker/build.gradle | 7 +-
.../control/DefaultJobBundleFactory.java | 7 +
.../org/apache/beam/runners/jet/JetRunner.java | 7 +
.../runners/jet/processors/AbstractParDoP.java | 2 +-
.../beam/runners/jet/processors/AssignWindowP.java | 7 +
.../beam/runners/jet/processors/WindowGroupP.java | 2 +-
runners/prism/java/build.gradle | 2 +
.../io/BoundedDatasetFactory.java | 2 +-
.../translation/PipelineTranslatorStreaming.java | 65 +-
.../StatefulParDoStreamingTranslator.java | 86 ++
.../streaming/state/BeamStatefulProcessor.java | 330 ++++++
.../io/streaming/TestUnboundedSource.java | 16 +-
.../PipelineTranslatorStreamingTest.java | 97 +-
.../streaming/StatefulParDoStreamingTest.java | 242 ++++
runners/spark/job-server/spark_job_server.gradle | 5 +
runners/spark/spark_runner.gradle | 5 +
.../beam/runners/spark/io/SourceDStream.java | 2 +-
.../spark/metrics/SparkBeamMetricSource.java | 7 +
.../beam/runners/spark/metrics/sink/CsvSink.java | 7 +
.../runners/spark/metrics/sink/GraphiteSink.java | 7 +
.../spark/stateful/SparkStateInternals.java | 90 +-
.../spark/stateful/SparkTimerInternals.java | 5 +
.../io/BoundedDatasetFactory.java | 2 +-
.../metrics/SparkBeamMetricSource.java | 7 +
.../metrics/sink/CodahaleCsvSink.java | 7 +
.../metrics/sink/CodahaleGraphiteSink.java | 7 +
.../translation/batch/DoFnRunnerFactory.java | 15 +-
.../batch/StatefulDoFnGroupFunction.java | 118 +-
.../translation/batch/StatefulTaskRunner.java | 145 +++
.../SparkBatchPortablePipelineTranslator.java | 7 +
.../SparkDatasetPortablePipelineTranslator.java | 7 +
.../SparkStreamingPortablePipelineTranslator.java | 7 +
scripts/ci/pr-bot/README.md | 1 +
scripts/ci/pr-bot/processNewPrs.ts | 19 +-
scripts/ci/pr-bot/processPrUpdate.ts | 10 +-
scripts/ci/pr-bot/shared/checks.ts | 7 +-
sdks/go.mod | 40 +-
sdks/go.sum | 88 +-
sdks/go/pkg/beam/core/graph/coder/int.go | 31 +
sdks/go/pkg/beam/core/graph/coder/int_test.go | 32 +
sdks/go/pkg/beam/core/graph/coder/row_decoder.go | 26 +-
sdks/go/pkg/beam/core/graph/coder/row_encoder.go | 14 +-
sdks/go/pkg/beam/core/graph/coder/row_test.go | 40 +
sdks/go/pkg/beam/core/runtime/harness/datamgr.go | 47 +-
.../pkg/beam/core/runtime/harness/datamgr_test.go | 289 +++++
sdks/go/pkg/beam/core/runtime/harness/harness.go | 2 +
sdks/go/pkg/beam/core/runtime/harness/statemgr.go | 29 +-
.../pkg/beam/core/runtime/harness/statemgr_test.go | 101 +-
sdks/go/pkg/beam/core/runtime/xlangx/expand.go | 47 +-
.../go/pkg/beam/core/runtime/xlangx/expand_test.go | 148 +++
sdks/go/pkg/beam/core/util/protox/any.go | 4 +-
sdks/go/pkg/beam/core/util/protox/any_test.go | 40 +
sdks/go/pkg/beam/core/util/protox/base64.go | 2 +-
sdks/go/pkg/beam/core/util/protox/protox.go | 2 +-
sdks/go/pkg/beam/options/jobopts/options.go | 2 +
sdks/go/pkg/beam/options/jobopts/options_test.go | 24 +-
sdks/go/pkg/beam/options/resource/hint.go | 39 +
sdks/go/pkg/beam/options/resource/hint_test.go | 45 +
sdks/go/pkg/beam/transforms/wait/wait.go | 172 +++
sdks/go/pkg/beam/transforms/wait/wait_test.go | 223 ++++
sdks/go/test/integration/integration.go | 13 +
sdks/go/test/integration/primitives/wait.go | 307 +++++
.../primitives/{state_test.go => wait_test.go} | 50 +-
.../src/main/resources/beam/spotbugs-filter.xml | 3 -
sdks/java/core/build.gradle | 1 +
.../org/apache/beam/sdk/jmh/schemas/RowBundle.java | 7 +
.../org/apache/beam/sdk/coders/CoderProviders.java | 2 +-
.../sdk/fn/data/BeamFnDataOutboundAggregator.java | 7 +
.../org/apache/beam/sdk/io/CompressedSource.java | 7 +
.../java/org/apache/beam/sdk/io/Compression.java | 2 +-
.../java/org/apache/beam/sdk/io/FileSystems.java | 11 +-
.../src/main/java/org/apache/beam/sdk/io/Read.java | 12 +-
.../apache/beam/sdk/options/StreamingOptions.java | 10 +
.../schemas/transforms/providers/JavaRowUdf.java | 6 +
.../beam/sdk/schemas/utils/ByteBuddyUtils.java | 9 +-
.../beam/sdk/schemas/utils/ConvertHelpers.java | 7 +
.../beam/sdk/testing/BeamParallelJunit4Runner.java | 1 +
.../java/org/apache/beam/sdk/testing/PAssert.java | 18 +-
.../beam/sdk/testing/SerializableMatchers.java | 4 +-
.../apache/beam/sdk/transforms/AsyncWrapper.java | 7 +
.../beam/sdk/transforms/InferableFunction.java | 7 +
.../org/apache/beam/sdk/transforms/Partition.java | 2 +-
.../org/apache/beam/sdk/transforms/Sample.java | 7 +
.../apache/beam/sdk/transforms/SimpleFunction.java | 7 +
.../java/org/apache/beam/sdk/transforms/View.java | 2 +-
.../java/org/apache/beam/sdk/transforms/Watch.java | 2 +-
.../beam/sdk/transforms/join/CoGbkResult.java | 7 +
.../reflect/ByteBuddyDoFnInvokerFactory.java | 2 +-
.../sdk/transforms/windowing/FixedWindows.java | 7 +
.../sdk/transforms/windowing/SlidingWindows.java | 7 +
.../apache/beam/sdk/util/ExplicitShardedFile.java | 7 +
.../apache/beam/sdk/util/MutationDetectors.java | 2 +-
.../util/construction/DefaultArtifactResolver.java | 7 +
.../util/construction/WriteFilesTranslation.java | 2 +-
.../util/construction/graph/QueryablePipeline.java | 7 +
.../apache/beam/sdk/values/PCollectionViews.java | 13 +
.../org/apache/beam/sdk/values/TypeDescriptor.java | 7 +
.../java/org/apache/beam/sdk/io/FileIOTest.java | 4 +-
.../org/apache/beam/sdk/io/FileSystemsTest.java | 86 ++
.../apache/beam/sdk/io/GenerateSequenceTest.java | 4 +-
.../test/java/org/apache/beam/sdk/io/ReadTest.java | 36 +
.../org/apache/beam/sdk/io/TFRecordIOTest.java | 4 +-
.../io/TFRecordSchemaTransformProviderTest.java | 4 +-
.../org/apache/beam/sdk/io/TextIOReadTest.java | 9 +-
.../org/apache/beam/sdk/io/WriteFilesTest.java | 5 +-
.../beam/sdk/schemas/AutoValueSchemaTest.java | 4 +-
.../beam/sdk/schemas/transforms/ConvertTest.java | 4 +-
.../sdk/transforms/ApproximateQuantilesTest.java | 4 +-
.../beam/sdk/transforms/AsyncWrapperTest.java | 72 +-
.../org/apache/beam/sdk/transforms/WaitTest.java | 25 +-
.../beam/sdk/values/PCollectionViewsTest.java | 4 +-
.../sdk/expansion/service/ExpansionServer.java | 7 +
.../ExpansionServiceSchemaTransformProvider.java | 7 +
.../service/JavaClassLookupTransformProvider.java | 2 +-
.../sdk/extensions/gcp/options/GcsOptions.java | 31 +-
.../beam/sdk/extensions/gcp/util/GcsUtilV1.java | 8 +-
.../beam/sdk/extensions/gcp/util/GcsUtilV2.java | 251 +++-
.../sdk/extensions/gcp/util/GcsUtilV2Test.java | 75 ++
.../beam/sdk/extensions/ml/AnnotateImages.java | 7 +
.../ordered/GlobalSequencesProcessorDoFn.java | 2 +
.../sdk/extensions/ordered/ProcessingState.java | 4 +
.../beam/sdk/extensions/ordered/ProcessorDoFn.java | 20 +-
.../ordered/SequencePerKeyProcessorDoFn.java | 2 +
.../OrderedEventProcessorPerKeySequenceTest.java | 48 +
.../python/PythonExternalTransformTest.java | 1 +
.../extensions/sorter/HadoopExternalSorter.java | 2 +-
.../sdk/extensions/sorter/NativeFileSorter.java | 2 +-
.../sorter/BufferedExternalSorterTest.java | 4 +-
.../sql/meta/provider/delta/DeltaTable.java | 2 +-
.../sql/meta/provider/iceberg/IcebergTable.java | 2 +-
.../fn/harness/jmh/ProcessBundleBenchmark.java | 19 +
.../jmh/logging/BeamFnLoggingClientBenchmark.java | 7 +
.../beam/fn/harness/BeamFnDataReadRunner.java | 7 +
.../apache/beam/fn/harness/FnApiDoFnRunner.java | 9 +-
.../SplittablePairWithRestrictionDoFnRunner.java | 7 +
...littableSplitAndSizeRestrictionsDoFnRunner.java | 7 +
...ittableTruncateSizedRestrictionsDoFnRunner.java | 7 +
.../fn/harness/control/BeamFnControlClient.java | 19 +-
.../fn/harness/control/ExecutionStateSampler.java | 7 +
.../harness/data/PCollectionConsumerRegistry.java | 17 +-
.../harness/data/PTransformFunctionRegistry.java | 7 +
.../fn/harness/logging/BeamFnLoggingClient.java | 7 +
.../harness/state/BeamFnStateGrpcClientCache.java | 2 +
.../fn/harness/state/FnApiTimerBundleTracker.java | 7 +
.../fn/harness/state/WorkCancelledException.java} | 32 +-
.../beam/fn/harness/status/MemoryMonitor.java | 40 +-
.../harness/control/BeamFnControlClientTest.java | 22 +
.../data/PCollectionConsumerRegistryTest.java | 84 ++
.../state/BeamFnStateGrpcClientCacheTest.java | 21 +-
.../beam/fn/harness/status/MemoryMonitorTest.java | 11 +
sdks/java/io/amqp/build.gradle | 16 +
.../components/throttling/AdaptiveThrottler.java | 7 +
.../components/throttling/ReactiveThrottler.java | 7 +
.../io/contextualtextio/ContextualTextIOTest.java | 9 +-
sdks/java/io/debezium/build.gradle | 6 -
.../apache/beam/io/debezium/SourceRecordJson.java | 7 +
.../apache/beam/sdk/io/delta/SerializableRow.java | 7 +
.../io/elasticsearch/ElasticsearchIOTestUtils.java | 2 +-
.../FileWriteSchemaTransformProvider.java | 2 +-
.../BigtableReadSchemaTransformProvider.java | 2 +-
...gtableSimpleWriteSchemaTransformProviderIT.java | 5 +-
.../beam/sdk/io/gcp/bigtable/BigtableWriteIT.java | 5 +-
.../BigtableWriteSchemaTransformProviderIT.java | 5 +-
sdks/java/io/iceberg/build.gradle | 13 +-
.../apache/beam/sdk/io/iceberg/BeamRowWrapper.java | 7 +
.../apache/beam/sdk/io/iceberg/BundleLifter.java | 7 +
.../apache/beam/sdk/io/iceberg/RecordWriter.java | 7 +
.../beam/sdk/io/iceberg/RecordWriterManager.java | 2 +-
.../beam/sdk/io/iceberg/cdc/DeleteReader.java | 7 +
.../io/iceberg/cdc/SerializableChangelogTask.java | 6 +-
.../io/iceberg/cdc/sink/RecordDeltaTaskWriter.java | 7 +
.../org/apache/beam/sdk/io/iceberg/AddFilesIT.java | 219 +++-
.../IcebergCdcReadSchemaTransformProviderTest.java | 71 ++
.../beam/sdk/io/iceberg/LakehouseTestCatalog.java | 198 ++++
.../io/iceberg/catalog/IcebergCatalogBaseIT.java | 21 +-
.../io/iceberg/catalog/IcebergCdcWriteBaseIT.java | 12 +-
.../catalog/LakehouseCatalogCdcWriteIT.java | 24 +-
.../sdk/io/iceberg/catalog/RESTCatalogBLMSIT.java | 121 +-
.../beam/sdk/io/influxdb/ShardInformation.java | 2 +-
.../io/jdbc/JdbcReadSchemaTransformProvider.java | 8 +-
.../io/jdbc/JdbcWriteSchemaTransformProvider.java | 8 +-
.../ReadFromSqlServerSchemaTransformProvider.java | 7 +
.../WriteToSqlServerSchemaTransformProvider.java | 7 +
sdks/java/io/jms/build.gradle | 8 +-
.../io/jms/BeamGenericJmsConnectionFactory.java | 2 +-
.../beam/sdk/io/jms/ConnectionConfiguration.java | 6 +-
.../apache/beam/sdk/io/jms/JmsCheckpointMark.java | 8 +-
.../java/org/apache/beam/sdk/io/jms/JmsIO.java | 38 +-
.../java/org/apache/beam/sdk/io/jms/JmsRecord.java | 2 +-
.../apache/beam/sdk/io/jms/TextMessageMapper.java | 12 +-
.../java/org/apache/beam/sdk/io/jms/CommonJms.java | 8 +-
.../sdk/io/jms/ConnectionConfigurationTest.java | 19 +-
.../java/org/apache/beam/sdk/io/jms/JmsIOIT.java | 14 +-
.../java/org/apache/beam/sdk/io/jms/JmsIOTest.java | 32 +-
.../org/apache/beam/sdk/io/jms/JmsLocalTest.java | 14 +-
.../jms/MockNonSerializableConnectionFactory.java | 8 +-
.../KafkaReadSchemaTransformConfiguration.java | 19 +
.../io/kafka/KafkaReadSchemaTransformProvider.java | 28 +-
.../beam/sdk/io/kafka/KafkaUnboundedReader.java | 181 +--
.../kafka/KafkaWriteSchemaTransformProvider.java | 52 +-
.../org/apache/beam/sdk/io/kafka/ProducerSpEL.java | 2 +-
.../org/apache/beam/sdk/io/kafka/KafkaIOTest.java | 27 -
.../org/apache/beam/sdk/io/kafka/KafkaMocks.java | 37 -
.../KafkaReadSchemaTransformProviderTest.java | 107 +-
.../KafkaWriteSchemaTransformProviderTest.java | 13 +-
.../apache/beam/sdk/io/kudu/KuduServiceImpl.java | 2 +-
.../io/messaging-expansion-service/build.gradle | 1 +
sdks/java/io/mqtt/build.gradle | 15 +
.../org/apache/beam/sdk/io/mqtt/MqttIOTest.java | 47 +-
.../org/apache/beam/sdk/io/parquet/ParquetIO.java | 54 +-
.../sdk/io/pulsar/NaiveReadFromPulsarDoFn.java | 2 +-
.../apache/beam/sdk/io/rabbitmq/RabbitMqIO.java | 2 +-
.../beam/sdk/io/rabbitmq/RabbitMqMessage.java | 7 +
.../org/apache/beam/io/requestresponse/Cache.java | 2 +-
.../io/snowflake/data/text/SnowflakeBinary.java | 7 +
.../io/snowflake/data/text/SnowflakeVarchar.java | 7 +
.../beam/sdk/io/solace/broker/BrokerResponse.java | 7 +
.../sdk/io/sparkreceiver/SparkReceiverIOTest.java | 17 +-
.../beam/sdk/io/splunk/CustomX509TrustManager.java | 7 +
.../io/synthetic/SyntheticBoundedSourceTest.java | 4 +-
.../beam/sdk/io/synthetic/SyntheticStepTest.java | 4 +-
.../java/org/apache/beam/sdk/io/xml/XmlSource.java | 2 +-
.../managed/ManagedSchemaTransformProvider.java | 2 +-
.../org/apache/beam/sdk/loadtests/LoadTest.java | 7 +
sdks/java/testing/nexmark/build.gradle | 5 +
.../nexmark/queries/BoundedSideInputJoinTest.java | 4 +-
.../apache/beam/sdk/nexmark/queries/QueryTest.java | 4 +-
.../nexmark/queries/SessionSideInputJoinTest.java | 4 +-
.../beam/sdk/nexmark/queries/SqlQueryTest.java | 3 +-
.../queries/sql/SqlBoundedSideInputJoinTest.java | 3 +-
.../launcher/TransformServiceLauncher.java | 7 +
sdks/python/apache_beam/__init__.py | 2 +-
.../examples/complete/game/leader_board.py | 19 +-
.../examples/complete/game/leader_board_test.py | 4 +-
.../apache_beam/io/external/xlang_jmsio_it_test.py | 4 +-
sdks/python/apache_beam/io/gcp/bigquery.py | 47 +-
sdks/python/apache_beam/io/gcp/bigquery_test.py | 52 +
sdks/python/apache_beam/io/gcp/bigquery_tools.py | 100 +-
.../apache_beam/io/gcp/bigquery_tools_test.py | 63 +
sdks/python/apache_beam/io/textio.py | 2 +-
sdks/python/apache_beam/io/textio_test.py | 9 +
sdks/python/apache_beam/io/unbounded_source.py | 13 +-
.../python/apache_beam/io/unbounded_source_test.py | 21 +
sdks/python/apache_beam/ml/inference/base.py | 7 +-
sdks/python/apache_beam/ml/inference/base_test.py | 47 +
.../apache_beam/ml/inference/openai_inference.py | 392 +++++++
.../ml/inference/openai_inference_it_test.py | 211 ++++
.../ml/inference/openai_inference_test.py | 624 ++++++++++
.../inference/openai_tests_requirements.txt} | 2 +-
.../vertex_ai_custom_prediction/Dockerfile | 2 +-
.../apache_beam/ml/rag/embeddings/huggingface.py | 23 +-
.../ml/rag/embeddings/huggingface_test.py | 56 +
.../ml/transforms/embeddings/huggingface.py | 48 +-
.../ml/transforms/embeddings/huggingface_test.py | 142 +++
.../python/apache_beam/options/pipeline_options.py | 5 +
.../runners/dataflow/internal/apiclient.py | 4 +-
.../runners/dataflow/internal/apiclient_test.py | 6 +-
.../runners/portability/fn_api_runner/execution.py | 17 +-
.../apache_beam/testing/benchmarks/README.md | 2 +-
sdks/python/apache_beam/testing/test_utils.py | 61 +
sdks/python/apache_beam/testing/test_utils_test.py | 31 +
sdks/python/apache_beam/transforms/async_dofn.py | 67 +-
.../apache_beam/transforms/async_dofn_test.py | 221 +++-
sdks/python/apache_beam/transforms/core.py | 4 +-
sdks/python/apache_beam/transforms/trigger.py | 15 +-
.../apache_beam/transforms/userstate_test.py | 42 +
sdks/python/apache_beam/transforms/util.py | 11 +-
sdks/python/apache_beam/transforms/util_test.py | 39 +
sdks/python/apache_beam/utils/timestamp.py | 5 +-
sdks/python/apache_beam/utils/timestamp_test.py | 10 +
.../yaml/examples/testing/examples_test.py | 2 +
..._to_bigquery.yaml => cloudsql_to_bigquery.yaml} | 23 +-
.../transforms/blueprint/postgres_to_bigquery.yaml | 5 +-
.../extended_tests/databases/bigquery_cdc.yaml | 91 ++
.../extended_tests/e2e/cloudsql_to_bigquery.yaml | 82 ++
.../yaml/extended_tests/messaging/kafka.yaml | 5 +
sdks/python/apache_beam/yaml/integration_tests.py | 129 +-
sdks/python/apache_beam/yaml/standard_io.yaml | 45 +-
sdks/python/apache_beam/yaml/tests/bigtable.yaml | 72 +-
sdks/python/apache_beam/yaml/yaml_ml_test.py | 40 +-
sdks/python/apache_beam/yaml/yaml_provider.py | 26 +-
.../apache_beam/yaml/yaml_provider_unit_test.py | 44 +
sdks/python/apache_beam/yaml/yaml_testing.py | 10 +-
sdks/python/apache_beam/yaml/yaml_transform.py | 5 +
.../python/apache_beam/yaml/yaml_transform_test.py | 40 +
sdks/python/build.gradle | 2 +-
sdks/python/container/Dockerfile | 2 +-
sdks/python/container/run_generate_requirements.sh | 4 +-
sdks/python/expansion-service-container/Dockerfile | 4 +-
sdks/python/pyproject.toml | 4 +-
sdks/python/pytest.ini | 1 +
sdks/python/setup.py | 9 +-
sdks/python/test-suites/dataflow/common.gradle | 30 +-
sdks/python/test-suites/gradle.properties | 24 +-
sdks/python/test-suites/tox/py311/build.gradle | 186 +++
sdks/python/tox.ini | 48 +-
settings.gradle.kts | 2 +-
website/build.gradle | 19 +-
website/www/site/config.toml | 2 +-
website/www/site/content/en/blog/beam-2.77.0.md | 74 ++
.../site/content/en/blog/beam-modernization-10y.md | 243 ++++
.../site/content/en/documentation/io/managed-io.md | 1240 +++++++++++++-------
.../content/en/documentation/runners/flaredb.md | 210 ++++
.../en/documentation/runtime/environments.md | 4 +-
.../documentation/sdks/python-sdk-image-build.md | 6 +-
.../www/site/content/en/get-started/downloads.md | 16 +-
website/www/site/content/en/security/_index.md | 93 +-
.../content/en/security/{_index.md => archive.md} | 13 +-
website/www/site/data/capability_matrix.yaml | 156 ++-
.../www/site/data/en/documentation_runners.yaml | 2 +
website/www/site/data/works_with.yaml | 3 +
.../layouts/partials/section-menu/en/runners.html | 1 +
website/www/site/static/images/logo_flaredb.png | Bin 0 -> 39481 bytes
728 files changed, 12571 insertions(+), 6116 deletions(-)
delete mode 100644 .github/actions/rerun-job-action/action.yml
copy .github/gh-actions-self-hosted-runners/arc/config/{arc_certificate.tpl =>
arc_frontend_config.tpl} (86%)
create mode 100644
runners/spark/4/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/streaming/StatefulParDoStreamingTranslator.java
create mode 100644
runners/spark/4/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/streaming/state/BeamStatefulProcessor.java
create mode 100644
runners/spark/4/src/test/java/org/apache/beam/runners/spark/structuredstreaming/translation/streaming/StatefulParDoStreamingTest.java
create mode 100644
runners/spark/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/StatefulTaskRunner.java
create mode 100644 sdks/go/pkg/beam/core/runtime/xlangx/expand_test.go
create mode 100644 sdks/go/pkg/beam/transforms/wait/wait.go
create mode 100644 sdks/go/pkg/beam/transforms/wait/wait_test.go
create mode 100644 sdks/go/test/integration/primitives/wait.go
copy sdks/go/test/integration/primitives/{state_test.go => wait_test.go} (53%)
copy
sdks/java/{io/solace/src/main/java/org/apache/beam/sdk/io/solace/read/ActiveReadersRegistry.java
=>
harness/src/main/java/org/apache/beam/fn/harness/state/WorkCancelledException.java}
(55%)
create mode 100644
sdks/java/io/iceberg/src/test/java/org/apache/beam/sdk/io/iceberg/LakehouseTestCatalog.java
create mode 100644 sdks/python/apache_beam/ml/inference/openai_inference.py
create mode 100644
sdks/python/apache_beam/ml/inference/openai_inference_it_test.py
create mode 100644
sdks/python/apache_beam/ml/inference/openai_inference_test.py
copy
sdks/python/apache_beam/{transforms/enrichment_handlers/feast_tests_requirements.txt
=> ml/inference/openai_tests_requirements.txt} (98%)
copy
sdks/python/apache_beam/yaml/examples/transforms/blueprint/{spanner_to_bigquery.yaml
=> cloudsql_to_bigquery.yaml} (76%)
create mode 100644
sdks/python/apache_beam/yaml/extended_tests/databases/bigquery_cdc.yaml
create mode 100644
sdks/python/apache_beam/yaml/extended_tests/e2e/cloudsql_to_bigquery.yaml
create mode 100644 website/www/site/content/en/blog/beam-2.77.0.md
create mode 100644 website/www/site/content/en/blog/beam-modernization-10y.md
create mode 100644 website/www/site/content/en/documentation/runners/flaredb.md
copy website/www/site/content/en/security/{_index.md => archive.md} (75%)
create mode 100644 website/www/site/static/images/logo_flaredb.png
