This is an automated email from the ASF dual-hosted git repository.
derrickaw pushed a change to branch 20260604_deltalakeYaml
in repository https://gitbox.apache.org/repos/asf/beam.git
from 39ec9cecf42 switch to only java version 17
add eaba570c569 Bump github.com/golang-cz/devslog from 0.0.16 to 0.0.17 in
/sdks (#39054)
add 54231e89266 try to fix forward tests for java 17 (#39058)
add 9aed894f5d4 Update pandas version upper bound to support python 3.14
(#39056)
add 9868954ce8c Fix playground precommit test failure (#39062)
add c2474960db7 [yaml] - expand jinja method (#38547)
add d031bddfdc9 Bump actions/checkout from 6 to 7 (#39033)
add 117bb893ca8 Bump google.golang.org/api from 0.285.0 to 0.286.0 in
/sdks (#39068)
add 9a9f48b9542 Fix Flink 2.1+ dependency resolution (#39061)
add c77b2d40626 Update Python Dependencies (#39070)
add a467fdc437f Enable precommit yaml xlang test on code push. (#39071)
add f61d0b5eeff Release Note: Rename Dataflow Runner to Dataflow Portable
Runner (#38747)
add f29d9af139a init (#39025)
add 447899e46e9 [mqtt] Add Python Xlang Messaging PostCommit with MQTT
integration tests (#38966)
add cedfd0bb3e8 [ReduceFnRunner] Fix Prefetches (#39082)
add c3d9a731b42 [Dataflow Streaming] Increase stuck commit invalidation
timeout to 1 hour (#39083)
add 24e9ed4e0e9 Bump actions/cache from 4 to 6 (#39081)
add f7f0e1655d5 Bump actions/checkout from 6 to 7 (#39080)
add 0165344ab13 Add UnboundedCountingSource::to for bounded reads from an
UnboundedCountingSource (#39084)
add f33feafb2b2 [Dataflow Java Streaming] Add a timeout to how long
commits will retry to the service. (#39085)
add 857dfc9a875 [mqtt] Fix streaming xlang IT failing the Messaging
PostCommit (#39088)
add 0802263e48e Add a github workflow to publish vllm image. (#39089)
add 8e4ea736213 [Python] Optimize BigQuery copy jobs in file loads using
multi-source copy (#38983)
add 5afb93c8c09 [OpenTelemetry] Turn off gcp otel auth extension by
default. (#38940)
add 4e9e483a51a Fix Flink XVR for Flink 2
add 49fd8745223 Fix Go Flink VR conf path
add 16acfa8a2c6 Update CHANGES.md and website
add f13ccf7afbe Merge pull request #39077 from Abacn/fix-flink-xvr
add 9e28c9d2150 Disable flaky SDF finalization tests
add 6095fd0de62 Merge pull request #39092 from
arunpandianp/disablesdffinalization
add 70b72adffee Fix Install xmllint
add 0aa808dba09 Merge pull request #39094 from apache/cut-release-branch
add cad9f1ebe76 Moving to 2.76.0-SNAPSHOT on master branch.
add beeb715739f Update beam-master (#39095)
add c531095fc40 [GSOC 2026] Adding logic to validate and generate an error
if there are key… (#38992)
add 8d252c4659b update changes log for yaml improvements etc (#39093)
add 3fba9950feb Update beam-master container
add e56ed7718a5 Trigger failed postcommit tests.
add 7454aa6a0bf Merge pull request #39100 from
shunping/update-python-beam-master
add d71c4aca4f4 Fix race condition in DirectRunner executor shutdown
add 0af28db1142 Apply suggested fix
add 23521069470 Merge pull request #39096 from shunping/fix-flaky-rrio-test
add 96733599e6c Fix JsonToRow swallowing downstream errors when runners
fuse transforms. (#39098)
add 8092349ca28 Enable playground tests triggered on cron schedule (#39069)
add 8ecdab8da94 [Iceberg CDC] Import BaseIncrementalChangelogScan (#38823)
add 8c311e45672 [Iceberg CDC] Add utils base, delete reader, and
serializable files (#38821)
add c1898ff68b8 Fix pip ResolutionTooDeep (#39017)
add 9d3694165e1 [Python] Add UnboundedSource SDF wrapper (#19137) (#38724)
add 5eefc931b5d remove infra change (#39104)
add eef26ca8b71 Revert "[Python] Optimize BigQuery copy jobs in file loads
using multi-source copy" (#39106)
add 0c51b0d288b Write append tables can be asynchronous (#39120)
add b642c5cbd0e Fix typo (#39123)
add 56b5a52294d Bump google-cloud-aiplatform (#39113)
add d77d12a47b2 Update cython requirement from <4,>=3.2.5 to >=3.2.6,<4 in
/sdks/python (#39099)
add 5cb2f09298c [Website] Update Python version compatibility table
(#38969)
add 698420027d5 Docs: update Python version support in Python quickstart
(#39122)
add 8ccd1d70c04 including GitHub Actions to schedule daily audits and Add
documentati… (#39116)
add 5a2e98bb712 Fix a flaky test in ApproximateQuantilesTest (#39132)
add e3b4d8c5d0e Fix flaky ML RunInference tests by disabling reshuffle on
beam.Create (#39118)
add 3cca8fa3dac [Dataflow Streaming] Access state/timerinternals via
StepContext (#38991)
add 2e8f75aaf4b Bump actions/checkout from 6 to 7 (#39147)
add b63ea65d6a3 Bump github.com/aws/smithy-go from 1.27.2 to 1.27.3 in
/sdks (#39145)
add 31cd92f5f55 Bump cloud.google.com/go/storage from 1.62.3 to 1.63.0 in
/sdks (#39144)
add edd3bfca48e Fix flaky GcsIOIntegrationTest and handle GCS bucket
creation race conditions (#39130)
add af005ef679c fix post commit yaml test (#39142)
add 3d8d8102acd Merge remote-tracking branch 'upstream' into
20260604_deltalakeYaml
No new revisions were added by this update.
Summary of changes:
.asf.yaml | 1 +
.../IO_Iceberg_Integration_Tests.json | 2 +-
.../beam_CloudML_Benchmarks_Dataflow.json | 2 +-
.../trigger_files/beam_PostCommit_Go_VR_Flink.json | 2 +-
.../beam_PostCommit_Java_Nexmark_Flink.json | 2 +-
....json => beam_PostCommit_Java_Tpcds_Flink.json} | 0
..._ValidatesRunner_Dataflow_Streaming_Engine.json | 2 +-
.github/trigger_files/beam_PostCommit_Python.json | 4 +-
.../beam_PostCommit_Python_Examples_Dataflow.json | 4 +-
..._PostCommit_Python_Xlang_Messaging_Direct.json} | 2 +-
.../trigger_files/beam_PostCommit_XVR_Flink.json | 2 +-
.../trigger_files/beam_PostCommit_XVR_Samza.json | 1 -
.../beam_PostCommit_Yaml_Xlang_Direct.json | 2 +-
.github/workflows/IO_Iceberg_Integration_Tests.yml | 2 +-
.../IO_Iceberg_Integration_Tests_Dataflow.yml | 2 +-
..._Iceberg_Managed_Integration_Tests_Dataflow.yml | 2 +-
.github/workflows/IO_Iceberg_Performance_Tests.yml | 2 +-
.github/workflows/IO_Iceberg_Unit_Tests.yml | 2 +-
.github/workflows/README.md | 3 +
.github/workflows/assign_milestone.yml | 2 +-
.github/workflows/beam_CancelStaleDataflowJobs.yml | 2 +-
.../workflows/beam_CleanUpDataprocResources.yml | 2 +-
.github/workflows/beam_CleanUpGCPResources.yml | 2 +-
.../workflows/beam_CleanUpPrebuiltSDKImages.yml | 2 +-
.../workflows/beam_CloudML_Benchmarks_Dataflow.yml | 2 +-
.../beam_IODatastoresCredentialsRotation.yml | 2 +-
.../beam_Inference_Python_Benchmarks_Dataflow.yml | 2 +-
.../beam_Infrastructure_AuditUnmanagedKeys.yml | 69 +
.../beam_Infrastructure_PolicyEnforcer.yml | 2 +-
.../beam_Infrastructure_SecurityLogging.yml | 2 +-
.../beam_Infrastructure_ServiceAccountKeys.yml | 2 +-
.../beam_Infrastructure_UsersPermissions.yml | 2 +-
.github/workflows/beam_Java_JMH.yml | 2 +-
.../beam_Java_LoadTests_Combine_Smoke.yml | 2 +-
.../beam_LoadTests_Go_CoGBK_Dataflow_Batch.yml | 2 +-
.../beam_LoadTests_Go_CoGBK_Flink_batch.yml | 2 +-
.../beam_LoadTests_Go_Combine_Dataflow_Batch.yml | 2 +-
.../beam_LoadTests_Go_Combine_Flink_Batch.yml | 2 +-
.../beam_LoadTests_Go_GBK_Dataflow_Batch.yml | 2 +-
.../beam_LoadTests_Go_GBK_Flink_Batch.yml | 2 +-
.../beam_LoadTests_Go_ParDo_Dataflow_Batch.yml | 2 +-
.../beam_LoadTests_Go_ParDo_Flink_Batch.yml | 2 +-
.../beam_LoadTests_Go_SideInput_Dataflow_Batch.yml | 2 +-
.../beam_LoadTests_Go_SideInput_Flink_Batch.yml | 2 +-
.../beam_LoadTests_Java_CoGBK_Dataflow_Batch.yml | 2 +-
...eam_LoadTests_Java_CoGBK_Dataflow_Streaming.yml | 2 +-
...s_Java_CoGBK_Dataflow_V2_Batch_JavaVersions.yml | 2 +-
...va_CoGBK_Dataflow_V2_Streaming_JavaVersions.yml | 2 +-
...s_Java_CoGBK_SparkStructuredStreaming_Batch.yml | 2 +-
.../beam_LoadTests_Java_Combine_Dataflow_Batch.yml | 2 +-
...m_LoadTests_Java_Combine_Dataflow_Streaming.yml | 2 +-
...Java_Combine_SparkStructuredStreaming_Batch.yml | 2 +-
.../beam_LoadTests_Java_GBK_Dataflow_Batch.yml | 2 +-
.../beam_LoadTests_Java_GBK_Dataflow_Streaming.yml | 2 +-
.../beam_LoadTests_Java_GBK_Dataflow_V2_Batch.yml | 2 +-
...LoadTests_Java_GBK_Dataflow_V2_Batch_Java17.yml | 2 +-
...am_LoadTests_Java_GBK_Dataflow_V2_Streaming.yml | 2 +-
...Tests_Java_GBK_Dataflow_V2_Streaming_Java17.yml | 2 +-
.../workflows/beam_LoadTests_Java_GBK_Smoke.yml | 2 +-
...sts_Java_GBK_SparkStructuredStreaming_Batch.yml | 2 +-
.../beam_LoadTests_Java_ParDo_Dataflow_Batch.yml | 2 +-
...eam_LoadTests_Java_ParDo_Dataflow_Streaming.yml | 2 +-
...s_Java_ParDo_Dataflow_V2_Batch_JavaVersions.yml | 2 +-
...va_ParDo_Dataflow_V2_Streaming_JavaVersions.yml | 2 +-
...s_Java_ParDo_SparkStructuredStreaming_Batch.yml | 2 +-
.github/workflows/beam_LoadTests_Java_PubsubIO.yml | 2 +-
.../beam_LoadTests_Python_CoGBK_Dataflow_Batch.yml | 2 +-
...m_LoadTests_Python_CoGBK_Dataflow_Streaming.yml | 2 +-
.../beam_LoadTests_Python_CoGBK_Flink_Batch.yml | 2 +-
...eam_LoadTests_Python_Combine_Dataflow_Batch.yml | 2 +-
...LoadTests_Python_Combine_Dataflow_Streaming.yml | 2 +-
.../beam_LoadTests_Python_Combine_Flink_Batch.yml | 2 +-
...am_LoadTests_Python_Combine_Flink_Streaming.yml | 2 +-
...LoadTests_Python_FnApiRunner_Microbenchmark.yml | 2 +-
.../beam_LoadTests_Python_GBK_Dataflow_Batch.yml | 2 +-
...eam_LoadTests_Python_GBK_Dataflow_Streaming.yml | 2 +-
.../beam_LoadTests_Python_GBK_Flink_Batch.yml | 2 +-
...adTests_Python_GBK_reiterate_Dataflow_Batch.yml | 2 +-
...sts_Python_GBK_reiterate_Dataflow_Streaming.yml | 2 +-
.../beam_LoadTests_Python_ParDo_Dataflow_Batch.yml | 2 +-
...m_LoadTests_Python_ParDo_Dataflow_Streaming.yml | 2 +-
.../beam_LoadTests_Python_ParDo_Flink_Batch.yml | 2 +-
...beam_LoadTests_Python_ParDo_Flink_Streaming.yml | 2 +-
...m_LoadTests_Python_SideInput_Dataflow_Batch.yml | 2 +-
.github/workflows/beam_LoadTests_Python_Smoke.yml | 2 +-
.../workflows/beam_MetricsCredentialsRotation.yml | 2 +-
.github/workflows/beam_Metrics_Report.yml | 2 +-
.../workflows/beam_PerformanceTests_AvroIOIT.yml | 2 +-
.../beam_PerformanceTests_AvroIOIT_HDFS.yml | 2 +-
...PerformanceTests_BigQueryIO_Batch_Java_Avro.yml | 2 +-
...PerformanceTests_BigQueryIO_Batch_Java_Json.yml | 2 +-
..._PerformanceTests_BigQueryIO_Streaming_Java.yml | 2 +-
...eam_PerformanceTests_BiqQueryIO_Read_Python.yml | 2 +-
...formanceTests_BiqQueryIO_Write_Python_Batch.yml | 2 +-
.github/workflows/beam_PerformanceTests_Cdap.yml | 2 +-
.../beam_PerformanceTests_Compressed_TextIOIT.yml | 2 +-
...m_PerformanceTests_Compressed_TextIOIT_HDFS.yml | 2 +-
.../beam_PerformanceTests_HadoopFormat.yml | 2 +-
.github/workflows/beam_PerformanceTests_JDBC.yml | 2 +-
.../workflows/beam_PerformanceTests_Kafka_IO.yml | 2 +-
.../beam_PerformanceTests_ManyFiles_TextIOIT.yml | 2 +-
...am_PerformanceTests_ManyFiles_TextIOIT_HDFS.yml | 2 +-
.../beam_PerformanceTests_MongoDBIO_IT.yml | 2 +-
.../beam_PerformanceTests_ParquetIOIT.yml | 2 +-
.../beam_PerformanceTests_ParquetIOIT_HDFS.yml | 2 +-
...erformanceTests_PubsubIOIT_Python_Streaming.yml | 2 +-
...m_PerformanceTests_SQLBigQueryIO_Batch_Java.yml | 2 +-
.../beam_PerformanceTests_SingleStoreIO.yml | 2 +-
..._PerformanceTests_SpannerIO_Read_2GB_Python.yml | 2 +-
...manceTests_SpannerIO_Write_2GB_Python_Batch.yml | 2 +-
.../beam_PerformanceTests_SparkReceiver_IO.yml | 2 +-
.../beam_PerformanceTests_TFRecordIOIT.yml | 2 +-
.../beam_PerformanceTests_TFRecordIOIT_HDFS.yml | 2 +-
.../workflows/beam_PerformanceTests_TextIOIT.yml | 2 +-
.../beam_PerformanceTests_TextIOIT_HDFS.yml | 2 +-
.../beam_PerformanceTests_TextIOIT_Python.yml | 2 +-
...PerformanceTests_WordCountIT_PythonVersions.yml | 2 +-
.../workflows/beam_PerformanceTests_XmlIOIT.yml | 2 +-
.../beam_PerformanceTests_XmlIOIT_HDFS.yml | 2 +-
.../beam_PerformanceTests_xlang_KafkaIO_Python.yml | 2 +-
.github/workflows/beam_Playground_CI_Nightly.yml | 2 +-
.github/workflows/beam_Playground_Precommit.yml | 5 +-
.github/workflows/beam_PostCommit_Go.yml | 2 +-
.../workflows/beam_PostCommit_Go_Dataflow_ARM.yml | 2 +-
.github/workflows/beam_PostCommit_Go_VR_Flink.yml | 2 +-
.github/workflows/beam_PostCommit_Go_VR_Spark.yml | 2 +-
.github/workflows/beam_PostCommit_Java.yml | 2 +-
.../beam_PostCommit_Java_Avro_Versions.yml | 2 +-
.../beam_PostCommit_Java_BigQueryEarlyRollout.yml | 2 +-
.../workflows/beam_PostCommit_Java_DataflowV1.yml | 2 +-
.../workflows/beam_PostCommit_Java_DataflowV2.yml | 2 +-
.../beam_PostCommit_Java_Examples_Dataflow.yml | 2 +-
.../beam_PostCommit_Java_Examples_Dataflow_ARM.yml | 2 +-
...beam_PostCommit_Java_Examples_Dataflow_Java.yml | 2 +-
.../beam_PostCommit_Java_Examples_Dataflow_V2.yml | 2 +-
...m_PostCommit_Java_Examples_Dataflow_V2_Java.yml | 2 +-
.../beam_PostCommit_Java_Examples_Direct.yml | 2 +-
.../beam_PostCommit_Java_Examples_Flink.yml | 2 +-
.../beam_PostCommit_Java_Examples_Spark.yml | 2 +-
.../beam_PostCommit_Java_Hadoop_Versions.yml | 2 +-
.../beam_PostCommit_Java_IO_Performance_Tests.yml | 4 +-
.../beam_PostCommit_Java_InfluxDbIO_IT.yml | 2 +-
.../beam_PostCommit_Java_Jpms_Dataflow.yml | 2 +-
...beam_PostCommit_Java_Jpms_Dataflow_Versions.yml | 2 +-
.../workflows/beam_PostCommit_Java_Jpms_Direct.yml | 2 +-
.../beam_PostCommit_Java_Jpms_Direct_Versions.yml | 2 +-
.../beam_PostCommit_Java_Jpms_Flink_Java11.yml | 2 +-
.../beam_PostCommit_Java_Jpms_Spark_Java11.yml | 2 +-
.../beam_PostCommit_Java_Nexmark_Dataflow.yml | 2 +-
.../beam_PostCommit_Java_Nexmark_Dataflow_V2.yml | 2 +-
...am_PostCommit_Java_Nexmark_Dataflow_V2_Java.yml | 2 +-
.../beam_PostCommit_Java_Nexmark_Direct.yml | 2 +-
.../beam_PostCommit_Java_Nexmark_Flink.yml | 5 +-
.../beam_PostCommit_Java_Nexmark_Spark.yml | 2 +-
.../beam_PostCommit_Java_PVR_Flink_Batch.yml | 2 +-
.../beam_PostCommit_Java_PVR_Flink_Streaming.yml | 2 +-
.../beam_PostCommit_Java_PVR_Spark3_Streaming.yml | 2 +-
.../beam_PostCommit_Java_PVR_Spark4_Batch.yml | 2 +-
.../beam_PostCommit_Java_PVR_Spark4_Streaming.yml | 2 +-
.../beam_PostCommit_Java_PVR_Spark_Batch.yml | 2 +-
.../beam_PostCommit_Java_SingleStoreIO_IT.yml | 2 +-
.../beam_PostCommit_Java_Tpcds_Dataflow.yml | 2 +-
.../workflows/beam_PostCommit_Java_Tpcds_Flink.yml | 2 +-
.../workflows/beam_PostCommit_Java_Tpcds_Spark.yml | 2 +-
...am_PostCommit_Java_ValidatesRunner_Dataflow.yml | 2 +-
..._Java_ValidatesRunner_Dataflow_JavaVersions.yml | 2 +-
...mit_Java_ValidatesRunner_Dataflow_Streaming.yml | 2 +-
...a_ValidatesRunner_Dataflow_Streaming_Engine.yml | 2 +-
...atesRunner_Dataflow_Streaming_TagEncodingV2.yml | 2 +-
...PostCommit_Java_ValidatesRunner_Dataflow_V2.yml | 2 +-
..._Java_ValidatesRunner_Dataflow_V2_Streaming.yml | 2 +-
...beam_PostCommit_Java_ValidatesRunner_Direct.yml | 2 +-
...it_Java_ValidatesRunner_Direct_JavaVersions.yml | 2 +-
.../beam_PostCommit_Java_ValidatesRunner_Flink.yml | 2 +-
.../beam_PostCommit_Java_ValidatesRunner_Spark.yml | 2 +-
...beam_PostCommit_Java_ValidatesRunner_Spark4.yml | 2 +-
...va_ValidatesRunner_SparkStructuredStreaming.yml | 2 +-
...am_PostCommit_Java_ValidatesRunner_Twister2.yml | 2 +-
.../beam_PostCommit_Java_ValidatesRunner_ULR.yml | 2 +-
.github/workflows/beam_PostCommit_Javadoc.yml | 2 +-
.../beam_PostCommit_PortableJar_Flink.yml | 2 +-
.../beam_PostCommit_PortableJar_Spark.yml | 2 +-
.github/workflows/beam_PostCommit_Python.yml | 2 +-
.github/workflows/beam_PostCommit_Python_Arm.yml | 2 +-
.../beam_PostCommit_Python_Dependency.yml | 2 +-
.../beam_PostCommit_Python_Examples_Dataflow.yml | 2 +-
.../beam_PostCommit_Python_Examples_Direct.yml | 2 +-
.../beam_PostCommit_Python_Examples_Flink.yml | 2 +-
.../beam_PostCommit_Python_Examples_Spark.yml | 2 +-
.../beam_PostCommit_Python_MongoDBIO_IT.yml | 2 +-
.../beam_PostCommit_Python_Nexmark_Direct.yml | 2 +-
.../beam_PostCommit_Python_Portable_Flink.yml | 2 +-
...stCommit_Python_ValidatesContainer_Dataflow.yml | 2 +-
..._Python_ValidatesContainer_Dataflow_With_RC.yml | 2 +-
..._PostCommit_Python_ValidatesRunner_Dataflow.yml | 2 +-
...eam_PostCommit_Python_ValidatesRunner_Flink.yml | 2 +-
...eam_PostCommit_Python_ValidatesRunner_Spark.yml | 2 +-
.../workflows/beam_PostCommit_Python_Versions.yml | 2 +-
.../beam_PostCommit_Python_Xlang_Gcp_Dataflow.yml | 2 +-
.../beam_PostCommit_Python_Xlang_Gcp_Direct.yml | 2 +-
.../beam_PostCommit_Python_Xlang_IO_Dataflow.yml | 2 +-
.../beam_PostCommit_Python_Xlang_IO_Direct.yml | 2 +-
...m_PostCommit_Python_Xlang_Messaging_Direct.yml} | 25 +-
.github/workflows/beam_PostCommit_SQL.yml | 2 +-
.../beam_PostCommit_TransformService_Direct.yml | 2 +-
.github/workflows/beam_PostCommit_Website_Test.yml | 2 +-
.github/workflows/beam_PostCommit_XVR_Direct.yml | 2 +-
.github/workflows/beam_PostCommit_XVR_Flink.yml | 6 +-
.../beam_PostCommit_XVR_GoUsingJava_Dataflow.yml | 6 +-
...eam_PostCommit_XVR_JavaUsingPython_Dataflow.yml | 6 +-
..._PostCommit_XVR_PythonUsingJavaSQL_Dataflow.yml | 6 +-
...eam_PostCommit_XVR_PythonUsingJava_Dataflow.yml | 6 +-
.github/workflows/beam_PostCommit_XVR_Spark3.yml | 6 +-
.../beam_PostCommit_Yaml_Xlang_Direct.yml | 2 +-
.../workflows/beam_PostRelease_NightlySnapshot.yml | 2 +-
.../workflows/beam_PreCommit_CommunityMetrics.yml | 2 +-
.../workflows/beam_PreCommit_Flink_Container.yml | 2 +-
.github/workflows/beam_PreCommit_GHA.yml | 2 +-
.github/workflows/beam_PreCommit_Go.yml | 2 +-
.github/workflows/beam_PreCommit_GoPortable.yml | 2 +-
.github/workflows/beam_PreCommit_GoPrism.yml | 2 +-
.github/workflows/beam_PreCommit_ItFramework.yml | 2 +-
.github/workflows/beam_PreCommit_Java.yml | 2 +-
...eCommit_Java_Amazon-Web-Services2_IO_Direct.yml | 2 +-
.../beam_PreCommit_Java_Amqp_IO_Direct.yml | 2 +-
.../beam_PreCommit_Java_Azure_IO_Direct.yml | 2 +-
.../beam_PreCommit_Java_Cassandra_IO_Direct.yml | 2 +-
.../beam_PreCommit_Java_Cdap_IO_Direct.yml | 2 +-
.../beam_PreCommit_Java_Clickhouse_IO_Direct.yml | 2 +-
.../beam_PreCommit_Java_Csv_IO_Direct.yml | 2 +-
.../beam_PreCommit_Java_Datadog_IO_Direct.yml | 2 +-
.github/workflows/beam_PreCommit_Java_Dataflow.yml | 2 +-
.../beam_PreCommit_Java_Debezium_IO_Direct.yml | 2 +-
.../beam_PreCommit_Java_Delta_IO_Direct.yml | 2 +-
...beam_PreCommit_Java_ElasticSearch_IO_Direct.yml | 2 +-
.../beam_PreCommit_Java_Examples_Dataflow.yml | 2 +-
...eam_PreCommit_Java_Examples_Dataflow_Java21.yml | 2 +-
...Commit_Java_File-schema-transform_IO_Direct.yml | 2 +-
.../beam_PreCommit_Java_Flink_Versions.yml | 2 +-
.../beam_PreCommit_Java_GCP_IO_Direct.yml | 2 +-
.../beam_PreCommit_Java_Google-ads_IO_Direct.yml | 2 +-
.../beam_PreCommit_Java_HBase_IO_Direct.yml | 2 +-
.../beam_PreCommit_Java_HCatalog_IO_Direct.yml | 2 +-
.../beam_PreCommit_Java_Hadoop_IO_Direct.yml | 2 +-
.../workflows/beam_PreCommit_Java_IOs_Direct.yml | 2 +-
.../beam_PreCommit_Java_InfluxDb_IO_Direct.yml | 2 +-
.../beam_PreCommit_Java_JDBC_IO_Direct.yml | 2 +-
.../beam_PreCommit_Java_Jms_IO_Direct.yml | 2 +-
.../beam_PreCommit_Java_Kafka_IO_Direct.yml | 2 +-
.../beam_PreCommit_Java_Kudu_IO_Direct.yml | 2 +-
.../beam_PreCommit_Java_MongoDb_IO_Direct.yml | 2 +-
.../beam_PreCommit_Java_Mqtt_IO_Direct.yml | 2 +-
.../beam_PreCommit_Java_Neo4j_IO_Direct.yml | 2 +-
.../beam_PreCommit_Java_PVR_Flink_Batch.yml | 2 +-
.../beam_PreCommit_Java_PVR_Flink_Docker.yml | 2 +-
.../beam_PreCommit_Java_PVR_Prism_Loopback.yml | 2 +-
.../beam_PreCommit_Java_Parquet_IO_Direct.yml | 2 +-
.../beam_PreCommit_Java_Pulsar_IO_Direct.yml | 2 +-
.../beam_PreCommit_Java_RabbitMq_IO_Direct.yml | 2 +-
.../beam_PreCommit_Java_Redis_IO_Direct.yml | 2 +-
...am_PreCommit_Java_RequestResponse_IO_Direct.yml | 2 +-
.../beam_PreCommit_Java_SingleStore_IO_Direct.yml | 2 +-
.../beam_PreCommit_Java_Snowflake_IO_Direct.yml | 2 +-
.../beam_PreCommit_Java_Solace_IO_Direct.yml | 2 +-
.../beam_PreCommit_Java_Solr_IO_Direct.yml | 2 +-
.../beam_PreCommit_Java_Spark_Versions.yml | 2 +-
.../beam_PreCommit_Java_Splunk_IO_Direct.yml | 2 +-
.../beam_PreCommit_Java_Thrift_IO_Direct.yml | 2 +-
.../beam_PreCommit_Java_Tika_IO_Direct.yml | 2 +-
.../workflows/beam_PreCommit_Kotlin_Examples.yml | 2 +-
.../workflows/beam_PreCommit_Portable_Python.yml | 2 +-
.github/workflows/beam_PreCommit_Prism_Python.yml | 2 +-
.github/workflows/beam_PreCommit_Python.yml | 2 +-
.github/workflows/beam_PreCommit_PythonDocker.yml | 2 +-
.github/workflows/beam_PreCommit_PythonDocs.yml | 2 +-
.../workflows/beam_PreCommit_PythonFormatter.yml | 2 +-
.github/workflows/beam_PreCommit_PythonLint.yml | 2 +-
.../workflows/beam_PreCommit_Python_Coverage.yml | 2 +-
.../workflows/beam_PreCommit_Python_Dataframes.yml | 2 +-
.github/workflows/beam_PreCommit_Python_Dill.yml | 2 +-
.../workflows/beam_PreCommit_Python_Examples.yml | 2 +-
.../beam_PreCommit_Python_Integration.yml | 2 +-
.github/workflows/beam_PreCommit_Python_ML.yml | 2 +-
.../workflows/beam_PreCommit_Python_PVR_Flink.yml | 2 +-
.../workflows/beam_PreCommit_Python_Runners.yml | 2 +-
.../workflows/beam_PreCommit_Python_Transforms.yml | 2 +-
.github/workflows/beam_PreCommit_RAT.yml | 2 +-
.github/workflows/beam_PreCommit_SQL.yml | 2 +-
.github/workflows/beam_PreCommit_SQL_Java17.yml | 2 +-
.github/workflows/beam_PreCommit_Spotless.yml | 2 +-
.github/workflows/beam_PreCommit_Typescript.yml | 2 +-
.github/workflows/beam_PreCommit_Website.yml | 2 +-
.../workflows/beam_PreCommit_Website_Stage_GCS.yml | 2 +-
.github/workflows/beam_PreCommit_Whitespace.yml | 2 +-
.../beam_PreCommit_Xlang_Generated_Transforms.yml | 2 +-
.../workflows/beam_PreCommit_Yaml_Xlang_Direct.yml | 3 +-
.github/workflows/beam_Prober_CommunityMetrics.yml | 2 +-
.github/workflows/beam_Publish_BeamMetrics.yml | 2 +-
.../workflows/beam_Publish_Beam_SDK_Snapshots.yml | 2 +-
.../workflows/beam_Publish_Docker_Snapshots.yml | 2 +-
...Jobs.yml => beam_Publish_Python_VLLM_Image.yml} | 71 +-
.github/workflows/beam_Publish_Website.yml | 4 +-
.../beam_Python_CostBenchmarks_Dataflow.yml | 2 +-
...beam_Python_ValidatesContainer_Dataflow_ARM.yml | 2 +-
.github/workflows/beam_Release_NightlySnapshot.yml | 2 +-
.../beam_Release_Python_NightlySnapshot.yml | 2 +-
.../workflows/beam_StressTests_Java_BigQueryIO.yml | 2 +-
.../workflows/beam_StressTests_Java_BigTableIO.yml | 2 +-
.../workflows/beam_StressTests_Java_KafkaIO.yml | 2 +-
.../workflows/beam_StressTests_Java_PubSubIO.yml | 2 +-
.../workflows/beam_StressTests_Java_SpannerIO.yml | 2 +-
.github/workflows/build_release_candidate.yml | 14 +-
.github/workflows/build_runner_image.yml | 2 +-
.github/workflows/build_wheels.yml | 6 +-
.github/workflows/cancel.yml | 2 +-
.github/workflows/code_completion_plugin_tests.yml | 4 +-
.github/workflows/codeql.yml | 2 +-
.github/workflows/cut_release_branch.yml | 9 +-
.github/workflows/dask_runner_tests.yml | 4 +-
.../workflows/deploy_release_candidate_pypi.yaml | 2 +-
.github/workflows/finalize_release.yml | 6 +-
.github/workflows/flaky_test_detection.yml | 2 +-
.github/workflows/git_tag_released_version.yml | 2 +-
.github/workflows/go_tests.yml | 2 +-
.github/workflows/issue-tagger.yml | 2 +-
.github/workflows/java_tests.yml | 4 +-
.github/workflows/local_env_tests.yml | 4 +-
.github/workflows/playground_frontend_test.yml | 4 +-
.github/workflows/pr-bot-new-prs.yml | 2 +-
.github/workflows/pr-bot-pr-updates.yml | 2 +-
.github/workflows/pr-bot-prs-needing-attention.yml | 2 +-
.github/workflows/publish_github_release_notes.yml | 4 +-
.github/workflows/python_dependency_tests.yml | 2 +-
.github/workflows/python_tests.yml | 8 +-
.github/workflows/refresh_looker_metrics.yml | 2 +-
.github/workflows/reportGenerator.yml | 2 +-
.../republish_released_docker_containers.yml | 2 +-
.github/workflows/run_perf_alert_tool.yml | 2 +-
.../workflows/run_rc_validation_go_wordcount.yml | 2 +-
.../run_rc_validation_java_mobile_gaming.yml | 2 +-
.../run_rc_validation_java_quickstart.yml | 2 +-
.../run_rc_validation_python_mobile_gaming.yml | 2 +-
.../workflows/run_rc_validation_python_yaml.yml | 2 +-
.github/workflows/tour_of_beam_backend.yml | 2 +-
.../workflows/tour_of_beam_backend_integration.yml | 2 +-
.github/workflows/tour_of_beam_frontend_test.yml | 4 +-
.github/workflows/typescript_tests.yml | 8 +-
.github/workflows/update_python_dependencies.yml | 4 +-
CHANGES.md | 10 +-
.../org/apache/beam/gradle/BeamModulePlugin.groovy | 36 +-
examples/java/common.gradle | 12 +-
gradle.properties | 4 +-
infra/enforcement/README.md | 12 +-
infra/enforcement/account_keys.py | 103 +-
infra/enforcement/sending.py | 105 +-
.../life_cycle/life_cycle_setuper_test.go | 3 +-
.../org/apache/beam/runners/core/DoFnRunners.java | 4 +-
.../runners/core/LateDataDroppingDoFnRunner.java | 17 +-
.../apache/beam/runners/core/ReduceFnRunner.java | 3 +-
.../core/LateDataDroppingDoFnRunnerTest.java | 4 +-
.../apache/beam/runners/core/ReduceFnTester.java | 20 +-
.../direct/ExecutorServiceParallelExecutor.java | 28 +-
runners/flink/2.1/build.gradle | 10 +-
runners/flink/2.1/job-server/build.gradle | 10 +-
runners/flink/2.2/build.gradle | 10 +-
runners/flink/2.2/job-server/build.gradle | 10 +-
runners/flink/job-server/flink_job_server.gradle | 20 +-
.../wrappers/streaming/WindowDoFnOperator.java | 2 +-
runners/google-cloud-dataflow-java/build.gradle | 7 +-
.../options/DataflowStreamingPipelineOptions.java | 19 +-
.../dataflow/worker/GroupAlsoByWindowsParDoFn.java | 2 +-
.../dataflow/worker/StreamingDataflowWorker.java | 8 +
.../worker/StreamingModeExecutionContext.java | 20 +
.../dataflow/worker/WindmillTimerInternals.java | 23 +
.../windmill/client/grpc/GrpcCommitWorkStream.java | 94 +-
.../client/grpc/GrpcWindmillStreamFactory.java | 10 +
.../windmill/state/WindmillStateInternals.java | 17 +
.../worker/StreamingGroupAlsoByWindowFnsTest.java | 3 +-
.../worker/StreamingModeExecutionContextTest.java | 60 +
.../client/grpc/GrpcCommitWorkStreamTest.java | 82 ++
scripts/beam-sql.sh | 2 +-
sdks/go.mod | 12 +-
sdks/go.sum | 28 +-
sdks/go/pkg/beam/core/core.go | 2 +-
sdks/go/test/build.gradle | 2 +-
.../org/apache/beam/sdk/io/CountingSource.java | 40 +-
.../org/apache/beam/sdk/transforms/JsonToRow.java | 8 +-
.../org/apache/beam/sdk/io/CountingSourceTest.java | 36 +
.../apache/beam/sdk/transforms/JsonToRowTest.java | 26 +
.../java/org/apache/beam/fn/harness/FnHarness.java | 6 +
.../beam/sdk/io/iceberg/IcebergScanConfig.java | 78 +-
.../apache/beam/sdk/io/iceberg/PartitionUtils.java | 100 +-
.../apache/beam/sdk/io/iceberg/ReadFromTasks.java | 4 +-
.../org/apache/beam/sdk/io/iceberg/ReadUtils.java | 137 +-
.../apache/beam/sdk/io/iceberg/ScanTaskReader.java | 4 +-
.../beam/sdk/io/iceberg/SerializableDataFile.java | 100 +-
.../sdk/io/iceberg/SerializableDeleteFile.java | 335 +++++
.../beam/sdk/io/iceberg/cdc/CdcOutputUtils.java | 178 +++
.../beam/sdk/io/iceberg/cdc/CdcReadUtils.java | 698 ++++++++++
.../beam/sdk/io/iceberg/cdc/DeleteReader.java | 309 +++++
.../io/iceberg/cdc/IcebergCdcMetadataColumns.java | 93 ++
.../io/iceberg/cdc/SerializableChangelogTask.java | 282 ++++
.../beam/sdk/io/iceberg/cdc}/package-info.java | 4 +-
.../iceberg/BaseIncrementalChangelogScan.java | 1014 ++++++++++++++
.../java/org/apache/iceberg}/package-info.java | 4 +-
.../IcebergCdcReadSchemaTransformProviderTest.java | 12 +-
.../beam/sdk/io/iceberg/IcebergIOReadTest.java | 83 +-
.../IcebergSchemaTransformTranslationTest.java | 33 +-
.../beam/sdk/io/iceberg/PartitionUtilsTest.java | 68 +
.../apache/beam/sdk/io/iceberg/ReadUtilsTest.java | 59 +-
.../sdk/io/iceberg/SerializableDataFileTest.java | 3 +
.../sdk/io/iceberg/SerializableDeleteFileTest.java | 222 ++++
.../beam/sdk/io/iceberg/TestDataWarehouse.java | 2 +-
.../io/iceberg/catalog/IcebergCatalogBaseIT.java | 7 +-
.../beam/sdk/io/iceberg/cdc/CdcReadUtilsTest.java | 381 ++++++
.../beam/sdk/io/iceberg/cdc/DeleteReaderTest.java | 419 ++++++
.../iceberg/cdc/SerializableChangelogTaskTest.java | 256 ++++
.../org/apache/beam/sdk/io/parquet/ParquetIO.java | 47 +
sdks/java/testing/nexmark/build.gradle | 15 +
sdks/java/testing/tpcds/build.gradle | 15 +
.../controller-container/Dockerfile | 2 +-
sdks/python/apache_beam/dataframe/frames.py | 7 +-
sdks/python/apache_beam/dataframe/frames_test.py | 4 +-
.../io/external/xlang_mqttio_it_test.py | 268 ++++
.../apache_beam/io/gcp/bigquery_file_loads.py | 2 +-
sdks/python/apache_beam/io/gcp/gcsio.py | 13 +-
.../apache_beam/io/gcp/gcsio_integration_test.py | 21 +-
sdks/python/apache_beam/io/gcp/gcsio_test.py | 40 +
sdks/python/apache_beam/io/iobase.py | 14 +-
sdks/python/apache_beam/io/iobase_test.py | 49 +-
sdks/python/apache_beam/io/unbounded_source.py | 996 ++++++++++++++
.../python/apache_beam/io/unbounded_source_test.py | 1380 ++++++++++++++++++++
sdks/python/apache_beam/ml/inference/base_test.py | 2 +-
.../ml/inference/pytorch_inference_test.py | 5 +-
.../ml/inference/sklearn_inference_test.py | 4 +-
.../ml/inference/tensorflow_inference_test.py | 4 +-
.../ml/inference/tensorrt_inference_test.py | 11 +-
.../apache_beam/runners/dataflow/internal/names.py | 2 +-
.../testing/benchmarks/cloudml/constraints.txt} | 46 +-
.../testing/benchmarks/cloudml/requirements.txt | 17 +-
sdks/python/apache_beam/transforms/stats_test.py | 20 +-
sdks/python/apache_beam/transforms/util_test.py | 49 +
sdks/python/apache_beam/version.py | 2 +-
sdks/python/apache_beam/yaml/examples/README.md | 4 +
.../yaml/extended_tests/databases/iceberg.yaml | 54 +-
sdks/python/apache_beam/yaml/integration_tests.py | 73 ++
sdks/python/apache_beam/yaml/main.py | 7 +-
sdks/python/apache_beam/yaml/yaml_provider.py | 11 +-
sdks/python/apache_beam/yaml/yaml_transform.py | 56 +-
.../python/apache_beam/yaml/yaml_transform_test.py | 119 ++
.../container/ml/py310/base_image_requirements.txt | 55 +-
.../container/ml/py310/gpu_image_requirements.txt | 73 +-
.../container/ml/py311/base_image_requirements.txt | 57 +-
.../container/ml/py311/gpu_image_requirements.txt | 75 +-
.../container/ml/py312/base_image_requirements.txt | 59 +-
.../container/ml/py312/gpu_image_requirements.txt | 77 +-
.../container/ml/py313/base_image_requirements.txt | 59 +-
.../container/py310/base_image_requirements.txt | 51 +-
.../container/py311/base_image_requirements.txt | 53 +-
.../container/py312/base_image_requirements.txt | 55 +-
.../container/py313/base_image_requirements.txt | 55 +-
.../container/py314/base_image_requirements.txt | 55 +-
sdks/python/pyproject.toml | 2 +-
sdks/python/pytest.ini | 1 +
sdks/python/setup.py | 6 +-
sdks/python/test-suites/dataflow/common.gradle | 10 +-
sdks/python/test-suites/direct/build.gradle | 6 +
sdks/python/test-suites/xlang/build.gradle | 8 +-
sdks/typescript/package.json | 2 +-
.../site/content/en/documentation/runners/flink.md | 4 +-
.../site/content/en/documentation/sdks/python.md | 6 +-
.../www/site/content/en/documentation/sdks/yaml.md | 34 +
.../site/content/en/get-started/quickstart-py.md | 2 +-
473 files changed, 9682 insertions(+), 1243 deletions(-)
copy .github/trigger_files/{beam_PostCommit_Java_Nexmark_Flink.json =>
beam_PostCommit_Java_Tpcds_Flink.json} (100%)
copy .github/trigger_files/{beam_PostCommit_SQL.json =>
beam_PostCommit_Python_Xlang_Messaging_Direct.json} (82%)
delete mode 100644 .github/trigger_files/beam_PostCommit_XVR_Samza.json
create mode 100644 .github/workflows/beam_Infrastructure_AuditUnmanagedKeys.yml
copy .github/workflows/{beam_PostCommit_XVR_PythonUsingJavaSQL_Dataflow.yml =>
beam_PostCommit_Python_Xlang_Messaging_Direct.yml} (82%)
copy .github/workflows/{beam_CancelStaleDataflowJobs.yml =>
beam_Publish_Python_VLLM_Image.yml} (55%)
create mode 100644
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/SerializableDeleteFile.java
create mode 100644
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/cdc/CdcOutputUtils.java
create mode 100644
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/cdc/CdcReadUtils.java
create mode 100644
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/cdc/DeleteReader.java
create mode 100644
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/cdc/IcebergCdcMetadataColumns.java
create mode 100644
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/cdc/SerializableChangelogTask.java
copy sdks/java/{core/jmh/src/main/java/org/apache/beam/sdk/jmh/io =>
io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/cdc}/package-info.java
(91%)
create mode 100644
sdks/java/io/iceberg/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java
copy
{runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/state
=> sdks/java/io/iceberg/src/main/java/org/apache/iceberg}/package-info.java
(90%)
create mode 100644
sdks/java/io/iceberg/src/test/java/org/apache/beam/sdk/io/iceberg/SerializableDeleteFileTest.java
create mode 100644
sdks/java/io/iceberg/src/test/java/org/apache/beam/sdk/io/iceberg/cdc/CdcReadUtilsTest.java
create mode 100644
sdks/java/io/iceberg/src/test/java/org/apache/beam/sdk/io/iceberg/cdc/DeleteReaderTest.java
create mode 100644
sdks/java/io/iceberg/src/test/java/org/apache/beam/sdk/io/iceberg/cdc/SerializableChangelogTaskTest.java
create mode 100644 sdks/python/apache_beam/io/external/xlang_mqttio_it_test.py
create mode 100644 sdks/python/apache_beam/io/unbounded_source.py
create mode 100644 sdks/python/apache_beam/io/unbounded_source_test.py
copy sdks/python/{scripts/run_eslint.sh =>
apache_beam/testing/benchmarks/cloudml/constraints.txt} (53%)
mode change 100755 => 100644
