This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a change to branch
dependabot/go_modules/sdks/cloud.google.com/go/bigtable-1.42.0
in repository https://gitbox.apache.org/repos/asf/beam.git
discard d537d94afe3 Bump cloud.google.com/go/bigtable from 1.41.0 to 1.42.0 in
/sdks
add 5240736a9bc [#37279] Document Ubuntu 24.04 Python version requirements
(#37297)
add beec6a3e156 Update logback version to address vulnerability (#37501)
add 15ba4874d72 Add schedule trigger for beam playground precommit (#37518)
add e2be9fd8e36 Plumb custom batch parameters for autosharding from
WriteFiles to FileIO. (#37463)
add d6920de30fe Add schedule trigger to Playground PreCommit (#37519)
add f504c5309bc Revert "[Python] Add take(n) convenience for PCollection
(#37470)" (#37493)
add dd92b8fc179 [Flink 2.0] Exercise validate runner test for classic
Flink runner (#37537)
add 99541ff4336 Fix flakiness in model manager and multiprocessshared
tests (#37529)
add 6b1d2cffca2 Align go rc validation inputs with other workflows (#37320)
add 37a9c1a14dd Bump go.mongodb.org/mongo-driver from 1.17.7 to 1.17.8 in
/sdks (#37469)
add 6602f1e125f Update RunInference to work with model manager (#37506)
add a0d27de943f Update queuing logic to avoid lock notify contention
(#37528)
add b7577e74a6a Bump golang.org/x/oauth2 from 0.34.0 to 0.35.0 in /sdks
(#37548)
add f52cf933446 feat: Introduce agent skills for various Apache Beam
developments (#37467)
add 4b478a70738 Exercise Flink PVR tests on Flink 2.0 (#37538)
add f96db3733c2 [SQL] Alter Catalog and Table statements (#36571)
add dd0f41d3fe7 Fix _OrderedUnionCoder is_deterministic check. (#37550)
add 25d3a2e46bf Add States for Pause (#37546)
add 257fa5a46d3 Fix Components visibility (#37551)
add 6dcb908e9fe Add back flex API option with improved pydoc (#37491)
add 5142be92511 [Java FnApi] Improve FinalizeCallbackManager to cleanup
timeout queue and use recommended ReentrantLock+Condition (#37304)
add f5aa2af52bd Bump pip from 25.3 to 26.0 in
/sdks/python/container/ml/py313 (#37485)
add 44eb7f63cdc Add RateLimiter Abstractions (#37547)
add d51ea3afb88 Add access to pipeline options during pipeline
construction and translation. (#37522)
add e8920c79a0d Add logic to handle end timestamp of mutable change stream
bounded query (#37459)
add 801bf89e3da Parse proto bytes change stream record in
ChangeStreamRecordMapper (#37427)
add d9adfa9c34f fix unparse bug (#37562)
add 195cc593590 Update python-pipeline-dependencies.md (#37561)
add 7df489cb7c6 add datadog it support (#37517)
add 91d3b6b1d39 move enum to top level class
add 4a2afe85f9f move enum to sdk core changed boolean causedByDrain in
WindowedValue interface to enum. spotless
add aaecfa8f545 Merge pull request #37549: Move CausedByDrain to top
level, move types
add 0228126c1b7 Update reviewers (#37572)
add d6390b86f0f Download performance looks to static while building
website (#37559)
add 22d1c576272 Add OOM handling for RunInference with model manager
(#37557)
add bafb270b48e Update go linter version (#37579)
add cea7923eef6 Use onlyif (#37578)
add 8c9d75567e7 Replace VarInt.getLength with Protobuf's branchless
implementation (#36959)
add 9ea7380de85 Refactor update compatibility checks and pipeline options
access (#37566)
add 72ce2074caf GCS client library migration in Java SDK - part 2 (#37502)
add cbf36740a53 Add Support for EnvoyRateLimiter Implementation (#37573)
add 695498f03a4 Update testcontainers version (#37591)
add eba1ec36adc Bigtable: skip reading large rows (#37586)
add 0adeae8b771 Optimize data size estimation for proto coder.
add 9a1dedfea25 Use existing _get_nested_size utility.
add 6e81f250977 Merge pull request #37584 Optimize data size estimation
for proto coder.
add c7d13c4d56d add aws and azure dependencies (#37602)
add 87aee9a58f6 Upgrade to debezium-io 3.1.3 (#37571)
add 25518f07fe2 Upgrade Spanner to the latest version (#37552)
add c69e8a18601 remove list_prefix deprecated method (#37587)
add e2a2b291921 Fix spelling mistakes in Go comments (#37617)
add 9a60847ad6b Update CHANGES.md for PR 37331 (#37599)
add 8da478be0d7 switch from v19 to v23 (#37620)
add 5849f2e38ea [yaml] : fix validate compatible method (#37588)
add 5d6cb04ea05 [yaml] : switch js2py to pythonmonkey (#37560)
add f2aa7912401 Exercise Python PVR tests on Flink 2.0 (#37313)
add 2ac0fe431e6 [yaml] : fix create different type elements issue (#37585)
add 66134f7107f Fix infer_result_type for pcollection with tags. (#37608)
add 97e5ccf0c6f Upgrade ClickHouseIO to use ClickHouse Java Client V2
(#37611)
add 90253bf5857 Tune Ratelimiter config (#37598)
add 76a53610898 Rename Chunk to EmbeddableItem (#37623)
add a9c14d1698d Add Ubuntu 24.04 runner pools (#37624)
add 0337ecefc61 Fix iceberg nested filter (#37340)
add 1ee2c717f3e Don't prematurely close FileIO (#37626)
add bbaace8b6be update changes for 2.72 - yaml etc (#37627)
add 0699f3594e6 Add post submit test for dataflow streaming with windmill
tag encoding v2 (#37603)
add bab2374552d Add Pause and Paused states (#37581)
add 3729c75d027 Improve RateLimiter Config
add 11586e46c39 add flag
add 9ac321abca9 fix readme
add 46b27eee882 Merge pull request #37630: Improve RateLimiter Config in
terraform example
add b46a6352ba9 Add HTTPS detection for ClickHouse JDBC URL parsing
(#37635)
add 51eec3c8658 Update to latest go version (#37638)
add 30bdbafcc95 feat: Improve YamlUtils.dumpAsMap error handling (#37468)
add f2765cab203 Fix bom upgrader and upgrade to latest (#37636)
add 808d23877bd Fix DoFnInvoker cache collision for generic types (#37355)
add 8dc5da5f500 Skip test for dataflow (#37612)
add e87bb292545 Bigtable: add an option to pass in row filter text proto
for use in template (#37632)
add 74fc46c404d GCS client library migration in Java SDK - part 2b (#37592)
add 9d528015cb6 Moving to 2.73.0-SNAPSHOT on master branch.
add f73bd6a0919 Update CHANGES.md
add cecc2a6ae0c Add image embedding support to ml/rag (#37628)
add 9690853b873 Bump golang.org/x/net from 0.49.0 to 0.50.0 in /sdks
(#37558)
add 19116300798 Bump cloud.google.com/go/bigtable from 1.41.0 to 1.42.0 in
/sdks
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (d537d94afe3)
\
N -- N -- N
refs/heads/dependabot/go_modules/sdks/cloud.google.com/go/bigtable-1.42.0
(19116300798)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.agent/skills/README.md | 62 ++
.agent/skills/beam-concepts/SKILL.md | 246 ++++++
.agent/skills/ci-cd/SKILL.md | 188 +++++
.agent/skills/contributing/SKILL.md | 149 ++++
.agent/skills/gradle-build/SKILL.md | 224 ++++++
.agent/skills/io-connectors/SKILL.md | 197 +++++
.agent/skills/java-development/SKILL.md | 153 ++++
.agent/skills/license-compliance/SKILL.md | 199 +++++
.agent/skills/python-development/SKILL.md | 183 +++++
.agent/skills/runners/SKILL.md | 244 ++++++
.asf.yaml | 1 +
.github/REVIEWERS.yml | 1 -
.../arc/environments/beam.env | 70 ++
.../arc/images/Dockerfile | 2 +-
.../IO_Iceberg_Integration_Tests.json | 2 +-
.../trigger_files/beam_PostCommit_Go_VR_Flink.json | 2 +-
.github/trigger_files/beam_PostCommit_Java.json | 4 +-
...m_PostCommit_Java_ValidatesRunner_Dataflow.json | 1 +
...esRunner_Dataflow_Streaming_TagEncodingV2.json} | 2 +-
...eam_PostCommit_Java_ValidatesRunner_Direct.json | 1 +
...beam_PostCommit_Java_ValidatesRunner_Flink.json | 1 +
...json => beam_PostCommit_PortableJar_Flink.json} | 0
.github/trigger_files/beam_PostCommit_Python.json | 4 +-
.../beam_PostCommit_Python_Examples_Flink.json | 3 +
.github/trigger_files/beam_PostCommit_SQL.json | 2 +-
.../trigger_files/beam_PreCommit_Python_Dill.json | 2 +-
.github/trigger_files/beam_PreCommit_SQL.json | 2 +-
.github/workflows/README.md | 1 +
.github/workflows/beam_Playground_Precommit.yml | 6 +-
.../beam_PostCommit_Java_PVR_Flink_Batch.yml | 21 +-
.../beam_PostCommit_Java_PVR_Flink_Streaming.yml | 8 +-
...tesRunner_Dataflow_Streaming_TagEncodingV2.yml} | 18 +-
.../beam_PostCommit_Java_ValidatesRunner_Flink.yml | 8 +-
.../beam_PostCommit_Python_Portable_Flink.yml | 5 +-
...eam_PostCommit_Python_ValidatesRunner_Flink.yml | 2 +-
.../workflows/beam_PreCommit_Python_PVR_Flink.yml | 1 +
.../workflows/beam_PreCommit_Website_Stage_GCS.yml | 12 +-
.github/workflows/beam_Publish_Website.yml | 8 +-
.../workflows/run_rc_validation_go_wordcount.yml | 24 +-
.github/workflows/stale.yml | 1 +
.gitignore | 7 +
CHANGES.md | 45 +-
CONTRIBUTING.md | 5 +-
build.gradle.kts | 5 +-
.../org/apache/beam/gradle/BeamModulePlugin.groovy | 24 +-
examples/java/build.gradle | 1 +
.../apache/beam/examples/RateLimiterSimple.java | 131 +++
examples/terraform/envoy-ratelimiter/README.md | 9 +-
examples/terraform/envoy-ratelimiter/gke.tf | 4 +-
examples/terraform/envoy-ratelimiter/ratelimit.tf | 138 +++-
examples/terraform/envoy-ratelimiter/variables.tf | 40 +-
gradle.properties | 4 +-
it/{mongodb => datadog}/build.gradle | 40 +-
.../beam/it/datadog/DatadogClientFactory.java} | 28 +-
.../apache/beam/it/datadog/DatadogLogEntry.java | 31 +-
.../beam/it/datadog/DatadogResourceManager.java | 335 ++++++++
.../datadog/DatadogResourceManagerException.java} | 10 +-
.../it/datadog/DatadogResourceManagerUtils.java | 70 ++
.../datadog/conditions/DatadogLogEntriesCheck.java | 91 +++
.../beam/it/datadog/conditions}/package-info.java | 4 +-
.../beam/it/datadog/matchers/DatadogAsserts.java} | 36 +-
.../beam/it/datadog/matchers}/package-info.java | 8 +-
.../org/apache/beam/it/datadog}/package-info.java | 8 +-
.../beam/it/datadog/DatadogResourceManagerIT.java | 77 ++
.../it/datadog/DatadogResourceManagerTest.java | 188 +++++
.../datadog/DatadogResourceManagerUtilsTest.java | 67 ++
playground/backend/cmd/server/controller.go | 2 +-
playground/backend/cmd/server/server.go | 4 +-
playground/backend/env_setup.sh | 2 +-
.../beam/runners/core/ReduceFnContextFactory.java | 10 +-
.../core/SplittableParDoViaKeyedWorkItems.java | 3 +-
.../apache/beam/runners/core/TimerInternals.java | 12 +-
.../runners/core/InMemoryTimerInternalsTest.java | 37 +-
.../beam/runners/core/KeyedWorkItemCoderTest.java | 3 +-
.../apache/beam/runners/core/ReduceFnTester.java | 5 +-
.../beam/runners/core/SimpleDoFnRunnerTest.java | 3 +-
.../SimplePushbackSideInputDoFnRunnerTest.java | 5 +-
.../beam/runners/core/TimerInternalsTest.java | 59 +-
.../core/triggers/TriggerStateMachineTester.java | 10 +-
.../runners/direct/DirectTimerInternalsTest.java | 13 +-
...cycleManagerRemovingTransformEvaluatorTest.java | 3 +-
.../beam/runners/direct/EvaluationContextTest.java | 3 +-
.../beam/runners/direct/WatermarkManagerTest.java | 85 +-
.../FlinkStreamingPortablePipelineTranslator.java | 26 +-
.../FlinkStreamingPortablePipelineTranslator.java | 21 +-
runners/google-cloud-dataflow-java/build.gradle | 63 +-
.../beam/runners/dataflow/BatchViewOverrides.java | 5 +-
.../beam/runners/dataflow/util/MonitoringUtil.java | 2 +
.../runners/dataflow/util/MonitoringUtilTest.java | 2 +
.../dataflow/worker/UngroupedWindmillReader.java | 7 +-
.../dataflow/worker/WindmillKeyedWorkItem.java | 7 +-
.../dataflow/worker/WindmillTimerInternals.java | 5 +-
.../dataflow/worker/util/ValueInEmptyWindows.java | 5 +-
.../worker/StreamingGroupAlsoByWindowFnsTest.java | 3 +-
...eamingKeyedWorkItemSideInputDoFnRunnerTest.java | 3 +-
.../worker/StreamingModeExecutionContextTest.java | 3 +-
.../worker/StreamingSideInputFetcherTest.java | 3 +-
.../dataflow/worker/UserParDoFnFactoryTest.java | 7 +-
.../dataflow/worker/WindmillKeyedWorkItemTest.java | 9 +-
.../windmill/state/WindmillTagEncodingV1Test.java | 15 +-
.../windmill/state/WindmillTagEncodingV2Test.java | 3 +-
.../samza/runtime/ClassicBundleManager.java | 3 +-
.../samza/runtime/PortableBundleManager.java | 3 +-
.../runners/samza/runtime/KeyedTimerDataTest.java | 3 +-
.../runtime/SamzaTimerInternalsFactoryTest.java | 25 +-
.../apache/beam/runners/spark/util/TimerUtils.java | 5 +-
scripts/beam-sql.sh | 2 +-
scripts/tools/bomupgrader.py | 1 -
sdks/go.mod | 18 +-
sdks/go.sum | 40 +-
sdks/go/pkg/beam/core/core.go | 2 +-
sdks/go/pkg/beam/core/runtime/harness/datamgr.go | 2 +-
.../runners/prism/internal/engine/teststream.go | 2 +-
sdks/go/test/build.gradle | 17 +-
sdks/go/test/run_validatesrunner_tests.sh | 6 +-
sdks/java/bom/gcp/build.gradle | 14 +-
.../resources/beam/checkstyle/suppressions.xml | 1 +
.../container/license_scripts/dep_urls_java.yaml | 2 +-
sdks/java/core/build.gradle | 1 -
.../main/java/org/apache/beam/sdk/io/FileIO.java | 66 ++
.../apache/beam/sdk/schemas/utils/YamlUtils.java | 33 +-
.../reflect/ByteBuddyDoFnInvokerFactory.java | 141 +++-
.../main/java/org/apache/beam/sdk/util/VarInt.java | 10 +-
.../org/apache/beam/sdk/values/CausedByDrain.java | 6 +-
.../org/apache/beam/sdk/values/OutputBuilder.java | 2 +-
.../org/apache/beam/sdk/values/WindowedValue.java | 2 +-
.../org/apache/beam/sdk/values/WindowedValues.java | 58 +-
.../java/org/apache/beam/sdk/io/FileIOTest.java | 139 ++++
.../sdk/transforms/reflect/DoFnInvokersTest.java | 53 +-
.../apache/beam/sdk/util/WindowedValueTest.java | 5 +-
.../org/apache/beam/sdk/util/YamlUtilsTest.java | 30 +
sdks/java/extensions/avro/build.gradle | 2 +
.../sdk/extensions/avro/coders/AvroCoderTest.java | 1 +
sdks/java/extensions/euphoria/build.gradle | 2 +-
.../google-cloud-platform-core/build.gradle | 3 +
.../beam/sdk/extensions/gcp/util/GcsUtil.java | 183 ++++-
.../beam/sdk/extensions/gcp/util/GcsUtilV1.java | 20 +-
.../beam/sdk/extensions/gcp/util/GcsUtilV2.java | 484 ++++++++++++
.../sdk/extensions/gcp/util/gcsfs/GcsPath.java | 15 +
.../sdk/extensions/gcp/GcpCoreApiSurfaceTest.java | 15 +-
.../gcp/util/GcsUtilParameterizedIT.java | 590 ++++++++++++++
.../sdk/extensions/gcp/util/gcsfs/GcsPathTest.java | 22 +
sdks/java/extensions/kryo/build.gradle | 2 +-
.../extensions/sql/expansion-service/build.gradle | 1 -
sdks/java/extensions/sql/iceberg/build.gradle | 2 +
.../provider/iceberg/IcebergAlterTableOps.java | 50 ++
.../sql/meta/provider/iceberg/IcebergCatalog.java | 50 +-
.../meta/provider/iceberg/IcebergMetastore.java | 16 +-
.../iceberg/BeamSqlCliIcebergAlterTest.java | 231 ++++++
.../extensions/sql/src/main/codegen/config.fmpp | 7 +
.../sql/src/main/codegen/includes/parserImpls.ftl | 120 ++-
.../sdk/extensions/sql/impl/CatalogSchema.java | 4 +
...{SqlCreateCatalog.java => SqlAlterCatalog.java} | 164 ++--
.../extensions/sql/impl/parser/SqlAlterTable.java | 254 ++++++
.../sql/impl/parser/SqlCreateExternalTable.java | 2 +-
.../extensions/sql/impl/parser/SqlDdlNodes.java | 41 +
.../sql/impl/parser/SqlDropDatabase.java | 6 +-
.../extensions/sql/impl/parser/SqlDropTable.java | 6 +-
.../extensions/sql/impl/parser/SqlUseDatabase.java | 2 +-
.../sdk/extensions/sql/meta/catalog/Catalog.java | 3 +
.../sql/meta/catalog/InMemoryCatalog.java | 8 +-
.../sql/meta/provider/AlterTableOps.java | 45 ++
.../sql/meta/provider/TableProvider.java | 5 +
.../sql/meta/provider/test/AlterTestTableOps.java | 101 +++
.../sql/meta/provider/test/TestTableProvider.java | 21 +
.../sql/meta/store/InMemoryMetaStore.java | 11 +
.../sdk/extensions/sql/BeamSqlCliCatalogTest.java | 93 +++
.../beam/sdk/extensions/sql/BeamSqlCliTest.java | 249 ++++++
.../fn/harness/control/FinalizeBundleHandler.java | 182 +++--
.../harness/control/FinalizeBundleHandlerTest.java | 49 ++
sdks/java/io/clickhouse/build.gradle | 7 +-
.../beam/sdk/io/clickhouse/ClickHouseIO.java | 393 +++++++--
.../sdk/io/clickhouse/ClickHouseJdbcUrlParser.java | 276 +++++++
.../apache/beam/sdk/io/clickhouse/TableSchema.java | 8 +-
.../beam/sdk/io/clickhouse/AtomicInsertTest.java | 28 +-
.../beam/sdk/io/clickhouse/BaseClickHouseTest.java | 140 +++-
.../ClickHouseIOJdbcBackwardCompatibilityTest.java | 98 +++
.../ClickHouseIOPropertyMergingTest.java | 211 +++++
.../beam/sdk/io/clickhouse/ClickHouseIOTest.java | 142 ++--
.../io/clickhouse/ClickHouseJdbcUrlParserTest.java | 429 ++++++++++
sdks/java/io/components/build.gradle | 5 +
.../components/deadletterqueue}/package-info.java | 4 +-
.../deadletterqueue/sinks}/package-info.java | 4 +-
.../components/ratelimiter/EnvoyRateLimiter.java} | 32 +-
.../ratelimiter/EnvoyRateLimiterContext.java | 65 ++
.../ratelimiter/EnvoyRateLimiterFactory.java | 239 ++++++
.../io/components/ratelimiter/RateLimiter.java} | 25 +-
.../ratelimiter/RateLimiterClientCache.java | 103 +++
.../ratelimiter/RateLimiterContext.java} | 14 +-
.../components/ratelimiter/RateLimiterFactory.java | 57 ++
.../components/ratelimiter/RateLimiterOptions.java | 68 ++
.../io/components/ratelimiter}/package-info.java | 4 +-
.../io/components/throttling}/package-info.java | 4 +-
.../ratelimiter/EnvoyRateLimiterTest.java | 168 ++++
.../ratelimiter/RateLimiterClientCacheTest.java | 115 +++
.../ratelimiter/RateLimiterOptionsTest.java | 81 ++
sdks/java/io/debezium/build.gradle | 8 +-
.../io/debezium/expansion-service/build.gradle | 2 +-
.../io/debezium/DebeziumIOMySqlConnectorIT.java | 4 +-
.../debezium/DebeziumIOPostgresSqlConnectorIT.java | 4 +-
sdks/java/io/expansion-service/build.gradle | 7 +-
sdks/java/io/google-ads/build.gradle | 4 +-
.../apache/beam/sdk/io/googleads/GoogleAdsIO.java | 6 +-
.../{GoogleAdsV19.java => GoogleAdsV23.java} | 56 +-
.../sdk/io/googleads/DummyRateLimitPolicy.java | 4 +-
.../beam/sdk/io/googleads/GoogleAdsIOTest.java | 40 +-
.../io/googleads/MockGoogleAdsClientFactory.java | 10 +-
sdks/java/io/google-cloud-platform/build.gradle | 16 +-
.../beam/sdk/io/gcp/bigtable/BigtableIO.java | 42 +-
.../sdk/io/gcp/bigtable/BigtableReadOptions.java | 12 +
.../io/gcp/bigtable/BigtableServiceFactory.java | 7 +-
.../sdk/io/gcp/bigtable/BigtableServiceImpl.java | 31 +-
.../beam/sdk/io/gcp/spanner/SpannerAccessor.java | 5 +
.../apache/beam/sdk/io/gcp/spanner/SpannerIO.java | 6 +-
.../changestreams/action/ActionFactory.java | 6 +-
.../action/QueryChangeStreamAction.java | 33 +-
.../changestreams/dao/ChangeStreamResultSet.java | 31 +
.../gcp/spanner/changestreams/dao/DaoFactory.java | 4 +
.../dofn/ReadChangeStreamPartitionDoFn.java | 5 +-
.../mapper/ChangeStreamRecordMapper.java | 13 +-
.../beam/sdk/io/gcp/bigtable/BigtableIOTest.java | 91 ++-
.../beam/sdk/io/gcp/bigtable/BigtableReadIT.java | 41 +
.../io/gcp/bigtable/BigtableServiceImplTest.java | 3 +-
.../beam/sdk/io/gcp/spanner/SpannerReadIT.java | 7 +-
.../beam/sdk/io/gcp/spanner/SpannerWriteIT.java | 7 +-
.../action/QueryChangeStreamActionTest.java | 120 ++-
.../dao/ChangeStreamResultSetTest.java | 60 ++
.../dofn/ReadChangeStreamPartitionDoFnTest.java | 25 +-
.../changestreams/it/IntegrationTestEnv.java | 7 +-
...annerChangeStreamPlacementTablePostgresIT.java} | 4 +-
.../mapper/ChangeStreamRecordMapperTest.java | 105 +++
sdks/java/io/iceberg/build.gradle | 4 +
.../apache/beam/sdk/io/iceberg/FilterUtils.java | 30 +-
.../beam/sdk/io/iceberg/IcebergCatalogConfig.java | 103 ++-
.../apache/beam/sdk/io/iceberg/PartitionUtils.java | 37 +
.../org/apache/beam/sdk/io/iceberg/ReadUtils.java | 10 +-
.../beam/sdk/io/iceberg/FilterUtilsTest.java | 22 +-
.../io/iceberg/catalog/IcebergCatalogBaseIT.java | 19 +
sdks/python/apache_beam/coders/coder_impl.py | 6 +
sdks/python/apache_beam/coders/coders.py | 34 +-
.../apache_beam/coders/coders_test_common.py | 208 ++---
sdks/python/apache_beam/coders/typecoders.py | 1 -
sdks/python/apache_beam/io/aws/s3io.py | 28 +-
sdks/python/apache_beam/io/aws/s3io_test.py | 36 +-
sdks/python/apache_beam/io/azure/blobstorageio.py | 26 +-
.../io/external/xlang_bigqueryio_it_test.py | 2 +-
.../io/external/xlang_debeziumio_it_test.py | 2 +-
.../apache_beam/io/gcp/bigquery_file_loads.py | 4 +-
.../apache_beam/io/gcp/bigquery_file_loads_test.py | 15 +-
sdks/python/apache_beam/io/gcp/gcsio.py | 26 +-
sdks/python/apache_beam/io/gcp/gcsio_test.py | 5 +-
sdks/python/apache_beam/ml/inference/base.py | 139 +++-
sdks/python/apache_beam/ml/inference/base_test.py | 104 ++-
.../apache_beam/ml/inference/gemini_inference.py | 31 +-
.../ml/inference/gemini_inference_test.py | 25 +
.../apache_beam/ml/inference/model_manager.py | 129 ++-
.../ml/inference/model_manager_it_test.py | 191 +++++
.../apache_beam/ml/inference/model_manager_test.py | 21 +-
sdks/python/apache_beam/ml/rag/embeddings/base.py | 66 +-
.../apache_beam/ml/rag/embeddings/base_test.py | 47 +-
.../apache_beam/ml/rag/embeddings/huggingface.py | 151 +++-
.../ml/rag/embeddings/huggingface_test.py | 135 ++++
.../apache_beam/ml/rag/embeddings/vertex_ai.py | 133 +++-
.../ml/rag/embeddings/vertex_ai_test.py | 128 +++
.../ml/rag/enrichment/bigquery_vector_search.py | 129 +--
.../apache_beam/ml/rag/enrichment/milvus_search.py | 71 +-
.../ml/rag/enrichment/milvus_search_test.py | 4 +-
.../python/apache_beam/ml/rag/ingestion/alloydb.py | 2 +-
sdks/python/apache_beam/ml/rag/ingestion/base.py | 34 +-
.../apache_beam/ml/rag/ingestion/bigquery.py | 58 +-
.../apache_beam/ml/rag/ingestion/cloudsql.py | 2 +-
.../apache_beam/ml/rag/ingestion/milvus_search.py | 32 +-
sdks/python/apache_beam/ml/rag/ingestion/mysql.py | 15 +-
.../apache_beam/ml/rag/ingestion/mysql_common.py | 133 ++--
.../apache_beam/ml/rag/ingestion/postgres.py | 14 +-
.../ml/rag/ingestion/postgres_common.py | 155 ++--
.../python/apache_beam/ml/rag/ingestion/spanner.py | 92 +--
sdks/python/apache_beam/ml/rag/types.py | 140 +++-
sdks/python/apache_beam/ml/rag/utils.py | 4 +-
sdks/python/apache_beam/ml/transforms/base_test.py | 4 +-
.../ml/transforms/embeddings/huggingface_test.py | 2 +-
.../transforms/embeddings/tensorflow_hub_test.py | 2 +-
.../ml/transforms/embeddings/vertex_ai.py | 10 +-
.../python/apache_beam/options/pipeline_options.py | 19 +
.../options/pipeline_options_context.py | 65 ++
.../options/pipeline_options_context_test.py | 335 ++++++++
.../apache_beam/options/pipeline_options_test.py | 66 ++
sdks/python/apache_beam/pipeline.py | 22 +-
sdks/python/apache_beam/pipeline_test.py | 2 +
sdks/python/apache_beam/pvalue.py | 19 -
.../runners/dataflow/dataflow_runner.py | 6 +-
.../runners/dataflow/dataflow_runner_test.py | 42 +
.../clients/dataflow/dataflow_v1b3_client.py | 515 ++++++------
.../clients/dataflow/dataflow_v1b3_messages.py | 879 +++++++++++++--------
.../runners/interactive/cache_manager.py | 2 +-
.../runners/portability/flink_runner_test.py | 29 +-
sdks/python/apache_beam/runners/runner.py | 2 +
sdks/python/apache_beam/transforms/external.py | 7 +-
sdks/python/apache_beam/transforms/ptransform.py | 6 +-
sdks/python/apache_beam/transforms/util.py | 110 +--
sdks/python/apache_beam/transforms/util_test.py | 117 +--
.../typehints/tagged_output_typehints_test.py | 84 ++
.../apache_beam/utils/multi_process_shared_test.py | 6 +-
sdks/python/apache_beam/version.py | 2 +-
sdks/python/apache_beam/yaml/json_utils.py | 30 +-
sdks/python/apache_beam/yaml/json_utils_test.py | 124 +++
sdks/python/apache_beam/yaml/standard_io.yaml | 1 +
sdks/python/apache_beam/yaml/tests/create.yaml | 18 +
sdks/python/apache_beam/yaml/yaml_mapping.py | 249 ++++--
sdks/python/apache_beam/yaml/yaml_provider.py | 32 +
.../apache_beam/yaml/yaml_provider_unit_test.py | 13 +
sdks/python/apache_beam/yaml/yaml_udf_test.py | 21 +-
.../container/ml/py310/base_image_requirements.txt | 88 ++-
.../container/ml/py310/gpu_image_requirements.txt | 120 +--
.../container/ml/py311/base_image_requirements.txt | 90 +--
.../container/ml/py311/gpu_image_requirements.txt | 120 +--
.../container/ml/py312/base_image_requirements.txt | 87 +-
.../container/ml/py312/gpu_image_requirements.txt | 116 +--
.../container/ml/py313/base_image_requirements.txt | 93 +--
.../container/py310/base_image_requirements.txt | 73 +-
.../container/py311/base_image_requirements.txt | 75 +-
.../container/py312/base_image_requirements.txt | 72 +-
.../container/py313/base_image_requirements.txt | 78 +-
sdks/python/setup.py | 7 +-
sdks/python/test-suites/portable/common.gradle | 64 +-
sdks/typescript/package.json | 2 +-
settings.gradle.kts | 2 +
website/Dockerfile | 17 +
website/build.gradle | 55 ++
.../sdks/python-pipeline-dependencies.md | 14 +
website/www/site/data/performance.yaml | 1 +
.../site/layouts/shortcodes/performance_looks.html | 21 +-
332 files changed, 14686 insertions(+), 3367 deletions(-)
create mode 100644 .agent/skills/README.md
create mode 100644 .agent/skills/beam-concepts/SKILL.md
create mode 100644 .agent/skills/ci-cd/SKILL.md
create mode 100644 .agent/skills/contributing/SKILL.md
create mode 100644 .agent/skills/gradle-build/SKILL.md
create mode 100644 .agent/skills/io-connectors/SKILL.md
create mode 100644 .agent/skills/java-development/SKILL.md
create mode 100644 .agent/skills/license-compliance/SKILL.md
create mode 100644 .agent/skills/python-development/SKILL.md
create mode 100644 .agent/skills/runners/SKILL.md
copy .github/trigger_files/{beam_PostCommit_Java_ValidatesRunner_Dataflow.json
=> beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming_TagEncodingV2.json}
(80%)
copy .github/trigger_files/{beam_PostCommit_Java_Examples_Dataflow_ARM.json =>
beam_PostCommit_PortableJar_Flink.json} (100%)
create mode 100644
.github/trigger_files/beam_PostCommit_Python_Examples_Flink.json
copy
.github/workflows/{beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming.yml
=> beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming_TagEncodingV2.yml}
(86%)
create mode 100644
examples/java/src/main/java/org/apache/beam/examples/RateLimiterSimple.java
copy it/{mongodb => datadog}/build.gradle (53%)
copy
it/{splunk/src/main/java/org/apache/beam/it/splunk/SplunkClientFactory.java =>
datadog/src/main/java/org/apache/beam/it/datadog/DatadogClientFactory.java}
(60%)
copy
sdks/java/io/datadog/src/main/java/org/apache/beam/sdk/io/datadog/DatadogEvent.java
=> it/datadog/src/main/java/org/apache/beam/it/datadog/DatadogLogEntry.java
(78%)
create mode 100644
it/datadog/src/main/java/org/apache/beam/it/datadog/DatadogResourceManager.java
copy
it/{mongodb/src/main/java/org/apache/beam/it/mongodb/MongoDBResourceManagerException.java
=>
datadog/src/main/java/org/apache/beam/it/datadog/DatadogResourceManagerException.java}
(75%)
create mode 100644
it/datadog/src/main/java/org/apache/beam/it/datadog/DatadogResourceManagerUtils.java
create mode 100644
it/datadog/src/main/java/org/apache/beam/it/datadog/conditions/DatadogLogEntriesCheck.java
copy it/{google-cloud-platform/src/main/java/org/apache/beam/it/gcp/pubsub =>
datadog/src/main/java/org/apache/beam/it/datadog/conditions}/package-info.java
(88%)
copy
it/{mongodb/src/main/java/org/apache/beam/it/mongodb/matchers/MongoDBAsserts.java
=>
datadog/src/main/java/org/apache/beam/it/datadog/matchers/DatadogAsserts.java}
(56%)
copy
{examples/java/src/main/java/org/apache/beam/examples/complete/datatokenization
=>
it/datadog/src/main/java/org/apache/beam/it/datadog/matchers}/package-info.java
(70%)
copy
{examples/java/src/main/java/org/apache/beam/examples/complete/datatokenization
=> it/datadog/src/main/java/org/apache/beam/it/datadog}/package-info.java (70%)
create mode 100644
it/datadog/src/test/java/org/apache/beam/it/datadog/DatadogResourceManagerIT.java
create mode 100644
it/datadog/src/test/java/org/apache/beam/it/datadog/DatadogResourceManagerTest.java
create mode 100644
it/datadog/src/test/java/org/apache/beam/it/datadog/DatadogResourceManagerUtilsTest.java
copy learning/tour-of-beam/frontend/lib/constants/storage_keys.dart =>
sdks/java/core/src/main/java/org/apache/beam/sdk/values/CausedByDrain.java (89%)
create mode 100644
sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtilV2.java
create mode 100644
sdks/java/extensions/google-cloud-platform-core/src/test/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtilParameterizedIT.java
create mode 100644
sdks/java/extensions/sql/iceberg/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/iceberg/IcebergAlterTableOps.java
create mode 100644
sdks/java/extensions/sql/iceberg/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/iceberg/BeamSqlCliIcebergAlterTest.java
copy
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/parser/{SqlCreateCatalog.java
=> SqlAlterCatalog.java} (54%)
create mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/parser/SqlAlterTable.java
create mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/AlterTableOps.java
create mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/test/AlterTestTableOps.java
create mode 100644
sdks/java/io/clickhouse/src/main/java/org/apache/beam/sdk/io/clickhouse/ClickHouseJdbcUrlParser.java
create mode 100644
sdks/java/io/clickhouse/src/test/java/org/apache/beam/sdk/io/clickhouse/ClickHouseIOJdbcBackwardCompatibilityTest.java
create mode 100644
sdks/java/io/clickhouse/src/test/java/org/apache/beam/sdk/io/clickhouse/ClickHouseIOPropertyMergingTest.java
create mode 100644
sdks/java/io/clickhouse/src/test/java/org/apache/beam/sdk/io/clickhouse/ClickHouseJdbcUrlParserTest.java
copy {it/splunk/src/main/java/org/apache/beam/it/splunk =>
sdks/java/io/components/src/main/java/org/apache/beam/sdk/io/components/deadletterqueue}/package-info.java
(86%)
copy {it/google-cloud-platform/src/main/java/org/apache/beam/it/gcp/dataflow
=>
sdks/java/io/components/src/main/java/org/apache/beam/sdk/io/components/deadletterqueue/sinks}/package-info.java
(86%)
copy
sdks/java/{core/src/test/java/org/apache/beam/sdk/util/FastNanoClockAndSleeper.java
=>
io/components/src/main/java/org/apache/beam/sdk/io/components/ratelimiter/EnvoyRateLimiter.java}
(55%)
create mode 100644
sdks/java/io/components/src/main/java/org/apache/beam/sdk/io/components/ratelimiter/EnvoyRateLimiterContext.java
create mode 100644
sdks/java/io/components/src/main/java/org/apache/beam/sdk/io/components/ratelimiter/EnvoyRateLimiterFactory.java
copy sdks/java/{core/src/main/java/org/apache/beam/sdk/util/ShardedFile.java
=>
io/components/src/main/java/org/apache/beam/sdk/io/components/ratelimiter/RateLimiter.java}
(50%)
create mode 100644
sdks/java/io/components/src/main/java/org/apache/beam/sdk/io/components/ratelimiter/RateLimiterClientCache.java
copy sdks/java/{core/src/main/java/org/apache/beam/sdk/metrics/Metric.java =>
io/components/src/main/java/org/apache/beam/sdk/io/components/ratelimiter/RateLimiterContext.java}
(77%)
create mode 100644
sdks/java/io/components/src/main/java/org/apache/beam/sdk/io/components/ratelimiter/RateLimiterFactory.java
create mode 100644
sdks/java/io/components/src/main/java/org/apache/beam/sdk/io/components/ratelimiter/RateLimiterOptions.java
copy {it/kafka/src/main/java/org/apache/beam/it/kafka =>
sdks/java/io/components/src/main/java/org/apache/beam/sdk/io/components/ratelimiter}/package-info.java
(87%)
copy
{examples/java/cdap/hubspot/src/main/java/org/apache/beam/examples/complete/cdap/hubspot/options
=>
sdks/java/io/components/src/main/java/org/apache/beam/sdk/io/components/throttling}/package-info.java
(87%)
create mode 100644
sdks/java/io/components/src/test/java/org/apache/beam/sdk/io/components/ratelimiter/EnvoyRateLimiterTest.java
create mode 100644
sdks/java/io/components/src/test/java/org/apache/beam/sdk/io/components/ratelimiter/RateLimiterClientCacheTest.java
create mode 100644
sdks/java/io/components/src/test/java/org/apache/beam/sdk/io/components/ratelimiter/RateLimiterOptionsTest.java
rename
sdks/java/io/google-ads/src/main/java/org/apache/beam/sdk/io/googleads/{GoogleAdsV19.java
=> GoogleAdsV23.java} (93%)
create mode 100644
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/dao/ChangeStreamResultSetTest.java
copy
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/it/{SpannerChangeStreamPostgresIT.java
=> SpannerChangeStreamPlacementTablePostgresIT.java} (98%)
create mode 100644
sdks/python/apache_beam/ml/inference/model_manager_it_test.py
create mode 100644 sdks/python/apache_beam/options/pipeline_options_context.py
create mode 100644
sdks/python/apache_beam/options/pipeline_options_context_test.py
