This is an automated email from the ASF dual-hosted git repository.
Amar3tto pushed a change to branch oss-image-detection
in repository https://gitbox.apache.org/repos/asf/beam.git
discard a7ba9dbae6f Resolve comments
discard d3a8653ce7c Move to one yml file
discard 5f31795342a Refactoring
discard fb4243e0614 Add beam_Inference_Python_Benchmarks_Dataflow_2 to
README.md
discard a1b01046d0e Refactor model and process init
discard 27424cb825d Remove rate limit
discard 9ce6c8e3fbd Fix formatting
discard 7dfdc4a38c2 Fix conflicts
discard d6def59aa38 Resolve comments [4]
discard 5379bffb469 Resolve comments [3]
discard 06384d55a59 Resolve comments [2]
discard 27428d2e965 Resolve comments [1]
discard 33d596b8a62 Add Looker IDs
discard e2711b20969 Fix lint
discard a5ca2a055f1 Fix formatting
discard 648aaebf124 Fix quotes
discard 564ecfb1c3b Fix clip blip
discard bc770f77a1d Fix streaming mode
discard ff93f0d42f4 Add all pipelines to separate workflow
discard cdeec42fdfb Fix float for imagenet rightfit
discard 3c8a7f42a40 Add Looker ids
discard f07d2fc9db3 Fix inference
discard 470bee51a23 Update torch for image captioning
discard 6e43258b2d1 Fix inference result
discard 2a00fc5ab7b Fix inference_fn
discard 63f8e420021 Add project argument
discard 30fa3f26cf6 Rename files
discard 2e85e64be64 Fix lint
discard 1315e8f1fc5 Add PyTorch Image Classification EfficientNet-B0 Streaming
(Right-fitting) ML pipeline
discard 5214df9caa0 Refactoring
discard d5834176aec Add PyTorch Image Captioning BLIP + CLIP Batch
discard c6fa34d5c69 Refactoring
discard ebecdb91cb9 ML pipelines: RunInference - OSS Image Object detection
add 034d7bf2190 Bump actions/github-script from 8 to 9 (#38402)
add 032cbe2d436 Bump actions/download-artifact from 7 to 8 (#38401)
add 5bc2d6da706 Bump actions/upload-artifact from 4 to 7 (#38400)
add 0972151ba60 Bump google.golang.org/grpc from 1.80.0 to 1.81.0 in /sdks
(#38399)
add 0975d7bc83a Bump github.com/lib/pq from 1.11.1 to 1.12.3 in /sdks
(#38398)
add bc900805f84 Bump github.com/nats-io/nats-server/v2 from 2.12.6 to
2.14.0 in /sdks (#38397)
add 2190c9fb001 Bump cloud.google.com/go/datastore from 1.22.0 to 1.23.0
in /sdks (#38392)
add b78e11ee2aa Stabilize StorageApiDataTriggeredSchemaUpdateIT assertion
(#38339)
add e9a2b6008aa Consolidate linting and type checking configs into
pyproject.toml (#38306)
add b30af9eb79c BatchElements transform for Java SDK (#38369)
add 422f6304af1 [yaml] - Add huggingface model handler (#38110)
add 02ac93ed697 Add DiskProvisionedIops/ThroughputMibps pipeline options
for the Python SDK (#38370)
add 22c43bf2bd5 Mitigate test broken after Avro Upgrade due to AVRO-4110
(#38405)
add b7b070790b6 Bump cloud.google.com/go/pubsub from 1.50.1 to 1.50.2 in
/sdks (#38415)
add c75b09d73d9 Bump cloud.google.com/go/storage from 1.62.0 to 1.62.1 in
/sdks (#38414)
add d273ee8a149 Bump github.com/go-sql-driver/mysql from 1.9.3 to 1.10.0
in /sdks (#38413)
add 878042d19da Bump github.com/aws/aws-sdk-go-v2 from 1.41.5 to 1.41.7 in
/sdks (#38416)
add 30e98f7fd52 yaml_transform_test - add some debug logs and increase row
count (#38367)
add e1b3769e7fb Fix test (#38418)
add a619a580802 fix rrio teardown executor cleanup path (#38417)
add a9cd0179508 Fix SystemError in _DeferredCall.get() under GC pressure
(#38355)
add 955b80aa694 Revert "Revert "Bump com.pswidersk.terraform-plugin from
1.0.0 to 1.1.1 (#325…" (#38420)
add b60082d17d4 Revert "[yaml] - Add huggingface model handler (#38110)"
(#38421)
add 81828fd2f0b [Dataflow] Added Portable Runner alias to java runners
(#38411)
add a8e7ffab371 Fix race conditions, error recovery, and exit handlers in
job servers (#38423)
add 9a6734bbe27 Fix PreCommit Java PVR Prism Loopback workflow (#38431)
add 81830f22515 Bump cloud.google.com/go/profiler from 0.4.3 to 0.6.0 in
/sdks (#38436)
add 2e6ff74e11e Bump golang.org/x/sys from 0.43.0 to 0.44.0 in /sdks
(#38435)
add 8363afe825f Bump cloud.google.com/go/spanner from 1.88.0 to 1.91.0 in
/sdks (#38434)
add 5733cc8115d Fix flaky BigQuery file loads by safely handling
concurrent mkdirs (#38426)
add a86f2ec49c8 upgrade test containers and fix issues (#38438)
add 399d9d7ea6b Introduce ValueKind to Java and add to WindowedValue
(#38315)
add 13bbd5c4d67 Reduce number of layers for Beam container images (#38440)
add f1bbb637e31 Fix deadlock in AsyncWrapper reset_state() (#38427)
add f01b9ddb05e Make Beartype use the default behavior in
is_consistent_with() (#38275)
add 1d008bad91f [runners-spark] Add Spark 4 runner (#38255)
add a72b78125ef Fix thread leak for LOOPBACK workers in external worker
pool (#38432)
add fba639a05ff Bump urllib3 from 2.6.3 to 2.7.0 in
/sdks/python/container/py312 (#38452)
add 81769cbfd13 Adds a new coder translator for Java SchemaCoder. (#37631)
add 118f2a3c3cf Bump github.com/nats-io/nats.go from 1.51.0 to 1.52.0 in
/sdks (#38463)
add fb2b8e8953c Bump actions/upload-artifact from 4 to 7 (#38460)
add abf0c7139d3 Bump github.com/aws/aws-sdk-go-v2/config from 1.32.7 to
1.32.17 in /sdks (#38459)
add 9bee04c92d3 Bump golang.org/x/net from 0.53.0 to 0.54.0 in /sdks
(#38462)
add 84a63ed6e85 Bump actions/checkout from 4 to 6 (#38464)
add 83706e1ad38 Bump github.com/aws/aws-sdk-go-v2/feature/s3/manager in
/sdks (#38461)
add 34c6e26112c Make SubprocessServer shared cache purging idempotent
(#38455)
add 4d683c0dd13 Exercise Spark 4 tests (#38453)
add fbe61eab131 [Python] Python] Bound the memory used for fnapi outbound
data messages and receiving messages. (#38407)
add cf5e517db67 update ruff to pyproject (#38474)
add 1f3d66546e7 Expose StorageWriteApiMaxRequestCallbackWaitTimeSec in BQ
storage write. (#38470)
add 89dde7c7a1b Upgrade gcsio to 3.1.16 (#38419)
add fa0eef96567 Add retry in connecting manager in MultiProcessShared
(#38456)
add ec65d642ccc [Prism] Fix gRPC deadline exceeded errors during bundle
failure by passing errgroup context (#38472)
add e7cb9f77973 [Website] add drain update to docs (#38450)
add 79ed7cc166d Remove legacy processContext usage across and replace it
with argument provider (#38366)
add e6fed0a2e08 Bump google.golang.org/api from 0.278.0 to 0.279.0 in
/sdks (#38480)
add 7ded0e0ce6f Setup Validates runner tests for Spark4 runner (#38478)
add bd52fce27a3 Moving to 2.75.0-SNAPSHOT on master branch.
add 13d36eb3b19 update port for jdbc test (#38485)
add 632f1e45d16 [runners-spark] Spark 4 follow-up: Announce in CHANGES.md
and address remaining review comments from #38255 (#38489)
add 3dbd7c8c35b Sickbay two failed tests due to new schema coder urn.
(#38497)
add f74b45af2cd Bump Java dev image for Dataflow (#38496)
add 10304c9cd78 Fix gradle command
add 442d6ff7bf2 Merge pull request #38499 from apache/fix-spark4-batch
add 39632ef2134 Fix iceberg gcs dependency after gcsio 3.0 upgrade (#38488)
add 1d6cdc0b4b6 Isolate tests in multi_process_shared with unique temp
path. (#38498)
add 5b5cd657c98 Bump @protobufjs/utf8 from 1.1.0 to 1.1.1 in
/sdks/typescript (#38475)
add 9a1a4b170e4 [Iceberg] pin hadoop to 3.3.6 (#38500)
add 7b4ae897cc7 Set torch upper bound
add 45790bbe2a2 Merge pull request #38505 from
shunping/set-torch-upper-bound
add 84a21684146 Bump google.golang.org/grpc from 1.81.0 to 1.81.1 in /sdks
(#38506)
add 5d2ff20e856 Enable multi-release attribute for Spark job server jar
(#38449)
add 6e23bc1ef3a Fix empty stream name encountered in
StorageApiFinalizeWrtiesDoFn (#38410)
add 6ae46e3c2d4 Bump jackson_version - Fix GHSA-72hv-8253-57qq (#37969)
add 4ac7598df14 Fix Dataflow archetype missing API client dependency
(#38509)
add b2d0ef66aa9 Fix pom involving maven-archetype and opentelemetry
(#38518)
add 357fd262211 Revert #37631 and #38497 on HEAD (#38516)
add 0496ea7117b Fix typo in README website description (#38469)
add 4c4a2c11e29 Revert "Fix Dataflow archetype missing API client
dependency (#38509)" (#38519)
add 1ca1fafcd62 Fix SubprocessServer cache thread-safety and test
isolation (#38501)
add 233ebd8e2bc [Dataflow Streaming] Add a job setting to limit value size
in windmill state cache (#38458)
add 2ea46979154 Fix non-unique job names in TextIOReadTest (#38242)
add 2c4d2c6de4d Rename run_pylint.sh (#38307)
add 26c6ec7cb74 add todo (#38517)
add 55eb624e5cd Fix interactive environment clean up failure at atexit.
(#38526)
add 79ae776fca7 Fix PipelineOptions deserialization NPE (#38531)
add 0318a2fa701 add yaml agent development skill (#38382)
add 01b09725ec3 Bump protobufjs in /sdks/typescript (#38538)
add c3335c887e6 Make sure session creation happens before starting agent
(#38477)
add 91f50ce5108 Remove Java 8 variant of
beam_PostCommit_Java_ValidatesRunner_Dataflow_JavaVersions (#38541)
add e83a1d5bad9 Pin tensor_rt digest for PyTorch sentiment Dataflow
benchmarks (#38374)
add b1f7115a633 Update Dataflow Dependency Java (#38542)
add 6b55c145853 Update Dataflow Proto (#38540)
add bb0bc772d4e Bump google.golang.org/api from 0.279.0 to 0.280.0 in
/sdks (#38563)
add ef703f65aff Bump cloud.google.com/go/datastore from 1.23.0 to 1.24.0
in /sdks (#38564)
add f22400a21a9 [Dataflow Streaming] Add experiment to use trigger state
to know if a window is new and optimize reads (#37574)
add 2105eaa4d23 Disable grpc fork support on some test suites. (#38566)
add c33bc972f15 Fix race condition in UserPipelineTracker.clear() and
various problems (#38537)
add 582bb452985 [Java SDK] Infer Beam logical types for JSR-310 and UUID
fields (#38194)
add ba63482cad5 introduce private method to remove clones (#38245)
add 2375fed9735 Refresh Iceberg partition specs periodically (#38408)
add 290e3721b50 WriteToJson - force num_shards (#38484)
add 4f2411e999e Revert "[Java Portable SDK] Configure JVM so that it exits
upon OutOfMemoryEr…" (#38567)
add 32ec9bb0946 Bump com.gradle.common-custom-user-data-gradle-plugin
(#38574)
add b9b9e78eeae Bump github.com/nats-io/nats-server/v2 from 2.14.0 to
2.14.1 in /sdks (#38576)
add 930b94cceb6 Fix test hang in subprocess expansion service on port bind
failure (#38572)
add ad93dd5ad64 Install wget and use repo token (#38340)
add 65e8b655195 huggingface model handler for yaml - retry (#38451)
add 707b48d4c6d Fix table row inference benchmark using wrong model path
(#38569)
add 898c1a8e22a Fix flaky MatrixPowerTest.test_basics by reading all
generated shards (#38585)
add 0c9b272c56b Update transform catalogue docs (#38457)
add b6aaf42e92e Initial skeleton for the Delta Lake source (#38571)
add 60620e788f0 Enforce Google Maven Mirror on CI environments (#38586)
add 1c0c02472c7 Adds backlog reporting support for non-fnapi based SDF's.
(#38346)
add 2203c8d4469 [Prism] Fix DEADLINE_EXCEEDED errors caused by worker
failures (#38523)
add 9d307e559eb Suppress log spams in gcsio 3.0 (#38588)
add 8f6e27117e1 Bump golang.org/x/net from 0.54.0 to 0.55.0 in /sdks
(#38595)
add 70d8f7dcfdd Revert "huggingface model handler for yaml - retry
(#38451)" (#38601)
add 63501f33968 Fix data_race condition in WindmillStreamSenderTest
(#38589)
add ef5f89f0508 [Iceberg] Fix manifest bounds being padded with trailing
0x00 bytes (#38580)
add 489efaa2400 Bump docker/build-push-action from 7.1.0 to 7.2.0 (#38597)
add 889ac54b236 Fix cancelled run handling in flaky test prefetcher
(#38579)
add a3e33da209c increase timeout (#38604)
add 978c37542c0 Revert "Bump golangci/golangci-lint-action from 3 to 8
(#36291)" (#38606)
add 590c9c27797 Report source lineage from HadoopFormatIO (#37265)
add 679adb85b14 Pipe Valuekind through DoFn and output builders (#38490)
add 7272ab5e5b8 [Python] Add type_overrides parameter to BigQuery I/O for
custom BigQuery-to-Python type mappings (#37253)
add d901481e851 Update Python Dependencies - manual trigger (#38610)
add db15fff4b9b Count cancelled runs in last-5 flaky check (#38614)
add 361c2c473a8 Reduce python expansion service startup time (#38611)
add bb5c486512d Update action.yml
add 94a9f824768 Bump docker/login-action from 4.1.0 to 4.2.0 (#38627)
add 8a30cc76045 Bump github.com/aws/aws-sdk-go-v2/config in /sdks (#38629)
add 35c70e5cb5b Bump docker/setup-buildx-action from 4.0.0 to 4.1.0
(#38628)
add 181424bb68d Bump github.com/aws/aws-sdk-go-v2/feature/s3/manager in
/sdks (#38630)
add 551a5a0c998 Update cython requirement from <4,>=3.0 to >=3.2.5,<4 in
/sdks/python (#38626)
add 5ad6df9f5a4 Fix flaky test_csv_to_json in Beam YAML SDK (#38617)
add 1858d2ef89a Update go dependencies (#38530)
add 1b261f0a399 Fix Fn API data plane deadlock when outbound queue is full
(#38581)
add 4049f00f165 [yaml] add support for matchall (#38512)
add eb29f866586 Update BEAM_DEV_SDK_CONTAINER_TAG to new version (#38699)
add d553b78ce2f [Dataflow Streaming] Add Operation::finishKey() and move
processTimers into it (#38430)
add 9583fe94755 [Cloud Spanner Change Streams] Fix inverted evaluation of
cancelQueryOnHeartbeat (#38695)
add b91538b4acc Bump cloud.google.com/go/bigtable from 1.47.0 to 1.48.0 in
/sdks (#38703)
add a7d2708d0f5 Bump google.golang.org/api from 0.280.0 to 0.281.0 in
/sdks (#38704)
add 45b79d23f9a Remove redundant HTTP->HTTPS redirect in .htaccess (#38522)
add 852d0720ac0 Refactor _SharedCache to handle context vs non-context
ownership (#38620)
add d2a3cee22e3 Enforce binary-only constraints during early requirements
cache attempts (#38688)
add 5096e63ecae Bump docker/setup-qemu-action from 4.0.0 to 4.1.0 (#38718)
add 3cb68c9bd68 Bump github.com/aws/aws-sdk-go-v2/service/s3 in /sdks
(#38719)
add 758042f0580 Bump google.golang.org/api from 0.281.0 to 0.282.0 in
/sdks (#38720)
add 789b278172c Bump github.com/aws/smithy-go from 1.25.1 to 1.26.0 in
/sdks (#38721)
add 54f7c7dd0d8 Bump github.com/aws/aws-sdk-go-v2/feature/s3/manager in
/sdks (#38722)
add 0302bf5529c Implemented MLTransform generate vocab Dataflow benchmark
(#38215)
add cd8b009d9fc [#38708] Add Flink 2.0 to SDK pipeline options validation
lists (#38726)
add a0494894c35 up timeout (#38732)
add 3f4bcb01570 remove playground annotations from BatchElementsExample
until next release (#38733)
add bdef23f826e Update CHANGES.md after release cut (#38608)
add 3e5a085bd0d Support infer types involving dataclass fields (#38548)
add a32774afe93 [Prism] Fix race condition in element manager (#38734)
add cbab4d7203a Add staged package hashes (#38311)
add d8d4cfc3e11 Updated All Externally Visible Logs/Docs from Runner V2 to
Portable Runner (#38532)
add 3d68e9df172 Bump github.com/aws/aws-sdk-go-v2 from 1.41.7 to 1.41.8 in
/sdks (#38740)
add dc3b2ccf8e3 Bump github.com/aws/aws-sdk-go-v2/credentials in /sdks
(#38737)
add 09a1e60fa38 fix race condition in sdb-operator (#38698)
add 5702872d9ae Disable build isolation for pip by default. (#38700)
add 07222b4dac6 Bump github.com/aws/aws-sdk-go-v2/service/s3 in /sdks
(#38741)
add 0a7df5c8ba4 Bump github.com/aws/aws-sdk-go-v2/config in /sdks (#38738)
add d9f07c5f6aa Bump setup gradle to allowlisted v6.1.0 (#38745)
add 5a5ef0cc02a Bump github.com/aws/aws-sdk-go-v2/feature/s3/manager in
/sdks (#38739)
add 6fba3cb34b2 Implement MLTransform One-Hot Encoding benchmark pipeline
(#38404)
add 2b5c1131af3 [yaml] - mongodb write normalization (#38376)
add f771827d9f1 Disable gRPC fork support in
PortableRunnerTestWithSubprocesses (#38744)
add cbc2c21edad Fix flaky
TestDataSampler/GetSamplesForPCollectionsTooManySamples (#38736)
add 44a2ff6146c Re-enable call-args check in yaml_io.py (#38730)
add c31f2327041 Fix Reshuffle handling in Prism to recursively remove
nested sub-transforms (#38735)
add 8ac0348beb9 Hugging face model handler #3 (#38696)
add 8364da70a4e Handle empty batches in Python worker counters (#38748)
add 8b07c7a1925 feat(ml): add qdrant ingestion (#38142)
add fe27feb2ab5 [Dataflow Streaming] Enable state tag encoding v2 (#38705)
add a1b0a51efca Bump nanasess/setup-chromedriver from 2 to 3 (#38757)
add 97ae60b0543 Bump github.com/aws/aws-sdk-go-v2/service/s3 in /sdks
(#38758)
add e23bf2d7e0b Bump github.com/golang-cz/devslog from 0.0.15 to 0.0.16 in
/sdks (#38759)
add 95c2161fc73 Bump github.com/aws/aws-sdk-go-v2 from 1.41.8 to 1.41.9 in
/sdks (#38761)
add 21d13388a94 Bump github.com/tetratelabs/wazero from 1.11.0 to 1.12.0
in /sdks (#38762)
add 6f2c8dd891f Bump github.com/aws/aws-sdk-go-v2/feature/s3/manager in
/sdks (#38760)
add 2400917d918 Add Worker hashes for Golang (#38754)
add 2ed225f3c4d Updates Beam master container tags for Dataflow (#38766)
add 98ca01f415c [Dataflow Streaming] Prepare BoundedQueueExecutor for
MultiKey bundles (#38592)
add c315ceacb5c Fix truncating file handle (#38425)
add 54accc32caf js2py to quickjs (#38473)
add fd0c3e48218 Migrate to Google Cloud Dataflow Client (#37639)
add 4f2d8c0ec2d Adding release-2.74.0-postrelease to protected branches in
.asf.yaml
add 8c93ab0efae Update managed-io.md for release 2.74.0-RC1. (#38527)
add ef3d8cccb43 Update Beam website to release 2.74.0 (#38536)
add 13a46356a02 Update Python Dependencies (#38775)
add 7916452e232 Fix internal test configuration for Dataflow client
(#38792)
add a83754a5360 Implements the Delta Lake source with support for
splitting (#38706)
add bec756f89b8 Fix offlineRepositoryRoot relative to Beam repo root
(#38790)
add 41d5dd4f55e Update republish released docker workflow defaults to
2.74.0 (#38796)
add 8346a68b5bc Update beam-master container (#38795)
add 96165fcbc52 Bump github.com/aws/aws-sdk-go-v2/service/s3 in /sdks
(#38801)
add a0c96065833 Bump github.com/aws/aws-sdk-go-v2/config in /sdks (#38804)
add fcdd87f0aad [OpenTelemetry] FNHarness, Dataflow Runner v1 - set open
telemetry settings. (#38785)
add 36c8db6c75f [OpenTelemetry] Default open telemetry configuration
behind experiment (#38784)
add 8fa601ae949 Bump google.golang.org/api from 0.282.0 to 0.283.0 in
/sdks (#38809)
add 2b3df10f21b Bump github.com/aws/aws-sdk-go-v2/feature/s3/manager in
/sdks (#38810)
add 26b21e964ae Bump github.com/nats-io/nats-server/v2 from 2.14.1 to
2.14.2 in /sdks (#38811)
add db1b852750c Upgrade Google Cloud libraries-bom to 26.83.0 (#38799)
add 3a99f94bd79 feat: Add LogElements transform to Java SDK (#38533)
add 53e4a8c2bfe Move google-auth lower bounds to >= 2.0.0 (#38778)
add dd3a4549d89 Normalize types in dataclass field type resolving (#38797)
add c01ceeab24c [Python] Honor disableCounterMetrics,
disableStringSetMetrics, and disableBoundedTrieMetrics experiments (#38749)
add aaf677b16c3 [Dataflow] Fix thread safety of HotKey logger (#38816)
add 0f5b5142c08 Fix BigQuery Storage Write API stream count for bounded
writes (#38776)
add 82f84e15ccf Bump github.com/aws/aws-sdk-go-v2/feature/s3/manager in
/sdks (#38825)
add dd2a86c178f Bump cloud.google.com/go/storage from 1.62.2 to 1.62.3 in
/sdks (#38826)
add 4fe3342c71f [OpenTelemetry] Add gcp otel auth extension (#38773)
add e353462e0a8 Revert "remove playground annotations from
BatchElementsExample until next re…" (#38830)
add 6c0db120f38 Automate OpenTelemetry license entries in bomupgrader
(#38815)
add 66e543997f1 support generics in from row and to row conversions
(#37347)
add d9117f423d9 [Dataflow Streaming] [Multi Key] Introduce
KeyGroupWorkQueue and integrate with BoundedQueueExecutor (#38767)
add 27341e2bdef Update SpannerIO.java - comment documentation (#38492)
add f88c2609d1a Bump github.com/aws/smithy-go from 1.27.1 to 1.27.2 in
/sdks (#38841)
add 34bedad38e3 [Experimental] Fix Yaml Xlang Test Timeout (#38798)
add 52d10870893 Fix parsing of dataflow api endpoint URLs to remove
trailing slash (#38847)
add b452ad63fb0 Update Gemini text classification to gemini-2.5-flash
(#38839)
add 726933b81db fix flaky Arm postcommit test collection on Python 3.14
(#38844)
add 3732cffb33c [Dataflow Streaming] Fix nullness supression in
StreamingModeExecutionContext (#38842)
add 2eff9f19340 Fix ensurepip bundled pip upgrade for Python 3.12+ (#38765)
add 0eac0389a4c Fix cloudML TFT install avoiding pip ResolutionTooDeep
(#38781)
add d521df8f1ed use local variable (#38850)
add 0bf63a295cc Fix Dataflow cost benchmark after Dataflow client
migration (#38788)
add 59c7a42257d Bump github.com/aws/aws-sdk-go-v2/service/s3 in /sdks
(#38856)
add 945e6e140d2 Bump golang.org/x/sys from 0.45.0 to 0.46.0 in /sdks
(#38857)
add 83a8138b825 Unpin google-cloud-bigtable version (fixes apache#37637)
(#38861)
add a9d7d2cb92e Bump distlib from 0.3.9 to 0.4.2 in /sdks/python (#38855)
add 7050d00f5fd Bump github.com/aws/aws-sdk-go-v2/credentials in /sdks
(#38858)
add 9cda5a33300 Bump golang.org/x/text from 0.37.0 to 0.38.0 in /sdks
(#38860)
add f9b4ee9cc82 Disable public ip for Dataflow Validates Runner test
(#38424)
add 9bcdedda196 increase timeout (#38864)
add 8e8327d6f90 Adds a Github action for running Delta Lake I/O unit tests
(#38869)
add fe7dd9e942d Enable code scanning alerts (#38817)
add 0856c22a9e9 Side Input improvements (#38363)
add fcfec9af467 Support global sort and improve UDF overload resolution in
Beam SQL (#38832)
add ca0615cc614 Update huggingface model handler info (#38789)
add 3b104b6d3b7 Fix yaml doc generation (#38874)
add ea2fde7a020 Fix race condition in statesampler_fast.pyx (#38851)
add a3cb642db76 Merge pull request #38878: Fix OnWindowExpirationContext.
add 872cb901dca Bump github.com/aws/aws-sdk-go-v2/config in /sdks (#38884)
add 530d0f91b10 Bump google.golang.org/api from 0.283.0 to 0.284.0 in
/sdks (#38887)
add d7948d07b80 Bump github.com/aws/aws-sdk-go-v2/feature/s3/manager in
/sdks (#38886)
add be50185053f Bump actions/checkout from 4 to 6 (#38885)
add 6eed130a176 update iceberg add files parameters (#38879)
add 20ac8c4f35a add datadog io normalization and yaml (#38362)
add 6a2dbb67985 Bump torch (#38896)
add 02168d3babe Revert "[Python] Honor disableCounterMetrics,
disableStringSetMetrics, and disableBoundedTrieMetrics experiments" (#38901)
add 043556c4b90 Bump torch (#38897)
add 3eb0031274e Bump torch (#38903)
add 081c61ee5b4 Fix Xlang IO Direct installGcpTest uv cache lock (#38862)
add 8051efce4c2 Turn to manual build for Go for CodeQL (#38907)
add 1cf3545e5b7 Fix Dataflow legacy worker abort loop thread death issue
(#38894)
add 8e859798a6f Bump cloud.google.com/go/bigtable from 1.48.0 to 1.49.0 in
/sdks (#38913)
add 48fe93cb0eb Bump github.com/aws/aws-sdk-go-v2/config in /sdks (#38916)
add 60621107046 Bump golang.org/x/net from 0.55.0 to 0.56.0 in /sdks
(#38915)
add 51e35e52440 [IcebergIO] Improve TableCache (#38882)
add c297c0dc4ab Bump github.com/aws/aws-sdk-go-v2/feature/s3/manager in
/sdks (#38912)
add 61a7368bdb7 Use p310_ml_test (#38890)
add 135a5593441 Replace ClassLoadingStrategy with custom loading strategy
(#38906)
add a40685ccc62 Don't preempt snapshot runs (#38904)
add 8ea81a9f289 Bump @grpc/grpc-js from 1.14.3 to 1.14.4 in
/sdks/typescript (#38922)
add 6f30f147462 Adds a new agent SKILL for developing new I/O connectors
(#38910)
add 782953e3422 Add Delta Lake source to the Java Managed API (#38902)
add afbbc31369b Implement Asynchronous wrapper for DoFn in Java SDK
(#38609)
add 8d15c63d551 Updates references to the I/O connector skill (#38923)
add 6a30a3d68de Updates CHANGES.md to mention the Delta Lake source
(#38924)
add 03f4f797953 [SQL] Support positional parameters (#38880)
add 5307c7138af Support MAP type in RowJson and fix datetime parsing with
spaces (#38865)
add 56a41422765 SQL DDL documentation (#37539)
add a88686c1594 Add DoFnRunner::finishKey() method (#38454)
add 705db250d51 [Dataflow Streaming] Activate SourceState Finalizers
before submitting workitem to harness threads (#38921)
add e8249afb1f7 Fix wordcount_rust requirements.txt and documentation
(#38877)
add 64bb91099a9 Exclude testSideInputNotReadyTimer from Spark batch suites
(#38939)
add 7d642a024a4 Add Gemini RunInference example notebook (#38943)
add 712d981ea79 update agent skills table (#38818)
add 6ffee9feebb [Infra] Add beam_viewer and beam_writer roles for GSoC
2026 participant (#38933)
add ebde16e3713 Add instrumentation for memory profiling in Python SDK
(#38853)
add 22ac580f827 [#38059] Fix GCS glob matching to support ** and / in
object names (#38099)
add 6c083df5f8e SQL Database DDL & Usability Improvements (#38952)
add 001093e5e0b [mqtt] Add portable MqttIO Read/Write transforms for batch
and streaming (revives #32385) (#38493)
add b3a62ee97c7 Upgrade expansion jar to java17 (#38931)
add 6ed0635429a Spanner: Extend SpannerIO to support Spanner Omni mTLS
setup (#38928)
add 378e10de4ea ClickHouseIO: Add DateTime64 support for sub-second
timestamp precision (#38510)
add 98e29d99fcd Revert "Upgrade expansion jar to java17 (#38931)" (#38965)
add bde545f91e4 Add support for STDDEV_POP and STDDEV_SAMP in VarianceFn
(#38871)
add 77348121ee9 Update BEAM_DEV_SDK_CONTAINER_TAG version (#38963)
add a996a9fb610 Bump protobufjs from 8.4.0 to 8.6.0 in /sdks/typescript
(#38968)
add 2c4822959e1 Bump form-data from 2.5.5 to 2.5.6 in /sdks/typescript
(#38967)
add b7bc937f592 Use Flink 2.0 for load tests (#38279)
add c0cc8897f4a Bump cloud.google.com/go/spanner from 1.91.0 to 1.92.0 in
/sdks (#38976)
add 468ae40dc8e Fix AsyncWrapperTest timeout and flakiness issues. (#38970)
add 4a10b29b7ea ML pipelines: RunInference - OSS Image Object detection
add cfa735c3c7c Refactoring
add d18a9600a29 Add PyTorch Image Captioning BLIP + CLIP Batch
add 8cb9c3afd0a Refactoring
add 69457d74324 Add PyTorch Image Classification EfficientNet-B0 Streaming
(Right-fitting) ML pipeline
add ee87ba0d223 Fix lint
add c6cd23b8807 Rename files
add 94acc5af830 Add project argument
add b3930842373 Fix inference_fn
add 3a1d079c25e Fix inference result
add b09cc145f04 Update torch for image captioning
add 7cdcf9fb8fd Fix inference
add a0c4b2e22d8 Add Looker ids
add 56f60759f2a Fix float for imagenet rightfit
add 5a5cc4f6483 Add all pipelines to separate workflow
add 1fe599edd26 Fix streaming mode
add 5c1694c2d03 Fix clip blip
add 91eded47323 Fix quotes
add 53a281f5494 Fix formatting
add 0544af5cf56 Fix lint
add 2fa46dc18f2 Add Looker IDs
add bcb5b4f50ca Resolve comments [1]
add 00955662b9d Resolve comments [2]
add 742a10ddb0c Resolve comments [3]
add d21f103b2a4 Resolve comments [4]
add 8983083861e Fix conflicts
add 1f407795f64 Fix formatting
add 32fe302a802 Remove rate limit
add 241091d531d Refactor model and process init
add 98a39cbc7f7 Add beam_Inference_Python_Benchmarks_Dataflow_2 to
README.md
add 7b78ff990de Refactoring
add 9274594e8f4 Move to one yml file
add 7b0a0df3ca5 Resolve comments
add 439b5aa7580 Resolve conflicts
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (a7ba9dbae6f)
\
N -- N -- N refs/heads/oss-image-detection (439b5aa7580)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.agent/skills/README.md | 4 +
.../skills/developing-new-io-connectors/SKILL.md | 347 +
.agent/skills/io-connectors/SKILL.md | 3 +-
.agent/skills/yaml-development/SKILL.md | 107 +
.asf.yaml | 2 +
.../test-properties.json | 2 +-
.../actions/setup-environment-action/action.yml | 2 +-
.github/autolabeler.yml | 1 +
.github/build.gradle | 10 +-
.../IO_Iceberg_Integration_Tests_Dataflow.json | 2 +-
.../beam_CloudML_Benchmarks_Dataflow.json | 2 +-
.github/trigger_files/beam_PostCommit_Go.json | 2 +-
.../beam_PostCommit_Java_DataflowV1.json | 2 +-
...m_PostCommit_Java_ValidatesRunner_Dataflow.json | 2 +-
..._ValidatesRunner_Dataflow_Streaming_Engine.json | 2 +-
...ostCommit_Java_ValidatesRunner_Dataflow_V2.json | 2 +-
.github/trigger_files/beam_PostCommit_Python.json | 2 +-
...PostCommit_Python_ValidatesRunner_Dataflow.json | 2 +-
...t.json => beam_PostCommit_Python_Versions.json} | 2 +-
.../beam_PostCommit_Python_Xlang_IO_Direct.json | 2 +-
.github/trigger_files/beam_PostCommit_SQL.json | 2 +-
.../trigger_files/beam_PostCommit_XVR_Direct.json | 2 +-
.../beam_PostCommit_Yaml_Xlang_Direct.json | 2 +-
...on => beam_PreCommit_Java_Delta_IO_Direct.json} | 0
...t.json => beam_PreCommit_Python_PVR_Flink.json} | 0
.../beam_PreCommit_Yaml_Xlang_Direct.json | 2 +-
.github/workflows/README.md | 6 +-
.github/workflows/assign_milestone.yml | 2 +-
.../beam_Inference_Python_Benchmarks_Dataflow.yml | 27 +-
.../beam_LoadTests_Go_CoGBK_Flink_batch.yml | 4 +-
.../beam_LoadTests_Go_Combine_Flink_Batch.yml | 4 +-
.../beam_LoadTests_Go_GBK_Flink_Batch.yml | 4 +-
.../beam_LoadTests_Go_ParDo_Flink_Batch.yml | 4 +-
.../beam_LoadTests_Go_SideInput_Flink_Batch.yml | 4 +-
.../beam_LoadTests_Python_CoGBK_Flink_Batch.yml | 4 +-
.../beam_LoadTests_Python_Combine_Flink_Batch.yml | 4 +-
...am_LoadTests_Python_Combine_Flink_Streaming.yml | 4 +-
.../beam_LoadTests_Python_GBK_Flink_Batch.yml | 4 +-
.../beam_LoadTests_Python_ParDo_Flink_Batch.yml | 4 +-
...beam_LoadTests_Python_ParDo_Flink_Streaming.yml | 4 +-
.../beam_PerformanceTests_SingleStoreIO.yml | 2 +-
.github/workflows/beam_PostCommit_Go.yml | 2 +-
.../workflows/beam_PostCommit_Go_Dataflow_ARM.yml | 2 +-
.../beam_PostCommit_Java_Examples_Dataflow_ARM.yml | 2 +-
...l => beam_PostCommit_Java_PVR_Spark4_Batch.yml} | 55 +-
... beam_PostCommit_Java_PVR_Spark4_Streaming.yml} | 45 +-
.../beam_PostCommit_Java_SingleStoreIO_IT.yml | 2 +-
..._Java_ValidatesRunner_Dataflow_JavaVersions.yml | 2 +-
...eam_PostCommit_Java_ValidatesRunner_Spark4.yml} | 40 +-
.github/workflows/beam_PostCommit_Python_Arm.yml | 2 +-
.../workflows/beam_PostCommit_Python_Versions.yml | 2 +-
.../beam_PostCommit_XVR_GoUsingJava_Dataflow.yml | 2 +-
.../beam_PostCommit_Yaml_Xlang_Direct.yml | 4 +-
.../workflows/beam_PreCommit_CommunityMetrics.yml | 2 +-
.../workflows/beam_PreCommit_Flink_Container.yml | 4 +-
.github/workflows/beam_PreCommit_Java_Dataflow.yml | 6 +-
...yml => beam_PreCommit_Java_Delta_IO_Direct.yml} | 78 +-
....yml => beam_PreCommit_Java_Spark_Versions.yml} | 21 +-
.github/workflows/beam_PreCommit_PythonDocker.yml | 4 +-
.../workflows/beam_PreCommit_Yaml_Xlang_Direct.yml | 4 +-
.../workflows/beam_Publish_Beam_SDK_Snapshots.yml | 7 +-
.../workflows/beam_Publish_Docker_Snapshots.yml | 4 +-
...beam_Python_ValidatesContainer_Dataflow_ARM.yml | 2 +-
.github/workflows/build_release_candidate.yml | 4 +-
.github/workflows/build_runner_image.yml | 6 +-
.github/workflows/build_wheels.yml | 10 +-
.github/workflows/codeql.yml | 134 +
.github/workflows/finalize_release.yml | 24 +-
...s_Dataflow_MLTransform_Generate_Vocab_Batch.txt | 42 +
...ataflow_MLTransform_One_Hot_Encoding_Batch.txt} | 19 +-
...rch_Sentiment_Batch_DistilBert_Base_Uncased.txt | 3 +
...Sentiment_Streaming_DistilBert_Base_Uncased.txt | 3 +
.github/workflows/playground_frontend_test.yml | 2 +-
.../republish_released_docker_containers.yml | 6 +-
.github/workflows/self-assign.yml | 2 +-
.github/workflows/tour_of_beam_backend.yml | 2 +-
.github/workflows/tour_of_beam_frontend_test.yml | 2 +-
.pre-commit-config.yaml | 2 +-
.test-infra/dataproc/flink_cluster.sh | 6 +-
.../github/github_runs_prefetcher/code/main.py | 12 +-
.test-infra/tools/refresh_looker_metrics.py | 4 +-
CHANGES.md | 68 +-
README.md | 2 +-
buildSrc/build.gradle.kts | 19 +
.../org/apache/beam/gradle/BeamModulePlugin.groovy | 194 +-
.../org/apache/beam/gradle/Repositories.groovy | 26 +-
.../examples/iceberg/IcebergBatchWriteExample.java | 15 +-
.../beam/examples/SchemaTransformExample.java | 10 +-
.../apache/beam/examples/SqlTransformExample.java | 10 +-
.../beam/examples/ApproximateQuantilesExample.java | 10 +-
...erKeyExample.java => BatchElementsExample.java} | 48 +-
.../beam/examples/CoCombineTransformExample.java | 19 +-
.../apache/beam/examples/CoGroupByKeyExample.java | 10 +-
.../org/apache/beam/examples/CombineExample.java | 10 +-
.../org/apache/beam/examples/CountExample.java | 10 +-
.../apache/beam/examples/CountPerKeyExample.java | 10 +-
.../org/apache/beam/examples/CreateExample.java | 10 +-
.../apache/beam/examples/DebuggingWordCount.java | 15 +-
.../org/apache/beam/examples/DistinctExample.java | 10 +-
.../beam/examples/FlatMapElementsExample.java | 10 +-
.../beam/examples/GroupIntoBatchesExample.java | 10 +-
.../beam/examples/KafkaPassengerCountJson.java | 21 +-
.../org/apache/beam/examples/KafkaStreaming.java | 33 +-
.../apache/beam/examples/KafkaWordCountAvro.java | 11 +-
.../apache/beam/examples/KafkaWordCountJson.java | 11 +-
.../java/org/apache/beam/examples/KeysExample.java | 10 +-
.../org/apache/beam/examples/KvSwapExample.java | 10 +-
.../org/apache/beam/examples/LatestExample.java | 10 +-
.../apache/beam/examples/MapElementsExample.java | 10 +-
.../java/org/apache/beam/examples/MaxExample.java | 10 +-
.../org/apache/beam/examples/MaxPerKeyExample.java | 10 +-
.../java/org/apache/beam/examples/MeanExample.java | 10 +-
.../apache/beam/examples/MeanPerKeyExample.java | 10 +-
.../java/org/apache/beam/examples/MinExample.java | 10 +-
.../org/apache/beam/examples/MinPerKeyExample.java | 10 +-
.../org/apache/beam/examples/PartitionExample.java | 10 +-
.../beam/examples/RateLimitedPubSubReader.java | 6 +-
.../apache/beam/examples/RateLimiterSimple.java | 8 +-
.../org/apache/beam/examples/RegexExample.java | 10 +-
.../org/apache/beam/examples/SampleExample.java | 10 +-
.../java/org/apache/beam/examples/SumExample.java | 10 +-
.../org/apache/beam/examples/SumPerKeyExample.java | 10 +-
.../org/apache/beam/examples/ToStringExample.java | 10 +-
.../java/org/apache/beam/examples/TopExample.java | 10 +-
.../org/apache/beam/examples/ValuesExample.java | 10 +-
.../java/org/apache/beam/examples/ViewExample.java | 19 +-
.../org/apache/beam/examples/WindowExample.java | 10 +-
.../beam/examples/complete/AutoComplete.java | 60 +-
.../examples/complete/StreamingWordExtract.java | 16 +-
.../org/apache/beam/examples/complete/TfIdf.java | 77 +-
.../examples/complete/TopWikipediaSessions.java | 32 +-
.../beam/examples/complete/TrafficMaxLaneFlow.java | 32 +-
.../beam/examples/complete/TrafficRoutes.java | 48 +-
.../transforms/DataProtectors.java | 21 +-
.../transforms/io/TokenizationBigQueryIO.java | 7 +-
.../transforms/io/TokenizationBigTableIO.java | 6 +-
.../datatokenization/utils/CsvConverters.java | 107 +-
.../datatokenization/utils/ErrorConverters.java | 30 +-
.../beam/examples/complete/game/GameStats.java | 45 +-
.../examples/complete/game/HourlyTeamScore.java | 6 +-
.../beam/examples/complete/game/LeaderBoard.java | 18 +-
.../examples/complete/game/StatefulTeamScore.java | 18 +-
.../beam/examples/complete/game/UserScore.java | 14 +-
.../complete/game/utils/WriteToBigQuery.java | 22 +-
.../examples/complete/game/utils/WriteToText.java | 21 +-
.../game/utils/WriteWindowedToBigQuery.java | 15 +-
.../cookbook/BigQueryStreamingTornadoes.java | 21 +-
.../beam/examples/cookbook/BigQueryTornadoes.java | 16 +-
.../examples/cookbook/CombinePerKeyExamples.java | 14 +-
.../beam/examples/cookbook/FilterExamples.java | 31 +-
.../beam/examples/cookbook/JoinExamples.java | 32 +-
.../beam/examples/cookbook/MaxPerKeyExamples.java | 17 +-
.../cookbook/MinimalBigQueryTornadoes.java | 20 +-
.../beam/examples/cookbook/TriggerExample.java | 47 +-
.../apache/beam/examples/snippets/Snippets.java | 40 +-
.../examples/subprocess/ExampleEchoPipeline.java | 10 +-
.../beam/examples/cookbook/TriggerExampleTest.java | 8 +-
.../subprocess/ExampleEchoPipelineTest.java | 10 +-
.../beam/examples/kotlin/DebuggingWordCount.kt | 12 +-
.../examples/kotlin/cookbook/BigQueryTornadoes.kt | 15 +-
.../kotlin/cookbook/CombinePerKeyExamples.kt | 13 +-
.../examples/kotlin/cookbook/FilterExamples.kt | 30 +-
.../beam/examples/kotlin/cookbook/JoinExamples.kt | 31 +-
.../examples/kotlin/cookbook/MaxPerKeyExamples.kt | 15 +-
.../examples/kotlin/cookbook/TriggerExample.kt | 42 +-
.../beam/examples/kotlin/snippets/Snippets.kt | 13 +-
.../notebooks/beam-ml/run_inference_gemini.ipynb | 609 ++
gradle.properties | 9 +-
infra/iam/users.yml | 8 +-
.../it/gcp/bigquery/BigQueryResourceManager.java | 42 +-
learning/tour-of-beam/terraform/build.gradle.kts | 2 +-
.../model/pipeline/v1/external_transforms.proto | 2 +
playground/terraform/build.gradle.kts | 2 +-
.../org/apache/beam/runners/core/DoFnRunner.java | 12 +
.../runners/core/LateDataDroppingDoFnRunner.java | 9 +-
...TimeBoundedSplittableProcessElementInvoker.java | 264 +-
.../apache/beam/runners/core/ReduceFnRunner.java | 49 +-
.../apache/beam/runners/core/SimpleDoFnRunner.java | 70 +-
.../core/SplittableParDoViaKeyedWorkItems.java | 12 +-
.../core/SplittableProcessElementInvoker.java | 25 +-
.../beam/runners/core/StatefulDoFnRunner.java | 6 +
.../org/apache/beam/runners/core/StepContext.java | 6 +
.../apache/beam/runners/core/WatermarkHold.java | 17 +
.../core/triggers/FinishedTriggersBitSet.java | 14 +
.../core/triggers/TriggerStateMachineRunner.java | 43 +-
...BoundedSplittableProcessElementInvokerTest.java | 47 +-
.../beam/runners/core/ReduceFnRunnerTest.java | 58 +
.../apache/beam/runners/core/ReduceFnTester.java | 13 +
.../SimplePushbackSideInputDoFnRunnerTest.java | 4 +
.../runners/core/SplittableParDoProcessFnTest.java | 124 +-
.../triggers/TriggerStateMachineRunnerTest.java | 117 +
.../flink/metrics/DoFnRunnerWithMetricsUpdate.java | 4 +
.../streaming/ExecutableStageDoFnOperator.java | 3 +
.../streaming/stableinput/BufferingDoFnRunner.java | 3 +
.../wrappers/streaming/DoFnOperatorTest.java | 3 +
.../google-cloud-dataflow-java/arm/build.gradle | 2 +-
runners/google-cloud-dataflow-java/build.gradle | 16 +-
.../beam/runners/dataflow/BatchViewOverrides.java | 6 +
.../beam/runners/dataflow/DataflowRunner.java | 52 +-
.../dataflow/RedistributeByKeyOverrideFactory.java | 1 +
.../options/DataflowStreamingPipelineOptions.java | 10 +
.../beam/runners/dataflow/util/PackageUtil.java | 3 +
.../beam/runners/dataflow/DataflowRunnerTest.java | 216 +-
.../runners/dataflow/util/PackageUtilTest.java | 4 +
.../worker/AssignWindowsParDoFnFactory.java | 3 +
.../worker/BatchModeUngroupingParDoFn.java | 3 +
.../CreateIsmShardKeyAndSortKeyDoFnFactory.java | 3 +
.../dataflow/worker/DataflowProcessFnRunner.java | 6 +
.../runners/dataflow/worker/ForwardingParDoFn.java | 6 +
.../dataflow/worker/GroupAlsoByWindowFnRunner.java | 4 +
.../dataflow/worker/GroupAlsoByWindowsParDoFn.java | 6 +
.../beam/runners/dataflow/worker/HotKeyLogger.java | 10 +-
.../worker/PairWithConstantKeyDoFnFactory.java | 3 +
.../dataflow/worker/PartialGroupByKeyParDoFns.java | 6 +
.../beam/runners/dataflow/worker/PubsubReader.java | 14 +-
.../ReifyTimestampAndWindowsParDoFnFactory.java | 3 +
.../dataflow/worker/SimpleDoFnRunnerFactory.java | 12 -
.../runners/dataflow/worker/SimpleParDoFn.java | 534 +-
...impleParDoFn.java => SimpleParDoFnHelpers.java} | 409 +-
.../worker/SplittableProcessFnFactory.java | 21 +-
.../dataflow/worker/StreamingDataflowWorker.java | 38 +-
.../StreamingKeyedWorkItemSideInputDoFnRunner.java | 6 +
.../StreamingKeyedWorkItemSideInputParDoFn.java | 249 +
.../worker/StreamingModeExecutionContext.java | 307 +-
.../StreamingPCollectionViewWriterParDoFn.java | 3 +
.../worker/StreamingSideInputDoFnRunner.java | 47 +-
.../dataflow/worker/StreamingSideInputFetcher.java | 34 +-
.../worker/StreamingSideInputProcessor.java | 132 +
.../worker/ToIsmRecordForMultimapDoFnFactory.java | 3 +
.../dataflow/worker/UngroupedWindmillReader.java | 23 +-
.../dataflow/worker/UserParDoFnFactory.java | 56 +-
.../runners/dataflow/worker/ValuesDoFnFactory.java | 3 +
.../dataflow/worker/WindmillKeyedWorkItem.java | 13 +-
.../worker/WindmillReaderIteratorBase.java | 11 +-
.../beam/runners/dataflow/worker/WindmillSink.java | 1 +
.../dataflow/worker/WindmillValueKindHelper.java | 60 +
.../dataflow/worker/WindowingWindmillReader.java | 2 +
.../WorkerCustomSourceOperationExecutor.java | 3 +
.../dataflow/worker/WorkerCustomSources.java | 9 +-
.../BoundedQueueExecutorWorkHandle.java} | 20 +-
.../worker/streaming/ComputationWorkExecutor.java | 2 +-
.../dataflow/worker/streaming/ExecutableWork.java | 63 +-
.../runners/dataflow/worker/streaming/Work.java | 71 +
.../dataflow/worker/util/BoundedQueueExecutor.java | 197 +-
.../dataflow/worker/util/ExceptionUtils.java | 36 +-
.../dataflow/worker/util/KeyGroupWorkQueue.java | 462 ++
.../dataflow/worker/util/SimpleByteHistogram.java} | 36 +-
.../dataflow/worker/util/ValueInEmptyWindows.java | 6 +
.../util/common/worker/FlattenOperation.java | 4 +
.../worker/util/common/worker/MapTaskExecutor.java | 10 +-
.../worker/util/common/worker/Operation.java | 5 +
.../worker/util/common/worker/ParDoFn.java | 4 +
.../worker/util/common/worker/ParDoOperation.java | 9 +-
.../worker/util/common/worker/ReadOperation.java | 3 +
.../worker/SimplePartialGroupByKeyParDoFn.java | 5 +
.../worker/util/common/worker/WorkExecutor.java | 3 +
.../worker/util/common/worker/WriteOperation.java | 4 +
.../client/grpc/GrpcWindmillStreamFactory.java | 6 +-
.../worker/windmill/state/WindmillStateCache.java | 122 +-
.../windmill/state/WindmillWatermarkHold.java | 94 +-
.../work/processing/StreamingCommitFinalizer.java | 2 +-
.../work/processing/StreamingWorkScheduler.java | 23 +-
.../worker/IntrinsicMapTaskExecutorTest.java | 15 +
.../dataflow/worker/SimpleParDoFnHelpersTest.java | 133 +
.../runners/dataflow/worker/SimpleParDoFnTest.java | 6 +-
.../worker/StreamingDataflowWorkerTest.java | 20 +-
.../worker/StreamingGroupAlsoByWindowFnsTest.java | 6 +
...eamingKeyedWorkItemSideInputDoFnRunnerTest.java | 3 +-
...StreamingKeyedWorkItemSideInputParDoFnTest.java | 488 ++
.../worker/StreamingModeExecutionContextTest.java | 111 +-
.../worker/StreamingSideInputDoFnRunnerTest.java | 4 +
.../worker/StreamingSideInputProcessorTest.java | 215 +
.../dataflow/worker/UserParDoFnFactoryTest.java | 27 +-
.../dataflow/worker/WindmillKeyedWorkItemTest.java | 43 +
.../worker/WindmillReaderIteratorBaseTest.java | 60 +-
.../dataflow/worker/WorkerCustomSourcesTest.java | 26 +-
.../worker/streaming/ActiveWorkStateTest.java | 4 +-
.../streaming/ComputationStateCacheTest.java | 2 +-
.../worker/util/BoundedQueueExecutorTest.java | 257 +-
.../worker/util/KeyGroupWorkQueueTest.java | 504 ++
.../worker/util/SimpleByteHistogramTest.java | 51 +
.../util/common/worker/ExecutorTestUtils.java | 3 +
.../util/common/worker/MapTaskExecutorTest.java | 15 +
.../util/common/worker/ParDoOperationTest.java | 6 +
.../windmill/state/WindmillStateCacheTest.java | 62 +
.../windmill/state/WindmillStateInternalsTest.java | 167 +-
.../processing/StreamingCommitFinalizerTest.java | 3 +-
.../failures/WorkFailureProcessorTest.java | 7 +-
.../work/refresh/ActiveWorkRefresherTest.java | 7 +-
.../worker/windmill/src/main/proto/windmill.proto | 11 +-
runners/prism/java/build.gradle | 4 +
runners/spark/4/README.md | 73 +
runners/spark/4/build.gradle | 63 +
.../4/job-server/build.gradle} | 27 +-
.../io/BoundedDatasetFactory.java | 13 +-
.../batch/CombinePerKeyTranslatorBatch.java | 6 +-
.../translation/batch/GroupByKeyHelpers.java | 8 +-
.../batch/GroupByKeyTranslatorBatch.java | 78 +-
.../translation/helpers/EncoderFactory.java | 333 +
.../translation/helpers/EncoderHelpers.java | 45 +-
.../translation/utils/ScalaInterop.java | 114 +
.../translation/helpers/EncoderHelpersTest.java | 298 +
runners/spark/job-server/spark_job_server.gradle | 6 +-
runners/spark/spark_runner.gradle | 25 +-
.../spark/coders/SparkRunnerKryoRegistrator.java | 28 +-
.../SparkGroupAlsoByWindowViaWindowSet.java | 4 +-
.../SparkStructuredStreamingPipelineResult.java | 1 +
.../io/BoundedDatasetFactory.java | 2 +-
.../translation/EvaluationContext.java | 10 +-
.../batch/CombinePerKeyTranslatorBatch.java | 2 +-
.../translation/batch/DoFnRunnerFactory.java | 4 +
.../translation/batch/DoFnRunnerWithMetrics.java | 4 +
.../translation/batch/GroupByKeyHelpers.java | 4 +-
.../batch/GroupByKeyTranslatorBatch.java | 2 +-
.../translation/helpers/EncoderHelpers.java | 21 +-
.../spark/translation/DoFnRunnerWithMetrics.java | 4 +
.../apache/beam/runners/spark/util/TimerUtils.java | 6 +
.../coders/SparkRunnerKryoRegistratorTest.java | 49 +
.../translation/EvaluationContextTest.java | 84 +
.../translation/SparkInputDataProcessorTest.java | 4 +
scripts/beam-sql.sh | 2 +-
scripts/tools/bomupgrader.py | 53 +-
sdks/go.mod | 189 +-
sdks/go.sum | 428 +-
sdks/go/container/Dockerfile | 10 +-
sdks/go/pkg/beam/core/core.go | 2 +-
.../pkg/beam/core/runtime/exec/datasampler_test.go | 13 +-
sdks/go/pkg/beam/io/filesystem/gcs/gcs.go | 88 +-
sdks/go/pkg/beam/io/filesystem/gcs/gcs_test.go | 146 +
sdks/go/pkg/beam/runners/dataflow/dataflow.go | 9 +-
sdks/go/pkg/beam/runners/dataflow/dataflow_test.go | 43 +-
.../beam/runners/dataflow/dataflowlib/execute.go | 1 +
.../pkg/beam/runners/dataflow/dataflowlib/job.go | 4 +
.../beam/runners/dataflow/dataflowlib/job_test.go | 16 +
.../prism/internal/engine/elementmanager.go | 7 +-
.../beam/runners/prism/internal/environments.go | 93 +-
sdks/go/pkg/beam/runners/prism/internal/execute.go | 6 +-
.../beam/runners/prism/internal/execute_test.go | 43 +
.../beam/runners/prism/internal/handlerunner.go | 4 +-
.../runners/prism/internal/handlerunner_test.go | 78 +
sdks/go/pkg/beam/runners/prism/internal/stage.go | 17 +-
.../beam/runners/prism/internal/testdofns_test.go | 39 +
.../beam/runners/prism/internal/worker/bundle.go | 58 +-
.../runners/prism/internal/worker/worker_test.go | 67 +
.../integration/internal/containers/containers.go | 3 +-
.../test/integration/io/spannerio/test_helpers.go | 3 +-
.../go/test/integration/io/xlang/jdbc/jdbc_test.go | 7 +-
.../src/main/resources/beam/beam-codestyle.xml | 6 +-
sdks/java/container/Dockerfile | 31 +-
sdks/java/container/boot.go | 15 +-
.../container/license_scripts/dep_urls_java.yaml | 12 +-
.../beam/sdk/options/PipelineOptionsFactory.java | 26 +-
.../sdk/schemas/FieldValueTypeInformation.java | 13 +-
.../beam/sdk/schemas/FromRowUsingCreator.java | 6 +-
.../sdk/schemas/GetterBasedSchemaProvider.java | 98 +-
.../beam/sdk/schemas/utils/AutoValueUtils.java | 9 +-
.../beam/sdk/schemas/utils/ByteBuddyUtils.java | 128 +
.../sdk/schemas/utils/StaticSchemaInference.java | 19 +-
.../apache/beam/sdk/state/WatermarkHoldState.java | 10 +
.../beam/sdk/testing/UsesSideInputsInTimer.java | 23 +-
.../apache/beam/sdk/transforms/AsyncWrapper.java | 779 ++
.../apache/beam/sdk/transforms/BatchElements.java | 608 ++
.../java/org/apache/beam/sdk/transforms/DoFn.java | 34 +
.../org/apache/beam/sdk/transforms/DoFnTester.java | 25 +-
.../apache/beam/sdk/transforms/LogElements.java | 234 +
.../apache/beam/sdk/transforms/Redistribute.java | 1 +
.../java/org/apache/beam/sdk/transforms/Reify.java | 5 +-
.../org/apache/beam/sdk/transforms/Reshuffle.java | 1 +
.../reflect/ByteBuddyDoFnInvokerFactory.java | 23 +
.../beam/sdk/transforms/reflect/DoFnInvoker.java | 32 +
.../beam/sdk/transforms/reflect/DoFnSignature.java | 35 +
.../sdk/transforms/reflect/DoFnSignatures.java | 18 +-
.../java/org/apache/beam/sdk/util/RowJson.java | 65 +-
.../beam/sdk/util/RowJsonValueExtractors.java | 9 +-
.../construction/SplittableParDoNaiveBounded.java | 17 +
.../org/apache/beam/sdk/values/OutputBuilder.java | 2 +
.../main/java/org/apache/beam/sdk/values/Row.java | 6 +-
.../org/apache/beam/sdk/values/RowWithGetters.java | 13 +-
.../beam/sdk/values/ValueInSingleWindow.java | 52 +-
.../java/org/apache/beam/sdk/values/ValueKind.java | 35 +-
.../org/apache/beam/sdk/values/ValueKindUtil.java | 54 +
.../org/apache/beam/sdk/values/WindowedValue.java | 4 +
.../org/apache/beam/sdk/values/WindowedValues.java | 233 +-
.../org/apache/beam/sdk/io/TextIOReadTest.java | 6 +-
.../beam/sdk/schemas/AutoValueSchemaTest.java | 426 +-
.../beam/sdk/schemas/JavaBeanSchemaTest.java | 250 +-
.../beam/sdk/schemas/JavaFieldSchemaTest.java | 318 +-
.../beam/sdk/schemas/transforms/ConvertTest.java | 99 +
.../beam/sdk/schemas/utils/JavaBeanUtilsTest.java | 10 +
.../beam/sdk/schemas/utils/POJOUtilsTest.java | 32 +
.../beam/sdk/schemas/utils/TestJavaBeans.java | 124 +
.../apache/beam/sdk/schemas/utils/TestPOJOs.java | 143 +
.../beam/sdk/transforms/AsyncWrapperTest.java | 877 +++
.../beam/sdk/transforms/BatchElementsTest.java | 598 ++
.../beam/sdk/transforms/LogElementsTest.java | 108 +
.../sdk/transforms/MetadataPropagationTest.java | 9 +-
.../org/apache/beam/sdk/transforms/ParDoTest.java | 153 +
.../apache/beam/sdk/transforms/ValueKindTest.java | 623 ++
.../sdk/transforms/reflect/DoFnSignaturesTest.java | 8 +-
.../java/org/apache/beam/sdk/util/RowJsonTest.java | 26 +-
.../UnboundedScheduledExecutorServiceTest.java | 2 +-
.../apache/beam/sdk/util/WindowedValueTest.java | 38 +-
.../beam/sdk/extensions/arrow/ArrowConversion.java | 4 +-
.../extensions/avro/schemas/utils/AvroUtils.java | 87 +-
.../beam/sdk/extensions/gcp/util/GcsUtilV1.java | 69 +-
.../beam/sdk/extensions/gcp/util/GcsUtilTest.java | 19 +-
.../opentelemetry-gcp-auth-extension/build.gradle | 50 +
.../opentelemetry/gcp/auth/ConfigurableOption.java | 163 +
...GcpAuthAutoConfigurationCustomizerProvider.java | 282 +
.../gcp/auth/GoogleAuthException.java | 69 +
.../opentelemetry/gcp/auth/package-info.java | 20 +-
...figure.spi.AutoConfigurationCustomizerProvider} | 8 +-
...uthAutoConfigurationCustomizerProviderTest.java | 1234 +++
.../extensions/protobuf/ProtoByteBuddyUtils.java | 49 +-
sdks/java/extensions/sql/iceberg/build.gradle | 1 +
.../meta/provider/iceberg/IcebergMetastore.java | 13 +-
.../sql/meta/provider/iceberg/IcebergTable.java | 19 +-
.../meta/provider/iceberg/PubsubToIcebergIT.java | 4 +-
.../sql/src/main/codegen/includes/parserImpls.ftl | 4 +-
.../extensions/sql/impl/CalciteQueryPlanner.java | 93 +-
.../beam/sdk/extensions/sql/impl/UdfImpl.java | 19 +-
.../extensions/sql/impl/parser/SqlDdlNodes.java | 7 +-
.../sdk/extensions/sql/impl/rel/BeamSortRel.java | 58 +-
.../impl/transform/BeamBuiltinAggregations.java | 2 +
.../sql/impl/transform/agg/VarianceFn.java | 38 +-
.../extensions/sql/impl/utils/CalciteUtils.java | 31 +-
.../sql/meta/catalog/InMemoryCatalog.java | 12 +-
.../{BeamSqlAliasTest => BeamSqlAliasTest.java} | 12 +-
.../sdk/extensions/sql/BeamSqlCliDatabaseTest.java | 52 +
.../sql/BeamSqlDslAggregationVarianceTest.java | 43 +-
.../extensions/sql/BeamSqlDslParametersTest.java | 78 +
.../sql/impl/LazyAggregateCombineFnTest.java | 5 +-
.../beam/sdk/extensions/sql/impl/UdfImplTest.java | 65 +
.../extensions/sql/impl/rel/BeamSortRelTest.java | 43 +
.../sql/impl/transform/agg/VarianceFnTest.java | 25 +-
.../sql/impl/utils/CalciteUtilsTest.java | 63 +
.../apache/beam/fn/harness/FnApiDoFnRunner.java | 75 +-
.../java/org/apache/beam/fn/harness/FnHarness.java | 11 +
.../sdk/io/aws2/schemas/AwsSchemaProvider.java | 7 +-
.../apache/beam/sdk/io/aws2/schemas/AwsTypes.java | 16 +
.../beam/sdk/io/clickhouse/ClickHouseIO.java | 3 +
.../beam/sdk/io/clickhouse/ClickHouseWriter.java | 40 +
.../apache/beam/sdk/io/clickhouse/TableSchema.java | 53 +-
.../clickhouse/src/main/javacc/ColumnTypeParser.jj | 27 +
.../beam/sdk/io/clickhouse/ClickHouseIOIT.java | 119 +
.../sdk/io/clickhouse/ClickHouseWriterTest.java | 141 +
.../beam/sdk/io/clickhouse/TableSchemaTest.java | 102 +
.../apache/beam/sdk/io/datadog/DatadogEvent.java | 6 +
.../DatadogWriteSchemaTransformConfiguration.java | 114 +
.../DatadogWriteSchemaTransformProvider.java | 294 +
.../DatadogWriteSchemaTransformProviderTest.java | 535 ++
sdks/java/io/delta/build.gradle | 84 +
.../org/apache/beam/sdk/io/delta/BeamEngine.java | 55 +
.../beam/sdk/io/delta/BeamParquetHandler.java | 376 +
.../beam/sdk/io/delta/CreateReadTasksDoFn.java | 140 +
.../java/org/apache/beam/sdk/io/delta/DeltaIO.java | 177 +
.../io/delta/DeltaReadSchemaTransformProvider.java | 142 +
.../apache/beam/sdk/io/delta/DeltaReadTask.java | 88 +
.../beam/sdk/io/delta/DeltaReadTaskTracker.java | 56 +
.../apache/beam/sdk/io/delta/DeltaSourceDoFn.java | 455 ++
.../apache/beam/sdk/io/delta/SerializableRow.java | 544 ++
.../beam/sdk/io/delta/SerializableStructType.java | 69 +
.../org/apache/beam/sdk/io/delta/package-info.java | 18 +-
.../org/apache/beam/sdk/io/delta/DeltaIOTest.java | 813 ++
.../beam/sdk/io/delta/SerializableRowTest.java | 498 ++
sdks/java/io/expansion-service/build.gradle | 21 +-
.../AvroReadSchemaTransformFormatProviderTest.java | 11 +
...WriteSchemaTransformFormatProviderTestData.java | 61 +
...rquetReadSchemaTransformFormatProviderTest.java | 11 +
.../beam/sdk/io/gcp/bigquery/BigQueryOptions.java | 7 +
.../sdk/io/gcp/bigquery/BigQueryServicesImpl.java | 10 +
.../bigquery/StorageApiWritesShardedRecords.java | 43 +-
...ueryStorageWriteApiSchemaTransformProvider.java | 9 +-
.../providers/BigQueryWriteConfiguration.java | 9 +-
.../beam/sdk/io/gcp/spanner/SpannerAccessor.java | 13 +-
.../beam/sdk/io/gcp/spanner/SpannerConfig.java | 37 +
.../apache/beam/sdk/io/gcp/spanner/SpannerIO.java | 179 +-
.../io/gcp/spanner/SpannerTransformRegistrar.java | 30 +
.../MetadataSpannerConfigFactory.java | 6 +
.../action/HeartbeatRecordAction.java | 2 +-
.../StorageApiDataTriggeredSchemaUpdateIT.java | 26 +-
...StorageWriteApiSchemaTransformProviderTest.java | 5 +-
.../beam/sdk/io/gcp/spanner/SpannerReadIT.java | 33 +-
.../beam/sdk/io/gcp/spanner/SpannerTestHelper.java | 110 +
.../beam/sdk/io/gcp/spanner/SpannerWriteIT.java | 110 +-
.../SpannerChangeStreamErrorTest.java | 41 +-
.../action/HeartbeatRecordActionTest.java | 4 +-
.../changestreams/it/IntegrationTestEnv.java | 17 +-
.../changestreams/it/SpannerChangeStreamIT.java | 19 +-
...StreamOrderedByTimestampAndTransactionIdIT.java | 10 +-
...nnerChangeStreamOrderedWithinKeyGloballyIT.java | 10 +-
.../it/SpannerChangeStreamOrderedWithinKeyIT.java | 10 +-
.../it/SpannerChangeStreamPlacementTableIT.java | 19 +-
...pannerChangeStreamPlacementTablePostgresIT.java | 10 +-
.../it/SpannerChangeStreamPostgresIT.java | 10 +-
...SpannerChangeStreamTransactionBoundariesIT.java | 10 +-
.../it/SpannerChangeStreamsSchemaTransformIT.java | 4 +
.../apache/beam/sdk/io/gcp/storage/GcsMatchIT.java | 8 +-
.../apache/beam/sdk/io/hdfs/HadoopFileSystem.java | 17 +
.../beam/sdk/io/hdfs/HadoopFileSystemTest.java | 20 +
.../beam/sdk/io/hadoop/format/HadoopFormatIO.java | 25 +
sdks/java/io/iceberg/build.gradle | 9 +-
.../iceberg/AssignDestinationsAndPartitions.java | 52 +-
.../beam/sdk/io/iceberg/CreateReadTasksDoFn.java | 7 +-
.../beam/sdk/io/iceberg/IcebergCatalogConfig.java | 10 +-
.../org/apache/beam/sdk/io/iceberg/IcebergIO.java | 2 +-
.../beam/sdk/io/iceberg/IcebergScanConfig.java | 5 +-
.../beam/sdk/io/iceberg/IncrementalScanSource.java | 6 +-
.../apache/beam/sdk/io/iceberg/ReadFromTasks.java | 7 +-
.../beam/sdk/io/iceberg/RecordWriterManager.java | 140 +-
.../org/apache/beam/sdk/io/iceberg/ScanSource.java | 3 +-
.../beam/sdk/io/iceberg/SerializableDataFile.java | 14 +-
.../org/apache/beam/sdk/io/iceberg/TableCache.java | 198 +-
.../beam/sdk/io/iceberg/WatchForSnapshots.java | 7 +-
.../sdk/io/iceberg/WriteDirectRowsToFiles.java | 12 +-
.../sdk/io/iceberg/WriteGroupedRowsToFiles.java | 12 +-
.../io/iceberg/WritePartitionedRowsToFiles.java | 166 +-
.../sdk/io/iceberg/WriteUngroupedRowsToFiles.java | 12 +-
.../sdk/io/iceberg/RecordWriterManagerTest.java | 121 +-
.../sdk/io/iceberg/SerializableDataFileTest.java | 65 +
.../apache/beam/sdk/io/iceberg/TableCacheTest.java | 126 +
.../io/messaging-expansion-service/build.gradle | 52 +
.../apache/beam/sdk/io/mongodb/MongoDbUtils.java | 74 +
.../MongoDbWriteSchemaTransformConfiguration.java | 85 +
.../MongoDbWriteSchemaTransformProvider.java | 168 +
.../beam/sdk/io/mongodb/MongoDbUtilsTest.java | 135 +
.../MongoDbWriteSchemaTransformProviderTest.java | 166 +
.../java/org/apache/beam/sdk/io/mqtt/MqttIO.java | 7 +
.../io/mqtt/MqttReadSchemaTransformProvider.java | 150 +
.../io/mqtt/MqttWriteSchemaTransformProvider.java | 132 +
.../io/mqtt/MqttSchemaTransformProviderTest.java | 252 +
.../org/apache/beam/io/requestresponse/Call.java | 55 +-
.../java/org/apache/beam/sdk/managed/Managed.java | 4 +
.../META-INF/maven/archetype-metadata.xml | 2 +-
.../resources/projects/basic/archetype.properties | 2 +-
.../resources/projects/basic/archetype.properties | 2 +-
sdks/python/.isort.cfg | 58 -
sdks/python/apache_beam/dataframe/frames_test.py | 33 +-
sdks/python/apache_beam/dataframe/io.py | 35 +-
sdks/python/apache_beam/dataframe/io_test.py | 45 +
.../cookbook/ordered_window_elements/streaming.py | 2 +-
.../anomaly_detection_pipeline/setup.py | 2 +-
.../examples/inference/gemini_image_generation.py | 4 +-
.../inference/gemini_text_classification.py | 4 +-
.../online_clustering/clustering_pipeline/setup.py | 2 +-
.../examples/inference/pytorch_sentiment.py | 80 +-
.../apache_beam/examples/kafkataxi/README.md | 2 +-
.../apache_beam/examples/matrix_power_test.py | 13 +-
.../kfp/components/train/requirements.txt | 2 +-
.../apache_beam/examples/ml_transform/README.md | 116 +
.../ml_transform/mltransform_generate_vocab.py | 265 +
.../mltransform_generate_vocab_requirements.txt} | 13 +-
.../mltransform_generate_vocab_test.py | 191 +
.../ml_transform/mltransform_one_hot_encoding.py | 266 +
.../mltransform_one_hot_encoding_test.py | 259 +
.../apache_beam/examples/wordcount_rust/README.md | 4 +-
.../examples/wordcount_rust/requirements.txt | 2 +-
.../io/external/xlang_bigqueryio_it_test.py | 2 +-
sdks/python/apache_beam/io/fileio.py | 3 +-
sdks/python/apache_beam/io/gcp/bigquery.py | 28 +-
.../apache_beam/io/gcp/bigquery_file_loads.py | 8 +-
.../apache_beam/io/gcp/bigquery_schema_tools.py | 63 +-
.../io/gcp/bigquery_schema_tools_test.py | 109 +
sdks/python/apache_beam/io/gcp/bigquery_tools.py | 13 +-
.../apache_beam/io/gcp/bigquery_tools_test.py | 161 +
sdks/python/apache_beam/metrics/metric.py | 2 +-
.../ml/inference/agent_development_kit.py | 36 +-
.../ml/inference/agent_development_kit_test.py | 38 +-
sdks/python/apache_beam/ml/rag/ingestion/qdrant.py | 328 +
.../apache_beam/ml/rag/ingestion/qdrant_it_test.py | 326 +
.../apache_beam/ml/rag/ingestion/qdrant_test.py | 480 ++
.../transforms/mltransform_tests_requirements.txt} | 14 +-
.../python/apache_beam/options/pipeline_options.py | 110 +-
.../apache_beam/options/pipeline_options_test.py | 43 +
.../options/pipeline_options_validator.py | 2 +
.../runners/dataflow/dataflow_metrics.py | 65 +-
.../runners/dataflow/dataflow_metrics_test.py | 314 +-
.../runners/dataflow/dataflow_runner.py | 72 +-
.../runners/dataflow/dataflow_runner_test.py | 59 +-
.../runners/dataflow/internal/apiclient.py | 528 +-
.../runners/dataflow/internal/apiclient_test.py | 341 +-
.../clients/dataflow/dataflow_v1b3_client.py | 1318 ----
.../clients/dataflow/dataflow_v1b3_messages.py | 8057 --------------------
.../internal/clients/dataflow/message_matchers.py | 118 -
.../clients/dataflow/message_matchers_test.py | 74 -
.../apache_beam/runners/dataflow/internal/names.py | 2 +-
.../runners/interactive/cache_manager.py | 22 +-
.../runners/interactive/interactive_environment.py | 10 +-
.../runners/interactive/user_pipeline_tracker.py | 109 +-
.../interactive/user_pipeline_tracker_test.py | 48 +
.../apache_beam/runners/portability/job_server.py | 14 +-
.../runners/portability/portable_runner_test.py | 17 +
.../runners/portability/prism_runner.py | 10 +-
.../runners/portability/prism_runner_test.py | 73 +
.../apache_beam/runners/portability/stager.py | 149 +-
.../apache_beam/runners/worker/data_plane.py | 114 +-
.../apache_beam/runners/worker/opcounters.py | 2 +
.../apache_beam/runners/worker/opcounters_test.py | 16 +
.../apache_beam/runners/worker/sdk_worker.py | 9 +-
.../apache_beam/runners/worker/sdk_worker_test.py | 32 +
.../runners/worker/statesampler_fast.pyx | 8 +-
.../runners/worker/statesampler_test.py | 43 +
.../apache_beam/runners/worker/worker_pool_main.py | 11 +
.../mltransform_generate_vocab_benchmark.py | 56 +
.../mltransform_one_hot_encoding_benchmark.py | 197 +
.../inference/table_row_inference_benchmark.py | 48 +-
.../testing/load_tests/dataflow_cost_benchmark.py | 48 +-
sdks/python/apache_beam/testing/test_pipeline.py | 4 +-
sdks/python/apache_beam/transforms/async_dofn.py | 29 +-
.../apache_beam/transforms/async_dofn_test.py | 35 +
sdks/python/apache_beam/transforms/core.py | 4 +-
.../python/apache_beam/transforms/external_test.py | 8 +-
sdks/python/apache_beam/typehints/opcodes.py | 8 +
sdks/python/apache_beam/typehints/row_type_test.py | 21 +
.../apache_beam/typehints/trivial_inference.py | 26 +
.../typehints/trivial_inference_test.py | 27 +
sdks/python/apache_beam/typehints/typehints.py | 13 +-
.../python/apache_beam/typehints/typehints_test.py | 27 +
.../__init__.py => utils/byte_limited_queue.pxd} | 27 +-
.../python/apache_beam/utils/byte_limited_queue.py | 204 +
.../apache_beam/utils/byte_limited_queue_test.py | 270 +
.../apache_beam/utils/multi_process_shared.py | 17 +-
.../apache_beam/utils/multi_process_shared_test.py | 63 +-
sdks/python/apache_beam/utils/subprocess_server.py | 87 +-
.../apache_beam/utils/subprocess_server_test.py | 296 +
sdks/python/apache_beam/version.py | 2 +-
sdks/python/apache_beam/yaml/integration_tests.py | 98 +-
sdks/python/apache_beam/yaml/readme_test.py | 16 +-
sdks/python/apache_beam/yaml/standard_io.yaml | 44 +-
.../apache_beam/yaml/standard_providers.yaml | 1 +
.../test_utils/__init__.py} | 3 +-
.../yaml/test_utils/datadog_test_utils.py | 131 +
sdks/python/apache_beam/yaml/tests/datadog.yaml | 63 +
.../apache_beam/yaml/tests/iceberg_add_files.yaml | 3 +
sdks/python/apache_beam/yaml/tests/match_all.yaml | 96 +
.../{version.py => yaml/tests/mongodb.yaml} | 28 +-
.../yaml/tests/runinference_huggingface.yaml | 62 +
...uninference.yaml => runinference_vertexai.yaml} | 0
sdks/python/apache_beam/yaml/yaml_io.py | 111 +-
sdks/python/apache_beam/yaml/yaml_io_test.py | 118 +
sdks/python/apache_beam/yaml/yaml_mapping.py | 171 +-
sdks/python/apache_beam/yaml/yaml_ml.py | 95 +-
.../python/apache_beam/yaml/yaml_transform_test.py | 26 +-
sdks/python/apache_beam/yaml/yaml_udf_test.py | 119 +-
sdks/python/build.gradle | 29 +-
sdks/python/container/Dockerfile | 17 +-
.../container/base_image_requirements_manual.txt | 2 +
sdks/python/container/boot.go | 100 +-
.../license_scripts/upgrade_bundled_pip.py | 65 +
.../container/ml/py310/base_image_requirements.txt | 172 +-
.../container/ml/py310/gpu_image_requirements.txt | 272 +-
.../container/ml/py311/base_image_requirements.txt | 176 +-
.../container/ml/py311/gpu_image_requirements.txt | 272 +-
.../container/ml/py312/base_image_requirements.txt | 173 +-
.../container/ml/py312/gpu_image_requirements.txt | 270 +-
.../container/ml/py313/base_image_requirements.txt | 173 +-
sdks/python/container/profiler.go | 336 +
.../container/py310/base_image_requirements.txt | 157 +-
.../container/py311/base_image_requirements.txt | 163 +-
.../container/py312/base_image_requirements.txt | 160 +-
.../container/py313/base_image_requirements.txt | 160 +-
.../container/py314/base_image_requirements.txt | 156 +-
sdks/python/pyproject.toml | 176 +-
sdks/python/pyrefly.toml | 60 -
sdks/python/ruff.toml | 96 -
sdks/python/scripts/{run_pylint.sh => run_lint.sh} | 6 +-
sdks/python/setup.py | 28 +-
sdks/python/test-suites/dataflow/common.gradle | 8 +-
sdks/python/test-suites/xlang/build.gradle | 3 +-
sdks/python/tox.ini | 2 +-
sdks/standard_expansion_services.yaml | 15 +
sdks/standard_external_transforms.yaml | 122 +-
sdks/typescript/package-lock.json | 238 +-
sdks/typescript/package.json | 6 +-
sdks/typescript/src/apache_beam/runners/flink.ts | 2 +-
settings.gradle.kts | 22 +-
website/www/site/config.toml | 2 +-
website/www/site/content/en/blog/beam-2.74.0.md | 70 +
website/www/site/content/en/blog/looping-timers.md | 11 +-
.../site/content/en/documentation/dsls/sql/ddl.md | 344 +
.../content/en/documentation/dsls/sql/overview.md | 2 +
.../documentation/io/built-in/google-bigquery.md | 2 +-
.../en/documentation/io/built-in/iceberg.md | 2 +-
.../site/content/en/documentation/io/managed-io.md | 551 +-
.../content/en/documentation/programming-guide.md | 83 +
.../transforms/java/aggregation/batchelements.md | 31 +
.../en/documentation/transforms/java/overview.md | 1 +
.../www/site/content/en/get-started/downloads.md | 22 +-
website/www/site/content/en/performance/_index.md | 2 +
.../en/performance/mltransformonehot/_index.md | 42 +
.../en/performance/mltransformvocab/_index.md | 37 +
website/www/site/data/capability_matrix.yaml | 2 +-
website/www/site/data/performance.yaml | 32 +
.../partials/section-menu/en/documentation.html | 1 +
.../layouts/partials/section-menu/en/sdks.html | 1 +
website/www/site/layouts/shortcodes/tab.html | 22 +
website/www/site/static/.htaccess | 14 -
697 files changed, 36706 insertions(+), 16398 deletions(-)
create mode 100644 .agent/skills/developing-new-io-connectors/SKILL.md
create mode 100644 .agent/skills/yaml-development/SKILL.md
copy .github/trigger_files/{beam_PostCommit_Yaml_Xlang_Direct.json =>
beam_PostCommit_Python_Versions.json} (84%)
copy .github/trigger_files/{beam_PostCommit_Java_Nexmark_Flink.json =>
beam_PreCommit_Java_Delta_IO_Direct.json} (100%)
copy .github/trigger_files/{beam_PreCommit_Yaml_Xlang_Direct.json =>
beam_PreCommit_Python_PVR_Flink.json} (100%)
copy .github/workflows/{beam_PreCommit_Java_Spark3_Versions.yml =>
beam_PostCommit_Java_PVR_Spark4_Batch.yml} (72%)
copy .github/workflows/{beam_PreCommit_Java_Spark3_Versions.yml =>
beam_PostCommit_Java_PVR_Spark4_Streaming.yml} (72%)
copy .github/workflows/{beam_PostCommit_XVR_GoUsingJava_Dataflow.yml =>
beam_PostCommit_Java_ValidatesRunner_Spark4.yml} (75%)
copy .github/workflows/{beam_PostCommit_XVR_GoUsingJava_Dataflow.yml =>
beam_PreCommit_Java_Delta_IO_Direct.yml} (61%)
rename .github/workflows/{beam_PreCommit_Java_Spark3_Versions.yml =>
beam_PreCommit_Java_Spark_Versions.yml} (85%)
create mode 100644 .github/workflows/codeql.yml
create mode 100644
.github/workflows/load-tests-pipeline-options/beam_Inference_Python_Benchmarks_Dataflow_MLTransform_Generate_Vocab_Batch.txt
copy
.github/workflows/load-tests-pipeline-options/{beam_Inference_Python_Benchmarks_Dataflow_Pytorch_Sentiment_Batch_DistilBert_Base_Uncased.txt
=>
beam_Inference_Python_Benchmarks_Dataflow_MLTransform_One_Hot_Encoding_Batch.txt}
(72%)
copy
examples/java/src/main/java/org/apache/beam/examples/{CountPerKeyExample.java
=> BatchElementsExample.java} (67%)
create mode 100644 examples/notebooks/beam-ml/run_inference_gemini.ipynb
create mode 100644
runners/core-java/src/test/java/org/apache/beam/runners/core/triggers/TriggerStateMachineRunnerTest.java
copy
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/{SimpleParDoFn.java
=> SimpleParDoFnHelpers.java} (76%)
create mode 100644
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingKeyedWorkItemSideInputParDoFn.java
create mode 100644
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingSideInputProcessor.java
create mode 100644
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WindmillValueKindHelper.java
copy
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/{util/common/worker/ParDoFn.java
=> streaming/BoundedQueueExecutorWorkHandle.java} (60%)
copy
sdks/java/core/src/main/java/org/apache/beam/sdk/state/WatermarkHoldState.java
=>
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/ExceptionUtils.java
(53%)
create mode 100644
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/KeyGroupWorkQueue.java
copy
runners/{core-java/src/main/java/org/apache/beam/runners/core/StepContext.java
=>
google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/SimpleByteHistogram.java}
(52%)
create mode 100644
runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/SimpleParDoFnHelpersTest.java
create mode 100644
runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/StreamingKeyedWorkItemSideInputParDoFnTest.java
create mode 100644
runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/StreamingSideInputProcessorTest.java
create mode 100644
runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/util/KeyGroupWorkQueueTest.java
create mode 100644
runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/util/SimpleByteHistogramTest.java
create mode 100644 runners/spark/4/README.md
create mode 100644 runners/spark/4/build.gradle
copy
runners/{google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/ParDoFn.java
=> spark/4/job-server/build.gradle} (59%)
copy runners/spark/{ =>
4}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/io/BoundedDatasetFactory.java
(95%)
copy runners/spark/{ =>
4}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/CombinePerKeyTranslatorBatch.java
(97%)
copy runners/spark/{ =>
4}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/GroupByKeyHelpers.java
(92%)
copy runners/spark/{ =>
4}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/GroupByKeyTranslatorBatch.java
(85%)
create mode 100644
runners/spark/4/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/helpers/EncoderFactory.java
copy runners/spark/{ =>
4}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/helpers/EncoderHelpers.java
(93%)
create mode 100644
runners/spark/4/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/utils/ScalaInterop.java
create mode 100644
runners/spark/4/src/test/java/org/apache/beam/runners/spark/structuredstreaming/translation/helpers/EncoderHelpersTest.java
create mode 100644
runners/spark/src/test/java/org/apache/beam/runners/spark/structuredstreaming/translation/EvaluationContextTest.java
create mode 100644 sdks/go/pkg/beam/runners/prism/internal/handlerunner_test.go
copy
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/ParDoFn.java
=>
sdks/java/core/src/main/java/org/apache/beam/sdk/testing/UsesSideInputsInTimer.java
(60%)
create mode 100644
sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/AsyncWrapper.java
create mode 100644
sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/BatchElements.java
create mode 100644
sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/LogElements.java
copy
runners/core-java/src/main/java/org/apache/beam/runners/core/StepContext.java
=> sdks/java/core/src/main/java/org/apache/beam/sdk/values/ValueKind.java (52%)
create mode 100644
sdks/java/core/src/main/java/org/apache/beam/sdk/values/ValueKindUtil.java
create mode 100644
sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/AsyncWrapperTest.java
create mode 100644
sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/BatchElementsTest.java
create mode 100644
sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/LogElementsTest.java
create mode 100644
sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ValueKindTest.java
create mode 100644
sdks/java/extensions/opentelemetry-gcp-auth-extension/build.gradle
create mode 100644
sdks/java/extensions/opentelemetry-gcp-auth-extension/src/main/java/org/apache/beam/sdk/extensions/opentelemetry/gcp/auth/ConfigurableOption.java
create mode 100644
sdks/java/extensions/opentelemetry-gcp-auth-extension/src/main/java/org/apache/beam/sdk/extensions/opentelemetry/gcp/auth/GcpAuthAutoConfigurationCustomizerProvider.java
create mode 100644
sdks/java/extensions/opentelemetry-gcp-auth-extension/src/main/java/org/apache/beam/sdk/extensions/opentelemetry/gcp/auth/GoogleAuthException.java
copy
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/ParDoFn.java
=>
sdks/java/extensions/opentelemetry-gcp-auth-extension/src/main/java/org/apache/beam/sdk/extensions/opentelemetry/gcp/auth/package-info.java
(59%)
copy sdks/{python/apache_beam/version.py =>
java/extensions/opentelemetry-gcp-auth-extension/src/main/resources/META-INF/services/io.opentelemetry.sdk.autoconfigure.spi.AutoConfigurationCustomizerProvider}
(83%)
create mode 100644
sdks/java/extensions/opentelemetry-gcp-auth-extension/src/test/java/org/apache/beam/sdk/extensions/opentelemetry/gcp/auth/GcpAuthAutoConfigurationCustomizerProviderTest.java
rename
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/{BeamSqlAliasTest
=> BeamSqlAliasTest.java} (92%)
create mode 100644
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslParametersTest.java
create mode 100644
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/UdfImplTest.java
create mode 100644
sdks/java/io/clickhouse/src/test/java/org/apache/beam/sdk/io/clickhouse/ClickHouseWriterTest.java
create mode 100644
sdks/java/io/datadog/src/main/java/org/apache/beam/sdk/io/datadog/DatadogWriteSchemaTransformConfiguration.java
create mode 100644
sdks/java/io/datadog/src/main/java/org/apache/beam/sdk/io/datadog/DatadogWriteSchemaTransformProvider.java
create mode 100644
sdks/java/io/datadog/src/test/java/org/apache/beam/sdk/io/datadog/DatadogWriteSchemaTransformProviderTest.java
create mode 100644 sdks/java/io/delta/build.gradle
create mode 100644
sdks/java/io/delta/src/main/java/org/apache/beam/sdk/io/delta/BeamEngine.java
create mode 100644
sdks/java/io/delta/src/main/java/org/apache/beam/sdk/io/delta/BeamParquetHandler.java
create mode 100644
sdks/java/io/delta/src/main/java/org/apache/beam/sdk/io/delta/CreateReadTasksDoFn.java
create mode 100644
sdks/java/io/delta/src/main/java/org/apache/beam/sdk/io/delta/DeltaIO.java
create mode 100644
sdks/java/io/delta/src/main/java/org/apache/beam/sdk/io/delta/DeltaReadSchemaTransformProvider.java
create mode 100644
sdks/java/io/delta/src/main/java/org/apache/beam/sdk/io/delta/DeltaReadTask.java
create mode 100644
sdks/java/io/delta/src/main/java/org/apache/beam/sdk/io/delta/DeltaReadTaskTracker.java
create mode 100644
sdks/java/io/delta/src/main/java/org/apache/beam/sdk/io/delta/DeltaSourceDoFn.java
create mode 100644
sdks/java/io/delta/src/main/java/org/apache/beam/sdk/io/delta/SerializableRow.java
create mode 100644
sdks/java/io/delta/src/main/java/org/apache/beam/sdk/io/delta/SerializableStructType.java
copy
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/ParDoFn.java
=>
sdks/java/io/delta/src/main/java/org/apache/beam/sdk/io/delta/package-info.java
(60%)
create mode 100644
sdks/java/io/delta/src/test/java/org/apache/beam/sdk/io/delta/DeltaIOTest.java
create mode 100644
sdks/java/io/delta/src/test/java/org/apache/beam/sdk/io/delta/SerializableRowTest.java
create mode 100644
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/spanner/SpannerTestHelper.java
create mode 100644
sdks/java/io/iceberg/src/test/java/org/apache/beam/sdk/io/iceberg/TableCacheTest.java
create mode 100644 sdks/java/io/messaging-expansion-service/build.gradle
create mode 100644
sdks/java/io/mongodb/src/main/java/org/apache/beam/sdk/io/mongodb/MongoDbUtils.java
create mode 100644
sdks/java/io/mongodb/src/main/java/org/apache/beam/sdk/io/mongodb/MongoDbWriteSchemaTransformConfiguration.java
create mode 100644
sdks/java/io/mongodb/src/main/java/org/apache/beam/sdk/io/mongodb/MongoDbWriteSchemaTransformProvider.java
create mode 100644
sdks/java/io/mongodb/src/test/java/org/apache/beam/sdk/io/mongodb/MongoDbUtilsTest.java
create mode 100644
sdks/java/io/mongodb/src/test/java/org/apache/beam/sdk/io/mongodb/MongoDbWriteSchemaTransformProviderTest.java
create mode 100644
sdks/java/io/mqtt/src/main/java/org/apache/beam/sdk/io/mqtt/MqttReadSchemaTransformProvider.java
create mode 100644
sdks/java/io/mqtt/src/main/java/org/apache/beam/sdk/io/mqtt/MqttWriteSchemaTransformProvider.java
create mode 100644
sdks/java/io/mqtt/src/test/java/org/apache/beam/sdk/io/mqtt/MqttSchemaTransformProviderTest.java
delete mode 100644 sdks/python/.isort.cfg
create mode 100644 sdks/python/apache_beam/examples/ml_transform/README.md
create mode 100644
sdks/python/apache_beam/examples/ml_transform/mltransform_generate_vocab.py
copy sdks/python/apache_beam/examples/{wordcount_rust/requirements.txt =>
ml_transform/mltransform_generate_vocab_requirements.txt} (67%)
create mode 100644
sdks/python/apache_beam/examples/ml_transform/mltransform_generate_vocab_test.py
create mode 100644
sdks/python/apache_beam/examples/ml_transform/mltransform_one_hot_encoding.py
create mode 100644
sdks/python/apache_beam/examples/ml_transform/mltransform_one_hot_encoding_test.py
create mode 100644 sdks/python/apache_beam/ml/rag/ingestion/qdrant.py
create mode 100644 sdks/python/apache_beam/ml/rag/ingestion/qdrant_it_test.py
create mode 100644 sdks/python/apache_beam/ml/rag/ingestion/qdrant_test.py
copy sdks/python/apache_beam/{examples/wordcount_rust/requirements.txt =>
ml/transforms/mltransform_tests_requirements.txt} (67%)
delete mode 100644
sdks/python/apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py
delete mode 100644
sdks/python/apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_messages.py
delete mode 100644
sdks/python/apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers.py
delete mode 100644
sdks/python/apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers_test.py
create mode 100644
sdks/python/apache_beam/testing/benchmarks/inference/mltransform_generate_vocab_benchmark.py
create mode 100644
sdks/python/apache_beam/testing/benchmarks/inference/mltransform_one_hot_encoding_benchmark.py
rename
sdks/python/apache_beam/{runners/dataflow/internal/clients/dataflow/__init__.py
=> utils/byte_limited_queue.pxd} (56%)
create mode 100644 sdks/python/apache_beam/utils/byte_limited_queue.py
create mode 100644 sdks/python/apache_beam/utils/byte_limited_queue_test.py
copy sdks/python/apache_beam/{examples/wordcount_rust/requirements.txt =>
yaml/test_utils/__init__.py} (93%)
create mode 100644
sdks/python/apache_beam/yaml/test_utils/datadog_test_utils.py
create mode 100644 sdks/python/apache_beam/yaml/tests/datadog.yaml
create mode 100644 sdks/python/apache_beam/yaml/tests/match_all.yaml
copy sdks/python/apache_beam/{version.py => yaml/tests/mongodb.yaml} (50%)
create mode 100644
sdks/python/apache_beam/yaml/tests/runinference_huggingface.yaml
rename sdks/python/apache_beam/yaml/tests/{runinference.yaml =>
runinference_vertexai.yaml} (100%)
create mode 100644 sdks/python/container/license_scripts/upgrade_bundled_pip.py
create mode 100644 sdks/python/container/profiler.go
delete mode 100644 sdks/python/pyrefly.toml
delete mode 100644 sdks/python/ruff.toml
rename sdks/python/scripts/{run_pylint.sh => run_lint.sh} (96%)
create mode 100644 website/www/site/content/en/blog/beam-2.74.0.md
create mode 100644 website/www/site/content/en/documentation/dsls/sql/ddl.md
create mode 100644
website/www/site/content/en/documentation/transforms/java/aggregation/batchelements.md
create mode 100644
website/www/site/content/en/performance/mltransformonehot/_index.md
create mode 100644
website/www/site/content/en/performance/mltransformvocab/_index.md
