This is an automated email from the ASF dual-hosted git repository.
chamikara pushed a change to branch java_expansion_service_container_snapshots
in repository https://gitbox.apache.org/repos/asf/beam.git
discard aff4921f473 Tags Java SNAPSHOT containers with the '.dev' version
discard af7828ddeb0 Tags Java SNAPSHOT containers with the '.dev' version
discard 4b35ea95ea7 Tags Java SNAPSHOT containers with the '.dev' version
omit d4dd58b2c4c sdks/python/apache_beam/io: add note on `output_fn` ctor
arg to indicate that It is unused and only retained for backward compatibility
(#32635)
omit decc997ea57 Improve wheels job name (#32644)
omit 2a917c71e20 Update CHANGES.md add 2.61.0 section and 2.59.0 known
issue (#32664)
omit a2710ed16e6 Spark Runner: Change to use partitioner in
GroupNonMergingWindowsFunctions#groupByKeyInGlobalWindow (#32610)
omit d84cfffd017 [Managed Iceberg] add GiB autosharding (#32612)
omit 0a71499f3f5 Fix counter metrics for
ParDo#with_exception_handling(timeout). (#32571)
omit 001ac59aa58 [#32601][prism] Initial Deep Dive Documentation (#32143)
omit 6561abf7beb Update groupbykey.py (#32359)
omit eb84ff6b962 Bump go.mongodb.org/mongo-driver from 1.17.0 to 1.17.1 in
/sdks (#32641)
omit 1d7b0bcde9e Rag opensearch usecase with Beam's MLTransform (#32018)
omit 2fb9efc6e52 Moving to 2.61.0-SNAPSHOT on master branch.
omit 3f76d11350e Optimized SparkRunner ParDo Operation (#32546)
omit 9095547bcd3 Update direct_runner.py (#32325)
omit bb09de7c183 Add Java documentation to IcebergIO (#32621)
omit d5b910c750d Remove unused testinfra pipelines module (#32560)
omit fbeba899071 Revert "Deepcopy combine_fn in PrecombineFn and
PostCombineFn." (#32634)
omit 3f79e30996e [yaml] package kafka_clients 3.1.2 in Kafka Provider jar
(#32623)
omit 6314775b2a3 Move remaining reference of python3.8 docker image to
python 3.9 (#32630)
omit 2829c140c63 Update building steps to use go 1.23.2 (#32629)
omit e91882f010c Fix pull_licenses_java script retry broken for tenacity
8.5 (#32626)
omit 8bdea3df66e Bump google.golang.org/api from 0.197.0 to 0.199.0 in
/sdks (#32605)
omit 721cbba4707 add virtualenv to bypass system packages requirement with
pip. Bump pyYAML (#32625)
omit 322667368c4 BigQuey fix invalid null checks in io translation (#32515)
omit d9ffc23ae13 Call out breaking assert_that change more explicitly
(#32624)
omit 6a09ca9f091 Bump cloud.google.com/go/bigtable from 1.31.0 to 1.33.0 in
/sdks (#32556)
omit a92eb38db67 Bump github.com/docker/docker in /sdks (#32554)
omit 36a8d39a3de [prism][java] Update Prism locator to match Python SDK
semantics. (#32619)
omit 415fdd32e7d Report Lineage metrics for SpannerIO (#32561)
omit 1bb357159d6 [prism][Java] Register option types (#32616)
omit 0ca3f19555e Merge pull request #32342 Fix writing raw messages to
pubsub.
omit 79b534c8941 fix typo
omit 00445adda91 Support string FQN as a way to add lineage information
(#32613)
omit eaf53e531b9 Deepcopy combine_fn in PrecombineFn and PostCombineFn.
(#32598)
omit 987c4c9a903 Bump dataflow java container version to
beam-master-20240930 (#32615)
omit 2438f43d604 [YAML] - Remove warning message (#32607)
omit 6a7ffa587df Update Release guide with new github release guidance.
(#32576)
omit 3dee026a2ab Update staticcheck version to fix breakage. (#32614)
omit 5aed5850619 Merge pull request #32567: Fixes a transform upgrade
compatibility issue related to BigqueryIO
omit dcc0bd2fb22 Build release candidate with Java 11 (#32573)
omit c8c674e1c04 Add support for dynamic write in `MqttIO` (#32470)
omit 76a600d25b8 Fix formatting typo.
omit 301286fd6a8 default null array to empty array (#32604)
omit aadb695348d Fix a bug in _get_function_body_without_inners for module
sdks.python.transforms.core (#32591)
omit deb4b4ada3a Managed Iceberg dynamic destinations (#32565)
omit 111f4c34ab2 Merge pull request #32602 from apache/tvalentyn-patch-2
omit 68263fcf507 Require bytes or strings.
omit 9b85f718cb8 [yaml] Preserve windowing for windowed input when using
FileIO Java providers (#32586)
omit 40a5524d65d Update environments.py
omit 543d2ce97de Keep string FQN as a way to add lineage information
(#32585)
omit 7da6c89592b [Go] Update Go version used by Beam repo to go1.23.1
(#32575)
omit 9487927adce [#32562] Incorporate Prism into the Beam Website. (#32563)
omit e640b2578fb Force BQIO to output elements in the correct row (#32584)
omit 48f836aae06 Bump github.com/aws/aws-sdk-go-v2/feature/s3/manager in
/sdks (#32595)
omit ac01dccfd28 represenation->representation (#32588)
omit e705e28ecfd Merge pull request #32589 from jaehyeon-kim/master
omit 9b28982fe4b Update python KafkaIO docstring to add use_deprecated_read
option
omit ee604a5eb8a Improve BQ <-> Avro conversions (#32482)
omit 271ea431156 BigQuery trim schema with selected fields (#32514)
omit 4309675e266 Merge pull request #32516 from hjtran/permissive_num_types
omit c2c640f8c33 Update CHANGES for recent fixes, improvements. (#32568)
omit 160dffd88e5 Revert "fix: add topic existing validation (#32465)"
(#32572)
omit 7c565c38c3c Add info log between executing work failed and cleanup
(#32570)
omit 91c9be4e9ce Log warning if process function return None (#32545)
omit 1eddbdca4c5 Tour of Beam markdown touchups (#32536)
omit 11318ae4211 Bump cloud.google.com/go/bigquery from 1.62.0 to 1.63.0 in
/sdks (#32555)
omit 9053dbf72e9 [website][Go SDK] Add Go testing advice to test your
pipeline. (#32549)
omit 8e8c9143ca5 [Dataflow Streaming] Use isolated windmill streams based
on job settings (#32503)
omit c7fb9a0d5da [Dataflow Streaming] Use separate heartbeat streams based
on job settings (#32511)
omit aabae2729c0 fix NPE and other exceptions thrown when work fails to
process before work execution (#32566)
omit 4d254a0863b Fixes a transform upgrade compatibility issue related to
BigqueryIO
omit 80960d19b61 Remove unused code (#32414)
omit 380ed7b7f74 fix: add topic existing validation (#32465)
omit 01c7caf92af SolaceIO: deduplicate records in the integration test
(#32557)
omit e3c6f4747be Add RowStringInterpolator utility (#32367)
omit faf884a7a1b Update Python Inference Benchmark Test to Python 3.10
(#32550)
omit d93f93abf0e Fix backlog reported by periodicsequence in python and go
sdks to not include future outputs (#32513)
omit 72d46ce0caf Revert "Bump com.pswidersk.terraform-plugin from 1.0.0 to
1.1.1 (#32541)" (#32551)
omit a79e7103450 Set a UUID when building a Schema object. (#32399)
omit c503ec4fda5 Bump github.com/aws/aws-sdk-go-v2/feature/s3/manager in
/sdks (#32543)
omit 5d8b0418cef Update build.gradle to use 3.12 for metrics report (#32547)
omit 2d5e72c615d Bump go.mongodb.org/mongo-driver from 1.16.1 to 1.17.0 in
/sdks (#32497)
omit d312e959fa0 Bump com.pswidersk.terraform-plugin from 1.0.0 to 1.1.1
(#32541)
omit cfcccbb4718 Bump github.com/aws/aws-sdk-go-v2/credentials in /sdks
(#32542)
omit 97cb452e4b9 Vllm model handler (#32410)
omit c3be9f096f0 Add RowFilter utility (#32366)
omit 65b29361eb1 Set np printoptions (#32539)
omit 878ccceb038 ReadFromKafkaDoFn: handle failed seek (#32456)
omit 42efefd9892 [GCSIO] Fix internal unit test failure (#32518)
omit 7c382df7d18 Merge pull request #32530 from apache/users/damccorm/assert
omit 6adb1453890 Add schrodinger logo (#32525)
omit 149f355cc84 Fix elements
omit 986ee964858 [Prism] Support BundleFinalization DoFn parameter (#32425)
omit 50dd0487724 Update beam_PostCommit_Python.json
omit 855bd54bba1 Move assert into pipeline
omit 818966c6163 [#32498][prism] Add split / progress back off + catch-up.
(#32526)
omit 6a095456aa3 [Python] Add a couple quality-of-life improvemenets to
`testing.util.assert_that` (#30771)
omit 7474e6a7427 Rollback Bigtable throttling counter (#32442)
omit 678104c55ec Remove debug leftover (#32524)
omit bbf72c2cf92 Include jamm as runtime dependency in expansion-service
(#32521)
omit 22ce9feddb9 update a couple more tests
omit f2d0558ad58 [Python] Fix bug where WorkerHandlerManager.close_all
re-initialized state with incorrect dict type (#32517)
omit ad55edc3249 Update failing unit test to use str instead of int
omit 52b86c52a29 Update is_consistent_with
omit 4661b14017f reverse ordering of test
omit 7f849eb49a9 Add test case
omit e1e6bdb2947 Change PeriodicSequence to report backlog accurately
(#32505)
omit 7581eb1667b Add runners/prism/java to java PreCommit (#32507)
omit 75a463790d7 Support Managed Iceberg streaming writes (#32451)
omit ec307a52496 Emit warning when Mqtt waiting for connection for extended
period of time (#32500)
omit bb96ac0456c Publish Java Expansion Service SNAPSHOT containers (#32491)
omit 83411302794 add test to verify kafka sdf commit expansion (#32438)
omit aeead3f7cd3 Disable a permared suite. (#32493)
omit 06e2170c1ab Update python SDK container (#32501)
omit eb8639b013e Add throttling counter in gcsio and refactor retrying
(#32428)
omit 475c98c8779 Merge pull request Better error message for incorrect
pipeline options flags.
omit 7daeadff032 Restore release-essential references to 3.8 (#32499)
omit ef532179d32 Fix JmsIO NPE (#32489)
omit 3105260ce58 Warn when using Beam with Python 3.8 (#32484)
omit 019efe131f9 [Dataflow Streaming] Add support to read user worker
settings from backend (#32408)
omit 70dd6473abd Bump github.com/docker/docker in /sdks (#32495)
omit e5f454b04b3 Fix mypy strictness.
omit 8690ddd6bfa Update python SDK container (#32487)
omit fc4db69dea6 Merge pull request #32463 from
shunping/readfromcsv-encoding
omit 4b8cc5fe082 Merge branch 'master' into readfromcsv-encoding
omit ec37750639b Flush the underlying file-like object.
omit e3f8c20eeea More conservative args parsing.
omit 2cd6fb8d758 Better error message for incorrect pipeline options flags.
omit 8088ecfffb3 Set default file mode to read for _TruncatingFileHandle.
omit f81c475f02a Apply yapf
omit 3cc00799c76 Fix non-utf8 encoding problem in ReadFromCsv and
WritetoCsv.
omit 3d90cffe6b0 Add case for exisiting bytes object
omit a0c9245e752 Fix writing raw messages to pubsub
add 05c7091625a Publish Java Expansion Service SNAPSHOT containers
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (aff4921f473)
\
N -- N -- N
refs/heads/java_expansion_service_container_snapshots (05c7091625a)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.asf.yaml | 1 -
.../test-properties.json | 2 +-
.../actions/setup-environment-action/action.yml | 2 +-
.github/autolabeler.yml | 7 +-
.github/build.gradle | 4 +-
.../arc/images/Dockerfile | 2 +-
.../IO_Iceberg_Integration_Tests.json | 2 +-
...beam_PostCommit_Java_ValidatesRunner_Spark.json | 3 +-
...a_ValidatesRunner_SparkStructuredStreaming.json | 3 +-
...stCommit_Java_ValidatesRunner_Spark_Java11.json | 3 +-
.github/trigger_files/beam_PostCommit_Python.json | 2 +-
...tCommit_Python_ValidatesContainer_Dataflow.json | 3 +-
.github/workflows/README.md | 4 +-
.../beam_Inference_Python_Benchmarks_Dataflow.yml | 12 +-
.../beam_PostCommit_XVR_GoUsingJava_Dataflow.yml | 11 +-
.../workflows/beam_Publish_Beam_SDK_Snapshots.yml | 11 +-
.github/workflows/build_release_candidate.yml | 24 +-
.github/workflows/build_wheels.yml | 6 +-
.github/workflows/go_tests.yml | 4 +-
...hon_Combine_Flink_Batch_2GB_10_byte_records.txt | 2 +-
.../python_Combine_Flink_Batch_2GB_Fanout_4.txt | 2 +-
.../python_Combine_Flink_Batch_2GB_Fanout_8.txt | 2 +-
.../python_GBK_Flink_Batch_2GB_of_10B_records.txt | 2 +-
.github/workflows/update_python_dependencies.yml | 1 -
.test-infra/jenkins/build.gradle | 3 +
.test-infra/pipelines/README.md | 34 +
.test-infra/pipelines/build.gradle | 85 +
.../core/core.go => .test-infra/pipelines/go.mod | 21 +-
.test-infra/pipelines/go.sum | 2 +
.../infrastructure/01.setup/.terraform.lock.hcl | 40 +
.../pipelines/infrastructure/01.setup/README.md | 83 +
.../01.setup/apache-beam-testing.tfbackend | 8 +-
.../01.setup/apache-beam-testing.tfvars | 8 +-
.../infrastructure/01.setup/artifactregistry.tf | 17 +-
.../infrastructure/01.setup/common.tfvars | 10 +-
.../pipelines/infrastructure/01.setup/iam.tf | 37 +
.../pipelines/infrastructure/01.setup/provider.tf | 11 +-
.../pipelines/infrastructure/01.setup/services.tf | 20 +-
.../pipelines/infrastructure/01.setup/state.tf | 13 +-
.../pipelines/infrastructure/01.setup/storage.tf | 34 +-
.../pipelines/infrastructure/01.setup/variables.tf | 22 +-
.../infrastructure/02.network/.terraform.lock.hcl | 22 +
.../pipelines/infrastructure/02.network/README.md | 81 +
.../02.network/apache-beam-testing.tfbackend | 8 +-
.../02.network/apache-beam-testing.tfvars | 8 +-
.../infrastructure/02.network/common.tfvars | 11 +-
.../pipelines/infrastructure/02.network/data.tf | 9 +-
.../pipelines/infrastructure/02.network/network.tf | 46 +
.../infrastructure/02.network}/provider.tf | 0
.../pipelines/infrastructure/02.network/state.tf | 13 +-
.../infrastructure/02.network/variables.tf | 31 +-
.../pipelines/infrastructure/03.io/README.md | 24 +
.../03.io/api-overuse-study/02.redis/README.md | 46 +
.../03.io/api-overuse-study/02.redis/common.tfvars | 8 +-
.../03.io/api-overuse-study/02.redis/data.tf | 13 +-
.../03.io/api-overuse-study/02.redis/provider.tf | 14 +-
.../03.io/api-overuse-study/02.redis/redis.tf | 24 +-
.../03.io/api-overuse-study/02.redis/variables.tf | 11 +-
.../03.io/api-overuse-study/README.md | 46 +
.../03.io/api-overuse-study/common.tfvars | 8 +-
.../03.io/api-overuse-study/namespace.tf | 13 +-
.../03.io/api-overuse-study/provider.tf | 10 +-
.../03.io/api-overuse-study/variables.tf | 11 +-
.../03.io/dataflow-to-bigquery/.terraform.lock.hcl | 21 +
.../03.io/dataflow-to-bigquery/README.md | 86 +
.../apache-beam-testing.tfbackend | 8 +-
.../apache-beam-testing.tfvars | 8 +-
.../03.io/dataflow-to-bigquery/bigquery.tf | 15 +-
.../03.io/dataflow-to-bigquery/common.tfvars | 11 +-
.../03.io/dataflow-to-bigquery/data.tf | 12 +-
.../03.io/dataflow-to-bigquery/iam.tf | 34 +-
.../03.io/dataflow-to-bigquery/provider.tf | 10 +-
.../03.io/dataflow-to-bigquery/pubsub.tf | 32 +-
.../03.io/dataflow-to-bigquery/state.tf | 13 +-
.../03.io/dataflow-to-bigquery/variables.tf | 35 +-
.../03.io/dataflow-to-bigquery/workflow.tf | 67 +
.../04.template/dataflow-to-bigquery/README.md | 66 +
.../apache-beam-testing.tfvars | 9 +-
.../04.template/dataflow-to-bigquery/common.tfvars | 13 +-
.../04.template/dataflow-to-bigquery/data.tf | 53 +
.../dataflow-to-bigquery/dataflow-template.json | 20 +
.../04.template/dataflow-to-bigquery/output.tf | 9 +-
.../04.template/dataflow-to-bigquery/provider.tf | 13 +-
.../04.template/dataflow-to-bigquery/template.tf | 58 +
.../04.template/dataflow-to-bigquery/variables.tf | 57 +
.test-infra/pipelines/infrastructure/README.md | 65 +
.../src/main/go/internal/environment/variable.go | 81 +
.../main/go/internal/environment/variable_test.go | 358 ++++
.../pipelines/ReadDataflowApiWriteBigQuery.java | 362 +++++
.../pipelines/bigquery/BigQueryWriteOptions.java | 16 +-
.../pipelines/bigquery/BigQueryWrites.java | 166 ++
.../bigquery/DatasetReferenceOptionValue.java | 58 +
.../testinfra/pipelines/bigquery/package-info.java | 9 +-
.../pipelines/conversions/ConversionError.java | 73 +
.../pipelines/conversions/EventarcConversions.java | 116 ++
.../testinfra/pipelines/conversions/JobsToRow.java | 89 +
.../pipelines/conversions/RowConversionResult.java | 81 +
.../conversions/WithAppendedDetailsToRow.java | 265 +++
.../pipelines/conversions/package-info.java | 9 +-
.../pipelines/dataflow/DataflowClientFactory.java | 102 ++
.../DataflowClientFactoryConfiguration.java | 54 +
.../dataflow/DataflowFilterEventarcJobs.java | 110 ++
.../dataflow/DataflowGetJobExecutionDetails.java | 160 ++
.../pipelines/dataflow/DataflowGetJobMetrics.java | 139 ++
.../pipelines/dataflow/DataflowGetJobs.java | 121 ++
.../dataflow/DataflowGetStageExecutionDetails.java | 166 ++
.../pipelines/dataflow/DataflowJobsOptions.java | 25 +-
.../pipelines/dataflow/DataflowReadResult.java | 90 +
.../pipelines/dataflow/DataflowRequestError.java | 95 ++
.../pipelines/dataflow/DataflowRequests.java | 52 +
.../dataflow/JobMetricsWithAppendedDetails.java | 84 +
.../dataflow/StageSummaryWithAppendedDetails.java | 84 +
.../testinfra/pipelines/dataflow/Throttle.java | 74 +
.../dataflow/WorkerDetailsWithAppendedDetails.java | 84 +
.../testinfra/pipelines/dataflow/package-info.java | 9 +-
.../beam/testinfra/pipelines/package-info.java | 10 +-
.../pipelines/pubsub/PubsubReadOptions.java | 16 +-
.../pubsub/SubscriptionPathOptionValue.java | 45 +
.../testinfra/pipelines/pubsub/package-info.java | 9 +-
.../schemas/DependencyDrivenDescriptorQueue.java | 110 ++
.../schemas/DescriptorSchemaRegistry.java | 153 ++
.../schemas/GeneratedMessageV3RowBuilder.java | 280 ++++
.../testinfra/pipelines/schemas/package-info.java | 9 +-
.../conversions/EventarcConversionsTest.java | 271 +++
.../JobMetricsWithAppendedDetailsTest.java | 143 ++
.../StageSummaryWithAppendedDetailsTest.java | 146 ++
.../conversions/WithAppendedDetailsToRowTest.java | 153 ++
.../WorkerDetailsWithAppendedDetailsTest.java | 153 ++
.../dataflow/DataflowFilterEventarcJobsTest.java | 145 ++
.../AbstractGeneratedMessageV3RowBuilderTest.java | 409 +++++
.../DependencyDrivenDescriptorQueueTest.java | 56 +
.../schemas/DescriptorSchemaRegistryTest.java | 79 +
.../schemas/EnvironmentRowBuilderTest.java | 140 ++
.../schemas/ExecutionStageSummaryTest.java | 84 +
.../schemas/TransformSummaryRowBuilderTest.java | 83 +
.../has_extra_data_payload_foo_property.json | 1 +
.../job_state_canceled_streaming.json | 1 +
.../job_state_canceling_streaming.json | 1 +
.../eventarc_data/job_state_done_batch.json | 1 +
.../eventarc_data/job_state_pending_batch.json | 1 +
.../eventarc_data/job_state_pending_streaming.json | 1 +
.../eventarc_data/job_state_queued_streaming.json | 1 +
.../eventarc_data/job_state_running_batch.json | 1 +
.../eventarc_data/job_state_running_streaming.json | 1 +
CHANGES.md | 55 +-
README.md | 1 -
build.gradle.kts | 2 -
.../org/apache/beam/gradle/BeamModulePlugin.groovy | 5 +-
contributor-docs/release-guide.md | 36 +-
dev-support/docker/Dockerfile | 2 +-
examples/multi-language/README.md | 6 +-
.../beam-ml/rag_usecase/opensearch_connector.py | 372 -----
.../beam-ml/rag_usecase/opensearch_enrichment.py | 134 --
.../rag_usecase/opensearch_rag_pipeline.ipynb | 1719 --------------------
gradle.properties | 6 +-
.../overview-pipeline/description.md | 2 +-
.../runner-concepts/description.md | 6 +-
.../introduction/introduction-guide/description.md | 35 +-
local-env-setup.sh | 4 +-
.../cloudbuild/playground_cd_examples.sh | 3 -
playground/infrastructure/requirements.txt | 2 +-
release/src/main/Dockerfile | 7 +-
.../python-release/python_release_automation.sh | 2 +-
runners/google-cloud-dataflow-java/build.gradle | 4 +-
.../options/DataflowStreamingPipelineOptions.java | 10 +-
.../runners/dataflow/worker/OperationalLimits.java | 39 +-
.../dataflow/worker/StreamingDataflowWorker.java | 137 +-
.../worker/StreamingModeExecutionContext.java | 37 +-
.../worker/streaming/ComputationWorkExecutor.java | 5 +-
.../worker/streaming/config/ComputationConfig.java | 3 +-
.../streaming/config/FakeGlobalConfigHandle.java | 52 -
.../streaming/config/FixedGlobalConfigHandle.java | 48 -
...StreamingApplianceComputationConfigFetcher.java | 10 +-
.../StreamingEngineComputationConfigFetcher.java | 69 +-
...fig.java => StreamingEnginePipelineConfig.java} | 41 +-
.../config/StreamingGlobalConfigHandleImpl.java | 113 --
.../harness/StreamingWorkerStatusPages.java | 22 +-
.../windmill/client/grpc/GrpcDispatcherClient.java | 74 +-
.../windmill/client/grpc/GrpcWindmillServer.java | 16 +-
.../grpc/stubs/WindmillStubFactoryFactoryImpl.java | 54 -
.../processing/ComputationWorkExecutorFactory.java | 7 +-
.../work/processing/StreamingWorkScheduler.java | 26 +-
.../work/refresh/StreamPoolHeartbeatSender.java | 45 +-
.../worker/StreamingDataflowWorkerTest.java | 45 +-
.../worker/StreamingModeExecutionContextTest.java | 8 +-
.../dataflow/worker/WorkerCustomSourcesTest.java | 11 +-
.../streaming/ComputationWorkExecutorTest.java | 72 -
.../config/FixedGlobalConfigHandleTest.java | 84 -
...amingApplianceComputationConfigFetcherTest.java | 4 +-
...treamingEngineComputationConfigFetcherTest.java | 102 +-
.../StreamingGlobalConfigHandleImplTest.java | 293 ----
.../FanOutStreamingEngineWorkerHarnessTest.java | 9 +-
.../client/grpc/GrpcDispatcherClientTest.java | 154 --
.../client/grpc/GrpcWindmillServerTest.java | 11 +-
.../refresh/StreamPoolHeartbeatSenderTest.java | 132 --
.../worker/windmill/src/main/proto/windmill.proto | 9 -
runners/prism/java/build.gradle | 5 +
.../beam/runners/prism/PrismArtifactResolver.java | 110 ++
.../beam/runners/prism/PrismArtifactStager.java | 173 ++
.../apache/beam/runners/prism/PrismJobManager.java | 160 ++
.../apache/beam/runners/prism/PrismLocator.java | 69 +-
.../beam/runners/prism/PrismPipelineOptions.java | 7 +-
.../apache/beam/runners/prism/PrismRegistrar.java | 53 -
.../beam/runners/prism/PrismRunnerRegistrar.java} | 24 +-
.../apache/beam/runners/prism/StateListener.java} | 12 +-
.../apache/beam/runners/prism/StateWatcher.java | 146 ++
.../apache/beam/runners/prism/WorkerService.java | 116 ++
.../runners/prism/PrismArtifactResolverTest.java | 45 +
.../runners/prism/PrismArtifactStagerTest.java | 143 ++
.../beam/runners/prism/PrismJobManagerTest.java | 211 +++
.../beam/runners/prism/PrismLocatorTest.java | 87 +-
.../beam/runners/prism/PrismRegistrarTest.java | 72 -
.../beam/runners/prism/StateWatcherTest.java | 136 ++
.../beam/runners/prism/WorkerServiceTest.java | 85 +
.../runners/spark/SparkTransformOverrides.java | 2 +-
.../GroupNonMergingWindowsFunctions.java | 25 +-
.../spark/translation/TransformTranslator.java | 28 +-
.../GroupNonMergingWindowsFunctionsTest.java | 57 -
.../runners/spark/translation/PassThrough.java | 91 --
.../beam/runners/spark/translation/RDDNode.java | 65 -
.../runners/spark/translation/RDDTreeParser.java | 55 -
.../spark/translation/TransformTranslatorTest.java | 96 +-
sdks/go.mod | 66 +-
sdks/go.sum | 132 +-
sdks/go/pkg/beam/core/core.go | 2 +-
sdks/go/pkg/beam/core/runtime/graphx/translate.go | 1 -
sdks/go/pkg/beam/core/runtime/harness/harness.go | 2 +-
sdks/go/pkg/beam/core/typex/special.go | 12 +-
sdks/go/pkg/beam/forward.go | 1 -
sdks/go/pkg/beam/runners/prism/internal/README.md | 492 +-----
.../pkg/beam/runners/prism/internal/handlepardo.go | 6 +-
.../beam/runners/prism/internal/jobservices/job.go | 1 -
.../pkg/beam/runners/prism/internal/preprocess.go | 1 -
.../beam/runners/prism/internal/preprocess_test.go | 6 +-
sdks/go/pkg/beam/runners/prism/internal/stage.go | 58 +-
.../runners/prism/internal/unimplemented_test.go | 1 -
.../beam/runners/prism/internal/worker/bundle.go | 11 -
sdks/go/pkg/beam/transforms/periodic/periodic.go | 13 +-
.../pkg/beam/transforms/periodic/periodic_test.go | 36 -
sdks/go/run_with_go_version.sh | 2 +-
sdks/go/test/integration/integration.go | 15 -
sdks/go/test/integration/primitives/pardo.go | 80 -
sdks/go/test/integration/primitives/pardo_test.go | 43 -
.../license_scripts/pull_licenses_java.py | 12 +-
.../java/org/apache/beam/sdk/metrics/Lineage.java | 15 +-
.../java/org/apache/beam/sdk/schemas/Schema.java | 75 +-
.../apache/beam/sdk/schemas/SchemaTranslation.java | 67 +-
.../beam/sdk/schemas/logicaltypes/OneOfType.java | 3 +-
.../beam/sdk/transforms/PeriodicSequence.java | 34 +-
.../java/org/apache/beam/sdk/util/RowFilter.java | 421 -----
.../beam/sdk/util/RowStringInterpolator.java | 167 --
.../beam/sdk/schemas/SchemaTranslationTest.java | 46 +-
.../sdk/schemas/logicaltypes/LogicalTypesTest.java | 10 -
.../beam/sdk/transforms/PeriodicSequenceTest.java | 22 -
.../org/apache/beam/sdk/util/RowFilterTest.java | 353 ----
.../beam/sdk/util/RowStringInterpolatorTest.java | 205 ---
sdks/java/expansion-service/build.gradle | 1 -
.../extensions/python/PythonExternalTransform.java | 3 +
.../csv/providers/CsvWriteTransformProvider.java | 17 +-
sdks/java/io/expansion-service/build.gradle | 6 +-
.../io/gcp/bigquery/BeamRowToStorageApiProto.java | 3 -
.../sdk/io/gcp/bigquery/BigQueryAvroUtils.java | 464 +++---
.../beam/sdk/io/gcp/bigquery/BigQueryIO.java | 33 +-
.../sdk/io/gcp/bigquery/BigQueryIOTranslation.java | 38 +-
.../io/gcp/bigquery/BigQueryQuerySourceDef.java | 6 +-
.../sdk/io/gcp/bigquery/BigQuerySourceBase.java | 3 +-
.../sdk/io/gcp/bigquery/BigQuerySourceDef.java | 7 +-
.../io/gcp/bigquery/BigQueryStorageSourceBase.java | 46 +-
.../io/gcp/bigquery/BigQueryTableSourceDef.java | 6 +-
.../beam/sdk/io/gcp/bigquery/BigQueryUtils.java | 103 +-
.../beam/sdk/io/gcp/bigtable/BigtableIO.java | 107 +-
.../sdk/io/gcp/bigtable/BigtableWriteOptions.java | 5 +
.../apache/beam/sdk/io/gcp/pubsub/PubsubIO.java | 1 +
.../beam/sdk/io/gcp/spanner/BatchSpannerRead.java | 24 -
.../beam/sdk/io/gcp/spanner/NaiveSpannerRead.java | 26 +-
.../beam/sdk/io/gcp/spanner/ReadOperation.java | 27 -
.../beam/sdk/io/gcp/spanner/SpannerAccessor.java | 26 +-
.../apache/beam/sdk/io/gcp/spanner/SpannerIO.java | 46 +-
.../gcp/bigquery/BeamRowToStorageApiProtoTest.java | 23 +-
.../sdk/io/gcp/bigquery/BigQueryAvroUtilsTest.java | 213 +--
.../gcp/bigquery/BigQueryIOStorageQueryTest.java | 12 +-
.../io/gcp/bigquery/BigQueryIOStorageReadTest.java | 15 +-
.../sdk/io/gcp/bigquery/BigQueryUtilsTest.java | 25 -
...QueryDirectReadSchemaTransformProviderTest.java | 3 +-
.../beam/sdk/io/gcp/bigtable/BigtableIOTest.java | 15 +
.../sdk/io/gcp/spanner/FakeServiceFactory.java | 24 -
.../beam/sdk/io/gcp/spanner/SpannerIOReadTest.java | 195 +--
.../sdk/io/gcp/spanner/SpannerIOWriteTest.java | 97 +-
sdks/java/io/iceberg/hive/exec/build.gradle | 15 +-
.../beam/sdk/io/iceberg/AppendFilesToTables.java | 19 +-
.../beam/sdk/io/iceberg/AssignDestinations.java | 45 +-
.../beam/sdk/io/iceberg/DynamicDestinations.java | 13 +-
.../org/apache/beam/sdk/io/iceberg/IcebergIO.java | 267 +--
.../IcebergWriteSchemaTransformProvider.java | 116 +-
.../io/iceberg/OneTableDynamicDestinations.java | 31 +-
.../io/iceberg/PortableIcebergDestinations.java | 81 -
.../apache/beam/sdk/io/iceberg/RecordWriter.java | 22 +-
.../beam/sdk/io/iceberg/RecordWriterManager.java | 29 +-
.../apache/beam/sdk/io/iceberg/SnapshotInfo.java | 5 +-
.../sdk/io/iceberg/WriteGroupedRowsToFiles.java | 16 +-
.../beam/sdk/io/iceberg/WriteToDestinations.java | 140 +-
.../sdk/io/iceberg/WriteUngroupedRowsToFiles.java | 62 +-
.../apache/beam/sdk/io/iceberg/IcebergIOIT.java | 240 +--
.../beam/sdk/io/iceberg/IcebergIOWriteTest.java | 105 +-
.../IcebergWriteSchemaTransformProviderTest.java | 219 +--
.../java/org/apache/beam/sdk/io/jms/JmsIO.java | 35 +-
.../json/providers/JsonWriteTransformProvider.java | 13 +-
sdks/java/io/kafka/build.gradle | 1 -
.../beam/sdk/io/kafka/ReadFromKafkaDoFn.java | 33 +-
.../org/apache/beam/sdk/io/kafka/KafkaIOTest.java | 6 +-
.../beam/sdk/io/kafka/ReadFromKafkaDoFnTest.java | 96 --
.../java/org/apache/beam/sdk/io/mqtt/MqttIO.java | 161 +-
.../org/apache/beam/sdk/io/mqtt/MqttIOTest.java | 227 +--
.../apache/beam/sdk/io/solace/it/SolaceIOIT.java | 1 -
.../ManagedSchemaTransformTranslationTest.java | 49 +-
sdks/python/apache_beam/__init__.py | 6 -
sdks/python/apache_beam/dataframe/doctests.py | 5 +-
sdks/python/apache_beam/dataframe/io.py | 16 +-
.../apache_beam/examples/inference/README.md | 80 -
.../examples/inference/vllm_text_completion.py | 162 --
.../transforms/aggregation/groupby_expr.py | 5 +-
.../aggregation/groupby_global_aggregate.py | 5 +-
.../aggregation/groupby_simple_aggregate.py | 5 +-
.../transforms/aggregation/groupby_test.py | 125 +-
.../snippets/transforms/aggregation/groupbykey.py | 8 +-
sdks/python/apache_beam/io/fileio.py | 2 -
sdks/python/apache_beam/io/gcp/bigquery.py | 59 +-
.../apache_beam/io/gcp/bigquery_write_it_test.py | 1 -
sdks/python/apache_beam/io/gcp/gcsio.py | 73 +-
.../apache_beam/io/gcp/gcsio_integration_test.py | 39 +-
sdks/python/apache_beam/io/gcp/gcsio_retry.py | 71 -
sdks/python/apache_beam/io/gcp/gcsio_retry_test.py | 84 -
sdks/python/apache_beam/io/gcp/gcsio_test.py | 16 +-
sdks/python/apache_beam/io/kafka.py | 12 -
sdks/python/apache_beam/io/textio_test.py | 42 -
sdks/python/apache_beam/metrics/metric.py | 26 +-
sdks/python/apache_beam/metrics/metric_test.py | 11 -
sdks/python/apache_beam/ml/gcp/visionml_test_it.py | 23 +-
.../ml/inference/test_resources/vllm.dockerfile | 47 -
.../apache_beam/ml/inference/vllm_inference.py | 312 ----
.../python/apache_beam/options/pipeline_options.py | 31 +-
.../apache_beam/options/pipeline_options_test.py | 47 +-
.../apache_beam/runners/dataflow/internal/names.py | 2 +-
.../apache_beam/runners/direct/direct_runner.py | 4 +-
.../portability/fn_api_runner/worker_handlers.py | 2 +-
.../fn_api_runner/worker_handlers_test.py | 52 -
sdks/python/apache_beam/testing/util.py | 22 -
sdks/python/apache_beam/testing/util_test.py | 13 -
sdks/python/apache_beam/transforms/core.py | 49 +-
sdks/python/apache_beam/transforms/core_test.py | 38 -
sdks/python/apache_beam/transforms/environments.py | 6 +-
.../apache_beam/transforms/periodicsequence.py | 21 +-
.../transforms/periodicsequence_test.py | 20 -
.../apache_beam/transforms/ptransform_test.py | 32 +-
sdks/python/apache_beam/transforms/trigger_test.py | 11 +-
sdks/python/apache_beam/transforms/util_test.py | 8 +-
.../apache_beam/typehints/typed_pipeline_test.py | 10 +-
sdks/python/apache_beam/typehints/typehints.py | 6 -
.../python/apache_beam/typehints/typehints_test.py | 10 +-
sdks/python/apache_beam/version.py | 2 +-
sdks/python/apache_beam/yaml/yaml_io.py | 17 +-
sdks/python/setup.py | 6 -
.../tensorrt_runinference/tensor_rt.dockerfile | 6 +-
sdks/python/test-suites/dataflow/common.gradle | 39 -
sdks/typescript/package.json | 2 +-
settings.gradle.kts | 2 +
website/www/site/content/en/blog/beam-2.59.0.md | 4 +-
.../site/content/en/blog/validate-beam-release.md | 31 -
.../site/content/en/case-studies/schrodinger.md | 19 -
.../documentation/pipelines/test-your-pipeline.md | 119 +-
.../content/en/documentation/runners/direct.md | 1 -
.../site/content/en/documentation/runners/prism.md | 145 --
website/www/site/content/en/roadmap/_index.md | 4 +-
.../www/site/content/en/roadmap/prism-runner.md | 35 -
.../www/site/data/en/documentation_runners.yaml | 3 -
.../layouts/partials/section-menu/en/roadmap.html | 1 -
.../layouts/partials/section-menu/en/runners.html | 1 -
.../static/images/logos/powered-by/schrodinger.png | Bin 145837 -> 0 bytes
378 files changed, 10016 insertions(+), 11706 deletions(-)
create mode 100644 .test-infra/pipelines/README.md
create mode 100644 .test-infra/pipelines/build.gradle
copy sdks/go/pkg/beam/core/core.go => .test-infra/pipelines/go.mod (52%)
create mode 100644 .test-infra/pipelines/go.sum
create mode 100644
.test-infra/pipelines/infrastructure/01.setup/.terraform.lock.hcl
create mode 100644 .test-infra/pipelines/infrastructure/01.setup/README.md
copy
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillStubFactoryFactory.java
=> .test-infra/pipelines/infrastructure/01.setup/apache-beam-testing.tfbackend
(75%)
copy
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillStubFactoryFactory.java
=> .test-infra/pipelines/infrastructure/01.setup/apache-beam-testing.tfvars
(75%)
copy
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillStubFactoryFactory.java
=> .test-infra/pipelines/infrastructure/01.setup/artifactregistry.tf (60%)
copy
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillStubFactoryFactory.java
=> .test-infra/pipelines/infrastructure/01.setup/common.tfvars (75%)
create mode 100644 .test-infra/pipelines/infrastructure/01.setup/iam.tf
copy
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillStubFactoryFactory.java
=> .test-infra/pipelines/infrastructure/01.setup/provider.tf (75%)
copy
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillStubFactoryFactory.java
=> .test-infra/pipelines/infrastructure/01.setup/services.tf (69%)
copy
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillStubFactoryFactory.java
=> .test-infra/pipelines/infrastructure/01.setup/state.tf (74%)
copy
runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/windmill/testing/FakeWindmillStubFactoryFactory.java
=> .test-infra/pipelines/infrastructure/01.setup/storage.tf (51%)
copy
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillStubFactoryFactory.java
=> .test-infra/pipelines/infrastructure/01.setup/variables.tf (60%)
create mode 100644
.test-infra/pipelines/infrastructure/02.network/.terraform.lock.hcl
create mode 100644 .test-infra/pipelines/infrastructure/02.network/README.md
copy
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillStubFactoryFactory.java
=>
.test-infra/pipelines/infrastructure/02.network/apache-beam-testing.tfbackend
(75%)
copy
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillStubFactoryFactory.java
=> .test-infra/pipelines/infrastructure/02.network/apache-beam-testing.tfvars
(75%)
copy
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillStubFactoryFactory.java
=> .test-infra/pipelines/infrastructure/02.network/common.tfvars (75%)
copy
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillStubFactoryFactory.java
=> .test-infra/pipelines/infrastructure/02.network/data.tf (75%)
create mode 100644 .test-infra/pipelines/infrastructure/02.network/network.tf
copy .test-infra/{kafka/proxy =>
pipelines/infrastructure/02.network}/provider.tf (100%)
copy
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillStubFactoryFactory.java
=> .test-infra/pipelines/infrastructure/02.network/state.tf (74%)
copy
runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/windmill/testing/FakeWindmillStubFactoryFactory.java
=> .test-infra/pipelines/infrastructure/02.network/variables.tf (50%)
create mode 100644 .test-infra/pipelines/infrastructure/03.io/README.md
create mode 100644
.test-infra/pipelines/infrastructure/03.io/api-overuse-study/02.redis/README.md
copy
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillStubFactoryFactory.java
=>
.test-infra/pipelines/infrastructure/03.io/api-overuse-study/02.redis/common.tfvars
(75%)
copy
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillStubFactoryFactory.java
=>
.test-infra/pipelines/infrastructure/03.io/api-overuse-study/02.redis/data.tf
(75%)
copy
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillStubFactoryFactory.java
=>
.test-infra/pipelines/infrastructure/03.io/api-overuse-study/02.redis/provider.tf
(75%)
copy
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillStubFactoryFactory.java
=>
.test-infra/pipelines/infrastructure/03.io/api-overuse-study/02.redis/redis.tf
(65%)
copy
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillStubFactoryFactory.java
=>
.test-infra/pipelines/infrastructure/03.io/api-overuse-study/02.redis/variables.tf
(75%)
create mode 100644
.test-infra/pipelines/infrastructure/03.io/api-overuse-study/README.md
copy
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillStubFactoryFactory.java
=> .test-infra/pipelines/infrastructure/03.io/api-overuse-study/common.tfvars
(75%)
copy
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillStubFactoryFactory.java
=> .test-infra/pipelines/infrastructure/03.io/api-overuse-study/namespace.tf
(75%)
copy
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillStubFactoryFactory.java
=> .test-infra/pipelines/infrastructure/03.io/api-overuse-study/provider.tf
(75%)
copy
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillStubFactoryFactory.java
=> .test-infra/pipelines/infrastructure/03.io/api-overuse-study/variables.tf
(75%)
create mode 100644
.test-infra/pipelines/infrastructure/03.io/dataflow-to-bigquery/.terraform.lock.hcl
create mode 100644
.test-infra/pipelines/infrastructure/03.io/dataflow-to-bigquery/README.md
copy
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillStubFactoryFactory.java
=>
.test-infra/pipelines/infrastructure/03.io/dataflow-to-bigquery/apache-beam-testing.tfbackend
(75%)
copy
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillStubFactoryFactory.java
=>
.test-infra/pipelines/infrastructure/03.io/dataflow-to-bigquery/apache-beam-testing.tfvars
(75%)
copy
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillStubFactoryFactory.java
=> .test-infra/pipelines/infrastructure/03.io/dataflow-to-bigquery/bigquery.tf
(60%)
copy
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillStubFactoryFactory.java
=>
.test-infra/pipelines/infrastructure/03.io/dataflow-to-bigquery/common.tfvars
(75%)
copy
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillStubFactoryFactory.java
=> .test-infra/pipelines/infrastructure/03.io/dataflow-to-bigquery/data.tf
(78%)
copy
runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/windmill/testing/FakeWindmillStubFactoryFactory.java
=> .test-infra/pipelines/infrastructure/03.io/dataflow-to-bigquery/iam.tf (50%)
copy
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillStubFactoryFactory.java
=> .test-infra/pipelines/infrastructure/03.io/dataflow-to-bigquery/provider.tf
(75%)
copy
runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/windmill/testing/FakeWindmillStubFactoryFactory.java
=> .test-infra/pipelines/infrastructure/03.io/dataflow-to-bigquery/pubsub.tf
(50%)
copy
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillStubFactoryFactory.java
=> .test-infra/pipelines/infrastructure/03.io/dataflow-to-bigquery/state.tf
(73%)
copy
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/DynamicDestinations.java
=>
.test-infra/pipelines/infrastructure/03.io/dataflow-to-bigquery/variables.tf
(52%)
create mode 100644
.test-infra/pipelines/infrastructure/03.io/dataflow-to-bigquery/workflow.tf
create mode 100644
.test-infra/pipelines/infrastructure/04.template/dataflow-to-bigquery/README.md
copy
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillStubFactoryFactory.java
=>
.test-infra/pipelines/infrastructure/04.template/dataflow-to-bigquery/apache-beam-testing.tfvars
(75%)
copy
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillStubFactoryFactory.java
=>
.test-infra/pipelines/infrastructure/04.template/dataflow-to-bigquery/common.tfvars
(69%)
create mode 100644
.test-infra/pipelines/infrastructure/04.template/dataflow-to-bigquery/data.tf
create mode 100644
.test-infra/pipelines/infrastructure/04.template/dataflow-to-bigquery/dataflow-template.json
copy
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillStubFactoryFactory.java
=>
.test-infra/pipelines/infrastructure/04.template/dataflow-to-bigquery/output.tf
(75%)
rename sdks/java/io/kafka/kafka-312/build.gradle =>
.test-infra/pipelines/infrastructure/04.template/dataflow-to-bigquery/provider.tf
(72%)
create mode 100644
.test-infra/pipelines/infrastructure/04.template/dataflow-to-bigquery/template.tf
create mode 100644
.test-infra/pipelines/infrastructure/04.template/dataflow-to-bigquery/variables.tf
create mode 100644 .test-infra/pipelines/infrastructure/README.md
create mode 100644
.test-infra/pipelines/src/main/go/internal/environment/variable.go
create mode 100644
.test-infra/pipelines/src/main/go/internal/environment/variable_test.go
create mode 100644
.test-infra/pipelines/src/main/java/org/apache/beam/testinfra/pipelines/ReadDataflowApiWriteBigQuery.java
copy
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillStubFactoryFactory.java
=>
.test-infra/pipelines/src/main/java/org/apache/beam/testinfra/pipelines/bigquery/BigQueryWriteOptions.java
(62%)
create mode 100644
.test-infra/pipelines/src/main/java/org/apache/beam/testinfra/pipelines/bigquery/BigQueryWrites.java
create mode 100644
.test-infra/pipelines/src/main/java/org/apache/beam/testinfra/pipelines/bigquery/DatasetReferenceOptionValue.java
copy
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillStubFactoryFactory.java
=>
.test-infra/pipelines/src/main/java/org/apache/beam/testinfra/pipelines/bigquery/package-info.java
(75%)
create mode 100644
.test-infra/pipelines/src/main/java/org/apache/beam/testinfra/pipelines/conversions/ConversionError.java
create mode 100644
.test-infra/pipelines/src/main/java/org/apache/beam/testinfra/pipelines/conversions/EventarcConversions.java
create mode 100644
.test-infra/pipelines/src/main/java/org/apache/beam/testinfra/pipelines/conversions/JobsToRow.java
create mode 100644
.test-infra/pipelines/src/main/java/org/apache/beam/testinfra/pipelines/conversions/RowConversionResult.java
create mode 100644
.test-infra/pipelines/src/main/java/org/apache/beam/testinfra/pipelines/conversions/WithAppendedDetailsToRow.java
copy
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillStubFactoryFactory.java
=>
.test-infra/pipelines/src/main/java/org/apache/beam/testinfra/pipelines/conversions/package-info.java
(75%)
create mode 100644
.test-infra/pipelines/src/main/java/org/apache/beam/testinfra/pipelines/dataflow/DataflowClientFactory.java
create mode 100644
.test-infra/pipelines/src/main/java/org/apache/beam/testinfra/pipelines/dataflow/DataflowClientFactoryConfiguration.java
create mode 100644
.test-infra/pipelines/src/main/java/org/apache/beam/testinfra/pipelines/dataflow/DataflowFilterEventarcJobs.java
create mode 100644
.test-infra/pipelines/src/main/java/org/apache/beam/testinfra/pipelines/dataflow/DataflowGetJobExecutionDetails.java
create mode 100644
.test-infra/pipelines/src/main/java/org/apache/beam/testinfra/pipelines/dataflow/DataflowGetJobMetrics.java
create mode 100644
.test-infra/pipelines/src/main/java/org/apache/beam/testinfra/pipelines/dataflow/DataflowGetJobs.java
create mode 100644
.test-infra/pipelines/src/main/java/org/apache/beam/testinfra/pipelines/dataflow/DataflowGetStageExecutionDetails.java
rename
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/config/StreamingGlobalConfigHandle.java
=>
.test-infra/pipelines/src/main/java/org/apache/beam/testinfra/pipelines/dataflow/DataflowJobsOptions.java
(58%)
create mode 100644
.test-infra/pipelines/src/main/java/org/apache/beam/testinfra/pipelines/dataflow/DataflowReadResult.java
create mode 100644
.test-infra/pipelines/src/main/java/org/apache/beam/testinfra/pipelines/dataflow/DataflowRequestError.java
create mode 100644
.test-infra/pipelines/src/main/java/org/apache/beam/testinfra/pipelines/dataflow/DataflowRequests.java
create mode 100644
.test-infra/pipelines/src/main/java/org/apache/beam/testinfra/pipelines/dataflow/JobMetricsWithAppendedDetails.java
create mode 100644
.test-infra/pipelines/src/main/java/org/apache/beam/testinfra/pipelines/dataflow/StageSummaryWithAppendedDetails.java
create mode 100644
.test-infra/pipelines/src/main/java/org/apache/beam/testinfra/pipelines/dataflow/Throttle.java
create mode 100644
.test-infra/pipelines/src/main/java/org/apache/beam/testinfra/pipelines/dataflow/WorkerDetailsWithAppendedDetails.java
copy
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillStubFactoryFactory.java
=>
.test-infra/pipelines/src/main/java/org/apache/beam/testinfra/pipelines/dataflow/package-info.java
(75%)
copy
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillStubFactoryFactory.java
=>
.test-infra/pipelines/src/main/java/org/apache/beam/testinfra/pipelines/package-info.java
(75%)
copy
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillStubFactoryFactory.java
=>
.test-infra/pipelines/src/main/java/org/apache/beam/testinfra/pipelines/pubsub/PubsubReadOptions.java
(61%)
create mode 100644
.test-infra/pipelines/src/main/java/org/apache/beam/testinfra/pipelines/pubsub/SubscriptionPathOptionValue.java
copy
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillStubFactoryFactory.java
=>
.test-infra/pipelines/src/main/java/org/apache/beam/testinfra/pipelines/pubsub/package-info.java
(75%)
create mode 100644
.test-infra/pipelines/src/main/java/org/apache/beam/testinfra/pipelines/schemas/DependencyDrivenDescriptorQueue.java
create mode 100644
.test-infra/pipelines/src/main/java/org/apache/beam/testinfra/pipelines/schemas/DescriptorSchemaRegistry.java
create mode 100644
.test-infra/pipelines/src/main/java/org/apache/beam/testinfra/pipelines/schemas/GeneratedMessageV3RowBuilder.java
copy
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillStubFactoryFactory.java
=>
.test-infra/pipelines/src/main/java/org/apache/beam/testinfra/pipelines/schemas/package-info.java
(75%)
create mode 100644
.test-infra/pipelines/src/test/java/org/apache/beam/testinfra/pipelines/conversions/EventarcConversionsTest.java
create mode 100644
.test-infra/pipelines/src/test/java/org/apache/beam/testinfra/pipelines/conversions/JobMetricsWithAppendedDetailsTest.java
create mode 100644
.test-infra/pipelines/src/test/java/org/apache/beam/testinfra/pipelines/conversions/StageSummaryWithAppendedDetailsTest.java
create mode 100644
.test-infra/pipelines/src/test/java/org/apache/beam/testinfra/pipelines/conversions/WithAppendedDetailsToRowTest.java
create mode 100644
.test-infra/pipelines/src/test/java/org/apache/beam/testinfra/pipelines/conversions/WorkerDetailsWithAppendedDetailsTest.java
create mode 100644
.test-infra/pipelines/src/test/java/org/apache/beam/testinfra/pipelines/dataflow/DataflowFilterEventarcJobsTest.java
create mode 100644
.test-infra/pipelines/src/test/java/org/apache/beam/testinfra/pipelines/schemas/AbstractGeneratedMessageV3RowBuilderTest.java
create mode 100644
.test-infra/pipelines/src/test/java/org/apache/beam/testinfra/pipelines/schemas/DependencyDrivenDescriptorQueueTest.java
create mode 100644
.test-infra/pipelines/src/test/java/org/apache/beam/testinfra/pipelines/schemas/DescriptorSchemaRegistryTest.java
create mode 100644
.test-infra/pipelines/src/test/java/org/apache/beam/testinfra/pipelines/schemas/EnvironmentRowBuilderTest.java
create mode 100644
.test-infra/pipelines/src/test/java/org/apache/beam/testinfra/pipelines/schemas/ExecutionStageSummaryTest.java
create mode 100644
.test-infra/pipelines/src/test/java/org/apache/beam/testinfra/pipelines/schemas/TransformSummaryRowBuilderTest.java
create mode 100644
.test-infra/pipelines/src/test/resources/eventarc_data/has_extra_data_payload_foo_property.json
create mode 100644
.test-infra/pipelines/src/test/resources/eventarc_data/job_state_canceled_streaming.json
create mode 100644
.test-infra/pipelines/src/test/resources/eventarc_data/job_state_canceling_streaming.json
create mode 100644
.test-infra/pipelines/src/test/resources/eventarc_data/job_state_done_batch.json
create mode 100644
.test-infra/pipelines/src/test/resources/eventarc_data/job_state_pending_batch.json
create mode 100644
.test-infra/pipelines/src/test/resources/eventarc_data/job_state_pending_streaming.json
create mode 100644
.test-infra/pipelines/src/test/resources/eventarc_data/job_state_queued_streaming.json
create mode 100644
.test-infra/pipelines/src/test/resources/eventarc_data/job_state_running_batch.json
create mode 100644
.test-infra/pipelines/src/test/resources/eventarc_data/job_state_running_streaming.json
delete mode 100644
examples/notebooks/beam-ml/rag_usecase/opensearch_connector.py
delete mode 100644
examples/notebooks/beam-ml/rag_usecase/opensearch_enrichment.py
delete mode 100644
examples/notebooks/beam-ml/rag_usecase/opensearch_rag_pipeline.ipynb
delete mode 100644
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/config/FakeGlobalConfigHandle.java
delete mode 100644
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/config/FixedGlobalConfigHandle.java
rename
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/config/{StreamingGlobalConfig.java
=> StreamingEnginePipelineConfig.java} (56%)
delete mode 100644
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/config/StreamingGlobalConfigHandleImpl.java
delete mode 100644
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillStubFactoryFactoryImpl.java
delete mode 100644
runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/streaming/ComputationWorkExecutorTest.java
delete mode 100644
runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/streaming/config/FixedGlobalConfigHandleTest.java
delete mode 100644
runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/streaming/config/StreamingGlobalConfigHandleImplTest.java
delete mode 100644
runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/GrpcDispatcherClientTest.java
delete mode 100644
runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/windmill/work/refresh/StreamPoolHeartbeatSenderTest.java
create mode 100644
runners/prism/java/src/main/java/org/apache/beam/runners/prism/PrismArtifactResolver.java
create mode 100644
runners/prism/java/src/main/java/org/apache/beam/runners/prism/PrismArtifactStager.java
create mode 100644
runners/prism/java/src/main/java/org/apache/beam/runners/prism/PrismJobManager.java
delete mode 100644
runners/prism/java/src/main/java/org/apache/beam/runners/prism/PrismRegistrar.java
rename
runners/{google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/windmill/testing/FakeWindmillStubFactoryFactory.java
=>
prism/java/src/main/java/org/apache/beam/runners/prism/PrismRunnerRegistrar.java}
(55%)
rename
runners/{google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillStubFactoryFactory.java
=> prism/java/src/main/java/org/apache/beam/runners/prism/StateListener.java}
(69%)
create mode 100644
runners/prism/java/src/main/java/org/apache/beam/runners/prism/StateWatcher.java
create mode 100644
runners/prism/java/src/main/java/org/apache/beam/runners/prism/WorkerService.java
create mode 100644
runners/prism/java/src/test/java/org/apache/beam/runners/prism/PrismArtifactResolverTest.java
create mode 100644
runners/prism/java/src/test/java/org/apache/beam/runners/prism/PrismArtifactStagerTest.java
create mode 100644
runners/prism/java/src/test/java/org/apache/beam/runners/prism/PrismJobManagerTest.java
delete mode 100644
runners/prism/java/src/test/java/org/apache/beam/runners/prism/PrismRegistrarTest.java
create mode 100644
runners/prism/java/src/test/java/org/apache/beam/runners/prism/StateWatcherTest.java
create mode 100644
runners/prism/java/src/test/java/org/apache/beam/runners/prism/WorkerServiceTest.java
delete mode 100644
runners/spark/src/test/java/org/apache/beam/runners/spark/translation/PassThrough.java
delete mode 100644
runners/spark/src/test/java/org/apache/beam/runners/spark/translation/RDDNode.java
delete mode 100644
runners/spark/src/test/java/org/apache/beam/runners/spark/translation/RDDTreeParser.java
delete mode 100644
sdks/java/core/src/main/java/org/apache/beam/sdk/util/RowFilter.java
delete mode 100644
sdks/java/core/src/main/java/org/apache/beam/sdk/util/RowStringInterpolator.java
delete mode 100644
sdks/java/core/src/test/java/org/apache/beam/sdk/util/RowFilterTest.java
delete mode 100644
sdks/java/core/src/test/java/org/apache/beam/sdk/util/RowStringInterpolatorTest.java
delete mode 100644
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/PortableIcebergDestinations.java
delete mode 100644
sdks/python/apache_beam/examples/inference/vllm_text_completion.py
delete mode 100644 sdks/python/apache_beam/io/gcp/gcsio_retry.py
delete mode 100644 sdks/python/apache_beam/io/gcp/gcsio_retry_test.py
delete mode 100644
sdks/python/apache_beam/ml/inference/test_resources/vllm.dockerfile
delete mode 100644 sdks/python/apache_beam/ml/inference/vllm_inference.py
delete mode 100644
sdks/python/apache_beam/runners/portability/fn_api_runner/worker_handlers_test.py
delete mode 100644 website/www/site/content/en/case-studies/schrodinger.md
delete mode 100644 website/www/site/content/en/documentation/runners/prism.md
delete mode 100644 website/www/site/content/en/roadmap/prism-runner.md
delete mode 100644
website/www/site/static/images/logos/powered-by/schrodinger.png
