This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a change to branch
dependabot/pip/sdks/python/container/py38/tqdm-4.66.3
in repository https://gitbox.apache.org/repos/asf/beam.git
omit 1a28c7f9bcb Bump tqdm from 4.66.2 to 4.66.3 in
/sdks/python/container/py38
add e64c3596253 Avoid json Unmarshal on proto CancelJobRequest (#31178)
add c531f898d6b Update hack to make Spark runner happy as well. (#31184)
add 58df36adda3 Update CHANGES.md (#31189)
add ea3c2a4dbbe Add guidance on fixing cloud build triggers (#31187)
add a22ee682fa8 Bump jinja2 from 2.11.3 to 3.1.4 in
/.test-infra/jenkins/metrics_report (#31191)
add 26c4ee435cb Suppress BigQuery read stream splitAtFraction when API
busy call or timeout (#31125)
add 070fc903571 Bump google.golang.org/grpc from 1.62.1 to 1.63.2 in /sdks
(#30900)
add 45fe4f96f22 [#30789] Add support for Flink 1.18 (#31062)
add bb51380f1b2 Add support for unaligned checkpoints for flink runner
(#31186)
add 2ca9af8ecbc Upgrade Pandas dependency to 2.1 (#31185)
add f84f191128e Move hardcoded version number as default version for
setup-environment-action (#31199)
add 14358f2f5eb Update website runner sections (#31205)
add 463f6871d51 Fix version issue (#31211)
add 30a1eb0e102 Updates the error message related to schema-transform
discovery
add e2e0ba67e27 Merge pull request #31208: Updates the error message
related to schema-transform discovery
add ebb1465942e Add jinja2 as a dependency for yaml preprocessing. (#31164)
add c0c8d7852a2 Restore pandas version constraint. (#31213)
add 815639b89ac Stop using KV as the Key for Histogram Metrics (#31166)
add 2ad45739acd Targeted dependency update for Pandas 2.1 (#31209)
add 0b36f8e23fc Use already-existing and more appropriately alarmist
experiment in Python
add 8328d073ab8 Merge pull request #31212: Use already-existing and more
appropriately alarmist experiment in Python
add 20409b5204d Fix python_test broke by #31199 (#31221)
add faaa68c1a1c Fix lint warning: JobId vs JobID (#31197)
add ace86e2d518 [YAML] Add the ability to pre-process yaml files with
jinja2.
add 95ac1306a56 Allow including from arbitrary Beam filesystems.
add 6bcf02b9218 Add tests for yaml main.
add 13c72e82abe Unconditionally invoke jinja and update release notes.
add 2e662d54347 Make test infra happy.
add 53e85aeb186 Use Sphinx-compatible jinja2.
add 94a6bbb40a8 Remove flags_as_jinja_variables.
add 2dcb12f4f3f Actually fix RowCoder tests.
add 5e058620c95 Also record template and jinja variables in display data.
add 037704b6f7d Merge branch 'master' into yaml-jinja
add 6197657c9c3 Merge pull request #30976 [YAML] Add the ability to
pre-process yaml files with jinja2.
add 287ed388a68 Fix trivial inference tests for Python 3.12 support
(#31170)
add 7ed94383d2e Don't hardcode python version in tox.ini for lint, mypy
and docs. (#31222)
add 057eb06514a Add a container for Python 3.12. (#31216)
add 736f2b963dd Turn off abandoned node enforcement in some Reshuffle unit
tests where it is not needed
add 13708eaedeb Merge pull request #31229: Turn off abandoned node
enforcement in some Reshuffle unit tests where it is not needed
add e0bc8e770a7 Add IcebergIO integration tests (#31220)
add 365c2d92965 Support Kafka Managed IO (#31172)
add 5ff55b1caf1 [flink] Add documentation for Flink runner versions updates
add 0afb52d94c5 Merge pull request #31217: [flink] Add documentation for
Flink runner versions updates
add cc1024cf164 Fix #31212 by removing the extra space in option string.
(#31233)
add 2d53926542f Avoid oversizing batch sizes with size estimation function
(#31228)
add 602f088b38e Add new BigQueryIO stress tests for AT_LEAST_ONCE writing
method (#31241)
add 929f41aea34 Publish snapshots for Python 3.12 container images.
(#31237)
add 5ee63c0e04e Exclude a test that doesn't pass due to an upstream issue.
(#31240)
add 0a58d7d2ff4 Remove a deprecated test helper. (#31236)
add 8b5d0009a1d Fix Python 3.12 python hinting errors on MacOS (#31246)
add fbb3ae1ebef add retries for flaky test (#31247)
add 45c730ad0c7 Update version tag for dataflow dev containers. (#31248)
add 2e5f69c4bb9 Update try-apache-beam-yaml.ipynb (#31257)
add 66da760357c Update yaml to non-beta link (#31210)
add a0c08272d42 Touch trigger files for all relevant ValidatesRunner and
PVR suites
add bf347fe898b Sickbay extremely flaky Flink ParDoLifecycleTest
add ef431ec90f2 Reapply "Add Redistribute transform to Java SDK"
add 737be230347 Merge pull request #31156: Reapply "Add Redistribute
transform to Java SDK"
add 466516b016c [Python] Use BufferedLogger instead of logger to log in
boot.go (#31265)
add 749c128bef1 Disclaim direct runner result can be different with remote
runners (#31250)
add c230655a1e5 Implement the hash_words TFT operation (#31249)
add 296c13a291d Avoid side input issue where Latest.globally produces 2
elements in a window (#31263)
add c88a863870c Add a subsuite of unit tests for ML package and remove
them from 'coverage' GHA workflow. (#31255)
add 31e81ffe12a add windmill computation state cache, config loaders, and
status pages (#31133)
add 459c87af706 Revert "[flink] Add documentation for Flink runner
versions updates"
add 557b5bad1b0 Revert "[#30789] Add support for Flink 1.18 (#31062)"
add 0542c83fc1a Un-revert doc fixes
add bcac88b815e Merge pull request #31274 from apache/debugpostrelease
add c7937001981 Beam YAML day zero notebook save file fix (#31266)
add c1d1bd642d1 Fix a typo in skip condition. (#31267)
add 4a15af019cc Touch trigger files for Dataflow ValidatesRunner tests
add 238792d7c65 Reapply "Add direct Dataflow translation of Redistribute
transform"
add 99714991b00 Merge pull request #31268: Reapply "Add direct Dataflow
translation of Redistribute transform"
add 8869175a887 Fix nullness errors in TestPipeline
add 7f9264ad91d Merge pull request #31230: Fix nullness errors in
TestPipeline
add ee170e142b9 Set backlog in gauge metric (#31137)
add 5c5bcccf516 Copy edit the code contribution guide (#31279)
add c061b2776b0 Revert "Don't specify an unnecessary config given that tox
task and gradle config already defines the Python version." (#31283)
add 785855e6c4c Exclude testRedistributePreservesMetadata from Spark PVR
batch (#31286)
add 54db453b032 Fix postrelease revert leftover (#31292)
add 5a42d54bfd1 Catch org.elasticsearch.client.ResponseException and
return document with withHasError(true)
add 29b47e69716 Remove status
add 84b1d92a73e Fix Style
add ff858b8b966 doc -> doc.withHasError(true)
add 3cbf7de5796 Add tests
add 1647872e054 Fix styles via spotlessApply
add 41772b015c2 rename INJECT_SOME_INVALID_DOCS and insert numDoc
add 00c59b251ec Catch with java.io.IOExecption
add 591d2c75679 explicit import
add 0f6a79cd2eb Removed unused import
add a7380297011 java util explicit import
add 035c88da6cf spotlessapply
add c6eabd19faf Fix max batch size
add 77da5add258 Rename message from
org.elasticsearch.client.ResponseException to java.io.IOException
add 1ee2f0ec5e6 Add test to verify response items message
add e010a3ecc66 spotlessapply
add 5ee569f930b fix comment
add ed84ba0e0dc Fix naming
add 1a399b96977 Verify the whole error message string
add e97064aa14c Remove mapToInputDoc
add 946f4959b7b retry 500-range http except 501
add 68790eb242f Add elasticResponseExceptionMessage predicate
add 3d897b27635 Merge pull request #31151 from andyzhangdialpad/master
add 6de9a60dc9e Skip tests using TFT in Beam ML unit test suite on Python
3.11+ (#31288)
add 041a8dd1173 Touch Samza ValidatesRunner trigger files
add f58a9d6680e Reapply "Add Redistribute translation to Samza runner"
add 2196758c20b Merge pull request #31270: Reapply "Add Redistribute
translation to Samza runner"
add 6cb30cc5c86 setup GCP auth before running tests (#31306)
add 118c51404d9 Implement DeduplicateTensorPerRow in MLTransform (#31307)
add 084f23b2a8d Install Beam from wheels in Dependency Compat Test Suite.
(#31308)
add 743e34e0098 Fix iceberg unit tests (#31314)
add 126d92249f9 Support newer versions of Pyarrow in Beam. (#31305)
add 840faea2262 Create option to specify temp query project, and wire into
source tab… (#31128)
add 999f7deb94e [Python] Add job name to GCS custom audit info (#31300)
add 2babd0aee5b Merge pull request #31120: Update Guava version to
33.1.0-jre to match the GCP Libraries BOM 26.38.0
add 93a5bc76577 [Python] Use retry decorator for retrying (#31273)
add 28a7199e80b [Java] Add job name to GCS custom audit info (#31316)
add 4f0cffff5fb Done with python scripts/tools/bomupgrader.py 26.39.0
based on
https://github.com/apache/beam/blob/master/contributor-docs/java-dependency-upgrades.md
(#31318)
add d15d98c0026 Update code-change-guide.md (#31333)
add d7165f940cc Upgrade cibuildwheel version. (#31321)
add 44177d1a9f3 [Python] BigQuery handler for enrichment transform (#31295)
add a435f452211 [Java] Disable soft delete policy when creating a default
bucket for a project. (#31324)
add f851cbb317e Update hamcrest dependency documentation version (#31309)
add 71f2278c79f Update
beam_Inference_Python_Benchmarks_Dataflow_Pytorch_Imagenet_Classification_Resnet_152_Tesla_T4_GPU.txt
(#31341)
add 092f769a4b4 add core transforms category to approximatequantiles.py
and approximateunique.py to pass playground validation (#31331)
add 2e4a152cecd start the connection before calling receive in JmsIOTest
(#31334)
add fed64891240 Implement CoderNullable pullDecoderNoAlloc case (#31340)
add 89795c0e7c7 Make GrpcCommitWorkStream thread-safe as documented by
moving batcher out of it. (#31304)
add 88af35e6df4 PubsubIO: Add
readMessagesWithAttributesWithCoderAndParseFn (#31206)
add 0e734fe1c38 [Go SDK] Make it clearer that timers and data don't
interact negatively in element batches. (#31319)
add f4119f564fb Upgrade jetty to latest 9.x for dataflow legacy worker
(#31351)
add 675dab26ec0 Enable BigQueryIO write throttling detection (#31253)
add 1a52285109c Logging loading filesystems failures. (#31219)
add f508b033c2b Bump com.gradle.enterprise from 3.17.2 to 3.17.4 (#31345)
add cfe206bb352 skip requests version 2.32.0,2.32.1 (#31359)
add c5b6475db2a [Python] Disable soft delete policy when creating new
default bucket. (#31344)
add a998107a1f5 [Java] Add warning to gcpTempLocation when its bucket has
soft delete enabled (#31358)
add 98b5d5b93d3 Upgrade cibuildwheel in another place as well. (#31337)
add 611676d108b Fix DebeziumIO testWrongHost trying to connect real
external address (#31365)
add 649e33c214d Don't set SETUPTOOLS_USE_DISTUTILS=stdlib (#31370)
add 33fccac00e0 Updates the Java Expansion Service container to support
gRPC ALTS authentication (#31352)
add bedba65a1f7 Update README.md (fix typo) (#31380)
add 2ee266bec03 Fixed the extension name in Java SDK Ordered Processing
extension (#31382)
add 5c8338a38c1 Reapply "[#30789] Add support for Flink 1.18 (#31062)"
add a37df32264b [runners-flink] use mvn exec:exec for
runQuickstartJavaFlinkLocal
add 736ed4a0a31 Merge pull request #31328: Reapply "[#30789] Add support
for Flink 1.18 (#31062)"
add 81d58d83d87 Fix beam quest link
add ec9e963735f Merge pull request #31383 from apache/users/damccorm/bql
add 1f6319624a4 move playground ci nightly to highmem due to mem exaustion
(#31384)
add 0e075b69f07 Define Python 3.12 test environments in tox.
add a0e950b454e Add a tox suite for Py312.
add 7e102700b52 Gradle definitions for Python 3.12
add ddb4e7f29ff Switch Py3.11 Gradle suites to use Python 3.12.
add dac22e3d074 Switch remaining GHA workflows to test Python 3.12.
add d12c628a388 Explicitly install now-optional setuptools package as some
tests depend on it.
add 6486591594a Build Python 3.12 wheels.
add 42c84822d42 Support Python 3.12 in Dataflow.
add ab460c1f5ba Support Python 3.12 in Beam.
add dc6b72caf73 Merge pull request #30828 - Add Python 3.12 support
add 6876734da12 Don't install TF on Python 3.12.
add 697c18fd6a5 Update setup.py
add 1ed29e1f94a Merge pull request #31386 from tvalentyn/py312_tf
add ca002724341 Update beam-master version to 20240524. (#31393)
add 3b9b6ed0038 added pytest.mark.flaky for
test_big_query_write_temp_table_append_sc… (#31364)
add cf3592bbfa0 Fix an incompatibility with hamcrest 2.2 (#31395)
add c10f5d40456 [runners-flink] Drop Flnk 1.14 and clean up code (#31112)
add 0790d698251 Merge pull request #31394: #31112 drop flink 1.14
add 5a822adc269 [flink] #31390 emit watermark with empty source
add 836e77e755b Merge pull request #31391: [flink] #31390 emit watermark
with empty source
add 5c30b1d6db2 Clean up stale code in BigtableService (#30172)
add 2719ca7f37a Changed the retry order for
test_big_query_write_temp_table_append_schema_update (#31407)
add e764fc9c17d remove unused field (#31293)
add 37b8c8a87b8 Add Iceberg workflows (#31401)
add 1f714760219 Add option to disable validation of cloud bigtable change
stream IO
add 10929b9ee7d Fix ordering of the validations
add 13a3e46f87f Add unit test for ReadChangeStream config and validation
add aa831b47330 Add additionl comment clarifying the effect of
withoutValidation
add 944273fe341 Merge pull request #31376 from
tonytanger/validate_cbt_change_stream
add bb4c1e6ded5 Include Python 3.12 in release validation steps. (#31415)
add fd4368f1c4a Add Backlog Metrics to Kafka Splittable DoFn
Implementation (#31281)
add e488f41b9bd Memoize some dataframes analysis operations. (#31377)
add 305e75359ad Better mixed Java/Python error messages for external
transforms. (#31284)
add 93cc6a521ee add PR trigger files (#31424)
add 791ead6a05b add pull_request_target event (#31426)
add 28443f803c8 Remove bad dialect option. (#31429)
add df067dba5dc pin the lower bound for typing-extensions (#31425)
add f7519774e3c BigQueryIO read throttling detection python (#31404)
add 7d281558dd8 [#29697] Add prism artifact building workflow. (#31369)
add 4daedbf5a8a [#30083] Add synthetic processing time to prism. (#30492)
add 49a4290426d Add options to specify read and write http timeout for gcs
as well as lower batching limit for rewrite operations which are copying.
(#31410)
add b1a6eb06051 [YAML] Fix simple YAML mappings type hinting (#31427)
add 6842136e0c9 Add SDK capability to detect if the SDK Fn Harness data
channel is busy.
add ad841c6004f Regenerate Go protos.
add 8b33e1f65c3 Merge pull request #31442 SDK protocol to detect if the
SDK Fn Harness data channel is busy
add df8bead5945 Refactor RowMutationInformation to use string type (#31323)
add 06e103d87e8 Add ApplyBucketsWithInterpolation TFTransform (#31291)
add 8d77c8fad07 Add try-excepts around data sampler encoding (#31396)
add 0b5ffd7d153 Add SDK capability to detect if the SDK Fn Harness data
channel is busy or not (#31420)
add 19630e576fe Add in-memory variants of side inputs. (#31232)
add 80d85aa38ff Add docs for YAML AssertThat. (#31448)
add 90f020921c1 Update bigquery_tools.py (#31444)
add 16d62827551 Update bigquery.py documentation (#31443)
add 74f014ee1b9 SpannerIO: support max commit delay
add 4a0849b15ee Merge pull request #31008: SpannerIO: support max commit
delay
add 9f3f1c93a60 Don't re-encode byte[] values in SortValues transform
(#31025)
add 3cadc83d348 flink portable client configurations (#31188)
add c22678b4efb Emit a warning when large elements are detected. (#31363)
add c90fd8320a5 Use bytes instead of mibs. (#31457)
add 5454489e29b Report prism changes. (#31433)
add b50ad0fe8fc [ManagedIO] pass underlying transform URN as an annotation
(#31398)
add 6911a1af720 Updating Commons Compress to 1.26.2
add f410b71c076 Merge pull request #31464: Updating Commons Compress to
1.26.2
add daf81143388 Merge pull request #31449 Pass through docs (and configs)
for SqlProviders.
add 19d57d01266 Change configurations for stress tests. Fix BigQueryIOST
(#31439)
add 7b6f9415c10 Updates Expansion Service Container to support upgrading
using the schema-transform ID (#31451)
add f93a67a6ae8 remove processing/scheduling logic from
StreamingDataflowWorker (#31317)
add 51525d34cfa Set retry policy to DEFAULT_RETRY when initializing
BlobWriter. (#31468)
add fa4ead99a6b Bump Jackson databind to 2.15 (#31473)
add 349d7b9f9c6 add support for ConnectionFactory ProviderFn in JmsIO
(#31264)
add fc003513805 Update JdbcIO.java documentation (#31454)
add 85650449827 Remove excess space from Programming Guide (#31469)
add bb4ad994304 Fix one small typo in build script. (#31413)
add fbe9427b0a9 Add Trigger_Never to Prepare features check (#31472)
add 44550eea721 Add test of Select type inference.
add 5914f272e01 yapf
add 64d9794f444 Merge pull request #31428 Add test of Select type
inference.
add f54fda9df73 Add a test for sql dialect.
add 59b0fdc9a3c Merge branch 'master' into yaml-sql-dialect
add 6dd32a0aa65 Merge pull request #31276 Add a test for sql dialect.
add 9cce412e302 Ensure that BigtableIO closes the stream after reading
(#31478)
add 4ad7037b55f Parse YAML ExpansionService configs directly using
SnakeYAML (#31406)
add b680fd58ffe Fix mangled license headers.
add f98b6617b1b Merge pull request #31480 Fix mangled license headers.
add bd17b74f442 Remove setup gcp credential step for beam_ workflows
running on self-hosted runner (#31483)
add bd0ba38ef98 Pins SnakeYAML version to 1.33 for Hadoop Format tests
(#31485)
add a5738c1cdd3 Extract latest beam version directly from the request
(#31487)
add e31e8855ad9 [#27839][Go SDK] Write pipeline options to a file, instead
reading from a flag. (#31482)
add 7b6554bb6e6 [YAML] mark Combine transform as stable
add 8fb9dc8c210 Merge pull request #31446 [YAML] mark Combine transform as
stable
add a7f5898f885 Default SchemaTransform configs to snake_case (#31374)
add b4b1cc05b5d Fix iceberg catalog validation (#31349)
add 1fea5f797e9 Support unknown repeated STRUCTs (#31447)
add 2bb80124472 Delete unused class
add cf01cba3aa0 Merge pull request #31502: Delete
RedistributeByKeyAllowingDuplicates from Redistribute
add 195dc3f0146 fix typescript syntax (#31493)
add 7ea8cd2608e Simplify Managed API to avoid dealing with
PCollectionRowTuple (#31470)
add 217ac6de253 Moving to 2.58.0-SNAPSHOT on master branch.
add 7f51d6a9bdf Add base _ImageEmbeddingHandler (#31501)
add bd00a69402f update the transform label error message (#31508)
add 5c6d23c5ede Update CHANGES.md: clean up 2.57.0 section
add 2a63d4b436e Update CHANGES.md: add 2.58.0 section
add f67f95c1553 Update CHANGES.md: clean up 2.57.0 section
add 9c9de4919b5 Update Go container build version to 1.21.11 (#31515)
add f2931d31246 Kafka SchemaTransform translation (#31362)
add 4098fce785c fix managed doc (#31517)
add 317b7c67760 Fix issue where manual subsequent label designations were
ignored.
add f98a42a3205 Merge pull request #31522 Fix issue where manual
subsequent label designations were ignored.
add 9cd00549dd8 Do not suppress downstream Exception in BigQueryIO Storage
write API FailedStorageApiInserts Handling (#31506)
add c37107a6ea6 filter out test stream using test from spark suite (#31509)
add 8c6e1a4654b Add missing import (#31533)
add 3cdbd3a908a Implement Vertex AI Image Embedding MLTransform (#31514)
add 03d553e1f9c retry test_big_query_legacy_sql (#31417)
add e5c2f6c5980 Skip newly added TestStream VR tests for flink runner
(#31538)
add 9a921d51054 Deduplicate Base Embedding Handler Code (#31534)
add b69c06991f0 Bump tqdm from 4.66.2 to 4.66.3 in
/sdks/python/container/py38
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (1a28c7f9bcb)
\
N -- N -- N
refs/heads/dependabot/pip/sdks/python/container/py38/tqdm-4.66.3 (b69c06991f0)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.asf.yaml | 1 +
.../test-properties.json | 6 +-
.../actions/setup-environment-action/action.yml | 6 +-
...owV1.json => IO_Iceberg_Integration_Tests.json} | 0
..._Gcp_Direct.json => IO_Iceberg_Unit_Tests.json} | 0
...aflow.json => beam_PostCommit_Go_VR_Flink.json} | 0
...n => beam_PostCommit_Java_Hadoop_Versions.json} | 2 +-
.../beam_PostCommit_Java_PVR_Flink_Batch.json | 4 +
.../beam_PostCommit_Java_PVR_Flink_Docker.json | 4 +
.../beam_PostCommit_Java_PVR_Flink_Streaming.json | 4 +
.../beam_PostCommit_Java_PVR_Samza.json | 5 +
.../beam_PostCommit_Java_PVR_Spark3_Streaming.json | 3 +-
.../beam_PostCommit_Java_PVR_Spark_Batch.json | 3 +-
...m_PostCommit_Java_ValidatesRunner_Dataflow.json | 4 +-
...Java_ValidatesRunner_Dataflow_JavaVersions.json | 4 +
...it_Java_ValidatesRunner_Dataflow_Streaming.json | 4 +-
...ostCommit_Java_ValidatesRunner_Dataflow_V2.json | 4 +-
...Java_ValidatesRunner_Dataflow_V2_Streaming.json | 4 +-
...eam_PostCommit_Java_ValidatesRunner_Direct.json | 4 +
...t_Java_ValidatesRunner_Direct_JavaVersions.json | 4 +
...beam_PostCommit_Java_ValidatesRunner_Flink.json | 3 +-
...stCommit_Java_ValidatesRunner_Flink_Java11.json | 3 +-
...beam_PostCommit_Java_ValidatesRunner_Samza.json | 4 +-
...beam_PostCommit_Java_ValidatesRunner_Spark.json | 3 +-
...a_ValidatesRunner_SparkStructuredStreaming.json | 3 +-
...stCommit_Java_ValidatesRunner_Spark_Java11.json | 4 +
...m_PostCommit_Java_ValidatesRunner_Twister2.json | 3 +-
.../beam_PostCommit_Java_ValidatesRunner_ULR.json | 3 +-
.github/trigger_files/beam_PostCommit_Python.json | 1 +
...json => beam_PostCommit_Python_Dependency.json} | 0
.../beam_PostCommit_Python_Xlang_Gcp_Direct.json | 2 +-
.../trigger_files/beam_PostCommit_XVR_Flink.json | 3 -
...oPrism.yml => IO_Iceberg_Integration_Tests.yml} | 73 +-
...it_RAT.yml => IO_Iceberg_Performance_Tests.yml} | 52 +-
.../{IO_Iceberg.yml => IO_Iceberg_Unit_Tests.yml} | 20 +-
.github/workflows/README.md | 10 +
.../beam_Inference_Python_Benchmarks_Dataflow.yml | 2 +-
.../beam_LoadTests_Python_CoGBK_Dataflow_Batch.yml | 2 +-
...m_LoadTests_Python_CoGBK_Dataflow_Streaming.yml | 2 +-
.../beam_LoadTests_Python_CoGBK_Flink_Batch.yml | 2 +-
...eam_LoadTests_Python_Combine_Dataflow_Batch.yml | 2 +-
...LoadTests_Python_Combine_Dataflow_Streaming.yml | 2 +-
.../beam_LoadTests_Python_Combine_Flink_Batch.yml | 2 +-
...am_LoadTests_Python_Combine_Flink_Streaming.yml | 2 +-
...LoadTests_Python_FnApiRunner_Microbenchmark.yml | 2 +-
.../beam_LoadTests_Python_GBK_Dataflow_Batch.yml | 2 +-
...eam_LoadTests_Python_GBK_Dataflow_Streaming.yml | 2 +-
.../beam_LoadTests_Python_GBK_Flink_Batch.yml | 2 +-
...adTests_Python_GBK_reiterate_Dataflow_Batch.yml | 2 +-
...sts_Python_GBK_reiterate_Dataflow_Streaming.yml | 2 +-
.../beam_LoadTests_Python_ParDo_Dataflow_Batch.yml | 2 +-
...m_LoadTests_Python_ParDo_Dataflow_Streaming.yml | 2 +-
.../beam_LoadTests_Python_ParDo_Flink_Batch.yml | 2 +-
...beam_LoadTests_Python_ParDo_Flink_Streaming.yml | 2 +-
...m_LoadTests_Python_SideInput_Dataflow_Batch.yml | 2 +-
.github/workflows/beam_LoadTests_Python_Smoke.yml | 2 +-
.github/workflows/beam_Metrics_Report.yml | 2 +-
...eam_PerformanceTests_BiqQueryIO_Read_Python.yml | 2 +-
...formanceTests_BiqQueryIO_Write_Python_Batch.yml | 2 +-
...erformanceTests_PubsubIOIT_Python_Streaming.yml | 2 +-
...m_PerformanceTests_SQLBigQueryIO_Batch_Java.yml | 2 +-
..._PerformanceTests_SpannerIO_Read_2GB_Python.yml | 2 +-
...manceTests_SpannerIO_Write_2GB_Python_Batch.yml | 2 +-
.../beam_PerformanceTests_TextIOIT_Python.yml | 2 +-
.../beam_PerformanceTests_xlang_KafkaIO_Python.yml | 2 +-
.github/workflows/beam_Playground_CI_Nightly.yml | 6 +-
.github/workflows/beam_PostCommit_Go.yml | 7 -
.../workflows/beam_PostCommit_Go_Dataflow_ARM.yml | 11 +-
.../beam_PostCommit_Java_BigQueryEarlyRollout.yml | 7 -
.../beam_PostCommit_Java_Examples_Dataflow_ARM.yml | 8 -
...m_PostCommit_Java_Examples_Dataflow_V2_Java.yml | 1 -
.../beam_PostCommit_Java_Examples_Direct.yml | 2 +-
.../beam_PostCommit_Java_Examples_Flink.yml | 4 +-
.../beam_PostCommit_Java_Examples_Spark.yml | 2 +-
.../beam_PostCommit_Java_Hadoop_Versions.yml | 2 +-
.../beam_PostCommit_Java_IO_Performance_Tests.yml | 2 +-
.../workflows/beam_PostCommit_Java_Tpcds_Flink.yml | 2 +-
...am_PostCommit_Java_ValidatesRunner_Dataflow.yml | 2 +-
...mit_Java_ValidatesRunner_Dataflow_Streaming.yml | 2 +-
...PostCommit_Java_ValidatesRunner_Dataflow_V2.yml | 2 +-
..._Java_ValidatesRunner_Dataflow_V2_Streaming.yml | 2 +-
...beam_PostCommit_Java_ValidatesRunner_Direct.yml | 2 +-
.../beam_PostCommit_Java_ValidatesRunner_Flink.yml | 6 +-
.../beam_PostCommit_Java_ValidatesRunner_Samza.yml | 2 +-
.../beam_PostCommit_Java_ValidatesRunner_Spark.yml | 2 +-
...va_ValidatesRunner_SparkStructuredStreaming.yml | 2 +-
...am_PostCommit_Java_ValidatesRunner_Twister2.yml | 2 +-
.../beam_PostCommit_Java_ValidatesRunner_ULR.yml | 4 +-
.../beam_PostCommit_PortableJar_Flink.yml | 2 +-
.../beam_PostCommit_PortableJar_Spark.yml | 2 +-
.github/workflows/beam_PostCommit_Python.yml | 2 +-
.github/workflows/beam_PostCommit_Python_Arm.yml | 2 +-
.../beam_PostCommit_Python_Dependency.yml | 5 +-
.../beam_PostCommit_Python_Examples_Dataflow.yml | 6 +-
.../beam_PostCommit_Python_Examples_Direct.yml | 4 +-
.../beam_PostCommit_Python_Examples_Flink.yml | 4 +-
.../beam_PostCommit_Python_Examples_Spark.yml | 4 +-
.../beam_PostCommit_Python_MongoDBIO_IT.yml | 8 +-
.../beam_PostCommit_Python_Nexmark_Direct.yml | 2 +-
...stCommit_Python_ValidatesContainer_Dataflow.yml | 2 +-
..._Python_ValidatesContainer_Dataflow_With_RC.yml | 2 +-
..._PostCommit_Python_ValidatesRunner_Dataflow.yml | 4 +-
...eam_PostCommit_Python_ValidatesRunner_Flink.yml | 4 +-
...eam_PostCommit_Python_ValidatesRunner_Samza.yml | 4 +-
...eam_PostCommit_Python_ValidatesRunner_Spark.yml | 4 +-
.../beam_PostCommit_Python_Xlang_Gcp_Dataflow.yml | 2 +-
.../beam_PostCommit_Python_Xlang_Gcp_Direct.yml | 2 +-
.../beam_PostCommit_Python_Xlang_IO_Dataflow.yml | 2 +-
.../workflows/beam_PostCommit_Sickbay_Python.yml | 2 +-
.../beam_PostCommit_TransformService_Direct.yml | 4 +-
.github/workflows/beam_PostCommit_XVR_Direct.yml | 4 +-
.github/workflows/beam_PostCommit_XVR_Flink.yml | 4 +-
.../beam_PostCommit_XVR_GoUsingJava_Dataflow.yml | 2 +-
...eam_PostCommit_XVR_JavaUsingPython_Dataflow.yml | 4 +-
..._PostCommit_XVR_PythonUsingJavaSQL_Dataflow.yml | 4 +-
...eam_PostCommit_XVR_PythonUsingJava_Dataflow.yml | 4 +-
.github/workflows/beam_PostCommit_XVR_Samza.yml | 4 +-
.github/workflows/beam_PostCommit_XVR_Spark3.yml | 4 +-
.../workflows/beam_PostRelease_NightlySnapshot.yml | 2 +-
.../workflows/beam_PreCommit_CommunityMetrics.yml | 10 +-
.github/workflows/beam_PreCommit_GHA.yml | 4 +-
.github/workflows/beam_PreCommit_Go.yml | 4 +-
.github/workflows/beam_PreCommit_GoPortable.yml | 4 +-
.github/workflows/beam_PreCommit_GoPrism.yml | 4 +-
.github/workflows/beam_PreCommit_ItFramework.yml | 2 +-
.../beam_PreCommit_Java_Examples_Dataflow.yml | 9 +-
.../beam_PreCommit_Java_Flink_Versions.yml | 4 +-
.../beam_PreCommit_Java_Spark3_Versions.yml | 9 +-
.../workflows/beam_PreCommit_Kotlin_Examples.yml | 2 +-
.../workflows/beam_PreCommit_Portable_Python.yml | 4 +-
.github/workflows/beam_PreCommit_Python.yml | 6 +-
.github/workflows/beam_PreCommit_PythonDocker.yml | 8 +-
.github/workflows/beam_PreCommit_PythonDocs.yml | 4 +-
.../workflows/beam_PreCommit_PythonFormatter.yml | 4 +-
.github/workflows/beam_PreCommit_PythonLint.yml | 6 +-
.../workflows/beam_PreCommit_Python_Coverage.yml | 4 +-
.../workflows/beam_PreCommit_Python_Dataframes.yml | 4 +-
.../workflows/beam_PreCommit_Python_Examples.yml | 4 +-
.../beam_PreCommit_Python_Integration.yml | 4 +-
...on_Runners.yml => beam_PreCommit_Python_ML.yml} | 22 +-
.../workflows/beam_PreCommit_Python_PVR_Flink.yml | 10 +-
.../workflows/beam_PreCommit_Python_Runners.yml | 4 +-
.../workflows/beam_PreCommit_Python_Transforms.yml | 4 +-
.github/workflows/beam_PreCommit_RAT.yml | 2 +-
.github/workflows/beam_PreCommit_SQL_Java11.yml | 4 +-
.github/workflows/beam_PreCommit_SQL_Java17.yml | 2 +-
.github/workflows/beam_PreCommit_Typescript.yml | 4 +-
.github/workflows/beam_PreCommit_Website.yml | 2 +-
.../workflows/beam_PreCommit_Website_Stage_GCS.yml | 11 +-
.github/workflows/beam_PreCommit_Whitespace.yml | 4 +-
.../beam_PreCommit_Xlang_Generated_Transforms.yml | 2 +-
.../workflows/beam_PreCommit_Yaml_Xlang_Direct.yml | 3 +-
.github/workflows/beam_Publish_BeamMetrics.yml | 8 -
.../workflows/beam_Publish_Beam_SDK_Snapshots.yml | 10 +-
.../workflows/beam_Publish_Docker_Snapshots.yml | 7 -
...beam_Python_ValidatesContainer_Dataflow_ARM.yml | 11 +-
.github/workflows/beam_Release_NightlySnapshot.yml | 2 +-
.../beam_Release_Python_NightlySnapshot.yml | 11 +-
.github/workflows/build_release_candidate.yml | 190 +-
.github/workflows/build_wheels.yml | 6 +-
.github/workflows/java_tests.yml | 6 +-
...enet_Classification_Resnet_152_Tesla_T4_GPU.txt | 4 +-
.github/workflows/python_dependency_tests.yml | 1 +
.github/workflows/python_tests.yml | 26 +-
.github/workflows/update_python_dependencies.yml | 13 +-
.test-infra/jenkins/CommonJobProperties.groovy | 3 -
.../jenkins/metrics_report/requirements.txt | 2 +-
CHANGES.md | 54 +-
build.gradle.kts | 19 +-
.../org/apache/beam/gradle/BeamModulePlugin.groovy | 28 +-
contributor-docs/code-change-guide.md | 441 +--
contributor-docs/release-guide.md | 6 +-
examples/java/build.gradle | 2 +-
.../apache/beam/examples/snippets/Snippets.java | 14 +-
examples/multi-language/README.md | 2 +-
.../get-started/try-apache-beam-yaml.ipynb | 24 +-
gradle.properties | 8 +-
.../apache/beam/it/gcp/bigquery/BigQueryIOST.java | 35 +-
.../apache/beam/it/gcp/bigtable/BigTableIOST.java | 2 +-
.../apache/beam/it/gcp/spanner/SpannerIOST.java | 2 +-
.../runner-concepts/description.md | 2 +-
.../beam/model/fn_execution/v1/beam_fn_api.proto | 5 +
.../beam/model/pipeline/v1/beam_runner_api.proto | 5 +
.../model/pipeline/v1/external_transforms.proto | 13 +
release/build.gradle.kts | 2 +-
release/src/main/Dockerfile | 7 +-
.../main/groovy/quickstart-java-flinklocal.groovy | 24 +-
.../python-release/python_release_automation.sh | 2 +-
.../src/main/scripts/build_release_candidate.sh | 2 +-
.../streaming/AbstractStreamOperatorCompat.java | 91 -
.../streaming/ProcessingTimeCallbackCompat.java | 22 -
.../beam/runners/flink/MiniClusterCompat.java | 29 -
.../runners/flink/metrics/MetricGroupWrapper.java | 31 -
.../translation/types/CoderTypeSerializer.java | 0
.../streaming/ProcessingTimeCallbackCompat.java | 22 -
.../beam/runners/flink/MiniClusterCompat.java | 30 -
runners/flink/{1.14 => 1.18}/build.gradle | 4 +-
.../job-server-container/build.gradle | 0
.../flink/{1.14 => 1.18}/job-server/build.gradle | 2 +-
runners/flink/flink_runner.gradle | 11 +-
runners/flink/job-server/flink_job_server.gradle | 2 +
.../runners/flink/FlinkExecutionEnvironments.java | 7 +
.../beam/runners/flink/FlinkPipelineOptions.java | 16 +
.../flink/FlinkPortableClientEntryPoint.java | 47 +-
.../flink/FlinkStreamingTransformTranslators.java | 8 +-
.../wrappers/streaming/DoFnOperator.java | 57 +-
.../streaming/io/UnboundedSourceWrapper.java | 8 +-
.../streaming/io/source/FlinkSourceReaderBase.java | 12 +-
.../io/source/FlinkSourceSplitEnumerator.java | 9 +-
.../unbounded/FlinkUnboundedSourceReader.java | 8 +-
.../beam/runners/flink/FlinkSavepointTest.java | 6 +-
.../beam/runners/flink/RemoteMiniClusterImpl.java | 0
.../flink/metrics/FlinkMetricContainerTest.java | 3 +-
.../runners/flink/streaming/StreamSources.java | 0
.../translation/types/CoderTypeSerializerTest.java | 0
.../streaming/io/UnboundedSourceWrapperTest.java | 4 +-
.../streaming/io/source/EmptyUnboundedSource.java | 4 +-
.../io/source/FlinkSourceReaderTestBase.java | 20 +-
.../streaming/io/source/SourceTestMetrics.java} | 2 +-
.../bounded/FlinkBoundedSourceReaderTest.java | 2 +-
.../unbounded/FlinkUnboundedSourceReaderTest.java | 56 +-
.../dataflow/DataflowPipelineTranslator.java | 37 +
.../beam/runners/dataflow/util/PropertyNames.java | 1 +
.../google-cloud-dataflow-java/worker/build.gradle | 6 +-
.../runners/dataflow/worker/LockFreeHistogram.java | 7 +-
.../beam/runners/dataflow/worker/PubsubReader.java | 2 +-
.../beam/runners/dataflow/worker/ReaderCache.java | 4 +-
.../dataflow/worker/StreamingDataflowWorker.java | 1365 ++------
.../worker/StreamingModeExecutionContext.java | 176 +-
.../worker/StreamingStepMetricsContainer.java | 19 +-
.../dataflow/worker/UngroupedWindmillReader.java | 2 +-
.../dataflow/worker/WindmillTimerInternals.java | 39 +-
.../dataflow/worker/WindowingWindmillReader.java | 2 +-
.../dataflow/worker/WorkerCustomSources.java | 2 +-
.../dataflow/worker/status/WorkerStatusPages.java | 4 +
.../dataflow/worker/streaming/ActiveWorkState.java | 77 +-
.../worker/streaming/ComputationState.java | 63 +-
.../worker/streaming/ComputationStateCache.java | 219 ++
.../worker/streaming/ComputationWorkExecutor.java | 118 +
.../dataflow/worker/streaming/ExecutableWork.java | 37 +-
.../dataflow/worker/streaming/ExecutionState.java | 54 -
.../dataflow/worker/streaming/Watermarks.java | 69 +
.../runners/dataflow/worker/streaming/Work.java | 289 +-
.../runners/dataflow/worker/streaming/WorkId.java | 8 +
.../worker/streaming/config/ComputationConfig.java | 59 +
...StreamingApplianceComputationConfigFetcher.java | 136 +
.../StreamingEngineComputationConfigFetcher.java | 283 ++
.../config/StreamingEnginePipelineConfig.java | 57 +
.../streaming/harness/MetricsDataProvider.java | 83 +
.../harness/StreamingWorkerStatusPages.java | 261 ++
.../streaming/sideinput/SideInputStateFetcher.java | 2 +
.../worker/util/common/worker/MapTaskExecutor.java | 2 +-
.../worker/util/common/worker/WorkExecutor.java | 2 +-
.../worker/windmill/WindmillServerBase.java | 2 +-
.../worker/windmill/WindmillServerStub.java | 3 +-
.../worker/windmill/client/WindmillStream.java | 39 +-
.../commits/StreamingApplianceWorkCommitter.java | 3 +-
.../commits/StreamingEngineWorkCommitter.java | 43 +-
.../windmill/client/grpc/GrpcCommitWorkStream.java | 72 +-
.../client/grpc/GrpcDirectGetWorkStream.java | 53 +-
.../windmill/client/grpc/GrpcDispatcherClient.java | 34 +-
.../windmill/client/grpc/GrpcGetDataStream.java | 2 +-
.../windmill/client/grpc/GrpcWindmillServer.java | 30 +-
.../client/grpc/GrpcWindmillStreamFactory.java | 11 +-
.../client/grpc/StreamingEngineClient.java | 161 +-
.../windmill/client/grpc/WindmillStreamSender.java | 37 +-
.../worker/windmill/state/WindmillStateReader.java | 32 +-
.../windmill/work/ProcessWorkItemClient.java | 52 -
...rkItemProcessor.java => WorkItemScheduler.java} | 29 +-
.../processing/ComputationWorkExecutorFactory.java | 291 ++
.../work/processing/StreamingCommitFinalizer.java | 85 +
.../work/processing/StreamingWorkScheduler.java | 428 +++
.../processing/failures/WorkFailureProcessor.java | 16 +-
.../dataflow/worker/FakeWindmillServer.java | 104 +-
.../dataflow/worker/LockFreeHistogramTest.java | 11 +-
...ricsToPerStepNamespaceMetricsConverterTest.java | 14 +-
.../runners/dataflow/worker/PubsubReaderTest.java | 2 +-
.../worker/StreamingDataflowWorkerTest.java | 579 ++--
.../worker/StreamingModeExecutionContextTest.java | 40 +-
.../dataflow/worker/WorkerCustomSourcesTest.java | 83 +-
.../worker/streaming/ActiveWorkStateTest.java | 213 +-
.../streaming/ComputationStateCacheTest.java | 289 ++
...amingApplianceComputationConfigFetcherTest.java | 142 +
...treamingEngineComputationConfigFetcherTest.java | 219 ++
.../StreamingApplianceWorkCommitterTest.java | 19 +-
.../commits/StreamingEngineWorkCommitterTest.java | 83 +-
.../client/grpc/GrpcWindmillServerTest.java | 121 +-
.../client/grpc/StreamingEngineClientTest.java | 37 +-
.../client/grpc/WindmillStreamSenderTest.java | 42 +-
.../budget/EvenGetWorkBudgetDistributorTest.java | 14 +-
.../failures/WorkFailureProcessorTest.java | 75 +-
.../refresh/DispatchedActiveWorkRefresherTest.java | 54 +-
.../runners/jobsubmission/JobServerDriver.java | 12 +
runners/portability/java/build.gradle | 2 +
runners/samza/job-server/build.gradle | 2 +
.../translation/RedistributeByKeyTranslator.java | 64 +
.../samza/translation/ReshuffleTranslator.java | 14 +-
.../samza/translation/SamzaPipelineTranslator.java | 1 +
runners/spark/job-server/spark_job_server.gradle | 11 +-
runners/spark/spark_runner.gradle | 7 +
sdks/go.mod | 3 +-
sdks/go.sum | 7 +-
sdks/go/container/boot.go | 4 +-
.../tools/pipeline_options.go} | 34 +-
sdks/go/examples/stringsplit/stringsplit.go | 2 +-
sdks/go/examples/wasm/README.md | 2 +-
sdks/go/pkg/beam/core/core.go | 2 +-
sdks/go/pkg/beam/core/runtime/exec/datasource.go | 48 +-
.../pkg/beam/core/runtime/exec/datasource_test.go | 8 +-
sdks/go/pkg/beam/core/runtime/graphx/translate.go | 12 +-
sdks/go/pkg/beam/core/runtime/harness/harness.go | 9 +-
sdks/go/pkg/beam/core/runtime/harness/init/init.go | 17 +-
.../go/pkg/beam/core/runtime/harness/monitoring.go | 8 +-
sdks/go/pkg/beam/io/xlang/bigtableio/bigtable.go | 6 +-
.../beam/model/fnexecution_v1/beam_fn_api.pb.go | 1582 ++++++----
.../beam/model/pipeline_v1/beam_runner_api.pb.go | 3245 ++++++++++----------
.../model/pipeline_v1/external_transforms.pb.go | 4 +
sdks/go/pkg/beam/runners/prism/internal/coders.go | 14 +
.../prism/internal/engine/elementmanager.go | 284 +-
.../runners/prism/internal/engine/engine_test.go | 43 +
.../beam/runners/prism/internal/engine/holds.go | 39 +-
.../prism/internal/engine/processingtime.go | 96 +
.../prism/internal/engine/processingtime_test.go | 139 +
.../runners/prism/internal/engine/teststream.go | 12 +-
.../beam/runners/prism/internal/engine/timers.go | 166 +
.../runners/prism/internal/engine/timers_test.go | 291 ++
sdks/go/pkg/beam/runners/prism/internal/execute.go | 10 +-
.../prism/internal/jobservices/management.go | 15 +-
sdks/go/pkg/beam/runners/prism/internal/stage.go | 25 +-
.../runners/prism/internal/unimplemented_test.go | 2 +-
sdks/go/pkg/beam/runners/prism/internal/web/web.go | 11 +-
.../beam/runners/prism/internal/worker/worker.go | 4 +-
sdks/go/test/integration/integration.go | 14 +-
sdks/go/test/integration/primitives/timers.go | 151 +
sdks/go/test/integration/primitives/timers_test.go | 10 +-
sdks/java/container/boot.go | 24 +-
.../container/license_scripts/dep_urls_java.yaml | 4 +-
sdks/java/core/build.gradle | 2 +-
.../sdk/fn/data/BeamFnDataInboundObserver.java | 14 +
.../transforms/TypedSchemaTransformProvider.java | 46 +-
.../apache/beam/sdk/schemas/utils/YamlUtils.java | 8 +
.../org/apache/beam/sdk/testing/TestPipeline.java | 85 +-
.../apache/beam/sdk/transforms/Redistribute.java | 287 ++
.../java/org/apache/beam/sdk/transforms/View.java | 200 +-
.../beam/sdk/util/construction/BeamUrns.java | 5 +
.../beam/sdk/util/construction/Environments.java | 1 +
.../construction/ExternalTranslationOptions.java | 4 +-
.../util/construction/PTransformTranslation.java | 45 +-
.../sdk/util/construction/TransformUpgrader.java | 54 +-
.../beam/sdk/values/PCollectionRowTuple.java | 17 +
.../apache/beam/sdk/values/PCollectionViews.java | 488 +++
.../TypedSchemaTransformProviderTest.java | 8 +-
.../beam/sdk/testing/CoderPropertiesTest.java | 6 +-
.../{ReshuffleTest.java => RedistributeTest.java} | 155 +-
.../apache/beam/sdk/transforms/ReshuffleTest.java | 4 +
.../org/apache/beam/sdk/transforms/ViewTest.java | 145 +
.../util/construction/TransformUpgraderTest.java | 139 +-
sdks/java/expansion-service/build.gradle | 4 +-
sdks/java/expansion-service/container/boot.go | 4 +
.../container/expansion_service_config.yml | 3 +
.../beam/sdk/expansion/service/Dependency.java | 5 +-
.../sdk/expansion/service/ExpansionService.java | 76 +-
.../expansion/service/ExpansionServiceConfig.java | 54 +-
.../expansion/service/ExpansionServiceOptions.java | 35 +-
.../service/JavaClassLookupTransformProvider.java | 60 +-
.../expansion/service/ExpansionServiceTest.java | 40 +
.../JavaClassLookupTransformProviderTest.java | 2 -
.../resources/test_expansion_service_config.yaml | 23 +
.../google-cloud-platform-core/build.gradle | 24 +
.../sdk/extensions/gcp/options/GcpOptions.java | 88 +-
.../sdk/extensions/gcp/options/GcsOptions.java | 18 +
.../beam/sdk/extensions/gcp/util/GcsUtil.java | 187 +-
.../gcp/util/RetryHttpRequestInitializer.java | 26 +-
.../beam/sdk/extensions/gcp/util/Transport.java | 51 +-
.../sdk/extensions/gcp/options/GcpOptionsIT.java | 93 +
.../sdk/extensions/gcp/options/GcpOptionsTest.java | 47 +
.../beam/sdk/extensions/gcp/util/GcsUtilTest.java | 83 +-
.../sdk/extensions/gcp/util/TransportTest.java | 35 +-
sdks/java/extensions/ordered/build.gradle | 2 +-
.../sdk/extensions/python/bootstrap_beam_venv.py | 10 +-
.../beam/sdk/extensions/sorter/SortValues.java | 26 +-
.../beam/sdk/extensions/sorter/SortValuesTest.java | 144 +-
.../fn/harness/control/ProcessBundleHandler.java | 3 +
sdks/java/io/cdap/build.gradle | 2 -
.../debezium/DebeziumReadSchemaTransformTest.java | 33 +-
.../sdk/io/elasticsearch/ElasticsearchIOTest.java | 6 +
.../elasticsearch/ElasticsearchIOTestCommon.java | 52 +
.../io/elasticsearch/ElasticsearchIOTestUtils.java | 26 +-
.../beam/sdk/io/elasticsearch/ElasticsearchIO.java | 54 +-
sdks/java/io/expansion-service/build.gradle | 9 +
.../beam/sdk/io/gcp/bigquery/AppendClientInfo.java | 2 +-
.../AvroGenericRecordToStorageApiProto.java | 17 +-
.../io/gcp/bigquery/BeamRowToStorageApiProto.java | 16 +-
.../beam/sdk/io/gcp/bigquery/BigQueryIO.java | 39 +-
.../sdk/io/gcp/bigquery/BigQueryIOTranslation.java | 10 +-
.../sdk/io/gcp/bigquery/BigQueryQueryHelper.java | 15 +-
.../io/gcp/bigquery/BigQueryQuerySourceDef.java | 26 +-
.../sdk/io/gcp/bigquery/BigQueryServicesImpl.java | 4 +-
.../sdk/io/gcp/bigquery/BigQuerySinkMetrics.java | 14 +-
.../gcp/bigquery/BigQueryStorageQuerySource.java | 8 +
.../gcp/bigquery/BigQueryStorageStreamSource.java | 31 +-
.../beam/sdk/io/gcp/bigquery/BigQueryUtils.java | 51 +
.../beam/sdk/io/gcp/bigquery/RowMutation.java | 27 +-
.../io/gcp/bigquery/RowMutationInformation.java | 111 +-
.../beam/sdk/io/gcp/bigquery/StorageApiCDC.java | 9 +
.../StorageApiDynamicDestinationsBeamRow.java | 4 +-
...StorageApiDynamicDestinationsGenericRecord.java | 7 +-
.../StorageApiDynamicDestinationsTableRow.java | 4 +-
.../bigquery/StorageApiWriteUnshardedRecords.java | 26 +-
.../bigquery/StorageApiWritesShardedRecords.java | 4 +-
.../io/gcp/bigquery/TableRowToStorageApiProto.java | 75 +-
.../beam/sdk/io/gcp/bigtable/BigtableIO.java | 121 +-
.../beam/sdk/io/gcp/bigtable/BigtableService.java | 7 +-
.../sdk/io/gcp/bigtable/BigtableServiceImpl.java | 93 +-
.../apache/beam/sdk/io/gcp/pubsub/PubsubIO.java | 11 +
.../beam/sdk/io/gcp/spanner/SpannerConfig.java | 20 +
.../apache/beam/sdk/io/gcp/spanner/SpannerIO.java | 41 +-
.../sdk/io/gcp/testing/FakeDatasetService.java | 3 +-
.../com/google/cloud/spanner/OptionsImposter.java | 15 +-
.../AvroGenericRecordToStorageApiProtoTest.java | 3 +-
.../gcp/bigquery/BeamRowToStorageApiProtoTest.java | 4 +-
.../sdk/io/gcp/bigquery/BigQueryIOReadTest.java | 26 +-
.../gcp/bigquery/BigQueryIOStorageQueryTest.java | 6 +
.../io/gcp/bigquery/BigQueryIOStorageReadTest.java | 85 +
.../io/gcp/bigquery/BigQueryIOTranslationTest.java | 1 +
.../sdk/io/gcp/bigquery/BigQueryIOWriteTest.java | 88 +-
.../io/gcp/bigquery/BigQuerySinkMetricsTest.java | 8 +-
.../beam/sdk/io/gcp/bigquery/BigQueryUtilTest.java | 60 +
.../gcp/bigquery/RowMutationInformationTest.java | 132 +
.../io/gcp/bigquery/StorageApiSinkRowUpdateIT.java | 63 +-
.../bigquery/TableRowToStorageApiProtoTest.java | 80 +-
.../beam/sdk/io/gcp/bigtable/BigtableIOTest.java | 102 +-
.../io/gcp/bigtable/BigtableServiceImplTest.java | 43 +-
.../beam/sdk/io/gcp/pubsub/PubsubIOTest.java | 43 +
.../sdk/io/gcp/spanner/SpannerIOWriteTest.java | 28 +
sdks/java/io/hadoop-format/build.gradle | 2 +
sdks/java/io/iceberg/build.gradle | 48 +-
.../IcebergReadSchemaTransformProvider.java | 12 -
.../IcebergSchemaTransformCatalogConfig.java | 2 +-
.../IcebergWriteSchemaTransformProvider.java | 11 -
.../sdk/io/iceberg/SchemaAndRowConversions.java | 29 +-
.../apache/beam/sdk/io/iceberg/IcebergIOIT.java | 287 ++
.../IcebergReadSchemaTransformProviderTest.java | 7 +-
.../IcebergSchemaTransformTranslationTest.java | 6 +-
.../IcebergWriteSchemaTransformProviderTest.java | 17 +-
.../io/iceberg/SchemaAndRowConversionsTest.java | 42 +
.../java/org/apache/beam/sdk/io/jdbc/JdbcIO.java | 8 +
.../java/org/apache/beam/sdk/io/jms/JmsIO.java | 134 +-
.../java/org/apache/beam/sdk/io/jms/CommonJms.java | 58 +-
.../java/org/apache/beam/sdk/io/jms/JmsIOIT.java | 35 +-
.../java/org/apache/beam/sdk/io/jms/JmsIOTest.java | 254 +-
.../jms/MockNonSerializableConnectionFactory.java | 19 +-
sdks/java/io/kafka/build.gradle | 1 +
.../java/org/apache/beam/sdk/io/kafka/KafkaIO.java | 4 +-
.../KafkaReadSchemaTransformConfiguration.java | 6 +
.../io/kafka/KafkaReadSchemaTransformProvider.java | 261 +-
.../io/kafka/KafkaSchemaTransformTranslation.java | 93 +
.../beam/sdk/io/kafka/KafkaUnboundedReader.java | 18 +-
.../kafka/KafkaWriteSchemaTransformProvider.java | 16 +
.../beam/sdk/io/kafka/ReadFromKafkaDoFn.java | 18 +
.../org/apache/beam/sdk/io/kafka/KafkaIOIT.java | 45 +-
.../KafkaReadSchemaTransformProviderTest.java | 116 +-
.../kafka/KafkaSchemaTransformTranslationTest.java | 216 ++
.../KafkaWriteSchemaTransformProviderTest.java | 56 +
sdks/java/managed/build.gradle | 1 +
.../java/org/apache/beam/sdk/managed/Managed.java | 63 +-
.../managed/ManagedSchemaTransformProvider.java | 105 +-
.../sdk/managed/ManagedTransformConstants.java | 38 +-
.../ManagedSchemaTransformProviderTest.java | 12 +-
.../ManagedSchemaTransformTranslationTest.java | 49 +-
.../org/apache/beam/sdk/managed/ManagedTest.java | 36 +-
.../managed/src/test/resources/test_config.yaml | 4 +-
sdks/java/transform-service/build.gradle | 4 +-
.../transformservice/TransformServiceConfig.java | 30 +-
.../transformservice/TransformServiceOptions.java | 17 +-
.../sdk/transformservice/ExpansionServiceTest.java | 15 +
.../resources/test_transform_service_config.yml | 15 +
sdks/python/apache_beam/__init__.py | 2 +-
sdks/python/apache_beam/dataframe/doctests.py | 16 +-
sdks/python/apache_beam/dataframe/expressions.py | 6 +-
sdks/python/apache_beam/dataframe/frames.py | 19 +-
sdks/python/apache_beam/dataframe/frames_test.py | 18 +
sdks/python/apache_beam/dataframe/io_test.py | 9 +
.../apache_beam/dataframe/pandas_doctests_test.py | 150 +-
sdks/python/apache_beam/dataframe/transforms.py | 1 +
.../transforms/aggregation/approximatequantiles.py | 1 +
.../transforms/aggregation/approximateunique.py | 1 +
sdks/python/apache_beam/io/filesystems.py | 45 +-
.../io/gcp/big_query_query_to_table_it_test.py | 3 +
sdks/python/apache_beam/io/gcp/bigquery.py | 49 +-
sdks/python/apache_beam/io/gcp/bigquery_tools.py | 2 +-
.../apache_beam/io/gcp/bigquery_write_it_test.py | 5 +-
sdks/python/apache_beam/io/gcp/bigtableio.py | 12 +-
sdks/python/apache_beam/io/gcp/gcsio.py | 65 +-
.../apache_beam/io/gcp/gcsio_integration_test.py | 40 +
sdks/python/apache_beam/io/gcp/gcsio_test.py | 80 +
sdks/python/apache_beam/io/requestresponse.py | 121 +-
sdks/python/apache_beam/io/requestresponse_test.py | 13 +
.../ml/inference/huggingface_inference.py | 2 +-
.../ml/inference/huggingface_inference_test.py | 2 +-
.../ml/inference/onnx_inference_test.py | 4 +
.../ml/inference/tensorflow_inference_test.py | 14 +-
.../ml/inference/tensorflow_tests_requirements.txt | 1 +
sdks/python/apache_beam/ml/transforms/base.py | 93 +-
sdks/python/apache_beam/ml/transforms/base_test.py | 88 +
.../ml/transforms/embeddings/huggingface_test.py | 2 +
.../ml/transforms/embeddings/vertex_ai.py | 96 +-
.../ml/transforms/embeddings/vertex_ai_test.py | 41 +
sdks/python/apache_beam/ml/transforms/tft.py | 115 +-
sdks/python/apache_beam/ml/transforms/tft_test.py | 169 +
.../python/apache_beam/options/pipeline_options.py | 2 +-
.../apache_beam/options/pipeline_options_test.py | 4 +-
sdks/python/apache_beam/pipeline.py | 6 +-
sdks/python/apache_beam/pipeline_test.py | 6 +-
.../runners/dataflow/dataflow_runner.py | 2 +-
.../runners/dataflow/internal/apiclient.py | 22 +-
.../runners/dataflow/internal/apiclient_test.py | 2 +-
.../apache_beam/runners/dataflow/internal/names.py | 2 +-
.../apache_beam/runners/interactive/utils.py | 18 +-
.../runners/portability/sdk_container_builder.py | 14 +-
.../apache_beam/runners/worker/bundle_processor.py | 9 +
.../apache_beam/runners/worker/data_plane.py | 42 +-
.../apache_beam/runners/worker/data_sampler.py | 35 +-
.../apache_beam/runners/worker/sdk_worker.py | 5 +-
.../apache_beam/runners/worker/sdk_worker_test.py | 4 +-
.../apache_beam/testing/test_pipeline_test.py | 5 +-
sdks/python/apache_beam/transforms/enrichment.py | 14 +-
.../transforms/enrichment_handlers/bigquery.py | 256 ++
.../enrichment_handlers/bigquery_it_test.py | 289 ++
.../enrichment_handlers/bigquery_test.py | 70 +
sdks/python/apache_beam/transforms/environments.py | 1 +
sdks/python/apache_beam/transforms/external.py | 32 +-
.../python/apache_beam/transforms/external_test.py | 30 +
.../transforms/external_transform_provider.py | 35 +-
.../external_transform_provider_it_test.py | 22 -
sdks/python/apache_beam/transforms/ptransform.py | 3 +
.../apache_beam/transforms/ptransform_test.py | 11 +
sdks/python/apache_beam/transforms/sql.py | 2 +-
sdks/python/apache_beam/transforms/util.py | 18 +-
sdks/python/apache_beam/transforms/util_test.py | 33 +-
.../apache_beam/typehints/intrinsic_one_ops.py | 100 +
.../intrinsic_one_ops_test.py} | 22 +-
sdks/python/apache_beam/typehints/opcodes.py | 65 +
sdks/python/apache_beam/typehints/schemas.py | 7 +
.../apache_beam/typehints/trivial_inference.py | 30 +-
.../typehints/trivial_inference_test.py | 1 +
sdks/python/apache_beam/version.py | 2 +-
.../apache_beam/yaml/examples/simple_filter.yaml | 4 +-
.../yaml/examples/simple_filter_and_combine.yaml | 7 +-
.../yaml/examples/wordcount_minimal.yaml | 4 +-
sdks/python/apache_beam/yaml/generate_yaml_docs.py | 24 +-
sdks/python/apache_beam/yaml/integration_tests.py | 1 -
sdks/python/apache_beam/yaml/main.py | 64 +-
sdks/python/apache_beam/yaml/main_test.py | 76 +
sdks/python/apache_beam/yaml/standard_io.yaml | 88 +-
.../apache_beam/yaml/standard_providers.yaml | 8 +-
sdks/python/apache_beam/yaml/tests/avro.yaml | 4 +-
sdks/python/apache_beam/yaml/tests/bigquery.yaml | 4 +-
sdks/python/apache_beam/yaml/tests/csv.yaml | 4 +-
sdks/python/apache_beam/yaml/tests/java-map.yaml | 4 +-
sdks/python/apache_beam/yaml/tests/join.yaml | 4 +-
sdks/python/apache_beam/yaml/tests/json.yaml | 4 +-
sdks/python/apache_beam/yaml/tests/parquet.yaml | 4 +-
sdks/python/apache_beam/yaml/tests/sql.yaml | 23 +-
sdks/python/apache_beam/yaml/tests/text.yaml | 4 +-
sdks/python/apache_beam/yaml/tests/windowing.yaml | 4 +-
sdks/python/apache_beam/yaml/yaml_combine.py | 2 +-
sdks/python/apache_beam/yaml/yaml_join.py | 45 +-
sdks/python/apache_beam/yaml/yaml_mapping.py | 23 +-
sdks/python/apache_beam/yaml/yaml_mapping_test.py | 30 +
sdks/python/apache_beam/yaml/yaml_provider.py | 54 +-
sdks/python/build.gradle | 3 +-
sdks/python/container/boot.go | 4 +-
sdks/python/container/build.gradle | 23 +-
.../container/license_scripts/dep_urls_py.yaml | 2 +
.../container/py310/base_image_requirements.txt | 2 +-
.../container/py311/base_image_requirements.txt | 2 +-
.../{py311 => py312}/base_image_requirements.txt | 64 +-
.../python/container/py312/build.gradle | 16 +-
.../container/py39/base_image_requirements.txt | 2 +-
sdks/python/gen_xlang_wrappers.py | 21 +-
sdks/python/scripts/generate_pydoc.sh | 1 +
sdks/python/scripts/run_pylint.sh | 2 +
sdks/python/setup.py | 55 +-
.../test-suites/dataflow/py312}/build.gradle | 11 +-
.../python/test-suites/direct/py312}/build.gradle | 11 +-
sdks/python/test-suites/gradle.properties | 22 +-
.../test-suites/portable/py312}/build.gradle | 12 +-
sdks/python/test-suites/tox/common.gradle | 3 +
.../tox/{pycommon => py312}/build.gradle | 11 +-
sdks/python/test-suites/tox/py38/build.gradle | 95 +-
sdks/python/test-suites/tox/pycommon/build.gradle | 22 +-
sdks/python/tox.ini | 72 +-
sdks/typescript/package.json | 2 +-
sdks/typescript/src/apache_beam/runners/flink.ts | 2 +-
settings.gradle.kts | 23 +-
website/www/site/content/en/blog/beamquest.md | 2 +-
.../documentation/pipelines/test-your-pipeline.md | 4 +-
.../content/en/documentation/programming-guide.md | 4 +-
.../content/en/documentation/runners/direct.md | 5 +
.../site/content/en/documentation/runners/flink.md | 17 +-
.../content/en/documentation/runners/jstorm.md | 111 -
.../content/en/documentation/runners/mapreduce.md | 91 -
.../content/en/documentation/sdks/yaml-combine.md | 2 +-
.../www/site/content/en/documentation/sdks/yaml.md | 2 +-
.../shortcodes/flink_java_pipeline_options.html | 10 +
.../shortcodes/flink_python_pipeline_options.html | 10 +
607 files changed, 18525 insertions(+), 7735 deletions(-)
copy .github/trigger_files/{beam_PostCommit_Java_DataflowV1.json =>
IO_Iceberg_Integration_Tests.json} (100%)
copy .github/trigger_files/{beam_PostCommit_Python_Xlang_Gcp_Direct.json =>
IO_Iceberg_Unit_Tests.json} (100%)
copy .github/trigger_files/{beam_PostCommit_Python_Xlang_IO_Dataflow.json =>
beam_PostCommit_Go_VR_Flink.json} (100%)
copy .github/trigger_files/{beam_PostCommit_Java_PVR_Spark_Batch.json =>
beam_PostCommit_Java_Hadoop_Versions.json} (97%)
create mode 100644
.github/trigger_files/beam_PostCommit_Java_PVR_Flink_Batch.json
create mode 100644
.github/trigger_files/beam_PostCommit_Java_PVR_Flink_Docker.json
create mode 100644
.github/trigger_files/beam_PostCommit_Java_PVR_Flink_Streaming.json
create mode 100644 .github/trigger_files/beam_PostCommit_Java_PVR_Samza.json
create mode 100644
.github/trigger_files/beam_PostCommit_Java_ValidatesRunner_Dataflow_JavaVersions.json
create mode 100644
.github/trigger_files/beam_PostCommit_Java_ValidatesRunner_Direct.json
create mode 100644
.github/trigger_files/beam_PostCommit_Java_ValidatesRunner_Direct_JavaVersions.json
create mode 100644
.github/trigger_files/beam_PostCommit_Java_ValidatesRunner_Spark_Java11.json
copy .github/trigger_files/{beam_PostCommit_Python_ValidatesRunner_Flink.json
=> beam_PostCommit_Python_Dependency.json} (100%)
delete mode 100644 .github/trigger_files/beam_PostCommit_XVR_Flink.json
copy .github/workflows/{beam_PreCommit_GoPrism.yml =>
IO_Iceberg_Integration_Tests.yml} (69%)
copy .github/workflows/{beam_PreCommit_RAT.yml =>
IO_Iceberg_Performance_Tests.yml} (74%)
rename .github/workflows/{IO_Iceberg.yml => IO_Iceberg_Unit_Tests.yml} (87%)
copy .github/workflows/{beam_PreCommit_Python_Runners.yml =>
beam_PreCommit_Python_ML.yml} (87%)
delete mode 100644
runners/flink/1.14/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/AbstractStreamOperatorCompat.java
delete mode 100644
runners/flink/1.14/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/ProcessingTimeCallbackCompat.java
delete mode 100644
runners/flink/1.14/src/test/java/org/apache/beam/runners/flink/MiniClusterCompat.java
delete mode 100644
runners/flink/1.14/src/test/java/org/apache/beam/runners/flink/metrics/MetricGroupWrapper.java
rename runners/flink/{1.14 =>
1.15}/src/main/java/org/apache/beam/runners/flink/translation/types/CoderTypeSerializer.java
(100%)
delete mode 100644
runners/flink/1.15/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/ProcessingTimeCallbackCompat.java
delete mode 100644
runners/flink/1.15/src/test/java/org/apache/beam/runners/flink/MiniClusterCompat.java
copy runners/flink/{1.14 => 1.18}/build.gradle (94%)
copy runners/flink/{1.14 => 1.18}/job-server-container/build.gradle (100%)
rename runners/flink/{1.14 => 1.18}/job-server/build.gradle (95%)
rename runners/flink/{1.14 =>
}/src/test/java/org/apache/beam/runners/flink/RemoteMiniClusterImpl.java (100%)
rename runners/flink/{1.14 =>
}/src/test/java/org/apache/beam/runners/flink/streaming/StreamSources.java
(100%)
rename runners/flink/{1.14 =>
}/src/test/java/org/apache/beam/runners/flink/translation/types/CoderTypeSerializerTest.java
(100%)
rename
runners/flink/{1.14/src/test/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/source/SourceTestCompat.java
=>
src/test/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/source/SourceTestMetrics.java}
(99%)
create mode 100644
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/ComputationStateCache.java
create mode 100644
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/ComputationWorkExecutor.java
copy
sdks/java/transform-service/src/main/java/org/apache/beam/sdk/transformservice/TransformServiceConfig.java
=>
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/ExecutableWork.java
(54%)
delete mode 100644
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/ExecutionState.java
create mode 100644
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/Watermarks.java
create mode 100644
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/config/ComputationConfig.java
create mode 100644
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/config/StreamingApplianceComputationConfigFetcher.java
create mode 100644
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/config/StreamingEngineComputationConfigFetcher.java
create mode 100644
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/config/StreamingEnginePipelineConfig.java
create mode 100644
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/harness/MetricsDataProvider.java
create mode 100644
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/harness/StreamingWorkerStatusPages.java
delete mode 100644
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/work/ProcessWorkItemClient.java
rename
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/work/{WorkItemProcessor.java
=> WorkItemScheduler.java} (61%)
create mode 100644
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/work/processing/ComputationWorkExecutorFactory.java
create mode 100644
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/work/processing/StreamingCommitFinalizer.java
create mode 100644
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/work/processing/StreamingWorkScheduler.java
create mode 100644
runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/streaming/ComputationStateCacheTest.java
create mode 100644
runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/streaming/config/StreamingApplianceComputationConfigFetcherTest.java
create mode 100644
runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/streaming/config/StreamingEngineComputationConfigFetcherTest.java
create mode 100644
runners/samza/src/main/java/org/apache/beam/runners/samza/translation/RedistributeByKeyTranslator.java
copy sdks/go/{test/integration/primitives/timers_test.go =>
container/tools/pipeline_options.go} (56%)
create mode 100644
sdks/go/pkg/beam/runners/prism/internal/engine/processingtime.go
create mode 100644
sdks/go/pkg/beam/runners/prism/internal/engine/processingtime_test.go
create mode 100644
sdks/go/pkg/beam/runners/prism/internal/engine/timers_test.go
create mode 100644
sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Redistribute.java
copy
sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/{ReshuffleTest.java
=> RedistributeTest.java} (74%)
create mode 100644
sdks/java/expansion-service/src/test/resources/test_expansion_service_config.yaml
create mode 100644
sdks/java/extensions/google-cloud-platform-core/src/test/java/org/apache/beam/sdk/extensions/gcp/options/GcpOptionsIT.java
rename
runners/flink/1.14/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/source/compat/FlinkSourceCompat.java
=>
sdks/java/io/google-cloud-platform/src/test/java/com/google/cloud/spanner/OptionsImposter.java
(66%)
create mode 100644
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/RowMutationInformationTest.java
create mode 100644
sdks/java/io/iceberg/src/test/java/org/apache/beam/sdk/io/iceberg/IcebergIOIT.java
rename
runners/flink/1.14/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/source/compat/SplitEnumeratorCompat.java
=>
sdks/java/io/jms/src/test/java/org/apache/beam/sdk/io/jms/MockNonSerializableConnectionFactory.java
(65%)
create mode 100644
sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaSchemaTransformTranslation.java
create mode 100644
sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaSchemaTransformTranslationTest.java
create mode 100644
sdks/java/transform-service/src/test/resources/test_transform_service_config.yml
create mode 100644
sdks/python/apache_beam/transforms/enrichment_handlers/bigquery.py
create mode 100644
sdks/python/apache_beam/transforms/enrichment_handlers/bigquery_it_test.py
create mode 100644
sdks/python/apache_beam/transforms/enrichment_handlers/bigquery_test.py
create mode 100644 sdks/python/apache_beam/typehints/intrinsic_one_ops.py
copy sdks/python/apache_beam/{version.py =>
typehints/intrinsic_one_ops_test.py} (58%)
create mode 100644 sdks/python/apache_beam/yaml/main_test.py
copy sdks/python/container/{py311 => py312}/base_image_requirements.txt (81%)
rename
runners/flink/1.14/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/source/compat/package-info.java
=> sdks/python/container/py312/build.gradle (69%)
copy {runners/flink/1.14 =>
sdks/python/test-suites/dataflow/py312}/build.gradle (81%)
rename {runners/flink/1.14 =>
sdks/python/test-suites/direct/py312}/build.gradle (82%)
rename {runners/flink/1.14/job-server-container =>
sdks/python/test-suites/portable/py312}/build.gradle (78%)
copy sdks/python/test-suites/tox/{pycommon => py312}/build.gradle (79%)
delete mode 100644 website/www/site/content/en/documentation/runners/jstorm.md
delete mode 100644
website/www/site/content/en/documentation/runners/mapreduce.md
