This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a change to branch
dependabot/go_modules/sdks/cloud.google.com/go/bigtable-1.40.1
in repository https://gitbox.apache.org/repos/asf/beam.git
discard 335f125d7c9 Bump cloud.google.com/go/bigtable from 1.39.0 to 1.40.1 in
/sdks
add d4438b63e18 Enforce deterministic field order in Schema generated from
KafkaIO classes. (#36295)
add 590ece2cd8e Fix Python CoGBK Flink Batch config
add 62df216296a Merge pull request #36472 from apache/fix-flink-cogbk
add 227a6323ead Add logger helper functions from detectron2 (licensed as
Apache 2.0) (#36432)
add fb8058454c0 Refactor GBEK tests to split out secret setup for tests
that dont need it (#36479)
add bf4bf81922f Update CHANGES.md to show issue #36470 is fixed in 2.69.0
release. (#36477)
add a5846889716 [Dataflow Streaming] Change GrpcGetDataStream to backoff
requests that have been cancelled. (#36475)
add 6be76ae8542 [Dataflow Streaming] Enforce that get data requests for
the same work item are not batched. (#36474)
add 7735e7a3bd1 Bump google.golang.org/grpc from 1.75.1 to 1.76.0 in /sdks
(#36488)
add 2a77c59b82e Bump golang.org/x/text from 0.29.0 to 0.30.0 in /sdks
(#36487)
add 090b17bf535 added missing schemaFieldNumber annotation (#36489)
add 041e12edd9d feat: add warnings for public repository downloads in
multiple SDKs (#36476)
add 08b480000ec add generics support to AutoValueUtils helpers (#32977)
add c54cc2b6ed9 Add flag for disabling dill check in coders. (#36453)
add 1c6f779bdf2 Add use_gbek service option when gbek option used (#36452)
add d54a661f47e Bump golang.org/x/net from 0.45.0 to 0.46.0 in /sdks
(#36466)
add b9a89722d9e External metadata for streaming runner v1 changes (#36373)
add 243a52c5b42 Move tests running pipeline into a separate class for
PythonExternalTransformTest (#36492)
add b5b91810b76 Move the logic to LP TestStream encoded bytes to
preprocess steps. (#36465)
add 554a73b4bf3 Enable real-time clock in prism by default. (#36473)
add 2b9827b6c2d Refactor prism and go sdk logging and clean up messages
(#36484)
add 6dbbaa687fe Fix Credentials issue while commit (#36494)
add 385271bea45 Softens the GBEK determinism requirement (#36495)
add 72557e58a06 Only run Py39 and Py313 tests for PostCommit Arm (#36508)
add 91f79c3a97f Add GroupByEncryptedKey to changes (#36510)
add 673309b328c Call out OutputBuilder change in CHANGES.md (#36511)
add 37c7e28231b ci(python-deps): update transformers version constraints
in tox (#36506)
add 75eda20a901 Revert "Per element schema parsing in ConvertToBeamRows
(#36393)" (#36507)
add 7a9a4e6afa5 x-lang GroupByEncryptedKey (Java to Python) (#36418)
add 95dcaeac932 docs: Expose ReadChangeStreamFromSpanner in Beam Spanner
documentation (#36428)
add e94579a3e6c Enhance JAXBCoder with XMLInputFactory support (#36446)
add 7b34ab75c46 Add some x-lang gbek tests (Python to Java) (#36457)
add ed39cbbf709 beam-sql.sh, a standalone launcher for Beam SQL Shell
(#36305)
add 12a34c8acf8 Change default timeout and add heartbeat logging (#36517)
add 4add79cab24 Updates ExpansionService so that managed transforms can
use specific dependencies during expansion. Behavior is guarded by an pipeline
option.
add 50f578aee32 Merge pull request #36515: Updates ExpansionService so
that managed transforms can use specific dependencies during expansion
add 9ed06d081ec Handle null keys in gbek (#36505)
add c703b7227de Bump github.com/nats-io/nats.go from 1.46.0 to 1.47.0 in
/sdks (#36521)
add 99ee1738e2b Add a flag to control whether to allow splitting on sdf.
(#36512)
add d687f4fe817 Add GRPC experiments to Python dockerfile (#36525)
add f0c92c7a772 revert outputWindowedValue changes as there is
outputBuilder
add abf1904759e Merge pull request #36523: revert outputWindowedValue
changes from KafkaIO as there is outputBuilder
add ed39503878e Skip TestTimers_ProcessingTime_Unbounded for spark.
(#36527)
add 6ad53078c48 Moving to 2.70.0-SNAPSHOT on master branch.
add f7619c789d0 Update CHANGES.md to have fields for 2.70.0 release
add 6562b5b677d Update CHANGES.md to mention breakign change around
ProcessContext (#36530)
add 30fd958f5fc feat(bigquery): add GEOGRAPHY type support for BigQuery
I/O (#36121)
add b6878702484 Fix flaky tests caused by secret overlap (#36526)
add 96e79cba3a6 Concat protos in BQStorageWriteAPI - solve edge cases
during mering of nested repeated fields (#34436)
add 19fef1bba24 add changes comment on yaml output_schema (#36497)
add 57e34b6906b Fix proto map access. (#36532)
add faae168fa34 Bump github.com/aws/smithy-go from 1.23.0 to 1.23.1 in
/sdks (#36533)
add 118b3c7a582 PortableRunner tests: surface worker-thread exceptions on
main thread after wait_until_finish() (fixes #35211) (#36485)
add 5d420c5f047 Add pickler.roundtrip() shortcut for testing pickle
(#36441)
add 2b43f8018ba Pin specifiable test to FnApiRunner (#36536)
add f07ccf37cbe Track bundle processors that are pending creation and
terminate SDK if creating a BP exceeds a timeout. (#36518)
add 581ec8bb17f Always mark the instruction as cleaned up in the GRPC data
channel when processing an instruction fails. (#36367)
add e87f8097e53 Move setup/teardown to class level to avoid flakiness
(#36546)
add 6ffc68778b9 Fix build release candidate workflow (#36541)
add d91fb6d6987 Timeout execution tree creation for SDK worker ops.
(#36200)
add d4dc3243303 Fix dill tox (#36543)
add 34a6f542a7f Update beam_PreCommit_Python_ML.yml (#36550)
add 87db35637a6 Add "return []" to PGBK to silence warning (#36535)
add 2b666dacf47 test(bigquery): skip geography test when expansion jars
not available (#36555)
add d4b841caa94 Update changes.md with pickler changes. (#36558)
add 9030ba8074c test(bigquery): mock client in geography type support
tests (#36559)
add af748d07a1a Update Python Dependencies (#36560)
add 1bf56295bc9 Fix publishing of ml/distroless images (#36548)
add 07b321e5811 Fix unsafe container cleanup that could delete images from
other runs (#36547)
add e081879a78f Fix BigQueryIO File load validate runtime value provider
(#36564)
add ee48e713282 [3/3] sdks/python: enrich data with Milvus Search [Vector,
Keyword, Hybrid] (#35467)
add f8901e3a4c5 Update the release notes. (#36566)
add 15e8f98fed6 Fix dependency version (#36568)
add d0d0cd8c2f1 Revert "Add GRPC experiments to Python dockerfile
(#36525)" (#36572)
add 944eef91344 Upload beam blog. (#36499)
add c22665c5111 Call now() once so start and end have exactly the same
base timestamp. (#36574)
add 0d52be60e88 Add image generation code to Gemini Model Handler (#36177)
add 179d4d1ee9c Fix flaky tests (#36579)
add 7e7d866d95a [python] add setup to BigQuery's convert row Map transform
(#36502)
add c7d920f26cf Update dev image. (#36582)
add 243d4077319 Integrate lambda name pickling with Cloudpickle (#35904)
add ab892e3dd09 Add logging for credential retrieval failures in
GcpCredentialFactory
add db92a3ad0c0 Merge pull request #36415: Add logging for credential
retrieval failures in GcpCredentialFactory
add b83c24e4d45 test(spannerio): make batch size validation more flexible
for non-deterministic execution (#36584)
add 4c08585626b [IcebergIO] Pass table props to data writers (#36542)
add 0ebf84b6b18 Add ib.collect support for raw records (#36516)
add 38481b58879 Address circular dependencies in Nexmark benchmark suite.
(#36513)
add 8bd92b5e376 Fix the soft-delete check and emit soft-delete log warning
at most once per bucket. (#36585)
add afeca4ea301 Increase timeouts (#36595)
add 66b7c7476ce Make SpannerChangeStreamPlacementTableIT against Spanner
prod. (#36071)
add fdfa6ec6338 Exclude a perma-red test suite
beam_PostCommit_XVR_GoUsingJava_Dataflow.yml (#36597)
add 6dedf8f0bab Update REVIEWERS.yml (#36598)
add fe71ab1b47b Add ordered window elements into example folder (#36575)
add ef07e40667b Fix proposal link (#36600)
add f2860fa2fe8 use utils._convert_to_result for huggingface_inference
(#36593)
add 05f6f01a33b Force torch to use cpu wheels (#36583)
add 6df5bd04185 Switch to use gcp-io expansion service for YAML BQIO
(#36604)
add a846bc92950 Bump github.com/aws/aws-sdk-go-v2/credentials in /sdks
(#36611)
add 1a4298f209c Bump github.com/avast/retry-go/v4 from 4.6.1 to 4.7.0 in
/sdks (#36519)
add a3d42aec4e0 Update woodstox package and version (#36613)
add 518b118bcaf Fix vLLM Gemma, add vLLM extra, fix getting throughput
(#36451)
add 7bfe0e19974 Do not panic if an element has no windows during
PersistBundle. (#36610)
add 64a92b2806d [python] Move histogram metric from internal to public
interface/files (#36405)
add 92a40213231 Force logback version in transitive dependencies
add f517b01c502 Merge pull request #36594: Force logback version in
transitive dependencies
add 85853a3edf0 proto change
add a6d2b7dabf5 add draining to output builder, encode draining
add 951943e9bba add draining to output builder
add 7ea3109a0e4 default impls
add 3d9e403b3ed comment
add 499039fe37d remove nullable
add cca50bff5aa remove nullable
add f8037f0f5bf remove nullable
add ac99abf3d79 rename field
add fc4b29c4256 rename field
add 2bd497f5d55 Merge pull request #36524: Drain - model part +
windowedValue changes
add 4e0dbb9f452 Upgrade joda-time to 2.14.0 (#36626)
add c6d92079b48 fix test (#36632)
add 9612583296a Bump github.com/linkedin/goavro/v2 from 2.14.0 to 2.14.1
in /sdks (#36628)
add 80f4564fbdc Bump github.com/nats-io/nats-server/v2 from 2.12.0 to
2.12.1 in /sdks (#36520)
add 471050c7cde add licences manually (#36635)
add 0f6b605f9e5 Fix a bug in streaming OrderedWindowElements. (#36642)
add 0c10658e9ba [Dataflow Streaming] Intern encoded tags across keys
(#36313)
add 949c87f0e20 Cache IntervalWindow hashCode (#36612)
add ecc840cd1e7 [Java SDK] Fix propagation of metrics set during onTimer
processing. (#36576)
add 942de54a14c Adding release-2.69.0-postrelease to protected branches in
.asf.yaml
add 00516e97235 Update managed-io.md for release 2.69.0-RC2.
add 3deba11b8a2 Merge pull request #36569 from
apache/updates_managed_io_docs_2.69.0_rc2
add bcf41e14aba Add missing py313 ML container requirements file (#36647)
add be7acf5eca2 Update Beam website to release 2.69.0
add be47decb542 Update date
add 37ca7d54090 Update breaking changes
add fdc89112719 Merge pull request #36554 from apache/release-269-website
add 5a8e2dc3996 remove empty file (#36648)
add 5fe0d4c084b Add batch version of OrderedWindowElements in examples
(#36621)
add 3432480129c Update website for roadmap and version compatibility
(#36602)
add 21dbf592f87 Adds Multimap support to JAVA FnApi (#36218)
add c60b8ae8671 Bump github.com/aws/aws-sdk-go-v2/config in /sdks (#36644)
add b33ab6e2e94 Bump cloud.google.com/go/bigquery from 1.70.0 to 1.71.0 in
/sdks (#36500)
add 0d9ebfbb67f Bump cloud.google.com/go/bigtable from 1.39.0 to 1.40.1 in
/sdks
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (335f125d7c9)
\
N -- N -- N
refs/heads/dependabot/go_modules/sdks/cloud.google.com/go/bigtable-1.40.1
(0d9ebfbb67f)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.asf.yaml | 2 +
.github/REVIEWERS.yml | 2 -
.../IO_Iceberg_Integration_Tests.json | 2 +-
...aming.json => beam_PostCommit_Go_VR_Spark.json} | 2 +-
.../beam_PostCommit_Java_PVR_Spark3_Streaming.json | 2 +-
.github/trigger_files/beam_PostCommit_Python.json | 2 +-
.../beam_PostCommit_Python_Dependency.json | 2 +-
...m_PostCommit_XVR_JavaUsingPython_Dataflow.json} | 2 +-
...m_PostCommit_XVR_PythonUsingJava_Dataflow.json} | 0
.../trigger_files/beam_PostCommit_XVR_Samza.json | 2 +-
...ow_ARM.json => beam_PostCommit_XVR_Spark3.json} | 0
...tainer.json => beam_PreCommit_Python_Dill.json} | 2 +-
.../beam_LoadTests_Python_CoGBK_Flink_Batch.yml | 17 +-
.github/workflows/beam_PostCommit_Python_Arm.yml | 13 +-
.../beam_PostCommit_Python_Dependency.yml | 2 +-
.../beam_PostCommit_XVR_GoUsingJava_Dataflow.yml | 6 +-
.github/workflows/beam_PreCommit_Python_Dill.yml | 6 +-
.github/workflows/beam_PreCommit_Python_ML.yml | 12 +
.../workflows/beam_Publish_Beam_SDK_Snapshots.yml | 5 +
.../workflows/beam_Publish_Docker_Snapshots.yml | 2 +-
.github/workflows/build_release_candidate.yml | 35 +-
...Sentiment_Streaming_DistilBert_Base_Uncased.txt | 3 +-
...Python_Benchmarks_Dataflow_VLLM_Gemma_Batch.txt | 4 +-
..._CoGBK_Dataflow_Flink_Batch_100b_Single_Key.txt | 28 -
.../python_CoGBK_Dataflow_Flink_Batch_10kB.txt | 28 -
...ython_CoGBK_Flink_Batch_100b_Multiple_Keys.txt} | 10 +-
...> python_CoGBK_Flink_Batch_100b_Single_Key.txt} | 17 +-
...out_1.txt => python_CoGBK_Flink_Batch_10kB.txt} | 17 +-
.github/workflows/refresh_looker_metrics.yml | 6 -
.../republish_released_docker_containers.yml | 17 +-
.../workflows/run_rc_validation_python_yaml.yml | 10 +-
CHANGES.md | 74 +-
.../org/apache/beam/gradle/BeamModulePlugin.groovy | 10 +-
.../beam-ml/milvus_enrichment_transform.ipynb | 2657 ++++++++++++++++++++
gradle.properties | 4 +-
.../beam/model/fn_execution/v1/beam_fn_api.proto | 34 +-
.../beam/model/pipeline/v1/beam_runner_api.proto | 10 +-
release/src/main/scripts/set_version.sh | 3 +
...TimeBoundedSplittableProcessElementInvoker.java | 35 +-
.../apache/beam/runners/core/SimpleDoFnRunner.java | 130 -
.../core/SplittableParDoViaKeyedWorkItems.java | 21 -
.../beam/runners/dataflow/BatchViewOverrides.java | 5 +
.../beam/runners/dataflow/DataflowRunner.java | 16 +
.../dataflow/RedistributeByKeyOverrideFactory.java | 1 +
.../dataflow/worker/StreamingDataflowWorker.java | 5 +
.../dataflow/worker/UngroupedWindmillReader.java | 5 +
.../dataflow/worker/WindmillKeyedWorkItem.java | 5 +-
.../beam/runners/dataflow/worker/WindmillSink.java | 46 +-
.../dataflow/worker/util/ValueInEmptyWindows.java | 5 +
.../util/common/worker/InternedByteString.java | 77 +
.../windmill/client/grpc/GrpcGetDataStream.java | 39 +-
.../client/grpc/GrpcGetDataStreamRequests.java | 211 +-
.../worker/windmill/state/CachingStateTable.java | 137 +-
.../worker/windmill/state/WindmillBag.java | 21 +-
.../windmill/state/WindmillCombiningState.java | 20 +-
.../worker/windmill/state/WindmillMap.java | 30 +-
.../worker/windmill/state/WindmillMultimap.java | 24 +-
.../worker/windmill/state/WindmillOrderedList.java | 17 +-
.../worker/windmill/state/WindmillStateCache.java | 82 +-
.../windmill/state/WindmillStateTagUtil.java | 10 +-
.../worker/windmill/state/WindmillValue.java | 21 +-
.../windmill/state/WindmillWatermarkHold.java | 30 +-
.../worker/StreamingGroupAlsoByWindowFnsTest.java | 8 +-
...reamingGroupAlsoByWindowsReshuffleDoFnTest.java | 8 +-
.../dataflow/worker/WindmillKeyedWorkItemTest.java | 9 +-
.../util/common/worker/InternedByteStringTest.java | 68 +
.../client/grpc/GrpcGetDataStreamRequestsTest.java | 160 +-
.../client/grpc/GrpcGetDataStreamTest.java | 51 +
.../windmill/state/WindmillStateCacheTest.java | 232 +-
.../windmill/state/WindmillStateTagUtilTest.java | 10 +-
.../apache/beam/runners/spark/util/TimerUtils.java | 5 +
scripts/beam-sql.sh | 448 ++++
sdks/go.mod | 57 +-
sdks/go.sum | 118 +-
sdks/go/cmd/prism/prism.go | 50 +-
sdks/go/pkg/beam/core/core.go | 2 +-
sdks/go/pkg/beam/core/runtime/harness/harness.go | 2 +-
.../core/runtime/xlangx/expansionx/download.go | 20 +
sdks/go/pkg/beam/forward.go | 5 +
sdks/go/pkg/beam/log/log.go | 61 +-
.../go/pkg/beam/log/{standard.go => structural.go} | 26 +-
sdks/go/pkg/beam/runners/prism/internal/coders.go | 6 +-
.../prism/internal/engine/elementmanager.go | 16 +-
.../runners/prism/internal/engine/teststream.go | 7 +
sdks/go/pkg/beam/runners/prism/internal/execute.go | 81 +-
.../beam/runners/prism/internal/handlerunner.go | 109 +
sdks/go/pkg/beam/runners/prism/internal/stage.go | 5 +-
.../runners/prism/internal/unimplemented_test.go | 3 +
.../beam/runners/prism/internal/worker/worker.go | 8 +-
.../go/pkg/beam/runners/universal/runnerlib/job.go | 12 +-
sdks/go/pkg/beam/runners/universal/universal.go | 3 +-
sdks/go/pkg/beam/x/debug/print_test.go | 9 +-
sdks/go/test/integration/integration.go | 6 +-
sdks/go/test/integration/primitives/timers.go | 36 +-
sdks/go/test/integration/primitives/timers_test.go | 5 +
.../resources/beam/checkstyle/suppressions.xml | 1 +
.../beam/sdk/fn/stream/PrefetchableIterables.java | 2 +-
.../apache/beam/sdk/schemas/AutoValueSchema.java | 2 +-
.../transforms/providers/ErrorHandling.java | 2 +
.../beam/sdk/schemas/utils/AutoValueUtils.java | 101 +-
.../java/org/apache/beam/sdk/transforms/DoFn.java | 31 -
.../org/apache/beam/sdk/transforms/DoFnTester.java | 64 -
.../beam/sdk/transforms/GroupByEncryptedKey.java | 30 +-
.../apache/beam/sdk/transforms/Redistribute.java | 1 +
.../java/org/apache/beam/sdk/transforms/Reify.java | 1 +
.../org/apache/beam/sdk/transforms/Reshuffle.java | 1 +
.../sdk/transforms/windowing/IntervalWindow.java | 17 +-
.../beam/sdk/transforms/windowing/PaneInfo.java | 58 +-
.../beam/sdk/util/construction/Environments.java | 26 +
.../construction/SplittableParDoNaiveBounded.java | 48 -
.../org/apache/beam/sdk/values/OutputBuilder.java | 2 +
.../org/apache/beam/sdk/values/TypeDescriptor.java | 5 +
.../beam/sdk/values/ValueInSingleWindow.java | 21 +-
.../org/apache/beam/sdk/values/WindowedValue.java | 2 +
.../org/apache/beam/sdk/values/WindowedValues.java | 171 +-
.../beam/sdk/schemas/utils/AutoValueUtilsTest.java | 166 ++
.../sdk/transforms/GroupByEncryptedKeyTest.java | 9 +-
.../apache/beam/sdk/transforms/GroupByKeyTest.java | 106 +-
.../org/apache/beam/sdk/transforms/ParDoTest.java | 67 +
.../sdk/transforms/windowing/PaneInfoTest.java | 20 +
.../apache/beam/sdk/util/WindowedValueTest.java | 30 +
.../sdk/util/construction/EnvironmentsTest.java | 3 +
.../util/construction/ValidateRunnerXlangTest.java | 129 +
.../container/expansion_service_config.yml | 30 +-
.../sdk/expansion/service/ExpansionService.java | 14 +
.../expansion/service/ExpansionServiceOptions.java | 7 +
.../sdk/expansion/service/TransformProvider.java | 55 +-
...xpansionServiceSchemaTransformProviderTest.java | 90 +-
.../expansion/service/ExpansionServiceTest.java | 2 +-
.../resources/test_expansion_service_config.yaml | 3 +
.../extensions/gcp/auth/GcpCredentialFactory.java | 5 +
.../python/PythonExternalTransformTest.java | 65 +-
.../sdk/extensions/sql/impl/JavaUdfLoader.java | 14 +
.../apache/beam/fn/harness/FnApiDoFnRunner.java | 171 +-
.../fn/harness/control/ProcessBundleHandler.java | 14 +-
.../harness/data/PCollectionConsumerRegistry.java | 79 +-
.../beam/fn/harness/state/FnApiStateAccessor.java | 121 +-
.../beam/fn/harness/state/MultimapUserState.java | 139 +-
.../harness/control/ProcessBundleHandlerTest.java | 149 ++
.../fn/harness/state/MultimapUserStateTest.java | 129 +
.../beam/sdk/io/gcp/bigquery/AppendClientInfo.java | 12 +
.../beam/sdk/io/gcp/bigquery/BatchLoads.java | 7 +-
.../sdk/io/gcp/bigquery/SplittingIterable.java | 19 +-
.../bigquery/StorageApiWriteUnshardedRecords.java | 9 +-
.../bigquery/StorageApiWritesShardedRecords.java | 4 +-
.../io/gcp/bigquery/TableRowToStorageApiProto.java | 154 ++
.../sdk/io/gcp/bigquery/BigQueryIOWriteTest.java | 2 +
.../bigquery/TableRowToStorageApiProtoTest.java | 138 +
.../it/SpannerChangeStreamPlacementTableIT.java | 16 +-
.../apache/beam/sdk/io/iceberg/RecordWriter.java | 10 +-
.../java/org/apache/beam/sdk/io/kafka/KafkaIO.java | 40 +-
.../KafkaReadSchemaTransformConfiguration.java | 18 +
.../beam/sdk/io/kafka/KafkaSourceDescriptor.java | 8 +
.../kafka/KafkaWriteSchemaTransformProvider.java | 9 +
.../org/apache/beam/sdk/io/kafka/KafkaIOTest.java | 59 +
.../KafkaReadSchemaTransformProviderTest.java | 112 +
.../KafkaWriteSchemaTransformProviderTest.java | 66 +
sdks/java/io/xml/build.gradle | 4 +-
.../java/org/apache/beam/sdk/io/xml/JAXBCoder.java | 28 +-
sdks/python/apache_beam/coders/coders.py | 16 +-
.../cookbook/ordered_window_elements}/__init__.py | 0
.../cookbook/ordered_window_elements/batch.py | 522 ++++
.../cookbook/ordered_window_elements/batch_test.py | 333 +++
.../cookbook/ordered_window_elements/streaming.py | 625 +++++
.../ordered_window_elements/streaming_test.py | 359 +++
...lassification.py => gemini_image_generation.py} | 49 +-
.../inference/gemini_text_classification.py | 12 +-
.../snippets/transforms/elementwise/enrichment.py | 77 +-
.../transforms/elementwise/enrichment_test.py | 171 +-
.../internal/cloudpickle/cloudpickle.py | 78 +
.../apache_beam/internal/cloudpickle_pickler.py | 61 +-
.../apache_beam/internal/code_object_pickler.py | 90 +-
.../internal/code_object_pickler_test.py | 27 +-
sdks/python/apache_beam/internal/dill_pickler.py | 42 +-
sdks/python/apache_beam/internal/metrics/cells.py | 165 --
.../apache_beam/internal/metrics/cells_test.py | 77 -
sdks/python/apache_beam/internal/metrics/metric.py | 44 -
.../apache_beam/internal/metrics/metric_test.py | 130 -
sdks/python/apache_beam/internal/module_test.py | 7 +
sdks/python/apache_beam/internal/pickler.py | 19 +-
sdks/python/apache_beam/internal/pickler_test.py | 44 +
sdks/python/apache_beam/io/filebasedsink.py | 14 +
sdks/python/apache_beam/io/filebasedsource.py | 4 +-
sdks/python/apache_beam/io/gcp/bigquery.py | 50 +-
.../apache_beam/io/gcp/bigquery_file_loads_test.py | 28 +-
.../io/gcp/bigquery_geography_it_test.py | 544 ++++
.../apache_beam/io/gcp/bigquery_schema_tools.py | 3 +-
.../io/gcp/bigquery_schema_tools_test.py | 134 +-
sdks/python/apache_beam/io/gcp/bigquery_tools.py | 15 +-
.../apache_beam/io/gcp/bigquery_tools_test.py | 154 ++
.../io/gcp/experimental/spannerio_test.py | 35 +-
sdks/python/apache_beam/io/gcp/gcsio.py | 2 +-
sdks/python/apache_beam/io/gcp/spanner.py | 56 +-
sdks/python/apache_beam/metrics/cells.py | 131 +
sdks/python/apache_beam/metrics/cells_test.py | 51 +
sdks/python/apache_beam/metrics/execution.py | 6 +-
sdks/python/apache_beam/metrics/metric.py | 44 +
sdks/python/apache_beam/metrics/metric_test.py | 38 +-
.../python/apache_beam/metrics/monitoring_infos.py | 2 +-
.../apache_beam/metrics/monitoring_infos_test.py | 4 +-
.../apache_beam/ml/inference/gemini_inference.py | 42 +-
.../ml/inference/gemini_inference_it_test.py | 25 +
.../ml/inference/gemini_tests_requirements.txt | 3 +-
.../ml/inference/huggingface_inference.py | 12 +-
.../ml/inference/huggingface_inference_test.py | 34 +-
.../ml/inference/test_resources/vllm.dockerfile | 14 +-
.../ml/inference/vllm_tests_requirements.txt | 2 +-
.../apache_beam/ml/rag/enrichment/milvus_search.py | 49 +-
.../ml/rag/enrichment/milvus_search_it_test.py | 27 +-
.../python/apache_beam/options/pipeline_options.py | 51 +-
.../apache_beam/options/pipeline_options_test.py | 21 +
.../runners/dataflow/dataflow_runner.py | 14 +-
.../apache_beam/runners/dataflow/internal/names.py | 2 +-
.../runners/direct/transform_evaluator.py | 2 +-
.../runners/interactive/interactive_beam.py | 30 +-
.../runners/interactive/interactive_beam_test.py | 85 +
.../python/apache_beam/runners/pipeline_context.py | 1 +
.../runners/portability/expansion_service.py | 12 +-
.../runners/portability/local_job_service.py | 1 -
.../runners/portability/portable_runner.py | 14 +-
.../runners/portability/prism_runner.py | 18 +-
.../apache_beam/runners/worker/bundle_processor.py | 27 +-
.../apache_beam/runners/worker/data_plane.py | 20 +-
.../apache_beam/runners/worker/sdk_worker.py | 19 +-
.../apache_beam/runners/worker/sdk_worker_main.py | 4 +
.../apache_beam/runners/worker/sdk_worker_test.py | 44 +-
.../apache_beam/runners/worker/worker_status.py | 118 +-
.../runners/worker/worker_status_test.py | 96 +-
.../benchmarks/inference/vllm_gemma_benchmarks.py | 2 +-
.../benchmarks/nexmark/models/auction_bid.py | 3 +-
.../benchmarks/nexmark/models/nexmark_model.py | 27 +-
.../testing/benchmarks/nexmark/nexmark_util.py | 18 -
sdks/python/apache_beam/transforms/combiners.py | 1 +
sdks/python/apache_beam/transforms/core.py | 4 +-
sdks/python/apache_beam/transforms/core_it_test.py | 43 +-
.../python/apache_beam/transforms/external_test.py | 22 +-
.../apache_beam/transforms/periodicsequence.py | 3 +-
sdks/python/apache_beam/transforms/ptransform.py | 8 +-
sdks/python/apache_beam/transforms/util.py | 24 +-
sdks/python/apache_beam/transforms/util_test.py | 64 +-
.../transforms/validate_runner_xlang_test.py | 96 +
sdks/python/apache_beam/transforms/window_test.py | 13 +
sdks/python/apache_beam/utils/logger.py | 137 +
sdks/python/apache_beam/utils/logger_test.py | 108 +
sdks/python/apache_beam/utils/subprocess_server.py | 26 +-
sdks/python/apache_beam/version.py | 2 +-
sdks/python/apache_beam/yaml/standard_io.yaml | 2 +-
sdks/python/apache_beam/yaml/yaml_provider.py | 26 +
.../apache_beam/yaml/yaml_specifiable_test.py | 7 +-
sdks/python/container/Dockerfile | 2 +-
sdks/python/container/build.gradle | 8 +-
sdks/python/container/common.gradle | 2 +-
.../container/license_scripts/dep_urls_py.yaml | 4 +
sdks/python/container/ml/common.gradle | 2 +-
.../ml/{py313 => py310}/ml_image_requirements.txt | 135 +-
.../ml/{py313 => py311}/ml_image_requirements.txt | 128 +-
.../ml/{py313 => py312}/ml_image_requirements.txt | 127 +-
.../{ => ml}/py313/base_image_requirements.txt | 78 +-
.../container/ml/py313/ml_image_requirements.txt | 117 +-
.../py39/ml_image_requirements.txt} | 116 +-
.../container/py310/base_image_requirements.txt | 74 +-
.../container/py311/base_image_requirements.txt | 72 +-
.../container/py312/base_image_requirements.txt | 72 +-
.../container/py313/base_image_requirements.txt | 78 +-
.../container/py39/base_image_requirements.txt | 68 +-
sdks/python/container/run_generate_requirements.sh | 5 +-
sdks/python/container/run_validatescontainer.sh | 16 +-
sdks/python/setup.py | 9 +-
sdks/python/tox.ini | 6 +-
sdks/typescript/package.json | 2 +-
website/www/site/config.toml | 2 +-
website/www/site/content/en/blog/beam-2.69.0.md | 86 +
.../en/blog/gsoc-25-jupyterlab-extensions.md | 74 +
.../en/documentation/dsls/sql/calcite/overview.md | 27 +
.../dsls/sql/calcite/scalar-functions.md | 9 +-
.../content/en/documentation/dsls/sql/shell.md | 114 +-
.../en/documentation/dsls/sql/zetasql/overview.md | 2 +-
.../site/content/en/documentation/io/managed-io.md | 1635 ++++++++++--
.../www/site/content/en/documentation/sdks/java.md | 29 +
.../content/en/documentation/sdks/java/euphoria.md | 2 +
.../site/content/en/documentation/sdks/python.md | 51 +-
.../python/elementwise/enrichment-cloudsql.md | 4 +-
.../python/elementwise/enrichment-milvus.md | 65 +
.../transforms/python/elementwise/enrichment.md | 3 +-
.../www/site/content/en/get-started/downloads.md | 18 +-
website/www/site/content/en/roadmap/_index.md | 19 +-
.../www/site/content/en/security/CVE-2020-1929.md | 16 -
website/www/site/data/authors.yml | 3 +
website/www/site/data/performance.yaml | 10 +-
.../partials/section-menu/en/documentation.html | 1 +
.../gsoc-25-jupyterlab-extensions/Yaml_main.png | Bin 0 -> 305413 bytes
291 files changed, 14197 insertions(+), 3387 deletions(-)
copy .github/trigger_files/{beam_PostCommit_Java_PVR_Flink_Streaming.json =>
beam_PostCommit_Go_VR_Spark.json} (53%)
copy .github/trigger_files/{beam_CloudML_Benchmarks_Dataflow.json =>
beam_PostCommit_XVR_JavaUsingPython_Dataflow.json} (98%)
copy .github/trigger_files/{beam_CloudML_Benchmarks_Dataflow.json =>
beam_PostCommit_XVR_PythonUsingJava_Dataflow.json} (100%)
copy .github/trigger_files/{beam_PostCommit_Java_Examples_Dataflow_ARM.json =>
beam_PostCommit_XVR_Spark3.json} (100%)
copy .github/trigger_files/{beam_PreCommit_Flink_Container.json =>
beam_PreCommit_Python_Dill.json} (80%)
delete mode 100644
.github/workflows/load-tests-pipeline-options/python_CoGBK_Dataflow_Flink_Batch_100b_Single_Key.txt
delete mode 100644
.github/workflows/load-tests-pipeline-options/python_CoGBK_Dataflow_Flink_Batch_10kB.txt
rename
.github/workflows/load-tests-pipeline-options/{python_CoGBK_Dataflow_Flink_Batch_100b_Multiple_Keys.txt
=> python_CoGBK_Flink_Batch_100b_Multiple_Keys.txt} (74%)
copy
.github/workflows/load-tests-pipeline-options/{python_Combine_Flink_Streaming_small_Fanout_1.txt
=> python_CoGBK_Flink_Batch_100b_Single_Key.txt} (77%)
copy
.github/workflows/load-tests-pipeline-options/{python_Combine_Flink_Streaming_small_Fanout_1.txt
=> python_CoGBK_Flink_Batch_10kB.txt} (76%)
create mode 100644 examples/notebooks/beam-ml/milvus_enrichment_transform.ipynb
create mode 100644
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/InternedByteString.java
create mode 100644
runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/util/common/worker/InternedByteStringTest.java
create mode 100755 scripts/beam-sql.sh
copy sdks/go/pkg/beam/log/{standard.go => structural.go} (64%)
create mode 100644
sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/utils/AutoValueUtilsTest.java
copy {examples/notebooks/notebook_test_scripts =>
sdks/python/apache_beam/examples/cookbook/ordered_window_elements}/__init__.py
(100%)
create mode 100644
sdks/python/apache_beam/examples/cookbook/ordered_window_elements/batch.py
create mode 100644
sdks/python/apache_beam/examples/cookbook/ordered_window_elements/batch_test.py
create mode 100644
sdks/python/apache_beam/examples/cookbook/ordered_window_elements/streaming.py
create mode 100644
sdks/python/apache_beam/examples/cookbook/ordered_window_elements/streaming_test.py
copy sdks/python/apache_beam/examples/inference/{gemini_text_classification.py
=> gemini_image_generation.py} (72%)
delete mode 100644 sdks/python/apache_beam/internal/metrics/cells.py
delete mode 100644 sdks/python/apache_beam/internal/metrics/cells_test.py
delete mode 100644 sdks/python/apache_beam/internal/metrics/metric_test.py
create mode 100644 sdks/python/apache_beam/io/gcp/bigquery_geography_it_test.py
create mode 100644 sdks/python/apache_beam/utils/logger.py
create mode 100644 sdks/python/apache_beam/utils/logger_test.py
copy sdks/python/container/ml/{py313 => py310}/ml_image_requirements.txt (70%)
copy sdks/python/container/ml/{py313 => py311}/ml_image_requirements.txt (71%)
copy sdks/python/container/ml/{py313 => py312}/ml_image_requirements.txt (71%)
copy sdks/python/container/{ => ml}/py313/base_image_requirements.txt (81%)
copy sdks/python/container/{py39/base_image_requirements.txt =>
ml/py39/ml_image_requirements.txt} (70%)
create mode 100644 website/www/site/content/en/blog/beam-2.69.0.md
create mode 100644
website/www/site/content/en/blog/gsoc-25-jupyterlab-extensions.md
create mode 100644
website/www/site/content/en/documentation/transforms/python/elementwise/enrichment-milvus.md
delete mode 100644 website/www/site/content/en/security/CVE-2020-1929.md
create mode 100644
website/www/site/static/images/blog/gsoc-25-jupyterlab-extensions/Yaml_main.png
