This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a change to branch
dependabot/pip/sdks/python/dill-gte-0.3.1.1-and-lt-0.4.1
in repository https://gitbox.apache.org/repos/asf/beam.git
discard 6cc13c0ef16 Update dill requirement in /sdks/python
add e410e34b067 Use consistent encoding for GBEK across languages (#36431)
add f973a4ed6c2 Add readme How to add a new ML benchmark pipeline
add 1b25848f658 Resolve comments
add 702d73ea50a Merge pull request #36437 from
apache/inference-benchmark-readme
add b5a0495e55b CombinePerKey with gbek (Java) (#36408)
add e3293e289e4 Minor changes on Managed JDBCIO (#36339)
add 67d469f43c1 Bump golang.org/x/sys from 0.36.0 to 0.37.0 in /sdks
(#36447)
add ff5eeeaf4b5 Bump golang.org/x/oauth2 from 0.30.0 to 0.31.0 in /sdks
(#36205)
add 4ff5fe7b5c8 Bump cloud.google.com/go/spanner from 1.85.1 to 1.86.0 in
/sdks (#36449)
add 116141a1f10 Bump gradle/wrapper-validation-action from 1.0.6 to 3.5.0
(#36278)
add ac4b5ab2aac Bump google.golang.org/api from 0.249.0 to 0.252.0 in
/sdks (#36450)
add 7e8ca06a1c9 Get viewer permissions for ksobrenat32 (#36458)
add 1e7167b6e6e Fix passing pipeline options to external transforms
(#36443)
add 6f31e56fcac Implement the member_type on the users.yml (#36460)
add 08c96f2c6e2 Fix XVR JavaUsingPython tests using dev Beam at expansion
(#36444)
add b2e123870f5 feat(pipeline_options): add support for custom maven
repository url (#36390)
add c5a61896433 [Dataflow Streaming] Fix outstanding bundle metric
reporting (#36455)
add 65ee22518ce [Dataflow Streaming] Move functionality creating windmill
tags to a common class. (#36283)
add b9f07c9dd29 [Spanner Change Streams] Ensure the partition watermark is
monotonic by reading within the transaction (#36463)
add b79d92fe01b Bump github.com/docker/docker in /sdks (#36468)
add 6683a1ae838 Bump golang.org/x/oauth2 from 0.31.0 to 0.32.0 in /sdks
(#36467)
add 893fc9a9b8f Update Go version to 1.25.2 (#36461)
add d4438b63e18 Enforce deterministic field order in Schema generated from
KafkaIO classes. (#36295)
add 590ece2cd8e Fix Python CoGBK Flink Batch config
add 62df216296a Merge pull request #36472 from apache/fix-flink-cogbk
add 227a6323ead Add logger helper functions from detectron2 (licensed as
Apache 2.0) (#36432)
add fb8058454c0 Refactor GBEK tests to split out secret setup for tests
that dont need it (#36479)
add bf4bf81922f Update CHANGES.md to show issue #36470 is fixed in 2.69.0
release. (#36477)
add a5846889716 [Dataflow Streaming] Change GrpcGetDataStream to backoff
requests that have been cancelled. (#36475)
add 6be76ae8542 [Dataflow Streaming] Enforce that get data requests for
the same work item are not batched. (#36474)
add 7735e7a3bd1 Bump google.golang.org/grpc from 1.75.1 to 1.76.0 in /sdks
(#36488)
add 2a77c59b82e Bump golang.org/x/text from 0.29.0 to 0.30.0 in /sdks
(#36487)
add 090b17bf535 added missing schemaFieldNumber annotation (#36489)
add 041e12edd9d feat: add warnings for public repository downloads in
multiple SDKs (#36476)
add 08b480000ec add generics support to AutoValueUtils helpers (#32977)
add c54cc2b6ed9 Add flag for disabling dill check in coders. (#36453)
add 1c6f779bdf2 Add use_gbek service option when gbek option used (#36452)
add d54a661f47e Bump golang.org/x/net from 0.45.0 to 0.46.0 in /sdks
(#36466)
add b9a89722d9e External metadata for streaming runner v1 changes (#36373)
add 243a52c5b42 Move tests running pipeline into a separate class for
PythonExternalTransformTest (#36492)
add b5b91810b76 Move the logic to LP TestStream encoded bytes to
preprocess steps. (#36465)
add 554a73b4bf3 Enable real-time clock in prism by default. (#36473)
add 2b9827b6c2d Refactor prism and go sdk logging and clean up messages
(#36484)
add 6dbbaa687fe Fix Credentials issue while commit (#36494)
add 385271bea45 Softens the GBEK determinism requirement (#36495)
add 72557e58a06 Only run Py39 and Py313 tests for PostCommit Arm (#36508)
add 91f79c3a97f Add GroupByEncryptedKey to changes (#36510)
add 673309b328c Call out OutputBuilder change in CHANGES.md (#36511)
add 37c7e28231b ci(python-deps): update transformers version constraints
in tox (#36506)
add 75eda20a901 Revert "Per element schema parsing in ConvertToBeamRows
(#36393)" (#36507)
add 7a9a4e6afa5 x-lang GroupByEncryptedKey (Java to Python) (#36418)
add 95dcaeac932 docs: Expose ReadChangeStreamFromSpanner in Beam Spanner
documentation (#36428)
add e94579a3e6c Enhance JAXBCoder with XMLInputFactory support (#36446)
add 7b34ab75c46 Add some x-lang gbek tests (Python to Java) (#36457)
add ed39cbbf709 beam-sql.sh, a standalone launcher for Beam SQL Shell
(#36305)
add 12a34c8acf8 Change default timeout and add heartbeat logging (#36517)
add 4add79cab24 Updates ExpansionService so that managed transforms can
use specific dependencies during expansion. Behavior is guarded by an pipeline
option.
add 50f578aee32 Merge pull request #36515: Updates ExpansionService so
that managed transforms can use specific dependencies during expansion
add 9ed06d081ec Handle null keys in gbek (#36505)
add c703b7227de Bump github.com/nats-io/nats.go from 1.46.0 to 1.47.0 in
/sdks (#36521)
add 99ee1738e2b Add a flag to control whether to allow splitting on sdf.
(#36512)
add d687f4fe817 Add GRPC experiments to Python dockerfile (#36525)
add f0c92c7a772 revert outputWindowedValue changes as there is
outputBuilder
add abf1904759e Merge pull request #36523: revert outputWindowedValue
changes from KafkaIO as there is outputBuilder
add ed39503878e Skip TestTimers_ProcessingTime_Unbounded for spark.
(#36527)
add 6ad53078c48 Moving to 2.70.0-SNAPSHOT on master branch.
add f7619c789d0 Update CHANGES.md to have fields for 2.70.0 release
add 6562b5b677d Update CHANGES.md to mention breakign change around
ProcessContext (#36530)
add 30fd958f5fc feat(bigquery): add GEOGRAPHY type support for BigQuery
I/O (#36121)
add b6878702484 Fix flaky tests caused by secret overlap (#36526)
add 96e79cba3a6 Concat protos in BQStorageWriteAPI - solve edge cases
during mering of nested repeated fields (#34436)
add 19fef1bba24 add changes comment on yaml output_schema (#36497)
add 57e34b6906b Fix proto map access. (#36532)
add faae168fa34 Bump github.com/aws/smithy-go from 1.23.0 to 1.23.1 in
/sdks (#36533)
add 118b3c7a582 PortableRunner tests: surface worker-thread exceptions on
main thread after wait_until_finish() (fixes #35211) (#36485)
add 5d420c5f047 Add pickler.roundtrip() shortcut for testing pickle
(#36441)
add 2b43f8018ba Pin specifiable test to FnApiRunner (#36536)
add f07ccf37cbe Track bundle processors that are pending creation and
terminate SDK if creating a BP exceeds a timeout. (#36518)
add 581ec8bb17f Always mark the instruction as cleaned up in the GRPC data
channel when processing an instruction fails. (#36367)
add e87f8097e53 Move setup/teardown to class level to avoid flakiness
(#36546)
add 6ffc68778b9 Fix build release candidate workflow (#36541)
add d91fb6d6987 Timeout execution tree creation for SDK worker ops.
(#36200)
add d4dc3243303 Fix dill tox (#36543)
add 34a6f542a7f Update beam_PreCommit_Python_ML.yml (#36550)
add 87db35637a6 Add "return []" to PGBK to silence warning (#36535)
add 2b666dacf47 test(bigquery): skip geography test when expansion jars
not available (#36555)
add d4b841caa94 Update changes.md with pickler changes. (#36558)
add 9030ba8074c test(bigquery): mock client in geography type support
tests (#36559)
add af748d07a1a Update Python Dependencies (#36560)
add 1bf56295bc9 Fix publishing of ml/distroless images (#36548)
add 07b321e5811 Fix unsafe container cleanup that could delete images from
other runs (#36547)
add e081879a78f Fix BigQueryIO File load validate runtime value provider
(#36564)
add ee48e713282 [3/3] sdks/python: enrich data with Milvus Search [Vector,
Keyword, Hybrid] (#35467)
add f8901e3a4c5 Update the release notes. (#36566)
add 15e8f98fed6 Fix dependency version (#36568)
add d0d0cd8c2f1 Revert "Add GRPC experiments to Python dockerfile
(#36525)" (#36572)
add 944eef91344 Upload beam blog. (#36499)
add c22665c5111 Call now() once so start and end have exactly the same
base timestamp. (#36574)
add 0d52be60e88 Add image generation code to Gemini Model Handler (#36177)
add 179d4d1ee9c Fix flaky tests (#36579)
add 7e7d866d95a [python] add setup to BigQuery's convert row Map transform
(#36502)
add c7d920f26cf Update dev image. (#36582)
add 243d4077319 Integrate lambda name pickling with Cloudpickle (#35904)
add ab892e3dd09 Add logging for credential retrieval failures in
GcpCredentialFactory
add db92a3ad0c0 Merge pull request #36415: Add logging for credential
retrieval failures in GcpCredentialFactory
add b83c24e4d45 test(spannerio): make batch size validation more flexible
for non-deterministic execution (#36584)
add 4c08585626b [IcebergIO] Pass table props to data writers (#36542)
add 0ebf84b6b18 Add ib.collect support for raw records (#36516)
add 38481b58879 Address circular dependencies in Nexmark benchmark suite.
(#36513)
add 8bd92b5e376 Fix the soft-delete check and emit soft-delete log warning
at most once per bucket. (#36585)
add afeca4ea301 Increase timeouts (#36595)
add 66b7c7476ce Make SpannerChangeStreamPlacementTableIT against Spanner
prod. (#36071)
add fdfa6ec6338 Exclude a perma-red test suite
beam_PostCommit_XVR_GoUsingJava_Dataflow.yml (#36597)
add 6dedf8f0bab Update REVIEWERS.yml (#36598)
add fe71ab1b47b Add ordered window elements into example folder (#36575)
add ef07e40667b Fix proposal link (#36600)
add f2860fa2fe8 use utils._convert_to_result for huggingface_inference
(#36593)
add 05f6f01a33b Force torch to use cpu wheels (#36583)
add b6d8b7dea69 Update dill requirement in /sdks/python
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (6cc13c0ef16)
\
N -- N -- N
refs/heads/dependabot/pip/sdks/python/dill-gte-0.3.1.1-and-lt-0.4.1
(b6d8b7dea69)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.asf.yaml | 1 +
.github/REVIEWERS.yml | 2 -
.../test-properties.json | 2 +-
.../actions/setup-environment-action/action.yml | 2 +-
.../arc/images/Dockerfile | 4 +-
.../IO_Iceberg_Integration_Tests.json | 2 +-
...aming.json => beam_PostCommit_Go_VR_Spark.json} | 2 +-
.github/trigger_files/beam_PostCommit_Java.json | 2 +-
.../beam_PostCommit_Java_DataflowV1.json | 2 +-
.../beam_PostCommit_Java_DataflowV2.json | 2 +-
.../beam_PostCommit_Java_PVR_Spark3_Streaming.json | 2 +-
.github/trigger_files/beam_PostCommit_Python.json | 2 +-
.../beam_PostCommit_Python_Dependency.json | 2 +-
...m_PostCommit_XVR_JavaUsingPython_Dataflow.json} | 2 +-
...m_PostCommit_XVR_PythonUsingJava_Dataflow.json} | 0
.../trigger_files/beam_PostCommit_XVR_Samza.json | 2 +-
...ow_ARM.json => beam_PostCommit_XVR_Spark3.json} | 0
...tainer.json => beam_PreCommit_Python_Dill.json} | 2 +-
.../beam_LoadTests_Python_CoGBK_Flink_Batch.yml | 17 +-
.github/workflows/beam_PostCommit_Python_Arm.yml | 13 +-
.../beam_PostCommit_Python_Dependency.yml | 2 +-
.../beam_PostCommit_XVR_GoUsingJava_Dataflow.yml | 6 +-
.github/workflows/beam_PreCommit_Python_Dill.yml | 6 +-
.github/workflows/beam_PreCommit_Python_ML.yml | 12 +
.../workflows/beam_Publish_Beam_SDK_Snapshots.yml | 5 +
.../workflows/beam_Publish_Docker_Snapshots.yml | 2 +-
.github/workflows/build_release_candidate.yml | 37 +-
.github/workflows/code_completion_plugin_tests.yml | 2 +-
..._CoGBK_Dataflow_Flink_Batch_100b_Single_Key.txt | 28 -
.../python_CoGBK_Dataflow_Flink_Batch_10kB.txt | 28 -
...ython_CoGBK_Flink_Batch_100b_Multiple_Keys.txt} | 10 +-
...> python_CoGBK_Flink_Batch_100b_Single_Key.txt} | 17 +-
...out_1.txt => python_CoGBK_Flink_Batch_10kB.txt} | 17 +-
.../republish_released_docker_containers.yml | 17 +-
.../workflows/run_rc_validation_python_yaml.yml | 10 +-
CHANGES.md | 71 +-
.../org/apache/beam/gradle/BeamModulePlugin.groovy | 2 +-
dev-support/docker/Dockerfile | 2 +-
.../beam-ml/milvus_enrichment_transform.ipynb | 2657 ++++++++++++++++++++
gradle.properties | 4 +-
infra/enforcement/iam.py | 4 +
infra/iam/README.md | 1 +
infra/iam/users.tf | 3 +-
infra/iam/users.yml | 198 +-
.../cloudbuild/playground_ci_examples.sh | 2 +-
release/src/main/scripts/set_version.sh | 3 +
...TimeBoundedSplittableProcessElementInvoker.java | 35 +-
.../apache/beam/runners/core/SimpleDoFnRunner.java | 130 -
.../core/SplittableParDoViaKeyedWorkItems.java | 21 -
runners/flink/job-server/flink_job_server.gradle | 1 +
runners/google-cloud-dataflow-java/build.gradle | 1 +
.../beam/runners/dataflow/DataflowRunner.java | 16 +
.../dataflow/RedistributeByKeyOverrideFactory.java | 1 +
.../dataflow/worker/StreamingDataflowWorker.java | 5 +
.../worker/StreamingModeExecutionContext.java | 4 +
.../dataflow/worker/UngroupedWindmillReader.java | 5 +
.../dataflow/worker/WindmillKeyedWorkItem.java | 5 +-
.../dataflow/worker/WindmillNamespacePrefix.java | 10 +-
.../beam/runners/dataflow/worker/WindmillSink.java | 46 +-
.../dataflow/worker/WindmillTimerInternals.java | 49 +-
.../harness/StreamingWorkerStatusReporter.java | 2 +-
.../windmill/client/grpc/GrpcGetDataStream.java | 39 +-
.../client/grpc/GrpcGetDataStreamRequests.java | 211 +-
.../worker/windmill/state/CachingStateTable.java | 61 +-
.../worker/windmill/state/WindmillBag.java | 5 +-
.../windmill/state/WindmillCombiningState.java | 16 +-
.../worker/windmill/state/WindmillMap.java | 7 +-
.../worker/windmill/state/WindmillMultimap.java | 7 +-
.../worker/windmill/state/WindmillOrderedList.java | 7 +-
.../windmill/state/WindmillStateInternals.java | 4 +-
...illStateUtil.java => WindmillStateTagUtil.java} | 57 +-
.../worker/windmill/state/WindmillValue.java | 7 +-
.../windmill/state/WindmillWatermarkHold.java | 7 +-
.../worker/StreamingGroupAlsoByWindowFnsTest.java | 8 +-
...reamingGroupAlsoByWindowsReshuffleDoFnTest.java | 8 +-
.../dataflow/worker/WindmillKeyedWorkItemTest.java | 9 +-
.../client/grpc/GrpcGetDataStreamRequestsTest.java | 160 +-
.../client/grpc/GrpcGetDataStreamTest.java | 51 +
.../windmill/state/WindmillStateInternalsTest.java | 3 +
...UtilTest.java => WindmillStateTagUtilTest.java} | 10 +-
runners/samza/job-server/build.gradle | 1 +
runners/spark/job-server/spark_job_server.gradle | 1 +
scripts/beam-sql.sh | 448 ++++
sdks/go.mod | 39 +-
sdks/go.sum | 74 +-
sdks/go/cmd/prism/prism.go | 50 +-
sdks/go/pkg/beam/core/core.go | 2 +-
sdks/go/pkg/beam/core/runtime/harness/harness.go | 2 +-
.../core/runtime/xlangx/expansionx/download.go | 20 +
sdks/go/pkg/beam/forward.go | 5 +
sdks/go/pkg/beam/log/log.go | 61 +-
.../go/pkg/beam/log/{standard.go => structural.go} | 26 +-
sdks/go/pkg/beam/runners/prism/internal/coders.go | 6 +-
.../prism/internal/engine/elementmanager.go | 10 +-
.../runners/prism/internal/engine/teststream.go | 7 +
sdks/go/pkg/beam/runners/prism/internal/execute.go | 81 +-
.../beam/runners/prism/internal/handlerunner.go | 109 +
sdks/go/pkg/beam/runners/prism/internal/stage.go | 5 +-
.../runners/prism/internal/unimplemented_test.go | 3 +
.../beam/runners/prism/internal/worker/worker.go | 8 +-
.../go/pkg/beam/runners/universal/runnerlib/job.go | 12 +-
sdks/go/pkg/beam/runners/universal/universal.go | 3 +-
sdks/go/pkg/beam/x/debug/print_test.go | 9 +-
sdks/go/run_with_go_version.sh | 2 +-
sdks/go/test/integration/integration.go | 6 +-
sdks/go/test/integration/primitives/timers.go | 36 +-
sdks/go/test/integration/primitives/timers_test.go | 5 +
.../resources/beam/checkstyle/suppressions.xml | 1 +
.../apache/beam/sdk/options/PipelineOptions.java | 11 +-
.../apache/beam/sdk/schemas/AutoValueSchema.java | 2 +-
.../transforms/providers/ErrorHandling.java | 2 +
.../beam/sdk/schemas/utils/AutoValueUtils.java | 101 +-
.../org/apache/beam/sdk/transforms/Combine.java | 19 +
.../java/org/apache/beam/sdk/transforms/DoFn.java | 31 -
.../org/apache/beam/sdk/transforms/DoFnTester.java | 64 -
.../beam/sdk/transforms/GroupByEncryptedKey.java | 40 +-
.../apache/beam/sdk/transforms/Redistribute.java | 1 +
.../java/org/apache/beam/sdk/transforms/Reify.java | 1 +
.../org/apache/beam/sdk/transforms/Reshuffle.java | 1 +
.../beam/sdk/transforms/windowing/PaneInfo.java | 58 +-
.../sdk/util/construction/CombineTranslation.java | 17 +-
.../beam/sdk/util/construction/Environments.java | 25 +
.../beam/sdk/util/construction/External.java | 2 +-
.../construction/SplittableParDoNaiveBounded.java | 48 -
.../org/apache/beam/sdk/values/TypeDescriptor.java | 5 +
.../beam/sdk/values/ValueInSingleWindow.java | 21 +-
.../org/apache/beam/sdk/values/WindowedValues.java | 24 +-
.../beam/sdk/schemas/utils/AutoValueUtilsTest.java | 166 ++
.../sdk/transforms/GroupByEncryptedKeyTest.java | 16 +-
.../apache/beam/sdk/transforms/GroupByKeyIT.java | 47 +-
.../apache/beam/sdk/transforms/GroupByKeyTest.java | 104 +-
.../sdk/transforms/windowing/PaneInfoTest.java | 20 +
.../apache/beam/sdk/util/WindowedValueTest.java | 26 +
.../util/construction/ValidateRunnerXlangTest.java | 129 +
sdks/java/expansion-service/container/Dockerfile | 1 -
.../container/expansion_service_config.yml | 30 +-
.../sdk/expansion/service/ExpansionService.java | 14 +
.../expansion/service/ExpansionServiceOptions.java | 7 +
.../sdk/expansion/service/TransformProvider.java | 55 +-
...xpansionServiceSchemaTransformProviderTest.java | 90 +-
.../expansion/service/ExpansionServiceTest.java | 2 +-
.../resources/test_expansion_service_config.yaml | 3 +
.../extensions/gcp/auth/GcpCredentialFactory.java | 5 +
.../python/PythonExternalTransformTest.java | 62 +-
.../schemaio-expansion-service/build.gradle | 6 +
.../sdk/extensions/sql/impl/JavaUdfLoader.java | 14 +
.../apache/beam/fn/harness/FnApiDoFnRunner.java | 171 +-
.../beam/sdk/io/gcp/bigquery/AppendClientInfo.java | 12 +
.../beam/sdk/io/gcp/bigquery/BatchLoads.java | 7 +-
.../sdk/io/gcp/bigquery/SplittingIterable.java | 19 +-
.../bigquery/StorageApiWriteUnshardedRecords.java | 9 +-
.../bigquery/StorageApiWritesShardedRecords.java | 4 +-
.../io/gcp/bigquery/TableRowToStorageApiProto.java | 154 ++
.../changestreams/dao/PartitionMetadataDao.java | 16 +-
.../sdk/io/gcp/bigquery/BigQueryIOWriteTest.java | 2 +
.../bigquery/TableRowToStorageApiProtoTest.java | 138 +
.../dao/PartitionMetadataDaoTest.java | 33 +-
.../it/SpannerChangeStreamPlacementTableIT.java | 16 +-
.../apache/beam/sdk/io/iceberg/RecordWriter.java | 10 +-
.../ReadFromPostgresSchemaTransformProvider.java | 10 +-
.../WriteToPostgresSchemaTransformProvider.java | 10 +-
.../java/org/apache/beam/sdk/io/kafka/KafkaIO.java | 40 +-
.../KafkaReadSchemaTransformConfiguration.java | 18 +
.../beam/sdk/io/kafka/KafkaSourceDescriptor.java | 8 +
.../kafka/KafkaWriteSchemaTransformProvider.java | 9 +
.../org/apache/beam/sdk/io/kafka/KafkaIOTest.java | 59 +
.../KafkaReadSchemaTransformProviderTest.java | 112 +
.../KafkaWriteSchemaTransformProviderTest.java | 66 +
.../java/org/apache/beam/sdk/io/xml/JAXBCoder.java | 28 +-
sdks/python/apache_beam/coders/coders.py | 16 +-
.../cookbook/ordered_window_elements}/__init__.py | 0
.../cookbook/ordered_window_elements/streaming.py | 625 +++++
.../ordered_window_elements/streaming_test.py | 359 +++
...lassification.py => gemini_image_generation.py} | 49 +-
.../inference/gemini_text_classification.py | 12 +-
.../snippets/transforms/elementwise/enrichment.py | 77 +-
.../transforms/elementwise/enrichment_test.py | 171 +-
.../internal/cloudpickle/cloudpickle.py | 78 +
.../apache_beam/internal/cloudpickle_pickler.py | 61 +-
.../apache_beam/internal/code_object_pickler.py | 90 +-
.../internal/code_object_pickler_test.py | 27 +-
sdks/python/apache_beam/internal/dill_pickler.py | 42 +-
sdks/python/apache_beam/internal/module_test.py | 7 +
sdks/python/apache_beam/internal/pickler.py | 19 +-
sdks/python/apache_beam/internal/pickler_test.py | 44 +
sdks/python/apache_beam/io/filebasedsink.py | 14 +
sdks/python/apache_beam/io/filebasedsource.py | 4 +-
sdks/python/apache_beam/io/gcp/bigquery.py | 50 +-
.../apache_beam/io/gcp/bigquery_file_loads_test.py | 28 +-
.../io/gcp/bigquery_geography_it_test.py | 544 ++++
.../apache_beam/io/gcp/bigquery_schema_tools.py | 3 +-
.../io/gcp/bigquery_schema_tools_test.py | 134 +-
sdks/python/apache_beam/io/gcp/bigquery_tools.py | 3 +-
.../apache_beam/io/gcp/bigquery_tools_test.py | 154 ++
.../io/gcp/experimental/spannerio_test.py | 35 +-
sdks/python/apache_beam/io/gcp/gcsio.py | 2 +-
sdks/python/apache_beam/io/gcp/spanner.py | 56 +-
.../apache_beam/ml/inference/gemini_inference.py | 42 +-
.../ml/inference/gemini_inference_it_test.py | 25 +
.../ml/inference/gemini_tests_requirements.txt | 3 +-
.../ml/inference/huggingface_inference.py | 12 +-
.../ml/inference/huggingface_inference_test.py | 34 +-
.../apache_beam/ml/rag/enrichment/milvus_search.py | 49 +-
.../ml/rag/enrichment/milvus_search_it_test.py | 27 +-
.../python/apache_beam/options/pipeline_options.py | 66 +-
.../apache_beam/options/pipeline_options_test.py | 21 +
.../runners/dataflow/dataflow_runner.py | 14 +-
.../apache_beam/runners/dataflow/internal/names.py | 2 +-
.../runners/direct/transform_evaluator.py | 2 +-
.../runners/interactive/interactive_beam.py | 30 +-
.../runners/interactive/interactive_beam_test.py | 85 +
.../python/apache_beam/runners/pipeline_context.py | 1 +
.../runners/portability/expansion_service.py | 12 +-
.../runners/portability/local_job_service.py | 1 -
.../runners/portability/portable_runner.py | 14 +-
.../runners/portability/prism_runner.py | 18 +-
.../apache_beam/runners/worker/bundle_processor.py | 27 +-
.../apache_beam/runners/worker/data_plane.py | 20 +-
.../apache_beam/runners/worker/sdk_worker.py | 19 +-
.../apache_beam/runners/worker/sdk_worker_main.py | 4 +
.../apache_beam/runners/worker/sdk_worker_test.py | 44 +-
.../apache_beam/runners/worker/worker_status.py | 118 +-
.../runners/worker/worker_status_test.py | 96 +-
.../testing/benchmarks/inference/README.md | 100 +-
.../benchmarks/nexmark/models/auction_bid.py | 3 +-
.../benchmarks/nexmark/models/nexmark_model.py | 27 +-
.../testing/benchmarks/nexmark/nexmark_util.py | 18 -
sdks/python/apache_beam/transforms/combiners.py | 1 +
sdks/python/apache_beam/transforms/core.py | 4 +-
sdks/python/apache_beam/transforms/core_it_test.py | 43 +-
sdks/python/apache_beam/transforms/external.py | 43 +-
.../python/apache_beam/transforms/external_test.py | 22 +-
sdks/python/apache_beam/transforms/managed.py | 49 +-
.../transforms/maven_repository_url_test.py | 224 ++
.../apache_beam/transforms/periodicsequence.py | 3 +-
sdks/python/apache_beam/transforms/ptransform.py | 8 +-
sdks/python/apache_beam/transforms/util.py | 24 +-
sdks/python/apache_beam/transforms/util_test.py | 64 +-
.../transforms/validate_runner_xlang_test.py | 96 +
sdks/python/apache_beam/utils/logger.py | 137 +
sdks/python/apache_beam/utils/logger_test.py | 108 +
sdks/python/apache_beam/utils/subprocess_server.py | 31 +-
sdks/python/apache_beam/version.py | 2 +-
sdks/python/apache_beam/yaml/yaml_provider.py | 26 +
.../apache_beam/yaml/yaml_specifiable_test.py | 7 +-
sdks/python/container/Dockerfile | 2 +-
sdks/python/container/build.gradle | 8 +-
sdks/python/container/common.gradle | 2 +-
sdks/python/container/ml/common.gradle | 2 +-
.../ml/{py313 => py310}/ml_image_requirements.txt | 135 +-
.../ml/{py313 => py311}/ml_image_requirements.txt | 128 +-
.../ml/{py313 => py312}/ml_image_requirements.txt | 127 +-
.../container/ml/py313/ml_image_requirements.txt | 117 +-
.../py39/ml_image_requirements.txt} | 116 +-
.../container/py310/base_image_requirements.txt | 74 +-
.../container/py311/base_image_requirements.txt | 72 +-
.../container/py312/base_image_requirements.txt | 72 +-
.../container/py313/base_image_requirements.txt | 78 +-
.../container/py39/base_image_requirements.txt | 68 +-
sdks/python/container/run_generate_requirements.sh | 5 +-
sdks/python/container/run_validatescontainer.sh | 16 +-
sdks/python/setup.py | 6 +
sdks/python/test-suites/direct/xlang/build.gradle | 1 +
sdks/python/tox.ini | 6 +-
sdks/typescript/package.json | 2 +-
.../en/blog/gsoc-25-jupyterlab-extensions.md | 74 +
.../content/en/documentation/dsls/sql/shell.md | 114 +-
.../python/elementwise/enrichment-cloudsql.md | 4 +-
.../python/elementwise/enrichment-milvus.md | 65 +
.../transforms/python/elementwise/enrichment.md | 3 +-
website/www/site/data/authors.yml | 3 +
.../partials/section-menu/en/documentation.html | 1 +
.../gsoc-25-jupyterlab-extensions/Yaml_main.png | Bin 0 -> 305413 bytes
273 files changed, 10972 insertions(+), 2329 deletions(-)
copy .github/trigger_files/{beam_PostCommit_Java_PVR_Flink_Streaming.json =>
beam_PostCommit_Go_VR_Spark.json} (53%)
copy .github/trigger_files/{beam_CloudML_Benchmarks_Dataflow.json =>
beam_PostCommit_XVR_JavaUsingPython_Dataflow.json} (98%)
copy .github/trigger_files/{beam_CloudML_Benchmarks_Dataflow.json =>
beam_PostCommit_XVR_PythonUsingJava_Dataflow.json} (100%)
copy .github/trigger_files/{beam_PostCommit_Java_Examples_Dataflow_ARM.json =>
beam_PostCommit_XVR_Spark3.json} (100%)
copy .github/trigger_files/{beam_PreCommit_Flink_Container.json =>
beam_PreCommit_Python_Dill.json} (80%)
delete mode 100644
.github/workflows/load-tests-pipeline-options/python_CoGBK_Dataflow_Flink_Batch_100b_Single_Key.txt
delete mode 100644
.github/workflows/load-tests-pipeline-options/python_CoGBK_Dataflow_Flink_Batch_10kB.txt
rename
.github/workflows/load-tests-pipeline-options/{python_CoGBK_Dataflow_Flink_Batch_100b_Multiple_Keys.txt
=> python_CoGBK_Flink_Batch_100b_Multiple_Keys.txt} (74%)
copy
.github/workflows/load-tests-pipeline-options/{python_Combine_Flink_Streaming_small_Fanout_1.txt
=> python_CoGBK_Flink_Batch_100b_Single_Key.txt} (77%)
copy
.github/workflows/load-tests-pipeline-options/{python_Combine_Flink_Streaming_small_Fanout_1.txt
=> python_CoGBK_Flink_Batch_10kB.txt} (76%)
create mode 100644 examples/notebooks/beam-ml/milvus_enrichment_transform.ipynb
rename
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/state/{WindmillStateUtil.java
=> WindmillStateTagUtil.java} (63%)
rename
runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/windmill/state/{WindmillStateUtilTest.java
=> WindmillStateTagUtilTest.java} (89%)
create mode 100755 scripts/beam-sql.sh
copy sdks/go/pkg/beam/log/{standard.go => structural.go} (64%)
create mode 100644
sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/utils/AutoValueUtilsTest.java
copy {examples/notebooks/notebook_test_scripts =>
sdks/python/apache_beam/examples/cookbook/ordered_window_elements}/__init__.py
(100%)
create mode 100644
sdks/python/apache_beam/examples/cookbook/ordered_window_elements/streaming.py
create mode 100644
sdks/python/apache_beam/examples/cookbook/ordered_window_elements/streaming_test.py
copy sdks/python/apache_beam/examples/inference/{gemini_text_classification.py
=> gemini_image_generation.py} (72%)
create mode 100644 sdks/python/apache_beam/io/gcp/bigquery_geography_it_test.py
create mode 100644
sdks/python/apache_beam/transforms/maven_repository_url_test.py
create mode 100644 sdks/python/apache_beam/utils/logger.py
create mode 100644 sdks/python/apache_beam/utils/logger_test.py
copy sdks/python/container/ml/{py313 => py310}/ml_image_requirements.txt (70%)
copy sdks/python/container/ml/{py313 => py311}/ml_image_requirements.txt (71%)
copy sdks/python/container/ml/{py313 => py312}/ml_image_requirements.txt (71%)
copy sdks/python/container/{py39/base_image_requirements.txt =>
ml/py39/ml_image_requirements.txt} (70%)
create mode 100644
website/www/site/content/en/blog/gsoc-25-jupyterlab-extensions.md
create mode 100644
website/www/site/content/en/documentation/transforms/python/elementwise/enrichment-milvus.md
create mode 100644
website/www/site/static/images/blog/gsoc-25-jupyterlab-extensions/Yaml_main.png
