This is an automated email from the ASF dual-hosted git repository.
jrmccluskey pushed a change to branch threeNineDep
in repository https://gitbox.apache.org/repos/asf/beam.git
from d477b91b5d0 add version removal to CHANGES.md
add 0d52be60e88 Add image generation code to Gemini Model Handler (#36177)
add 179d4d1ee9c Fix flaky tests (#36579)
add 7e7d866d95a [python] add setup to BigQuery's convert row Map transform
(#36502)
add c7d920f26cf Update dev image. (#36582)
add 243d4077319 Integrate lambda name pickling with Cloudpickle (#35904)
add ab892e3dd09 Add logging for credential retrieval failures in
GcpCredentialFactory
add db92a3ad0c0 Merge pull request #36415: Add logging for credential
retrieval failures in GcpCredentialFactory
add b83c24e4d45 test(spannerio): make batch size validation more flexible
for non-deterministic execution (#36584)
add 4c08585626b [IcebergIO] Pass table props to data writers (#36542)
add 0ebf84b6b18 Add ib.collect support for raw records (#36516)
add 38481b58879 Address circular dependencies in Nexmark benchmark suite.
(#36513)
add 8bd92b5e376 Fix the soft-delete check and emit soft-delete log warning
at most once per bucket. (#36585)
add afeca4ea301 Increase timeouts (#36595)
add 66b7c7476ce Make SpannerChangeStreamPlacementTableIT against Spanner
prod. (#36071)
add fdfa6ec6338 Exclude a perma-red test suite
beam_PostCommit_XVR_GoUsingJava_Dataflow.yml (#36597)
add 6dedf8f0bab Update REVIEWERS.yml (#36598)
add fe71ab1b47b Add ordered window elements into example folder (#36575)
add ef07e40667b Fix proposal link (#36600)
add f2860fa2fe8 use utils._convert_to_result for huggingface_inference
(#36593)
add 05f6f01a33b Force torch to use cpu wheels (#36583)
add 6df5bd04185 Switch to use gcp-io expansion service for YAML BQIO
(#36604)
add a846bc92950 Bump github.com/aws/aws-sdk-go-v2/credentials in /sdks
(#36611)
add 1a4298f209c Bump github.com/avast/retry-go/v4 from 4.6.1 to 4.7.0 in
/sdks (#36519)
add a3d42aec4e0 Update woodstox package and version (#36613)
add 518b118bcaf Fix vLLM Gemma, add vLLM extra, fix getting throughput
(#36451)
add 7bfe0e19974 Do not panic if an element has no windows during
PersistBundle. (#36610)
add 64a92b2806d [python] Move histogram metric from internal to public
interface/files (#36405)
add 92a40213231 Force logback version in transitive dependencies
add f517b01c502 Merge pull request #36594: Force logback version in
transitive dependencies
add 85853a3edf0 proto change
add a6d2b7dabf5 add draining to output builder, encode draining
add 951943e9bba add draining to output builder
add 7ea3109a0e4 default impls
add 3d9e403b3ed comment
add 499039fe37d remove nullable
add cca50bff5aa remove nullable
add f8037f0f5bf remove nullable
add ac99abf3d79 rename field
add fc4b29c4256 rename field
add 2bd497f5d55 Merge pull request #36524: Drain - model part +
windowedValue changes
add 4e0dbb9f452 Upgrade joda-time to 2.14.0 (#36626)
add c6d92079b48 fix test (#36632)
add 9612583296a Bump github.com/linkedin/goavro/v2 from 2.14.0 to 2.14.1
in /sdks (#36628)
add 80f4564fbdc Bump github.com/nats-io/nats-server/v2 from 2.12.0 to
2.12.1 in /sdks (#36520)
add 471050c7cde add licences manually (#36635)
add 0f6b605f9e5 Fix a bug in streaming OrderedWindowElements. (#36642)
add 0c10658e9ba [Dataflow Streaming] Intern encoded tags across keys
(#36313)
add 949c87f0e20 Cache IntervalWindow hashCode (#36612)
add ecc840cd1e7 [Java SDK] Fix propagation of metrics set during onTimer
processing. (#36576)
add 942de54a14c Adding release-2.69.0-postrelease to protected branches in
.asf.yaml
add 00516e97235 Update managed-io.md for release 2.69.0-RC2.
add 3deba11b8a2 Merge pull request #36569 from
apache/updates_managed_io_docs_2.69.0_rc2
add bcf41e14aba Add missing py313 ML container requirements file (#36647)
add be7acf5eca2 Update Beam website to release 2.69.0
add be47decb542 Update date
add 37ca7d54090 Update breaking changes
add fdc89112719 Merge pull request #36554 from apache/release-269-website
add 5a8e2dc3996 remove empty file (#36648)
add 5fe0d4c084b Add batch version of OrderedWindowElements in examples
(#36621)
add 3432480129c Update website for roadmap and version compatibility
(#36602)
add 21dbf592f87 Adds Multimap support to JAVA FnApi (#36218)
add c60b8ae8671 Bump github.com/aws/aws-sdk-go-v2/config in /sdks (#36644)
add b33ab6e2e94 Bump cloud.google.com/go/bigquery from 1.70.0 to 1.71.0 in
/sdks (#36500)
add b2960c9db3e fix(website): Use standard method for CSP frame-src
exceptions (#36653)
add 020335ee23b Fixed KafkaIO Python perf test: use existing kafka-cluster
config, fix broker setup, and correct test arguments
add 50477d8bd9b ix KafkaIO perf test assertion logic
add b9ec235e5ec fixed linting
add 54e4dcf737f Fixed the pylint error
add f5416c2bcc7 Fixed the YAPF formatting issue
add 8b2657bf80d fixed kafka stress tests
add 263510afbfd Fix Kafka stress test: remove invalid param
add ab4a80ddd22 fixed formatting
add f8c8dda8ed4 removed invalid params
add 0e4ac08e4b1 Merge pull request #36633 from
aIbrahiim/30960-performance-tests-xlang-kafka--python
add 1b34a6d7571 Bump go.mongodb.org/mongo-driver from 1.17.4 to 1.17.6 in
/sdks (#36659)
add 86bc45286d1 fixed issue in python precommit (#36561)
add 0697e54a543 Kerberos Authentication for KafkaIO (#36099)
add 1c5f0f3a505 Revert "Force logback version in transitive dependencies"
(#36649)
add 15615afe03c Merge branch 'master' into threeNineDep
No new revisions were added by this update.
Summary of changes:
.asf.yaml | 1 +
.github/REVIEWERS.yml | 2 -
.../IO_Iceberg_Integration_Tests.json | 2 +-
.../beam_PerformanceTests_xlang_KafkaIO_Python.yml | 82 +-
.../beam_PostCommit_XVR_GoUsingJava_Dataflow.yml | 6 +-
.../workflows/beam_StressTests_Java_KafkaIO.yml | 77 +-
...Sentiment_Streaming_DistilBert_Base_Uncased.txt | 3 +-
...Python_Benchmarks_Dataflow_VLLM_Gemma_Batch.txt | 4 +-
.github/workflows/refresh_looker_metrics.yml | 6 -
.../workflows/run_rc_validation_python_yaml.yml | 10 +-
.../kafka-cluster/03-zookeeper/50pzoo.yml | 121 +-
.../kubernetes/kafka-cluster/05-kafka/50kafka.yml | 156 +-
CHANGES.md | 10 +-
build.gradle.kts | 1 +
.../org/apache/beam/gradle/BeamModulePlugin.groovy | 4 +-
contributor-docs/python-tips.md | 12 +-
.../java/org/apache/beam/it/kafka/KafkaIOST.java | 28 +-
.../beam/model/fn_execution/v1/beam_fn_api.proto | 34 +-
.../beam/model/pipeline/v1/beam_runner_api.proto | 10 +-
.../beam/runners/dataflow/BatchViewOverrides.java | 5 +
.../dataflow/worker/util/ValueInEmptyWindows.java | 5 +
.../util/common/worker/InternedByteString.java | 77 +
.../worker/windmill/state/CachingStateTable.java | 137 +-
.../worker/windmill/state/WindmillBag.java | 21 +-
.../windmill/state/WindmillCombiningState.java | 20 +-
.../worker/windmill/state/WindmillMap.java | 30 +-
.../worker/windmill/state/WindmillMultimap.java | 24 +-
.../worker/windmill/state/WindmillOrderedList.java | 17 +-
.../worker/windmill/state/WindmillStateCache.java | 82 +-
.../windmill/state/WindmillStateTagUtil.java | 10 +-
.../worker/windmill/state/WindmillValue.java | 21 +-
.../windmill/state/WindmillWatermarkHold.java | 30 +-
.../util/common/worker/InternedByteStringTest.java | 68 +
.../windmill/state/WindmillStateCacheTest.java | 232 ++-
.../windmill/state/WindmillStateTagUtilTest.java | 10 +-
.../apache/beam/runners/spark/util/TimerUtils.java | 5 +
sdks/go.mod | 40 +-
sdks/go.sum | 80 +-
.../prism/internal/engine/elementmanager.go | 6 +-
.../beam/sdk/fn/stream/PrefetchableIterables.java | 2 +-
.../sdk/transforms/windowing/IntervalWindow.java | 17 +-
.../beam/sdk/util/construction/Environments.java | 1 +
.../org/apache/beam/sdk/values/OutputBuilder.java | 2 +
.../org/apache/beam/sdk/values/WindowedValue.java | 2 +
.../org/apache/beam/sdk/values/WindowedValues.java | 153 +-
.../org/apache/beam/sdk/transforms/ParDoTest.java | 67 +
.../apache/beam/sdk/util/WindowedValueTest.java | 6 +-
.../sdk/util/construction/EnvironmentsTest.java | 3 +
.../extensions/gcp/auth/GcpCredentialFactory.java | 5 +
.../{sbe => kafka-factories}/build.gradle | 27 +-
.../kafka/factories/FileAwareFactoryFn.java | 258 +++
.../kafka/factories/KerberosConsumerFactoryFn.java | 150 ++
.../extensions/kafka/factories}/package-info.java | 4 +-
.../kafka/factories/FileAwareFactoryFnTest.java | 268 ++++
.../factories/KerberosConsumerFactoryFnTest.java | 242 +++
.../fn/harness/control/ProcessBundleHandler.java | 14 +-
.../harness/data/PCollectionConsumerRegistry.java | 79 +-
.../beam/fn/harness/state/FnApiStateAccessor.java | 121 +-
.../beam/fn/harness/state/MultimapUserState.java | 139 +-
.../harness/control/ProcessBundleHandlerTest.java | 149 ++
.../fn/harness/state/MultimapUserStateTest.java | 129 ++
.../it/SpannerChangeStreamPlacementTableIT.java | 16 +-
.../apache/beam/sdk/io/iceberg/RecordWriter.java | 10 +-
sdks/java/io/xml/build.gradle | 4 +-
.../cookbook/ordered_window_elements}/__init__.py | 0
.../cookbook/ordered_window_elements/batch.py | 522 +++++++
.../cookbook/ordered_window_elements/batch_test.py | 333 ++++
.../cookbook/ordered_window_elements/streaming.py | 625 ++++++++
.../ordered_window_elements/streaming_test.py | 359 +++++
...lassification.py => gemini_image_generation.py} | 49 +-
.../inference/gemini_text_classification.py | 12 +-
.../transforms/elementwise/enrichment_test.py | 2 +-
.../internal/cloudpickle/cloudpickle.py | 78 +
.../apache_beam/internal/cloudpickle_pickler.py | 17 +-
.../apache_beam/internal/code_object_pickler.py | 90 +-
.../internal/code_object_pickler_test.py | 27 +-
sdks/python/apache_beam/internal/metrics/cells.py | 165 --
.../apache_beam/internal/metrics/cells_test.py | 77 -
sdks/python/apache_beam/internal/metrics/metric.py | 44 -
.../apache_beam/internal/metrics/metric_test.py | 130 --
sdks/python/apache_beam/internal/module_test.py | 7 +
sdks/python/apache_beam/internal/pickler.py | 14 +-
sdks/python/apache_beam/internal/pickler_test.py | 44 +
.../io/external/xlang_kafkaio_perf_test.py | 10 +-
sdks/python/apache_beam/io/filebasedsink.py | 14 +
sdks/python/apache_beam/io/gcp/bigquery.py | 34 +-
sdks/python/apache_beam/io/gcp/bigquery_tools.py | 12 +-
.../io/gcp/experimental/spannerio_test.py | 35 +-
sdks/python/apache_beam/io/gcp/gcsio.py | 2 +-
sdks/python/apache_beam/metrics/cells.py | 131 ++
sdks/python/apache_beam/metrics/cells_test.py | 51 +
sdks/python/apache_beam/metrics/execution.py | 6 +-
sdks/python/apache_beam/metrics/metric.py | 44 +
sdks/python/apache_beam/metrics/metric_test.py | 38 +-
.../python/apache_beam/metrics/monitoring_infos.py | 2 +-
.../apache_beam/metrics/monitoring_infos_test.py | 4 +-
.../apache_beam/ml/inference/gemini_inference.py | 42 +-
.../ml/inference/gemini_inference_it_test.py | 25 +
.../ml/inference/gemini_tests_requirements.txt | 3 +-
.../ml/inference/huggingface_inference.py | 12 +-
.../ml/inference/huggingface_inference_test.py | 34 +-
.../ml/inference/test_resources/vllm.dockerfile | 14 +-
.../ml/inference/vllm_tests_requirements.txt | 2 +-
.../apache_beam/ml/rag/enrichment/milvus_search.py | 49 +-
.../ml/rag/enrichment/milvus_search_it_test.py | 3 +-
.../python/apache_beam/options/pipeline_options.py | 16 +-
.../apache_beam/runners/dataflow/internal/names.py | 2 +-
.../runners/interactive/interactive_beam.py | 30 +-
.../runners/interactive/interactive_beam_test.py | 85 +
.../python/apache_beam/runners/pipeline_context.py | 1 +
.../benchmarks/inference/vllm_gemma_benchmarks.py | 2 +-
.../benchmarks/nexmark/models/auction_bid.py | 3 +-
.../benchmarks/nexmark/models/nexmark_model.py | 27 +-
.../testing/benchmarks/nexmark/nexmark_util.py | 18 -
.../python/apache_beam/transforms/external_test.py | 22 +-
sdks/python/apache_beam/transforms/ptransform.py | 2 +
sdks/python/apache_beam/transforms/window_test.py | 13 +
sdks/python/apache_beam/yaml/standard_io.yaml | 2 +-
sdks/python/container/Dockerfile | 2 +-
sdks/python/container/common.gradle | 2 +-
.../container/license_scripts/dep_urls_py.yaml | 4 +
.../container/ml/py310/ml_image_requirements.txt | 33 +-
.../container/ml/py311/ml_image_requirements.txt | 33 +-
.../container/ml/py312/ml_image_requirements.txt | 33 +-
.../{ => ml}/py313/base_image_requirements.txt | 0
.../container/ml/py313/ml_image_requirements.txt | 37 +-
.../container/ml/py39/ml_image_requirements.txt | 31 +-
sdks/python/container/run_generate_requirements.sh | 5 +-
sdks/python/scripts/run_pytest.sh | 2 +-
sdks/python/setup.py | 9 +-
sdks/python/test-suites/tox/common.gradle | 5 +-
settings.gradle.kts | 1 +
website/www/site/config.toml | 2 +-
website/www/site/content/en/blog/beam-2.69.0.md | 86 +
.../en/blog/gsoc-25-jupyterlab-extensions.md | 2 +-
.../en/documentation/dsls/sql/calcite/overview.md | 27 +
.../dsls/sql/calcite/scalar-functions.md | 9 +-
.../en/documentation/dsls/sql/zetasql/overview.md | 2 +-
.../site/content/en/documentation/io/managed-io.md | 1635 +++++++++++++++++---
.../www/site/content/en/documentation/sdks/java.md | 29 +
.../content/en/documentation/sdks/java/euphoria.md | 2 +
.../site/content/en/documentation/sdks/python.md | 51 +-
.../www/site/content/en/get-started/downloads.md | 18 +-
website/www/site/content/en/roadmap/_index.md | 19 +-
.../www/site/content/en/security/CVE-2020-1929.md | 16 -
website/www/site/data/performance.yaml | 10 +-
website/www/site/static/.htaccess | 4 +-
147 files changed, 7158 insertions(+), 1693 deletions(-)
create mode 100644
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/InternedByteString.java
create mode 100644
runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/util/common/worker/InternedByteStringTest.java
copy sdks/java/extensions/{sbe => kafka-factories}/build.gradle (53%)
create mode 100644
sdks/java/extensions/kafka-factories/src/main/java/org/apache/beam/sdk/extensions/kafka/factories/FileAwareFactoryFn.java
create mode 100644
sdks/java/extensions/kafka-factories/src/main/java/org/apache/beam/sdk/extensions/kafka/factories/KerberosConsumerFactoryFn.java
copy
{.test-infra/mock-apis/src/main/java/org/apache/beam/testinfra/mockapis/echo/v1
=>
sdks/java/extensions/kafka-factories/src/main/java/org/apache/beam/sdk/extensions/kafka/factories}/package-info.java
(85%)
create mode 100644
sdks/java/extensions/kafka-factories/src/test/java/org/apache/beam/sdk/extensions/kafka/factories/FileAwareFactoryFnTest.java
create mode 100644
sdks/java/extensions/kafka-factories/src/test/java/org/apache/beam/sdk/extensions/kafka/factories/KerberosConsumerFactoryFnTest.java
copy {examples/notebooks/notebook_test_scripts =>
sdks/python/apache_beam/examples/cookbook/ordered_window_elements}/__init__.py
(100%)
create mode 100644
sdks/python/apache_beam/examples/cookbook/ordered_window_elements/batch.py
create mode 100644
sdks/python/apache_beam/examples/cookbook/ordered_window_elements/batch_test.py
create mode 100644
sdks/python/apache_beam/examples/cookbook/ordered_window_elements/streaming.py
create mode 100644
sdks/python/apache_beam/examples/cookbook/ordered_window_elements/streaming_test.py
copy sdks/python/apache_beam/examples/inference/{gemini_text_classification.py
=> gemini_image_generation.py} (72%)
delete mode 100644 sdks/python/apache_beam/internal/metrics/cells.py
delete mode 100644 sdks/python/apache_beam/internal/metrics/cells_test.py
delete mode 100644 sdks/python/apache_beam/internal/metrics/metric_test.py
copy sdks/python/container/{ => ml}/py313/base_image_requirements.txt (100%)
create mode 100644 website/www/site/content/en/blog/beam-2.69.0.md
delete mode 100644 website/www/site/content/en/security/CVE-2020-1929.md
