This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a change to branch nightly-refs/heads/master
in repository https://gitbox.apache.org/repos/asf/beam.git
from 5394822d017 Remove usage of a (leaking) shared handles, which can also
accidentally release the last reference hodling a model in
_SharedMap._keepalive when all DoFns have been gc'ed. (#40441)
add 09fbdf9acdf fix leader board trigger
add d413595e717 trigger workflow
add da910f3bb09 fix leader board triggers
add 05725753096 shorten leader board team trigger delays
add c275c97b2e0 trigger workflow
add 76f6e6d7c0c fire leader board user scores every 10 seconds
add b7cfa6c95a2 Merge pull request #40370 from
aIbrahiim/fix-leaderboard-processing-time-triggers
add fd2bb97d76a Simplify pull request template (#40339)
add 483f14b6dd4 [Python] Isolate async work by key and window (#39996)
add 0f106b32122 [Python] Don't skip a delimiter right after an escaped one
in ReadFromText (#40460)
add 93a84b6a007 Remove kafka reviewers label (#40464)
add c5d866c50fe Let SortAndBatchElements batches reach max_batch_weight
(#40462)
add 7a5f89199d5 [Claude] Plumb through pipeline-level resource hints to
cross-lanugage transforms (#40379)
add af5d596ddbc Remove unused net.researchgate.release Gradle plugin
(#40465)
add 22abffca3a4 [Go SDK] Encode INT16 row fields as big endian to match
Java and Python (#40152)
add debfa0e6088 [#40250] Remove support for Python 3.10 (#40413)
add dabcf50ffbf [YAML] Fix flaky ErrorHandlingTest: don't pickle Scope
into user DoFn closures (#40467)
add 627d7fbb3fd Reduce Gradle build and test log spam in Java CI workflows
(#40458)
add 69bfcacf0be Make AsyncWrapperTest deterministic (#40448) (#40449)
add ddcd6544642 Add FlareDB to Beam runner docs (#40435)
add 40852a71e00 Update JmsIO to ActiveMQ 6.2.5 and jakarta.jms (#38729)
add 0dd464f9801 remove redundant Thank you (#40471)
add a2df8610996 Deflake MqttIOTest read tests (#40393)
add c692c225d97 add redistribute kafka io parameters (#40396)
add fadfc789376 Removes usages of the sentence-transformers package...
(#40439)
No new revisions were added by this update.
Summary of changes:
.github/PULL_REQUEST_TEMPLATE.md | 22 +-
.github/REVIEWERS.yml | 6 +-
.../test-properties.json | 16 +-
.../actions/setup-environment-action/action.yml | 4 +-
.github/trigger_files/beam_PostCommit_Python.json | 2 +-
.github/workflows/README.md | 2 +-
.../workflows/beam_CloudML_Benchmarks_Dataflow.yml | 3 +-
.../beam_Inference_Python_Benchmarks_Dataflow.yml | 58 +++---
.../beam_LoadTests_Python_CoGBK_Dataflow_Batch.yml | 8 +-
...m_LoadTests_Python_CoGBK_Dataflow_Streaming.yml | 8 +-
.../beam_LoadTests_Python_CoGBK_Flink_Batch.yml | 6 +-
...eam_LoadTests_Python_Combine_Dataflow_Batch.yml | 6 +-
...LoadTests_Python_Combine_Dataflow_Streaming.yml | 6 +-
.../beam_LoadTests_Python_Combine_Flink_Batch.yml | 6 +-
...am_LoadTests_Python_Combine_Flink_Streaming.yml | 4 +-
...LoadTests_Python_FnApiRunner_Microbenchmark.yml | 2 +-
.../beam_LoadTests_Python_GBK_Dataflow_Batch.yml | 10 +-
...eam_LoadTests_Python_GBK_Dataflow_Streaming.yml | 2 +-
...adTests_Python_GBK_reiterate_Dataflow_Batch.yml | 4 +-
...sts_Python_GBK_reiterate_Dataflow_Streaming.yml | 4 +-
.../beam_LoadTests_Python_ParDo_Dataflow_Batch.yml | 8 +-
...m_LoadTests_Python_ParDo_Dataflow_Streaming.yml | 8 +-
.../beam_LoadTests_Python_ParDo_Flink_Batch.yml | 6 +-
...beam_LoadTests_Python_ParDo_Flink_Streaming.yml | 10 +-
...m_LoadTests_Python_SideInput_Dataflow_Batch.yml | 20 +-
.github/workflows/beam_LoadTests_Python_Smoke.yml | 4 +-
...eam_PerformanceTests_BiqQueryIO_Read_Python.yml | 2 +-
...formanceTests_BiqQueryIO_Write_Python_Batch.yml | 2 +-
...erformanceTests_PubsubIOIT_Python_Streaming.yml | 2 +-
..._PerformanceTests_SpannerIO_Read_2GB_Python.yml | 2 +-
...manceTests_SpannerIO_Write_2GB_Python_Batch.yml | 2 +-
.../beam_PerformanceTests_TextIOIT_Python.yml | 2 +-
...PerformanceTests_WordCountIT_PythonVersions.yml | 2 +-
.../beam_PerformanceTests_xlang_KafkaIO_Python.yml | 2 +-
.github/workflows/beam_Playground_CI_Nightly.yml | 2 +-
.github/workflows/beam_Playground_Precommit.yml | 2 +-
.../beam_PostCommit_PortableJar_Flink.yml | 4 +-
.../beam_PostCommit_PortableJar_Spark.yml | 4 +-
.github/workflows/beam_PostCommit_Python.yml | 2 +-
.github/workflows/beam_PostCommit_Python_Arm.yml | 2 +-
.../beam_PostCommit_Python_Dependency.yml | 2 +-
.../beam_PostCommit_Python_Examples_Direct.yml | 2 +-
.../beam_PostCommit_Python_Examples_Flink.yml | 2 +-
.../beam_PostCommit_Python_Examples_Spark.yml | 2 +-
.../beam_PostCommit_Python_Nexmark_Direct.yml | 2 +-
.../beam_PostCommit_Python_Portable_Flink.yml | 6 +-
...stCommit_Python_ValidatesContainer_Dataflow.yml | 2 +-
..._Python_ValidatesContainer_Dataflow_With_RC.yml | 2 +-
..._PostCommit_Python_ValidatesRunner_Dataflow.yml | 2 +-
...eam_PostCommit_Python_ValidatesRunner_Flink.yml | 2 +-
...eam_PostCommit_Python_ValidatesRunner_Spark.yml | 2 +-
.../beam_PostCommit_Python_Xlang_Gcp_Dataflow.yml | 2 +-
.../beam_PostCommit_Python_Xlang_Gcp_Direct.yml | 2 +-
.../beam_PostCommit_Python_Xlang_IO_Dataflow.yml | 2 +-
.../beam_PostCommit_Python_Xlang_IO_Direct.yml | 2 +-
...am_PostCommit_Python_Xlang_Messaging_Direct.yml | 2 +-
.../beam_PostCommit_TransformService_Direct.yml | 4 +-
.github/workflows/beam_PostCommit_XVR_Direct.yml | 4 +-
.github/workflows/beam_PostCommit_XVR_Flink.yml | 4 +-
...eam_PostCommit_XVR_JavaUsingPython_Dataflow.yml | 4 +-
...eam_PostCommit_XVR_PythonUsingJava_Dataflow.yml | 4 +-
.github/workflows/beam_PostCommit_XVR_Spark3.yml | 4 +-
.../workflows/beam_PreCommit_Flink_Container.yml | 4 +-
.../workflows/beam_PreCommit_Portable_Python.yml | 4 +-
.github/workflows/beam_PreCommit_Prism_Python.yml | 4 +-
.github/workflows/beam_PreCommit_Python.yml | 2 +-
.github/workflows/beam_PreCommit_PythonDocker.yml | 2 +-
.../workflows/beam_PreCommit_Python_Coverage.yml | 4 +-
.../workflows/beam_PreCommit_Python_Dataframes.yml | 2 +-
.../workflows/beam_PreCommit_Python_Examples.yml | 2 +-
.../beam_PreCommit_Python_Integration.yml | 2 +-
.github/workflows/beam_PreCommit_Python_ML.yml | 4 +-
.../workflows/beam_PreCommit_Python_Runners.yml | 2 +-
.../workflows/beam_PreCommit_Python_Transforms.yml | 2 +-
.../beam_PreCommit_Xlang_Generated_Transforms.yml | 2 +-
.../workflows/beam_Publish_Beam_SDK_Snapshots.yml | 3 -
.../beam_Python_CostBenchmarks_Dataflow.yml | 6 +-
...beam_Python_ValidatesContainer_Dataflow_ARM.yml | 2 +-
.github/workflows/build_release_candidate.yml | 12 +-
.github/workflows/build_wheels.yml | 4 +-
.github/workflows/dask_runner_tests.yml | 4 +-
...python_CoGBK_Flink_Batch_100b_Multiple_Keys.txt | 2 +-
.../python_CoGBK_Flink_Batch_100b_Single_Key.txt | 2 +-
.../python_CoGBK_Flink_Batch_10kB.txt | 2 +-
...hon_Combine_Flink_Batch_2GB_10_byte_records.txt | 2 +-
.../python_Combine_Flink_Batch_2GB_Fanout_4.txt | 2 +-
.../python_Combine_Flink_Batch_2GB_Fanout_8.txt | 2 +-
...Combine_Flink_Streaming_2GB_10_byte_records.txt | 2 +-
...python_Combine_Flink_Streaming_2GB_Fanout_4.txt | 2 +-
...python_Combine_Flink_Streaming_2GB_Fanout_8.txt | 2 +-
...thon_Combine_Flink_Streaming_small_Fanout_1.txt | 2 +-
...thon_Combine_Flink_Streaming_small_Fanout_2.txt | 2 +-
.../python_GBK_Flink_Batch_2GB_of_100B_records.txt | 2 +-
.../python_GBK_Flink_Batch_2GB_of_10B_records.txt | 2 +-
...nout_4_times_with_2GB_10-byte_records_total.txt | 2 +-
...nout_8_times_with_2GB_10-byte_records_total.txt | 2 +-
...K_Flink_Batch_reiterate_4_times_10kB_values.txt | 2 +-
.../python_ParDo_Flink_Batch_10_Counters.txt | 2 +-
.../python_ParDo_Flink_Batch_10_Iterations.txt | 2 +-
.../python_ParDo_Flink_Batch_200_Iterations.txt | 2 +-
.../python_ParDo_Flink_Streaming_100_Counters.txt | 2 +-
.../python_ParDo_Flink_Streaming_10_Counters.txt | 2 +-
.../python_ParDo_Flink_Streaming_10_Iterations.txt | 2 +-
...python_ParDo_Flink_Streaming_200_Iterations.txt | 2 +-
.../python_ParDo_Flink_Streaming_5_Iterations.txt | 2 +-
.github/workflows/python_dependency_tests.yml | 1 -
.github/workflows/python_tests.yml | 3 +-
.../republish_released_docker_containers.yml | 6 +-
.github/workflows/run_perf_alert_tool.yml | 2 +-
.../run_rc_validation_python_mobile_gaming.yml | 2 +-
.github/workflows/typescript_tests.yml | 4 +-
.github/workflows/update_python_dependencies.yml | 1 -
.test-infra/mock-apis/pyproject.toml | 2 +-
.test-infra/tools/python_installer.sh | 2 +-
CHANGES.md | 10 +-
build.gradle.kts | 45 +----
.../org/apache/beam/gradle/BeamModulePlugin.groovy | 28 ++-
.../org/apache/beam/gradle/Repositories.groovy | 2 +-
contributor-docs/release-guide.md | 2 +-
gradle.properties | 2 +-
local-env-setup.sh | 4 +-
playground/backend/containers/python/Dockerfile | 2 +-
.../cloudbuild/playground_cd_examples.sh | 8 +-
.../cloudbuild/playground_ci_examples.sh | 16 +-
release/src/main/Dockerfile | 5 +-
.../python-release/python_release_automation.sh | 2 +-
runners/direct-java/build.gradle | 7 -
.../beam/runners/dataflow/DataflowRunnerTest.java | 12 +-
sdks/go/pkg/beam/core/graph/coder/int.go | 31 +++
sdks/go/pkg/beam/core/graph/coder/int_test.go | 32 +++
sdks/go/pkg/beam/core/graph/coder/row_decoder.go | 26 ++-
sdks/go/pkg/beam/core/graph/coder/row_encoder.go | 14 +-
sdks/go/pkg/beam/core/graph/coder/row_test.go | 40 ++++
sdks/go/pkg/beam/core/runtime/xlangx/expand.go | 47 ++++-
.../go/pkg/beam/core/runtime/xlangx/expand_test.go | 148 ++++++++++++++
sdks/go/pkg/beam/options/jobopts/options.go | 2 +
sdks/go/pkg/beam/options/jobopts/options_test.go | 24 ++-
sdks/go/pkg/beam/options/resource/hint.go | 39 ++++
sdks/go/pkg/beam/options/resource/hint_test.go | 45 +++++
.../beam/sdk/transforms/AsyncWrapperTest.java | 72 ++++---
sdks/java/io/amqp/build.gradle | 16 ++
sdks/java/io/debezium/build.gradle | 6 -
sdks/java/io/jms/build.gradle | 8 +-
.../io/jms/BeamGenericJmsConnectionFactory.java | 2 +-
.../beam/sdk/io/jms/ConnectionConfiguration.java | 6 +-
.../apache/beam/sdk/io/jms/JmsCheckpointMark.java | 8 +-
.../java/org/apache/beam/sdk/io/jms/JmsIO.java | 38 ++--
.../java/org/apache/beam/sdk/io/jms/JmsRecord.java | 2 +-
.../apache/beam/sdk/io/jms/TextMessageMapper.java | 12 +-
.../java/org/apache/beam/sdk/io/jms/CommonJms.java | 8 +-
.../sdk/io/jms/ConnectionConfigurationTest.java | 19 +-
.../java/org/apache/beam/sdk/io/jms/JmsIOIT.java | 14 +-
.../java/org/apache/beam/sdk/io/jms/JmsIOTest.java | 32 +--
.../org/apache/beam/sdk/io/jms/JmsLocalTest.java | 14 +-
.../jms/MockNonSerializableConnectionFactory.java | 8 +-
.../KafkaReadSchemaTransformProviderTest.java | 45 ++++-
.../io/messaging-expansion-service/build.gradle | 1 +
sdks/java/io/mqtt/build.gradle | 15 ++
.../org/apache/beam/sdk/io/mqtt/MqttIOTest.java | 47 ++++-
sdks/python/apache_beam/__init__.py | 2 +-
.../examples/complete/game/leader_board.py | 19 +-
.../examples/complete/game/leader_board_test.py | 4 +-
.../apache_beam/io/external/xlang_jmsio_it_test.py | 4 +-
sdks/python/apache_beam/io/textio.py | 2 +-
sdks/python/apache_beam/io/textio_test.py | 9 +
.../vertex_ai_custom_prediction/Dockerfile | 2 +-
.../apache_beam/ml/rag/embeddings/huggingface.py | 23 +--
.../ml/rag/embeddings/huggingface_test.py | 56 ++++++
.../ml/transforms/embeddings/huggingface.py | 48 +++--
.../ml/transforms/embeddings/huggingface_test.py | 142 +++++++++++++
.../runners/dataflow/internal/apiclient.py | 4 +-
.../runners/dataflow/internal/apiclient_test.py | 6 +-
.../runners/portability/fn_api_runner/execution.py | 17 +-
.../apache_beam/testing/benchmarks/README.md | 2 +-
sdks/python/apache_beam/testing/test_utils.py | 61 ++++++
sdks/python/apache_beam/testing/test_utils_test.py | 31 +++
sdks/python/apache_beam/transforms/async_dofn.py | 67 +++++--
.../apache_beam/transforms/async_dofn_test.py | 221 ++++++++++++++++++++-
sdks/python/apache_beam/transforms/trigger.py | 15 +-
sdks/python/apache_beam/transforms/util.py | 4 +-
sdks/python/apache_beam/transforms/util_test.py | 31 +++
.../yaml/extended_tests/messaging/kafka.yaml | 5 +
sdks/python/apache_beam/yaml/integration_tests.py | 2 +-
sdks/python/apache_beam/yaml/standard_io.yaml | 7 +-
sdks/python/apache_beam/yaml/yaml_ml_test.py | 40 +++-
sdks/python/apache_beam/yaml/yaml_transform.py | 5 +
.../python/apache_beam/yaml/yaml_transform_test.py | 40 ++++
sdks/python/build.gradle | 2 +-
sdks/python/container/Dockerfile | 2 +-
sdks/python/container/run_generate_requirements.sh | 4 +-
sdks/python/expansion-service-container/Dockerfile | 4 +-
sdks/python/pyproject.toml | 4 +-
sdks/python/setup.py | 6 +-
sdks/python/test-suites/dataflow/common.gradle | 2 +-
sdks/python/test-suites/gradle.properties | 24 +--
sdks/python/test-suites/tox/py311/build.gradle | 186 +++++++++++++++++
sdks/python/tox.ini | 48 ++---
settings.gradle.kts | 2 +-
.../content/en/documentation/runners/flaredb.md | 210 ++++++++++++++++++++
.../en/documentation/runtime/environments.md | 4 +-
.../documentation/sdks/python-sdk-image-build.md | 6 +-
website/www/site/data/capability_matrix.yaml | 156 ++++++++++++++-
.../www/site/data/en/documentation_runners.yaml | 2 +
website/www/site/data/works_with.yaml | 3 +
.../layouts/partials/section-menu/en/runners.html | 1 +
website/www/site/static/images/logo_flaredb.png | Bin 0 -> 39481 bytes
206 files changed, 2358 insertions(+), 608 deletions(-)
create mode 100644 sdks/go/pkg/beam/core/runtime/xlangx/expand_test.go
create mode 100644 website/www/site/content/en/documentation/runners/flaredb.md
create mode 100644 website/www/site/static/images/logo_flaredb.png
