This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a change to branch
dependabot/go_modules/sdks/github.com/aws/aws-sdk-go-v2/service/s3-1.61.2
in repository https://gitbox.apache.org/repos/asf/beam.git
discard a75c5568546 Bump github.com/aws/aws-sdk-go-v2/service/s3 in /sdks
add 696978018ed Bump actions/download-artifact from 4.1.7 to 4.1.8 (#32396)
add 3b4ecd7a987 Bump commons-cli:commons-cli from 1.8.0 to 1.9.0 (#32350)
add 4dddde4e6de Colab Notebook for Unit Tests in Beam (#32336)
add 6901d7c8623 Undo part of artifact action upgrade to fix workflow
(#32401)
add 1ee2f6bff4a Bump jinja2 version to resolve vulnerability (#32403)
add 50a6cd2b580 Add tests of using ib.collect(...) without
InteractiveRunner.
add d5dc20000d5 Allow ib.collect(...) to take multiple PCollections.
add a7852d9ddb1 Merge pull request #32392 Allow ib.collect(...) to take
multiple PCollections.
add d9a3d4457a0 Add keyrings.google-artifactregistry-auth (#32404)
add 2a7755b0f2c Use Knuth–Morris–Pratt algorithm for delimiter search in
TextIO (#32398)
add 7a88e6fa0bb [Dataflow Streaming] Move `throwExceptionOnLargeOutput`
out of OperationalLimits (#32407)
add 173dd486842 [yaml] Adding Spanner IO Providers for Beam YAML (#31987)
add 4aeb246596d [yaml] Move js2py package to yaml deps (#32377)
add 2573106f367 Bump webpack (#32374)
add 284004484c1 Add jobLabelsMap parameter to BigQueryOptions (#31698)
add 1d0e09a6d57 Support Zstd codec in SerializableAvroCodecFactory (#32352)
add f362045afb7 Bump micromatch from 4.0.2 to 4.0.8 in /website/www
add 2a0ce8f9103 Merge pull request #32375 from
apache/dependabot/npm_and_yarn/website/www/micromatch-4.0.8
add 5fbfdba926c Move PubsubIO source Lineage report to MapElements (#32381)
add c657507c082 Upgrade gcp-bom to 26.45.0 (#32413)
add ebcb2dbd160 Dont run flaky test on windows (#32419)
add a806bfc0526 [Python] Add the "no_wait_until_finish" option without
waiting fo the job completion
add e8c6a8c3aec Merge pull request #32409 from liferoad/wait_until_finish
add e76b51feb3a Fix RowCoderGenerator to use the encodingPositions when
encoding and decoding the bit set representing null fields. (#32389)
add 21abfb23f36 Disable permared go xlang tests. (#32423)
add 9c31e45d595 2.59.0 Beam site updates. (#32308)
add ef1bb52d3d3 Fixes link typo in 2.59, and fixes archive links. (#32437)
add 17718a97eaf Temporarily fix pip check failure on xgboost and grpcio
(#32432)
add ed84acb2967 Bump spanner max bound back to <4 (#32411)
add 4ee26065d9b Accept runner and options in ib.collect. (#32434)
add 02af7d4a218 Unit Testing in Beam Blog Post (#32412)
add bcf404238ab downgrade grpc (#32449)
add 45036736e0a Remove beam summit banner
add 98410caac19 Merge pull request #32331 from
apache/users/damccorm/beamsummit
add 9a524cc004e Add Lineage metrics to Python PubsubIO, BigtableIO, FileIO
(#32430)
add 83ae551cea1 Update Playground and other location usage of Go to use
latest released version by default. (#32444)
add ed8dc5df618 fix: use threadsafe data structure thats shared between
threads (#32452)
add 25196614070 fix: ensure that non ApiExceptions in
handleMutationException are not swallowed (#32455)
add a6f11c89667 Bump Python SDK container for Dataflow
add 1c307088a86 Merge pull request #32453: Bump Python SDK container for
Dataflow
add fb5424e5888 Bump cloud.google.com/go/pubsub from 1.41.0 to 1.43.0 in
/sdks (#32415)
add 7a6121a3231 fix docker compose command (#32459)
add e259f472add Don't override existing consumer config values KafkaIO
(#32443)
add 12974595cd0 Bump golang.org/x/text from 0.17.0 to 0.18.0 in /sdks
(#32466)
add 6514136c17e fix: ensure that all outstanding future have completed in
FinishBundle (#32454)
add bbd09d60665 Remove Python 3.8 GitHub Actions Runs (#32429)
add 8d1c8d2cb9d Add the script to extract docs from the dev list
add 56cb2c22e02 Add the script to extract docs from the dev list
add dd511fb22a6 Merge remote-tracking branch
'refs/remotes/origin/discussion-docs' into discussion-docs
add f2992149035 update license
add a65bed2ea23 updated the md content
add 6b3043ba100 run this for all years
add b808e9e90c4 fixed some author name errors
add c2b09e26ae6 standardized the urls
add e00559140df Merge pull request #32460 from liferoad/discussion-docs
add f8475c952c1 Fix macos tox.ini definition (#32472)
add 9f8a4b2d177 [yaml] Add examples for Spanner IO in YAML (#32288)
add d20c0b1bbb2 Add information on python metrics to the programming guide
(#32464)
add b949ac2ca43 Bump golang.org/x/sys from 0.24.0 to 0.25.0 in /sdks
(#32479)
add 1b2d21acf8b Fix Kafka with Redistribute and commits enabled (#32344)
add b1fd221a6d8 Restore mypy ignore[call-arg] in yaml_io.py (#32475)
add ad7d7bc973c Update Dataflow internal client test (#32471)
add 77913c59a17 Bump github.com/aws/aws-sdk-go-v2/config in /sdks (#32480)
add 6ab6b581c83 Bump github.com/aws/aws-sdk-go-v2/service/s3 in /sdks
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (a75c5568546)
\
N -- N -- N
refs/heads/dependabot/go_modules/sdks/github.com/aws/aws-sdk-go-v2/service/s3-1.61.2
(6ab6b581c83)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.../test-properties.json | 18 +-
.../actions/setup-environment-action/action.yml | 2 +-
...low_Java.json => beam_PreCommit_Python_ML.json} | 0
.../workflows/beam_CloudML_Benchmarks_Dataflow.yml | 2 +-
.../beam_Inference_Python_Benchmarks_Dataflow.yml | 10 +-
.../beam_LoadTests_Python_CoGBK_Dataflow_Batch.yml | 8 +-
...m_LoadTests_Python_CoGBK_Dataflow_Streaming.yml | 8 +-
...eam_LoadTests_Python_Combine_Dataflow_Batch.yml | 6 +-
...LoadTests_Python_Combine_Dataflow_Streaming.yml | 6 +-
.../beam_LoadTests_Python_Combine_Flink_Batch.yml | 6 +-
...am_LoadTests_Python_Combine_Flink_Streaming.yml | 4 +-
...LoadTests_Python_FnApiRunner_Microbenchmark.yml | 2 +-
.../beam_LoadTests_Python_GBK_Dataflow_Batch.yml | 10 +-
...eam_LoadTests_Python_GBK_Dataflow_Streaming.yml | 2 +-
...adTests_Python_GBK_reiterate_Dataflow_Batch.yml | 4 +-
...sts_Python_GBK_reiterate_Dataflow_Streaming.yml | 4 +-
.../beam_LoadTests_Python_ParDo_Dataflow_Batch.yml | 8 +-
...m_LoadTests_Python_ParDo_Dataflow_Streaming.yml | 8 +-
.../beam_LoadTests_Python_ParDo_Flink_Batch.yml | 6 +-
...beam_LoadTests_Python_ParDo_Flink_Streaming.yml | 10 +-
...m_LoadTests_Python_SideInput_Dataflow_Batch.yml | 20 +-
.github/workflows/beam_LoadTests_Python_Smoke.yml | 4 +-
...eam_PerformanceTests_BiqQueryIO_Read_Python.yml | 2 +-
...formanceTests_BiqQueryIO_Write_Python_Batch.yml | 2 +-
...erformanceTests_PubsubIOIT_Python_Streaming.yml | 2 +-
..._PerformanceTests_SpannerIO_Read_2GB_Python.yml | 2 +-
...manceTests_SpannerIO_Write_2GB_Python_Batch.yml | 2 +-
.../beam_PerformanceTests_TextIOIT_Python.yml | 2 +-
...PerformanceTests_WordCountIT_PythonVersions.yml | 2 +-
.../beam_PerformanceTests_xlang_KafkaIO_Python.yml | 2 +-
.../beam_PostCommit_PortableJar_Flink.yml | 4 +-
.../beam_PostCommit_PortableJar_Spark.yml | 4 +-
.github/workflows/beam_PostCommit_Python.yml | 2 +-
.github/workflows/beam_PostCommit_Python_Arm.yml | 2 +-
.../beam_PostCommit_Python_Dependency.yml | 2 +-
.../beam_PostCommit_Python_Examples_Direct.yml | 2 +-
.../beam_PostCommit_Python_Examples_Flink.yml | 2 +-
.../beam_PostCommit_Python_Examples_Spark.yml | 2 +-
.../beam_PostCommit_Python_Nexmark_Direct.yml | 2 +-
...stCommit_Python_ValidatesContainer_Dataflow.yml | 2 +-
..._Python_ValidatesContainer_Dataflow_With_RC.yml | 2 +-
..._PostCommit_Python_ValidatesRunner_Dataflow.yml | 2 +-
...eam_PostCommit_Python_ValidatesRunner_Flink.yml | 2 +-
...eam_PostCommit_Python_ValidatesRunner_Samza.yml | 2 +-
...eam_PostCommit_Python_ValidatesRunner_Spark.yml | 2 +-
.../beam_PostCommit_Python_Xlang_Gcp_Dataflow.yml | 2 +-
.../beam_PostCommit_Python_Xlang_Gcp_Direct.yml | 2 +-
.../beam_PostCommit_Python_Xlang_IO_Dataflow.yml | 2 +-
.../workflows/beam_PostCommit_Sickbay_Python.yml | 2 +-
.../beam_PostCommit_TransformService_Direct.yml | 4 +-
.github/workflows/beam_PostCommit_XVR_Direct.yml | 8 +-
.github/workflows/beam_PostCommit_XVR_Flink.yml | 8 +-
...eam_PostCommit_XVR_JavaUsingPython_Dataflow.yml | 4 +-
...eam_PostCommit_XVR_PythonUsingJava_Dataflow.yml | 4 +-
.github/workflows/beam_PostCommit_XVR_Samza.yml | 8 +-
.github/workflows/beam_PostCommit_XVR_Spark3.yml | 8 +-
.../workflows/beam_PreCommit_Portable_Python.yml | 4 +-
.github/workflows/beam_PreCommit_Python.yml | 2 +-
.github/workflows/beam_PreCommit_PythonDocker.yml | 2 +-
.../workflows/beam_PreCommit_Python_Coverage.yml | 2 +-
.../workflows/beam_PreCommit_Python_Dataframes.yml | 2 +-
.../workflows/beam_PreCommit_Python_Examples.yml | 2 +-
.../beam_PreCommit_Python_Integration.yml | 2 +-
.github/workflows/beam_PreCommit_Python_ML.yml | 2 +-
.../workflows/beam_PreCommit_Python_Runners.yml | 2 +-
.../workflows/beam_PreCommit_Python_Transforms.yml | 2 +-
.../beam_PreCommit_Xlang_Generated_Transforms.yml | 2 +-
.../workflows/beam_Publish_Beam_SDK_Snapshots.yml | 1 -
...beam_Python_ValidatesContainer_Dataflow_ARM.yml | 2 +-
.github/workflows/build_release_candidate.yml | 8 +-
.github/workflows/build_wheels.yml | 26 +-
.github/workflows/dask_runner_tests.yml | 3 +-
...Combine_Flink_Streaming_2GB_10_byte_records.txt | 2 +-
...python_Combine_Flink_Streaming_2GB_Fanout_4.txt | 2 +-
...python_Combine_Flink_Streaming_2GB_Fanout_8.txt | 2 +-
.../python_GBK_Flink_Batch_2GB_of_100B_records.txt | 2 +-
...nout_4_times_with_2GB_10-byte_records_total.txt | 2 +-
...nout_8_times_with_2GB_10-byte_records_total.txt | 2 +-
...K_Flink_Batch_reiterate_4_times_10kB_values.txt | 2 +-
.../python_ParDo_Flink_Batch_10_Counters.txt | 2 +-
.../python_ParDo_Flink_Batch_10_Iterations.txt | 2 +-
.../python_ParDo_Flink_Batch_200_Iterations.txt | 2 +-
.../python_ParDo_Flink_Streaming_100_Counters.txt | 2 +-
.../python_ParDo_Flink_Streaming_10_Counters.txt | 2 +-
.../python_ParDo_Flink_Streaming_10_Iterations.txt | 2 +-
...python_ParDo_Flink_Streaming_200_Iterations.txt | 2 +-
.../python_ParDo_Flink_Streaming_5_Iterations.txt | 2 +-
.github/workflows/playground_backend_precommit.yml | 2 +-
.github/workflows/python_dependency_tests.yml | 1 -
.github/workflows/python_tests.yml | 15 +-
.github/workflows/run_perf_alert_tool.yml | 2 +-
.github/workflows/run_rc_validation.yml | 14 +-
.../workflows/tour_of_beam_backend_integration.yml | 6 +-
.github/workflows/typescript_tests.yml | 4 +-
.github/workflows/update_python_dependencies.yml | 1 -
.test-infra/jenkins/PythonTestProperties.groovy | 12 +-
.test-infra/metrics/influxdb/Dockerfile | 2 +-
.test-infra/metrics/influxdb/gsutil/Dockerfile | 2 +-
.test-infra/metrics/sync/github/Dockerfile | 2 +-
.test-infra/metrics/sync/jenkins/Dockerfile | 2 +-
.test-infra/mock-apis/poetry.lock | 8 +-
CHANGES.md | 20 +-
build.gradle.kts | 15 +-
.../org/apache/beam/gradle/BeamModulePlugin.groovy | 25 +-
contributor-docs/discussion-docs/2016.md | 50 +++
contributor-docs/discussion-docs/2017.md | 90 ++++
contributor-docs/discussion-docs/2018.md | 116 +++++
contributor-docs/discussion-docs/2019.md | 103 +++++
contributor-docs/discussion-docs/2020.md | 80 ++++
contributor-docs/discussion-docs/2021.md | 69 +++
contributor-docs/discussion-docs/2022.md | 89 ++++
contributor-docs/discussion-docs/2023.md | 62 +++
contributor-docs/discussion-docs/2024.md | 45 ++
.../discussion-docs/generate_doc_md.py | 321 ++++++++++++++
examples/notebooks/blog/unittests_in_beam.ipynb | 370 ++++++++++++++++
gradle.properties | 2 +-
local-env-setup.sh | 4 +-
playground/README.md | 2 +-
playground/backend/containers/go/build.gradle | 2 +-
playground/backend/containers/java/Dockerfile | 2 +-
playground/backend/containers/python/Dockerfile | 2 +-
playground/backend/containers/python/build.gradle | 2 +-
playground/backend/containers/router/Dockerfile | 2 +-
playground/backend/containers/router/build.gradle | 2 +-
playground/backend/containers/scio/Dockerfile | 2 +-
playground/backend/playground_functions/Dockerfile | 2 +-
.../cloudbuild/playground_ci_examples.sh | 4 +-
release/build.gradle.kts | 4 +-
release/go-licenses/Dockerfile | 2 +-
release/src/main/Dockerfile | 3 +-
.../python-release/python_release_automation.sh | 2 +-
release/src/main/scripts/run_rc_validation.sh | 2 +-
.../beam/runners/dataflow/DataflowRunnerTest.java | 8 +-
.../runners/dataflow/worker/OperationalLimits.java | 14 +-
.../dataflow/worker/StreamingDataflowWorker.java | 5 -
.../worker/StreamingModeExecutionContext.java | 7 +-
.../processing/ComputationWorkExecutorFactory.java | 10 +-
.../worker/StreamingDataflowWorkerTest.java | 15 +-
.../worker/StreamingModeExecutionContextTest.java | 3 +-
.../dataflow/worker/WorkerCustomSourcesTest.java | 6 +-
scripts/tools/bomupgrader.py | 2 +-
sdks/go.mod | 74 ++--
sdks/go.sum | 158 +++----
.../container/license_scripts/dep_urls_java.yaml | 2 +-
.../java/org/apache/beam/sdk/coders/RowCoder.java | 8 +-
.../apache/beam/sdk/coders/RowCoderGenerator.java | 131 +++++-
.../java/org/apache/beam/sdk/io/TextSource.java | 226 ++++++----
.../java/org/apache/beam/sdk/metrics/Lineage.java | 12 +-
.../org/apache/beam/sdk/schemas/SchemaCoder.java | 7 +-
.../transforms/providers/ErrorHandling.java | 3 +-
.../org/apache/beam/sdk/coders/RowCoderTest.java | 134 +++++-
.../org/apache/beam/sdk/io/TextIOReadTest.java | 68 +++
.../org/apache/beam/sdk/io/TextSourceTest.java | 156 +++++++
.../avro/io/SerializableAvroCodecFactory.java | 12 +-
.../avro/io/SerializableAvroCodecFactoryTest.java | 48 ++-
.../beam/sdk/io/gcp/bigquery/BigQueryOptions.java | 8 +
.../sdk/io/gcp/bigquery/BigQueryServicesImpl.java | 28 +-
.../beam/sdk/io/gcp/bigtable/BigtableIO.java | 57 ++-
.../beam/sdk/io/gcp/bigtable/BigtableService.java | 4 +-
.../sdk/io/gcp/bigtable/BigtableServiceImpl.java | 5 +-
.../apache/beam/sdk/io/gcp/pubsub/PubsubIO.java | 61 +--
.../beam/sdk/io/gcp/spanner/MutationUtils.java | 60 +++
.../SpannerReadSchemaTransformProvider.java | 235 ++++++++++
.../SpannerWriteSchemaTransformProvider.java | 160 +++++--
.../beam/sdk/io/gcp/bigtable/BigtableIOTest.java | 8 +-
.../io/gcp/bigtable/BigtableServiceImplTest.java | 5 +-
.../java/org/apache/beam/sdk/io/kafka/KafkaIO.java | 30 +-
.../io/kafka/KafkaReadSchemaTransformProvider.java | 8 +-
...KafkaIOReadImplementationCompatibilityTest.java | 8 +-
.../org/apache/beam/sdk/io/kafka/KafkaIOTest.java | 118 ++++-
sdks/python/apache_beam/io/aws/s3filesystem.py | 8 +
.../apache_beam/io/azure/blobstoragefilesystem.py | 8 +
sdks/python/apache_beam/io/filebasedsink.py | 1 +
sdks/python/apache_beam/io/filebasedsource.py | 2 +
sdks/python/apache_beam/io/filesystem.py | 8 +
sdks/python/apache_beam/io/filesystems.py | 13 +
sdks/python/apache_beam/io/gcp/bigtableio.py | 7 +
sdks/python/apache_beam/io/gcp/bigtableio_test.py | 15 +
sdks/python/apache_beam/io/gcp/gcsfilesystem.py | 8 +
sdks/python/apache_beam/io/gcp/pubsub.py | 67 ++-
sdks/python/apache_beam/io/gcp/pubsub_test.py | 57 +++
sdks/python/apache_beam/io/gcp/spanner_wrapper.py | 76 ++++
sdks/python/apache_beam/metrics/metric.py | 14 +-
sdks/python/apache_beam/metrics/metric_test.py | 4 +-
sdks/python/apache_beam/ml/inference/base_test.py | 13 +-
.../python/apache_beam/options/pipeline_options.py | 8 +
sdks/python/apache_beam/pipeline.py | 8 +-
sdks/python/apache_beam/pipeline_test.py | 12 +-
.../runners/dataflow/internal/apiclient_test.py | 4 +-
.../apache_beam/runners/dataflow/internal/names.py | 2 +-
.../display/pcoll_visualization_test.py | 2 +-
.../apache-beam-jupyterlab-sidepanel/yarn.lock | 480 +++++++++++----------
.../runners/interactive/interactive_beam.py | 121 ++++--
.../interactive/non_interactive_runner_test.py | 292 +++++++++++++
.../runners/interactive/pipeline_fragment.py | 12 +-
.../runners/interactive/recording_manager.py | 17 +-
.../external_transform_provider_it_test.py | 2 +-
.../apache_beam/yaml/examples/io/spanner_read.yaml | 80 ++++
.../yaml/examples/io/spanner_write.yaml | 53 +++
sdks/python/apache_beam/yaml/integration_tests.py | 17 +
sdks/python/apache_beam/yaml/standard_io.yaml | 27 ++
sdks/python/apache_beam/yaml/tests/spanner.yaml | 95 ++++
.../container/base_image_requirements_manual.txt | 2 +-
.../container/py310/base_image_requirements.txt | 53 ++-
.../container/py311/base_image_requirements.txt | 53 ++-
.../container/py312/base_image_requirements.txt | 52 ++-
.../container/py38/base_image_requirements.txt | 51 ++-
.../container/py39/base_image_requirements.txt | 53 ++-
sdks/python/expansion-service-container/Dockerfile | 2 +-
sdks/python/mypy.ini | 3 +-
sdks/python/setup.py | 26 +-
.../tensorrt_runinference/tensor_rt.dockerfile | 4 +-
sdks/python/test-suites/gradle.properties | 20 +-
sdks/python/test-suites/tox/py39/build.gradle | 195 +++++++++
sdks/python/tox.ini | 53 ++-
sdks/standard_external_transforms.yaml | 5 +-
website/www/site/config.toml | 2 +-
website/www/site/content/en/blog/beam-2.59.0.md | 76 ++++
.../site/content/en/blog/unit-testing-in-beam.md | 198 +++++++++
.../content/en/documentation/programming-guide.md | 79 ++++
.../www/site/content/en/get-started/downloads.md | 25 +-
website/www/site/layouts/partials/header.html | 6 -
website/www/yarn.lock | 33 +-
223 files changed, 5239 insertions(+), 1102 deletions(-)
copy .github/trigger_files/{beam_PostCommit_Java_Examples_Dataflow_Java.json
=> beam_PreCommit_Python_ML.json} (100%)
create mode 100644 contributor-docs/discussion-docs/2016.md
create mode 100644 contributor-docs/discussion-docs/2017.md
create mode 100644 contributor-docs/discussion-docs/2018.md
create mode 100644 contributor-docs/discussion-docs/2019.md
create mode 100644 contributor-docs/discussion-docs/2020.md
create mode 100644 contributor-docs/discussion-docs/2021.md
create mode 100644 contributor-docs/discussion-docs/2022.md
create mode 100644 contributor-docs/discussion-docs/2023.md
create mode 100644 contributor-docs/discussion-docs/2024.md
create mode 100644 contributor-docs/discussion-docs/generate_doc_md.py
create mode 100644 examples/notebooks/blog/unittests_in_beam.ipynb
create mode 100644
sdks/java/core/src/test/java/org/apache/beam/sdk/io/TextSourceTest.java
create mode 100644
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerReadSchemaTransformProvider.java
create mode 100644 sdks/python/apache_beam/io/gcp/spanner_wrapper.py
create mode 100644
sdks/python/apache_beam/runners/interactive/non_interactive_runner_test.py
create mode 100644 sdks/python/apache_beam/yaml/examples/io/spanner_read.yaml
create mode 100644 sdks/python/apache_beam/yaml/examples/io/spanner_write.yaml
create mode 100644 sdks/python/apache_beam/yaml/tests/spanner.yaml
create mode 100644 website/www/site/content/en/blog/beam-2.59.0.md
create mode 100644 website/www/site/content/en/blog/unit-testing-in-beam.md
