This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a change to branch
dependabot/pip/sdks/python/pillow-gte-7.1.1-and-lt-10
in repository https://gitbox.apache.org/repos/asf/beam.git
omit d9156e2560d Update pillow requirement from <8,>=7.1.1 to >=7.1.1,<10
in /sdks/python
add 14ca840f346 Change the type of Samza Runner MaxBundleSize pipeline
option to match with Flink Runner (#25854)
add 1879587c01c Bump grpcio-tools from 1.37.0 to 1.51.3 in /sdks/python
(#25585)
add 831918f5713 Update sqlalchemy requirement in /sdks/python (#25855)
add eac926878c2 Use autoSharding from config when BigQueryIO.Write is
created. (#25862)
add 738825c90fb Revert "Adding support for DLQ for ZetaSQL (#25426)"
add b2222b5e5d0 Merge pull request #25863: Revert "Adding support for DLQ
for ZetaSQL (#25426)"
add 213b3ed6bf1 Use self hosted hadoop docker image in hdfsIntegrationTest
(#25867)
add cbc1a7661bd Revert "Update sqlalchemy requirement in /sdks/python
(#25855)" (#25872)
add b2afbe0b26f Minor fix: move the getattr out of the loop as it is not
needed to be inside (#25859)
add 821a77beb00 Multi language/runinference example (#25083)
add d9f81b8592d Upgrade tox from 3.x.x to 4.x.x (#25785)
add 167a4c5fb59 Remove changes from MANIFEST.in (#25877)
add b4ee5bd60e0 Update pytest-xdist requirement in /sdks/python (#25833)
add 6f82f34eeb1 Onboard Performance tests using the Change point analysis
tool. (#25754)
add a9826ee2db1 Update default splitting behavior to get fewer, larger,
splits
add de578e606e5 Merge pull request #25871: Update default splitting
behavior to get fewer, larger, splits
add 7555d989799 Make conversion to Iterable tighter (#25866)
add fad7f96dc85 Add support for CPython 3.11 opcodes (#25795)
add ac57b8709d3 Optimize change stream connector with more efficient
batching and blind writes, and add transaction/query tags (#25718)
add e8fad9ca2ba Adding DLQ support for ZetaSQL (#25873)
add 4e96e700c2d Disable firestore integration tests temporarily (#25891)
add 315b53d6611 [Go SDK]: Align filesystem List behaviors (#25811)
add 4ceb475ebe4 Google Cloud Bigtable Change Stream Connector (#25797)
add e8eccaf6290 Update beam.Pardo doc comment about SDFs (#25781)
add 476c43356db add write options with create disposition for bigqueryio
(#25352)
add b5ce1106a24 Add one example to learn beam by doing (#25719)
add 6cb7b8e5f82 Ensure truncate element is wrapped in *FullValue (#25908)
add 5f9bf8b74f3 Pubsub test client fixup (#25907)
add 9a5e5b8ab76 Raise the Runtime error when DoFn.process uses both yield
and return (#25743)
add 6746fbec7f7 Use the cord ctype for proto generation, where available.
add 8c90c64836c Merge pull request #25802 Use the cord ctype for proto
generation, where available.
add 2c075b3603f Advance DetectNewPartition's watermark by aggregating the
(#25906)
add ca0787642a6 Merge pull request #25723: #25722 Add option to propagate
successful storage-api writes
add b2bd2a8cbe2 Refactoring: Moved hard coded environment variables' names
into constants. (#25365)
add 942e3fa5b6d Adds multi-language Python and Java quickstart release
validations (#25910)
add fcf51fb7d98 Update gen_protos to work with new protobuf on os-x.
(#25914)
add 545cadc5199 Move inference tests from py37 to py38 (#25936)
add 92abdf636c5 [Playground] Resolve issue with SCIO examples failing on
start due to a timeout (#24946)
add ea50d98048f Increase logging buffer size to 256kb (#25922)
add 43b3f88c68c [Python] TF with TFHub Notebook Example (#25933)
add 99202b237e3 Fix Illegal to use destination for jms producer (#25886)
add 40ca181ebea Update chromedriver-binary requirement in /sdks/python
(#25898)
add 2a9d3be2bdb Update pillow requirement from <8,>=7.1.1 to >=7.1.1,<10
in /sdks/python
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (d9156e2560d)
\
N -- N -- N
refs/heads/dependabot/pip/sdks/python/pillow-gte-7.1.1-and-lt-10 (2a9d3be2bdb)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.github/workflows/python_tests.yml | 6 +-
.github/workflows/run_perf_alert_tool.yml | 24 +-
build.gradle.kts | 2 +-
.../org/apache/beam/gradle/BeamModulePlugin.groovy | 2 +-
...n_inference_tensorflow_with_tensorflowhub.ipynb | 693 +++++++++++++
.../get-started/learn_beam_basics_by_doing.ipynb | 1095 ++++++++++++++++++++
.../beam/model/fn_execution/v1/beam_fn_api.proto | 2 +-
playground/README.md | 35 +-
playground/backend/README.md | 61 +-
playground/backend/containers/scio/Dockerfile | 17 +-
.../internal/code_processing/code_processing.go | 8 +-
playground/backend/internal/fs_tool/fs.go | 36 +-
playground/backend/internal/fs_tool/fs_test.go | 6 +-
.../setup_tools/life_cycle/life_cycle_setuper.go | 49 +-
.../life_cycle/life_cycle_setuper_test.go | 8 +-
playground/backend/internal/utils/common.go | 51 +-
playground/backend/internal/utils/common_test.go | 87 --
.../backend/internal/utils/preparers_utils.go | 7 +-
.../backend/internal/utils/preparers_utils_test.go | 61 ++
.../internal/utils/test_data/JavaFileName.java | 14 +-
.../backend/internal/utils/test_data}/wordcount.py | 0
playground/backend/new_scio_project.sh | 4 +-
playground/docker-compose.local.yaml | 4 +
playground/infrastructure/checker.py | 9 +-
playground/infrastructure/ci_cd.py | 5 +-
playground/infrastructure/config.py | 5 +-
.../constants.py} | 11 +-
playground/infrastructure/grpc_client.py | 5 +-
playground/infrastructure/helper.py | 3 +-
.../src/main/scripts/build_release_candidate.sh | 3 +-
release/src/main/scripts/run_rc_validation.sh | 251 ++++-
release/src/main/scripts/script.config | 4 +
.../logging/DataflowWorkerLoggingHandler.java | 10 +-
.../beam/runners/samza/SamzaPipelineOptions.java | 8 +-
sdks/go/pkg/beam/core/runtime/exec/sdf.go | 27 +-
sdks/go/pkg/beam/io/bigqueryio/bigquery.go | 43 +-
sdks/go/pkg/beam/io/filesystem/filesystem.go | 1 +
sdks/go/pkg/beam/io/filesystem/gcs/gcs.go | 52 +-
sdks/go/pkg/beam/io/filesystem/local/local_test.go | 19 +
.../go/pkg/beam/io/filesystem/memfs/memory_test.go | 6 +
sdks/go/pkg/beam/io/filesystem/s3/s3.go | 7 +-
sdks/go/pkg/beam/io/filesystem/s3/s3_test.go | 2 +-
sdks/go/pkg/beam/io/filesystem/s3/util.go | 9 -
sdks/go/pkg/beam/io/filesystem/s3/util_test.go | 27 -
sdks/go/pkg/beam/pardo.go | 34 +-
.../filesystem/s3/util.go => util/fsx/prefix.go} | 43 +-
sdks/go/pkg/beam/util/fsx/prefix_test.go | 45 +
.../integration/io/bigqueryio/bigqueryio_test.go | 193 ++++
.../test/integration/io/bigqueryio/helper_test.go | 81 ++
sdks/go/test/integration/primitives/drain.go | 24 +-
.../apache/beam/sdk/coders/RowCoderGenerator.java | 12 +-
.../src/main/java/org/apache/beam/sdk/io/Read.java | 14 +-
.../beam/sdk/transforms/GroupIntoBatches.java | 64 +-
.../extensions/sql/zetasql/BeamZetaSqlCalcRel.java | 14 +-
.../sql/zetasql/BeamZetaSqlCalcRelTest.java | 70 --
.../sql/zetasql/ZetaSqlDialectSpecTest.java | 14 +-
sdks/java/io/google-cloud-platform/build.gradle | 6 +
.../beam/sdk/io/gcp/bigquery/BatchLoads.java | 2 +
.../beam/sdk/io/gcp/bigquery/BigQueryIO.java | 26 +-
.../io/gcp/bigquery/BigQuerySchemaIOProvider.java | 9 +-
.../sdk/io/gcp/bigquery/SplittingIterable.java | 46 +-
.../io/gcp/bigquery/StorageApiConvertMessages.java | 5 +-
.../StorageApiDynamicDestinationsBeamRow.java | 4 +-
...StorageApiDynamicDestinationsGenericRecord.java | 4 +-
.../beam/sdk/io/gcp/bigquery/StorageApiLoads.java | 78 +-
.../io/gcp/bigquery/StorageApiWritePayload.java | 26 +-
.../StorageApiWriteRecordsInconsistent.java | 18 +-
.../bigquery/StorageApiWriteUnshardedRecords.java | 123 ++-
.../bigquery/StorageApiWritesShardedRecords.java | 73 +-
.../sdk/io/gcp/bigquery/StreamingWriteTables.java | 2 +
.../beam/sdk/io/gcp/bigquery/WriteResult.java | 40 +-
...ueryStorageWriteApiSchemaTransformProvider.java | 23 +-
.../beam/sdk/io/gcp/bigtable/BigtableIO.java | 76 +-
.../changestreams/ByteStringRangeHelper.java | 158 +++
.../changestreams/ChangeStreamMetrics.java | 145 +++
.../bigtable/changestreams/TimestampConverter.java | 14 +-
.../changestreams/action/ActionFactory.java | 18 +-
.../changestreams/action/ChangeStreamAction.java | 116 ++-
.../action/DetectNewPartitionsAction.java | 119 ++-
.../action/GenerateInitialPartitionsAction.java | 46 +-
.../action/ReadChangeStreamPartitionAction.java | 126 ++-
.../dao/BigtableChangeStreamAccessor.java | 194 ++++
.../changestreams/dao/ChangeStreamDao.java | 71 +-
.../gcp/bigtable/changestreams/dao/DaoFactory.java | 18 +-
.../changestreams/dao/MetadataTableAdminDao.java | 95 +-
.../changestreams/dao/MetadataTableDao.java | 201 +++-
.../dofn/DetectNewPartitionsDoFn.java | 21 +-
.../changestreams/dofn/InitializeDoFn.java | 33 +-
.../dofn/ReadChangeStreamPartitionDoFn.java | 18 +-
.../encoder/MetadataTableEncoder.java | 62 ++
.../changestreams/encoder/package-info.java | 2 +
.../changestreams/model/PartitionRecord.java | 66 +-
.../ReadChangeStreamPartitionProgressTracker.java | 2 +-
.../changestreams/restriction/StreamProgress.java | 64 +-
.../beam/sdk/io/gcp/pubsub/PubsubTestClient.java | 6 +-
.../MetadataSpannerConfigFactory.java | 2 +-
.../action/ChildPartitionsRecordAction.java | 3 +-
.../action/DetectNewPartitionsAction.java | 19 +-
.../changestreams/dao/PartitionMetadataDao.java | 68 +-
.../apache/beam/sdk/io/gcp/GcpApiSurfaceTest.java | 12 +-
.../sdk/io/gcp/bigquery/BigQueryIOWriteTest.java | 6 +
.../changestreams/ByteStringRangeHelperTest.java | 291 ++++++
.../changestreams/TimestampConverterTest.java | 22 +-
.../action/ChangeStreamActionTest.java | 179 ++++
.../action/DetectNewPartitionsActionTest.java | 183 ++++
.../GenerateInitialPartitionsActionTest.java | 115 ++
.../ReadChangeStreamPartitionActionTest.java | 202 ++++
.../dao/MetadataTableAdminDaoTest.java | 126 +++
.../changestreams/dao/MetadataTableDaoTest.java | 211 ++++
.../changestreams/dofn/InitializeDoFnTest.java | 123 +++
...adChangeStreamPartitionProgressTrackerTest.java | 84 ++
.../action/ChildPartitionsRecordActionTest.java | 4 +-
.../dao/PartitionMetadataDaoTest.java | 29 +-
.../java/org/apache/beam/sdk/io/jms/JmsIO.java | 3 +-
.../inference/multi_language_inference/README.md | 69 ++
.../beam/examples/MultiLangRunInference.java | 92 ++
.../multi_language_custom_transform/__init__.py | 7 +-
.../composite_transform.py | 122 +++
.../multi_language_custom_transform/setup.py | 13 +-
.../io/hdfs_integration_test/docker-compose.yml | 4 +-
.../apache_beam/ml/inference/pytorch_inference.py | 2 +-
.../python/apache_beam/testing/analyzers/README.md | 9 +-
.../apache_beam/testing/analyzers/constants.py | 2 +-
.../testing/analyzers/github_issues_utils.py | 3 +-
.../apache_beam/testing/analyzers/perf_analysis.py | 17 +-
.../testing/analyzers/perf_analysis_utils.py | 20 +-
.../testing/analyzers/tests_config.yaml | 59 +-
.../benchmarks/cloudml/cloudml_benchmark_test.py | 24 -
sdks/python/apache_beam/transforms/core.py | 81 +-
sdks/python/apache_beam/transforms/core_test.py | 113 ++
.../typehints/native_type_compatibility.py | 2 +-
.../typehints/native_type_compatibility_test.py | 8 +
sdks/python/apache_beam/typehints/opcodes.py | 36 +-
.../apache_beam/typehints/trivial_inference.py | 129 ++-
sdks/python/build-requirements.txt | 2 +-
sdks/python/gen_protos.py | 73 +-
sdks/python/setup.py | 4 +-
sdks/python/tox.ini | 18 +-
.../site/content/en/contribute/release-guide.md | 18 +-
.../documentation/ml/multi-language-inference.md | 133 +++
.../site/content/en/documentation/ml/overview.md | 5 +-
.../partials/section-menu/en/documentation.html | 3 +-
142 files changed, 7342 insertions(+), 951 deletions(-)
create mode 100644
examples/notebooks/beam-ml/run_inference_tensorflow_with_tensorflowhub.ipynb
create mode 100644
examples/notebooks/get-started/learn_beam_basics_by_doing.ipynb
rename
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/changestreams/ChangeStreamMutation.java
=> playground/backend/internal/utils/test_data/JavaFileName.java (70%)
copy {sdks/python/apache_beam/examples =>
playground/backend/internal/utils/test_data}/wordcount.py (100%)
mode change 100644 => 100755 playground/backend/new_scio_project.sh
copy playground/{backend/new_scio_project.sh => infrastructure/constants.py}
(68%)
copy sdks/go/pkg/beam/{io/filesystem/s3/util.go => util/fsx/prefix.go} (51%)
create mode 100644 sdks/go/pkg/beam/util/fsx/prefix_test.go
create mode 100644 sdks/go/test/integration/io/bigqueryio/bigqueryio_test.go
create mode 100644 sdks/go/test/integration/io/bigqueryio/helper_test.go
create mode 100644
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/changestreams/dao/BigtableChangeStreamAccessor.java
create mode 100644
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/changestreams/encoder/MetadataTableEncoder.java
create mode 100644
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/changestreams/ByteStringRangeHelperTest.java
create mode 100644
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/changestreams/action/ChangeStreamActionTest.java
create mode 100644
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/changestreams/action/DetectNewPartitionsActionTest.java
create mode 100644
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/changestreams/action/GenerateInitialPartitionsActionTest.java
create mode 100644
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/changestreams/action/ReadChangeStreamPartitionActionTest.java
create mode 100644
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/changestreams/dao/MetadataTableAdminDaoTest.java
create mode 100644
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/changestreams/dao/MetadataTableDaoTest.java
create mode 100644
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/changestreams/dofn/InitializeDoFnTest.java
create mode 100644
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/changestreams/restriction/ReadChangeStreamPartitionProgressTrackerTest.java
create mode 100644
sdks/python/apache_beam/examples/inference/multi_language_inference/README.md
create mode 100644
sdks/python/apache_beam/examples/inference/multi_language_inference/last_word_prediction/src/main/java/org/apache/beam/examples/MultiLangRunInference.java
copy playground/backend/new_scio_project.sh =>
sdks/python/apache_beam/examples/inference/multi_language_inference/multi_language_custom_transform/multi_language_custom_transform/__init__.py
(87%)
create mode 100644
sdks/python/apache_beam/examples/inference/multi_language_inference/multi_language_custom_transform/multi_language_custom_transform/composite_transform.py
copy playground/backend/new_scio_project.sh =>
sdks/python/apache_beam/examples/inference/multi_language_inference/multi_language_custom_transform/setup.py
(79%)
create mode 100644 sdks/python/apache_beam/transforms/core_test.py
create mode 100644
website/www/site/content/en/documentation/ml/multi-language-inference.md
