This is an automated email from the ASF dual-hosted git repository.
lostluck pushed a change to branch prismDisconnect
in repository https://gitbox.apache.org/repos/asf/beam.git
omit 138b6329534 Add causes, extract bundle failures to RunPipeline
omit 2bfe351fe02 [prism] better workerID, warn on pre-bundle fail, buffer
done chan
omit 5e8498f31ff Reduce flaky short bame for passert test.
omit 072da96afd0 [prism] Fail jobs on SDK disconnect.
add c8f3b51cec1 Fix ConcurrentModificationException in OutputSampler
add 1eda6dbfcf8 Merge pull request #28150: Fix
ConcurrentModificationException in OutputSampler
add c416f4f6f28 Add per key inference integration test (#28026)
add 10e1480081c fix: fix a race condition in BigtableService cache (#28122)
add 770a4f8b355 Add GitHub Workflow Replacement for Jenkins
job_PreCommit_Java_File-schema-transform_IO_Direct (#28119)
add ffc7821511d Add BagOfWords to tft data transforms (#28188)
add e3c3e81af60 Use commas in passenv (#28204)
add 2a871357bca Add GitHub Workflow Replacement for Jenkins
job_PreCommit_Java_Debezium_IO_Direct and
job_PreCommit_Java_ElasticSearch_IO_Direct (#28027)
add 3b95d14228e Per key model updates (#28161)
add b149b4141d3 Add GitHub Workflow Replacement for Jenkins
job_PreCommit_Spotless (#27665)
add 3e2b9b25a42 Add GitHub Workflow Replacement for Jenkins
job_PreCommit_Java (#27999)
add 6de9565fdcd [Java][Schemas] Improve performance of
GetterBasedSchemaProvider#fromRowFunction (closes #27533) (#27534)
add 63f419726dc Add word count to BagOfWords (#28203)
add 00136b112bc Update KeyMhMapping to KeyModelMapping (#28209)
add 45998db60e9 beam metrics publish job (#28220)
add d616fc2cdcb [Go] Exclude "TestFhirIO.*" from Go PostCommit Dataflow
ARM test suite (#28215)
add 5ccee16106c Add GitHub Workflow Replacement for Jenkins
job_PreCommit_Java_Examples_Dataflow_Java17 (#28082)
add 100c06a0f4b Doc updates for 2.50.0 release (#28055)
add e5ed7aa1582 Add debug logs for updated schema in BigQueryIO (#28189)
add 5f1cc573cd3 Change calls to dockerPush to docker for multi-arch
containers (#28216)
add c360399f1a1 Add github issue link to TODO (#28230)
add 7bc5d1a99d7 SpannerIO: retry if concurrent transaction is aborted by
emulator (#28093)
add 7dfb0cebdd0 Improve Go SDK Staticcheck Compliance (#28224)
add ca0345b8acc Connect GitHub Actions builds to ge.apache.org and remote
build cache (#28212)
add c7b13f90874 adding --namespace to beam metrics gradle build (#28232)
add d359758c3ac Fix 2.50.0 release dates. (#28231)
add 93fd021e64a Added integration tests for GCSFileSystem (#28185)
add a636f25e139 [Go SDK]: Refactor avroio and parquetio Read to use fileio
abstractions (#28177)
add 51f25427270 [Python] refactor payload creation for display_data
(#28225)
add f91bb68b2e9 Add some simple annotations to Python transforms. (#28191)
add 29810a32164 added call to gcloud to setup the docker credential helper
(#28252)
add 9a06c9eca6d remove jenkins PostCommit BeamMetrics job (#28260)
add a43ee6fd983 remove sync stub list from GrpcWindmillServer (#28016)
add a65f55b4ec2 Add Java PVR Flink Batch action (#28221)
add 205083dd72f Keep track of metrics at the KeyedModelHandler level
(#28228)
add 262a2dac6c3 Upgrade pylint, re-enable bad-builtins check (#28239)
add c75975a569d Remove more legacy Runner v1 cruft.
add 0b11e2670f8 Make runner entrypoint more portable.
add 6ab14abc014 lint
add b776ce62a9c Lint, typing, test fix.
add 1cfc0fdc6ff Add an simple runner for pedagogical purposes.
add a13a584b986 Merge branch 'master' into trivial-runner
add 84922403b4f fix up some types
add a65779ca0fa rat
add d4c0dcd7172 cleanup
add b3cd8ecf45c import order
add 4fbc1c0a74f Add test clause.
add 4600517b4cd Merge pull request #27631 Add a simple runner for
pedagogical purposes.
add c1f3e129be6 Add Publish JUnit Test Results Action (#28075)
add 50392e97e8a Add Python PVR Flink action (#28222)
add 1294e730ef1 Python nightly snapshot release workflow (#28197)
add 913f00012c6 Generate separate tmp folder for TestSamzaRunner state
store (#28265)
add 31c6af443ca Implement initial version of the BufferedLogger (#28202)
add fd22129cee0 Fix java17 PostCommit ARM test config issue (#28234)
add 56d5341d201 Add guidance on container versioning (#28269)
add ddd74c68138 Use direct runner to avoid logging flake. (#28271)
add f32252832db Reference the memory leak in CHANGES.md (#28247)
add a472d1a004c Add support for limiting number of models in memory
(#28263)
add 140a187430e [prism] Fail jobs on SDK disconnect.
add fd6000fe5d3 Reduce flaky short bame for passert test.
add 68ff2c51d25 [prism] better workerID, warn on pre-bundle fail, buffer
done chan
add e9e42d2d1bb Add causes, extract bundle failures to RunPipeline
add ceb39d87758 Return bundle errors through execPipeline.
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (138b6329534)
\
N -- N -- N refs/heads/prismDisconnect (ceb39d87758)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.github/workflows/README.md | 9 +
...yml => beam_PostCommit_BeamMetrics_Publish.yml} | 53 ++-
.../workflows/beam_PostCommit_Go_Dataflow_ARM.yml | 5 +
.../beam_PostCommit_Java_Examples_Dataflow_ARM.yml | 5 +
.../workflows/beam_PreCommit_CommunityMetrics.yml | 5 +
.github/workflows/beam_PreCommit_Go.yml | 5 +
.github/workflows/beam_PreCommit_GoPortable.yml | 5 +
.github/workflows/beam_PreCommit_ItFramework.yml | 5 +
.github/workflows/beam_PreCommit_Java.yml | 183 +++++++++
...eCommit_Java_Amazon-Web-Services2_IO_Direct.yml | 5 +
...reCommit_Java_Amazon-Web-Services_IO_Direct.yml | 5 +
.../beam_PreCommit_Java_Amqp_IO_Direct.yml | 5 +
.../beam_PreCommit_Java_Azure_IO_Direct.yml | 5 +
.../beam_PreCommit_Java_Cassandra_IO_Direct.yml | 5 +
.../beam_PreCommit_Java_Cdap_IO_Direct.yml | 5 +
.../beam_PreCommit_Java_Clickhouse_IO_Direct.yml | 5 +
.../beam_PreCommit_Java_Csv_IO_Direct.yml | 5 +
... => beam_PreCommit_Java_Debezium_IO_Direct.yml} | 28 +-
...eam_PreCommit_Java_ElasticSearch_IO_Direct.yml} | 30 +-
.../beam_PreCommit_Java_Examples_Dataflow.yml | 5 +
...eam_PreCommit_Java_Examples_Dataflow_Java11.yml | 5 +
...am_PreCommit_Java_Examples_Dataflow_Java17.yml} | 57 +--
...ommit_Java_File-schema-transform_IO_Direct.yml} | 25 +-
.../beam_PreCommit_Java_Flink_Versions.yml | 5 +
.../beam_PreCommit_Java_HBase_IO_Direct.yml | 5 +
.../beam_PreCommit_Java_HCatalog_IO_Direct.yml | 5 +
.../beam_PreCommit_Java_Hadoop_IO_Direct.yml | 5 +
.../workflows/beam_PreCommit_Java_IOs_Direct.yml | 5 +
.../beam_PreCommit_Java_InfluxDb_IO_Direct.yml | 5 +
.../beam_PreCommit_Java_JDBC_IO_Direct.yml | 5 +
.../beam_PreCommit_Java_Jms_IO_Direct.yml | 5 +
.../beam_PreCommit_Java_Kinesis_IO_Direct.yml | 5 +
.../beam_PreCommit_Java_Kudu_IO_Direct.yml | 5 +
.../beam_PreCommit_Java_MongoDb_IO_Direct.yml | 5 +
.../beam_PreCommit_Java_Mqtt_IO_Direct.yml | 5 +
.../beam_PreCommit_Java_Neo4j_IO_Direct.yml | 5 +
...yml => beam_PreCommit_Java_PVR_Flink_Batch.yml} | 43 +--
.../beam_PreCommit_Java_PVR_Flink_Docker.yml | 5 +
.../beam_PreCommit_Java_Parquet_IO_Direct.yml | 5 +
.../beam_PreCommit_Java_Pulsar_IO_Direct.yml | 5 +
.../beam_PreCommit_Java_RabbitMq_IO_Direct.yml | 5 +
.../beam_PreCommit_Java_Redis_IO_Direct.yml | 5 +
.../beam_PreCommit_Java_SingleStore_IO_Direct.yml | 5 +
.../beam_PreCommit_Java_Snowflake_IO_Direct.yml | 5 +
.../beam_PreCommit_Java_Solr_IO_Direct.yml | 5 +
.../beam_PreCommit_Java_Spark3_Versions.yml | 5 +
.../beam_PreCommit_Java_Splunk_IO_Direct.yml | 5 +
.../beam_PreCommit_Java_Thrift_IO_Direct.yml | 5 +
.../beam_PreCommit_Java_Tika_IO_Direct.yml | 5 +
.../workflows/beam_PreCommit_Kotlin_Examples.yml | 5 +
.../workflows/beam_PreCommit_Portable_Python.yml | 5 +
.github/workflows/beam_PreCommit_Python.yml | 5 +
.github/workflows/beam_PreCommit_PythonDocker.yml | 5 +
.github/workflows/beam_PreCommit_PythonDocs.yml | 5 +
.../workflows/beam_PreCommit_PythonFormatter.yml | 5 +
.github/workflows/beam_PreCommit_PythonLint.yml | 5 +
.../workflows/beam_PreCommit_Python_Coverage.yml | 5 +
.../workflows/beam_PreCommit_Python_Dataframes.yml | 5 +
.../workflows/beam_PreCommit_Python_Examples.yml | 5 +
.../beam_PreCommit_Python_Integration.yml | 5 +
...hon.yml => beam_PreCommit_Python_PVR_Flink.yml} | 82 ++--
.../workflows/beam_PreCommit_Python_Runners.yml | 5 +
.../workflows/beam_PreCommit_Python_Transforms.yml | 5 +
.github/workflows/beam_PreCommit_RAT.yml | 5 +
.github/workflows/beam_PreCommit_SQL_Java11.yml | 5 +
.github/workflows/beam_PreCommit_SQL_Java17.yml | 18 +-
...link_Docker.yml => beam_PreCommit_Spotless.yml} | 83 ++--
.github/workflows/beam_PreCommit_Typescript.yml | 5 +
.github/workflows/beam_PreCommit_Website.yml | 5 +
.../workflows/beam_PreCommit_Website_Stage_GCS.yml | 5 +
.github/workflows/beam_PreCommit_Whitespace.yml | 5 +
...beam_Python_ValidatesContainer_Dataflow_ARM.yml | 5 +
...yml => beam_Release_Python_NightlySnapshot.yml} | 70 ++--
.github/workflows/build_release_candidate.yml | 5 +
.github/workflows/code_completion_plugin_tests.yml | 3 +
.github/workflows/java_tests.yml | 4 +
.github/workflows/local_env_tests.yml | 5 +
.github/workflows/playground_backend_precommit.yml | 6 +
.github/workflows/playground_frontend_test.yml | 5 +
.../workflows/tour_of_beam_backend_integration.yml | 5 +
.github/workflows/tour_of_beam_frontend_test.yml | 5 +
.github/workflows/update_python_dependencies.yml | 5 +
.pre-commit-config.yaml | 2 +-
.../job_PostCommit_BeamMetrics_Publish.groovy | 41 --
.../job_Release_Python_NightlySnapshot.groovy | 60 ---
.test-infra/jenkins/metrics_report/tox.ini | 4 +-
.test-infra/metrics/build.gradle | 6 +-
CHANGES.md | 15 +-
.../org/apache/beam/gradle/BeamModulePlugin.groovy | 13 +-
.../apache/beam/examples/snippets/Snippets.java | 11 +-
.../grpcclient/GrpcDeadlineClientInterceptor.java | 58 +++
.../windmill/grpcclient/GrpcWindmillServer.java | 154 +++-----
.../grpcclient/GrpcWindmillServerTest.java | 71 ++++
.../apache/beam/runners/samza/TestSamzaRunner.java | 39 +-
sdks/go/container/tools/buffered_logging.go | 78 ++++
sdks/go/container/tools/buffered_logging_test.go | 169 +++++++++
sdks/go/pkg/beam/core/graph/fn.go | 2 +-
sdks/go/pkg/beam/core/runtime/exec/sdf.go | 6 +-
sdks/go/pkg/beam/core/runtime/harness/datamgr.go | 3 +-
sdks/go/pkg/beam/io/avroio/avroio.go | 41 +-
sdks/go/pkg/beam/io/datastoreio/datastore.go | 10 +-
sdks/go/pkg/beam/io/parquetio/parquetio.go | 48 +--
sdks/go/pkg/beam/register/iter.go | 2 +-
.../beam/runners/prism/internal/config/config.go | 2 +-
sdks/go/pkg/beam/runners/prism/internal/stage.go | 10 +-
.../beam/runners/prism/internal/worker/bundle.go | 11 +-
.../pkg/beam/runners/universal/runnerlib/stage.go | 4 +-
sdks/go/pkg/beam/staticcheck.conf | 2 +-
sdks/go/pkg/beam/util/execx/exec.go | 2 +-
sdks/go/pkg/beam/x/debug/head_test.go | 4 +-
sdks/go/test/integration/integration.go | 8 +-
.../beam/sdk/schemas/FromRowUsingCreator.java | 320 +++++++++-------
sdks/java/expansion-service/container/build.gradle | 2 +-
.../beam/fn/harness/debug/OutputSampler.java | 12 +-
.../beam/fn/harness/debug/OutputSamplerTest.java | 8 +-
.../beam/sdk/io/gcp/bigquery/BigQueryHelpers.java | 2 +-
.../bigquery/StorageApiWriteUnshardedRecords.java | 5 +-
.../bigquery/StorageApiWritesShardedRecords.java | 2 +
.../beam/sdk/io/gcp/bigquery/WriteTables.java | 2 +
.../io/gcp/bigtable/BigtableServiceFactory.java | 43 ++-
.../apache/beam/sdk/io/gcp/spanner/SpannerIO.java | 10 +-
.../sdk/io/gcp/spanner/SpannerIOWriteTest.java | 29 +-
.../controller-container/build.gradle | 2 +-
sdks/python/.pylintrc | 23 +-
.../apache_beam/examples/inference/README.md | 56 +++
.../pytorch_model_per_key_image_segmentation.py | 311 +++++++++++++++
.../io/gcp/gcsfilesystem_integration_test.py | 139 +++++++
sdks/python/apache_beam/ml/inference/base.py | 418 ++++++++++++++++++---
sdks/python/apache_beam/ml/inference/base_test.py | 409 ++++++++++++++++++--
.../ml/inference/pytorch_inference_it_test.py | 48 +++
sdks/python/apache_beam/ml/transforms/tft.py | 84 ++++-
sdks/python/apache_beam/ml/transforms/tft_test.py | 175 +++++++++
sdks/python/apache_beam/runners/common.py | 8 +
.../runners/portability/fn_api_runner/execution.py | 9 +-
.../portability/fn_api_runner/translations.py | 31 +-
.../runners/portability/portable_runner.py | 33 +-
sdks/python/apache_beam/runners/runner.py | 38 +-
sdks/python/apache_beam/runners/trivial_runner.py | 416 ++++++++++++++++++++
.../apache_beam/runners/trivial_runner_test.py | 74 ++++
sdks/python/apache_beam/transforms/display.py | 2 +-
sdks/python/apache_beam/transforms/ptransform.py | 5 +-
.../apache_beam/yaml/yaml_transform_scope_test.py | 10 +-
.../expansion-service-container/build.gradle | 2 +-
sdks/python/tox.ini | 4 +-
website/www/site/config.toml | 2 +-
website/www/site/content/en/blog/beam-2.50.0.md | 243 ++++++++++++
.../en/documentation/runtime/environments.md | 8 +-
.../en/documentation/sdks/go-dependencies.md | 2 +-
.../en/documentation/sdks/java-dependencies.md | 2 +-
.../en/documentation/sdks/python-dependencies.md | 2 +-
.../www/site/content/en/get-started/downloads.md | 14 +-
151 files changed, 3983 insertions(+), 927 deletions(-)
copy .github/workflows/{beam_PostCommit_Go_Dataflow_ARM.yml =>
beam_PostCommit_BeamMetrics_Publish.yml} (69%)
create mode 100644 .github/workflows/beam_PreCommit_Java.yml
copy .github/workflows/{beam_PreCommit_Java_Neo4j_IO_Direct.yml =>
beam_PreCommit_Java_Debezium_IO_Direct.yml} (80%)
copy .github/workflows/{beam_PreCommit_Java_JDBC_IO_Direct.yml =>
beam_PreCommit_Java_ElasticSearch_IO_Direct.yml} (73%)
copy .github/workflows/{beam_PreCommit_SQL_Java17.yml =>
beam_PreCommit_Java_Examples_Dataflow_Java17.yml} (69%)
copy .github/workflows/{beam_PreCommit_Java_Cdap_IO_Direct.yml =>
beam_PreCommit_Java_File-schema-transform_IO_Direct.yml} (79%)
copy .github/workflows/{beam_PreCommit_Java_PVR_Flink_Docker.yml =>
beam_PreCommit_Java_PVR_Flink_Batch.yml} (75%)
copy .github/workflows/{beam_PreCommit_Portable_Python.yml =>
beam_PreCommit_Python_PVR_Flink.yml} (66%)
copy .github/workflows/{beam_PreCommit_Java_PVR_Flink_Docker.yml =>
beam_PreCommit_Spotless.yml} (50%)
copy .github/workflows/{beam_PostCommit_Go_Dataflow_ARM.yml =>
beam_Release_Python_NightlySnapshot.yml} (58%)
delete mode 100644
.test-infra/jenkins/job_PostCommit_BeamMetrics_Publish.groovy
delete mode 100644
.test-infra/jenkins/job_Release_Python_NightlySnapshot.groovy
create mode 100644
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/grpcclient/GrpcDeadlineClientInterceptor.java
create mode 100644 sdks/go/container/tools/buffered_logging.go
create mode 100644 sdks/go/container/tools/buffered_logging_test.go
create mode 100644
sdks/python/apache_beam/examples/inference/pytorch_model_per_key_image_segmentation.py
create mode 100644
sdks/python/apache_beam/io/gcp/gcsfilesystem_integration_test.py
create mode 100644 sdks/python/apache_beam/runners/trivial_runner.py
create mode 100644 sdks/python/apache_beam/runners/trivial_runner_test.py
create mode 100644 website/www/site/content/en/blog/beam-2.50.0.md
