This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a change to branch
dependabot/go_modules/sdks/cloud.google.com/go/datastore-1.14.0
in repository https://gitbox.apache.org/repos/asf/beam.git
discard a7284c4f8bb Bump cloud.google.com/go/datastore from 1.13.0 to 1.14.0
in /sdks
add 6ca4110e16d Always proxy RunInference with large models (#27989)
add d93200dde1e Free disk space before running build_release_candidate
(#28123)
add 2abff20dbe1 Add Java Neo4j IO Direct job (#28090)
add fef4cf8a028 Merge pull request #28124: Allow using CREATE_IF_NEEDED
when writing deletes or updates to BigQuery
add 44b639fdd7e [yaml] yaml_transform.py unit tests (#27338)
add 18908f5b777 adding highmem runner to ARC (#28157)
add a3b82f4b5ec Fix typo in bigqueryio/bigquery.go (#28156)
add 21b7e3abae6 Implement terraform code to provision Vertex AI resources
(#27979)
add b918dd4c169 [Python] Cleanup Python tests in pipeline options module
(#28158)
add 6e0ff033876 Add GitHub Workflow Replacement for Jenkins
job_PreCommit_Java_IOs_Direct (#28141)
add 235b7f955be add bullet points to SDK list on home page (resolves
#22307)
add 9989e632210 Merge pull request #28163: add bullet points to SDK list
on home page
add 98da8506952 [release automation] Add documentation building job to
build_release_candidate.yml workflow (#28136)
add e98bbfd367b Add Vertex AI RunInference Notebook (#28160)
add 0ab1e58146e Revert "Bump github.com/proullon/ramsql from 0.0.1 to
0.1.1 in /sdks (#28089)" (#28165)
add bc7655d7dfe Fix Bigquery read when selected_fields is used with beam
rows output.
add 9ace69a3956 Add some tests.
add fdf3dfc57b9 Merge pull request #28153 Fix Bigquery read when
selected_fields is used with beam rows output.
add db23583d022 [#28126] plumb coder errors with better context. (#28164)
add e26735d69f0 connect err (#28170)
add 505f9421387 Fix flaky test StreamingDataflowWorkerTest (#28173)
add 45ea758b615 [yaml] yaml_transform.py unit tests python - PR 3/3
(#27356)
add 48da8720a42 Add MLTransform IT test (#27620)
add 252c713e9f8 Shell out to model handlers to collect byte sizes (#28182)
add 2d5aced0126 [prism] Add a debugz page to standalone web ui. (#28183)
add 39bafd2ee1d Bump github.com/proullon/ramsql from 0.0.1 to 0.1.2 in
/sdks (#28180)
add 687ed79f1f7 [#28187][prism] worker shutdown, cleanup, log fail, port
spec, grpc recv size (#28184)
add 33aa34f819c Fix a bug that maxConnections is Short in config but read
back as Integer (#28174)
add b4bec8c9b06 add to style guide
add a0d7e53240b Merge pull request #28148: Add PTransform Design Doc to
website
add 6e00a27edcc Bump github.com/tetratelabs/wazero from 1.4.0 to 1.5.0 in
/sdks (#28154)
add f0c0685dbe0 Update ttypescript version. (#28133)
add c8f3b51cec1 Fix ConcurrentModificationException in OutputSampler
add 1eda6dbfcf8 Merge pull request #28150: Fix
ConcurrentModificationException in OutputSampler
add c416f4f6f28 Add per key inference integration test (#28026)
add 10e1480081c fix: fix a race condition in BigtableService cache (#28122)
add 770a4f8b355 Add GitHub Workflow Replacement for Jenkins
job_PreCommit_Java_File-schema-transform_IO_Direct (#28119)
add ffc7821511d Add BagOfWords to tft data transforms (#28188)
add e3c3e81af60 Use commas in passenv (#28204)
add 2a871357bca Add GitHub Workflow Replacement for Jenkins
job_PreCommit_Java_Debezium_IO_Direct and
job_PreCommit_Java_ElasticSearch_IO_Direct (#28027)
add 3b95d14228e Per key model updates (#28161)
add b149b4141d3 Add GitHub Workflow Replacement for Jenkins
job_PreCommit_Spotless (#27665)
add 3e2b9b25a42 Add GitHub Workflow Replacement for Jenkins
job_PreCommit_Java (#27999)
add 6de9565fdcd [Java][Schemas] Improve performance of
GetterBasedSchemaProvider#fromRowFunction (closes #27533) (#27534)
add 63f419726dc Add word count to BagOfWords (#28203)
add 00136b112bc Update KeyMhMapping to KeyModelMapping (#28209)
add 45998db60e9 beam metrics publish job (#28220)
add d616fc2cdcb [Go] Exclude "TestFhirIO.*" from Go PostCommit Dataflow
ARM test suite (#28215)
add 5ccee16106c Add GitHub Workflow Replacement for Jenkins
job_PreCommit_Java_Examples_Dataflow_Java17 (#28082)
add 100c06a0f4b Doc updates for 2.50.0 release (#28055)
add e5ed7aa1582 Add debug logs for updated schema in BigQueryIO (#28189)
add 5f1cc573cd3 Change calls to dockerPush to docker for multi-arch
containers (#28216)
add c360399f1a1 Add github issue link to TODO (#28230)
add 7bc5d1a99d7 SpannerIO: retry if concurrent transaction is aborted by
emulator (#28093)
add 7dfb0cebdd0 Improve Go SDK Staticcheck Compliance (#28224)
add ca0345b8acc Connect GitHub Actions builds to ge.apache.org and remote
build cache (#28212)
add c7b13f90874 adding --namespace to beam metrics gradle build (#28232)
add d359758c3ac Fix 2.50.0 release dates. (#28231)
add 93fd021e64a Added integration tests for GCSFileSystem (#28185)
add a636f25e139 [Go SDK]: Refactor avroio and parquetio Read to use fileio
abstractions (#28177)
add 51f25427270 [Python] refactor payload creation for display_data
(#28225)
add f91bb68b2e9 Add some simple annotations to Python transforms. (#28191)
add 29810a32164 added call to gcloud to setup the docker credential helper
(#28252)
add 9a06c9eca6d remove jenkins PostCommit BeamMetrics job (#28260)
add a43ee6fd983 remove sync stub list from GrpcWindmillServer (#28016)
add a65f55b4ec2 Add Java PVR Flink Batch action (#28221)
add 205083dd72f Keep track of metrics at the KeyedModelHandler level
(#28228)
add 262a2dac6c3 Upgrade pylint, re-enable bad-builtins check (#28239)
add c75975a569d Remove more legacy Runner v1 cruft.
add 0b11e2670f8 Make runner entrypoint more portable.
add 6ab14abc014 lint
add b776ce62a9c Lint, typing, test fix.
add 1cfc0fdc6ff Add an simple runner for pedagogical purposes.
add a13a584b986 Merge branch 'master' into trivial-runner
add 84922403b4f fix up some types
add a65779ca0fa rat
add d4c0dcd7172 cleanup
add b3cd8ecf45c import order
add 4fbc1c0a74f Add test clause.
add 4600517b4cd Merge pull request #27631 Add a simple runner for
pedagogical purposes.
add c1f3e129be6 Add Publish JUnit Test Results Action (#28075)
add 50392e97e8a Add Python PVR Flink action (#28222)
add 1294e730ef1 Python nightly snapshot release workflow (#28197)
add 913f00012c6 Generate separate tmp folder for TestSamzaRunner state
store (#28265)
add 31c6af443ca Implement initial version of the BufferedLogger (#28202)
add fd22129cee0 Fix java17 PostCommit ARM test config issue (#28234)
add 56d5341d201 Add guidance on container versioning (#28269)
add ddd74c68138 Use direct runner to avoid logging flake. (#28271)
add f32252832db Reference the memory leak in CHANGES.md (#28247)
add a472d1a004c Add support for limiting number of models in memory
(#28263)
add 3ff66d38c41 Add relative median threshold filtering to perf alert tool
(#28237)
add 325efced65d Return new ReflectData for both reflect datum reader and
writer (#28280)
add 5221fde659a Add Speech Emotion Recognition TensorFlow Notebook (#28172)
add 0b4302e5f95 [Documentation] Update docs to run SparkPipelineRunner on
a Kubernetes cluster (closes #27984)
add 3591283d722 Fix a couple notebook typos (#28166)
add 3c93f7d17ba Feature/add warnings to file watchers (#28206)
add dc16aa386fe Bump github.com/aws/aws-sdk-go-v2/credentials in /sdks
(#28278)
add 06cdd5ef1f7 Add PostCommit Go VR Samza action (#28251)
add 3463aa3fde5 [prism] Fail jobs on SDK disconnect. (#28193)
add 6b3ce095b0a Reduce the maximum size of input splits in Flink to better
distribute work (#28045)
add 6d5048e0508 Add ARM test suite trigger commands to mass comment script
(#27982)
add f6c64005c07 remove src from path and remove link step for src (#28284)
add 3364271e2c9 Pass runtime configs & variables to BeamSqlSeekableTable
(#28253)
add 547d70ae924 Bump Package (#28292)
add fc4b459cdfb Bump Vertex AI notebook to Beam 2.50 (#28306)
add 7c1c2509242 Add use_public_ips to Go Dataflow Runner (#28308)
add 55e10313341 Various render runner improvements. (#28285)
add 67e88d9fec9 adding nightly publish (#28255)
add 0f39661439f Allow customizing log level in beam.LogElements (#28313)
add 58a25c013b6 Bump golang.org/x/sys from 0.11.0 to 0.12.0 in /sdks
(#28296)
add de726e655ef Bump github.com/aws/aws-sdk-go-v2/feature/s3/manager in
/sdks (#28324)
add 72ab20cf41c Add per key models to changes.md (#28328)
add 99b2f7bd793 Add a note about a known issue. (#28321)
add 741e8c66b0d Better error messages on missing or malformed yaml
providers. (#28311)
add 0a6a6210f3e [Spark, TPCDS] Disable AQE with Spark 3.2 in TPCDS
benchmarks to reduce overhead for complex queries (closes #28249)
add 967e366e1be Merge pull request #28250: [Spark, TPCDS] Disable AQE with
Spark 3.2 in TPCDS benchmarks to reduce overhead for complex queries
add 73a9c5fb3f6 Fix Java 11 Example PreCommit GItHub Action (#28291)
add ce04f351c2b Migrate "beam_PreCommit_SQL" Jenkins job to Github Actions
(#28336)
add 9954d3ed5df added beam_PostCommit_Python_Examples jobs to GitHub
Actions (#28337)
add 24a47b467a7 Pin orjson dependency. (#28322)
add e3974d83274 Interpret yaml mapping elements as rows.
add f4721d58871 Better document Create transform.
add 214fb4fa030 Merge pull request #28334 Interpret yaml mapping elements
as rows.
add a1ac7521a06 [BEAM-11151]Adds the ToStringFn Bundle Processor to Golang
(#28213)
add 9c55daedb54 Bump golang.org/x/text from 0.12.0 to 0.13.0 in /sdks
(#28297)
add 61f0184c528 Retag 2.50.0 RC2 to publish multi-arch containers (#28343)
add 888fe9128d2 Refactor ConvertHelpers in "core" to not depend on Avro
add 96f3dfba4ce Address review comments
add 8037b06b613 Merge pull request #27991: Refactor ConvertHelpers in
"core" to not depend on Avro
add 70e29571358 Add GitHub Workflow Replacement for Jenkins
job_PostCommit_Java_Examples_Dataflow_Java11 (#28257)
add 60d397ecfbd adds maven to image (#28348)
add d559cfe8d57 Inline publish script (#28329)
add 8b2525bca68 Add buffered logger to the Python bootloader (#28317)
add 0612a1a0d8a Bump golang.org/x/net from 0.14.0 to 0.15.0 in /sdks
(#28347)
add 5cc87092b3f Add GitHub Workflow Replacement for Jenkins
job_PostCommit_Java_Examples_Dataflow_Java17 (#28314)
add 7764b832ce0 Allow users to pass inference args into vertex ai model
handler (#28354)
add 4016760cbd4 Bump golang.org/x/oauth2 from 0.11.0 to 0.12.0 in /sdks
(#28345)
add 72e6b9a9d4b Enable use-implicit-booleanness-not-comparison check
(#28353)
add 926774dd02b Direct Python SDK worker crash logs through Logging API
(#28349)
add 4bccd45a07b Apply the monkey-patch with backport of newer changes to
dill pickler also in Python 3.10. (#28355)
add b35ad6dffb7 updade beam arc env file: new image with maven (#28357)
add 5bf6fbf2670 Undo changes to BeamModulePlugin.groovy (#28356)
add 5afb54e8f29 Edit for clarity (#28360)
add 7ba4fa9142a Remove mad and tshift when Pandas >= 2.0 as they are
removed there. (#28332)
add a0e5bf2eb86 Bump classgraph version to 4.8.162 (#28116)
add a7bbded542a Add vertex ai llm example + integration test (#28358)
add ac936b0b89a Add descriptions for the 2 vertex examples (#28361)
add ffb9d113ad9 New MLTransform documentation (#28042)
add 9d452be986c More robust SchemaTransforms.
add 47f3f496c7d Don't lookup and cache id-less schemas using empty id.
add 6f88946ea83 Allow writing a schema'd collection directly with BigQuery
direct write.
add 80794ba1dac Add a Renaming YAML Provider.
add 9a785e93757 Add the ability to specify an inline Python provider from
yaml.
add 67f22527d88 Implement a consistent BigQuery read and write transform.
add 3c447baea57 Prefer the YAML type for naming external transforms.
add 367da322ba4 Bump version number to match.
add 2217cf4ea44 Remove unused import.
add 63c16bbbe42 Fix BigQuery Schema Provider test.
add 778c30eb9a3 Merge pull request #28169 [YAML] Implement a consistent
BigQuery read and write transform.
add d60d784236a Allow extra transforms block to preserve linear chains
with error handling.
add 8efa616cf16 Reasonable unions of compatible schema types.
add c216a87b5b8 Add some documentation on yaml error handling.
add c1a44becad7 Fix Python 3.10 error.
add a7726490917 Fix typos.
add 3a99c9d7900 Merge pull request #28190 Add some documentation on yaml
error handling.
add 19160f44c30 Informed choice of yaml provider for root transforms.
(#28310)
add ee8e53be408 Add standard {ReadFrom,WriteTo}{Csv,Json} yaml endpoints.
(#28268)
add 03b5588d4e6 Post 2.50.0 Streamline release guide improvements (#28287)
add 92fdd54ccb8 Add GitHub Workflow Replacement for Jenkins
job_PostCommit_Sickbay_Python38, job_PostCommit_Sickbay_Python39,
job_PostCommit_Sickbay_Python310 and job_PostCommit_Sickbay_Python311 (#28303)
add f86d9e2d012 update concurrency group definition (#28373)
add f2f5bf6a293 Don't improperly filter newly-added elements that overlap
with a delete.
add 5e4b9bbae5e Merge pull request #28171: Properly handle in-flight
deletes followed by adds in OrderedListState
add e663910627e Github Workflow Replacement for Jenkins Job,
beam_PostCommit_XVR_GoUsingJava_Dataflow (#28369)
add 5bdc450d662 Add GitHub Workflow Replacement for Jenkins
job_PostCommit_Java_Avro_Versions (#28305)
add d083b2081a6 Github Workflow Replacement for Jenkins Job,
beam_PreCommit_Java_Kafka_IO_Direct (#28367)
add 603b51711f3 Github Workflow Replacement for Jenkins Job,
beam_PreCommit_Java_GCP_IO_Direct (Cron, Commit, Phrase) (#28300)
add bec94d936e9 Bump google.golang.org/api from 0.138.0 to 0.139.0 in
/sdks (#28387)
add b8f5c64ed5f Bump cloud.google.com/go/datastore from 1.13.0 to 1.14.0
in /sdks
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (a7284c4f8bb)
\
N -- N -- N
refs/heads/dependabot/go_modules/sdks/cloud.google.com/go/datastore-1.14.0
(b8f5c64ed5f)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.../arc/environments/beam.env | 21 +-
.../arc/images/Dockerfile | 7 +
.github/workflows/README.md | 26 +-
...yml => beam_PostCommit_BeamMetrics_Publish.yml} | 55 +-
.../workflows/beam_PostCommit_Go_Dataflow_ARM.yml | 7 +-
...mit_RAT.yml => beam_PostCommit_Go_VR_Samza.yml} | 49 +-
....yml => beam_PostCommit_Java_Avro_Versions.yml} | 48 +-
.../beam_PostCommit_Java_Examples_Dataflow_ARM.yml | 7 +-
...m_PostCommit_Java_Examples_Dataflow_Java11.yml} | 63 +-
...m_PostCommit_Java_Examples_Dataflow_Java17.yml} | 57 +-
...> beam_PostCommit_Python_Examples_Dataflow.yml} | 49 +-
... => beam_PostCommit_Python_Examples_Direct.yml} | 42 +-
...l => beam_PostCommit_Python_Examples_Flink.yml} | 44 +-
...l => beam_PostCommit_Python_Examples_Spark.yml} | 44 +-
...cker.yml => beam_PostCommit_Sickbay_Python.yml} | 78 +-
...> beam_PostCommit_XVR_GoUsingJava_Dataflow.yml} | 60 +-
.../workflows/beam_PreCommit_CommunityMetrics.yml | 7 +-
.github/workflows/beam_PreCommit_Go.yml | 7 +-
.github/workflows/beam_PreCommit_GoPortable.yml | 7 +-
.github/workflows/beam_PreCommit_ItFramework.yml | 7 +-
.github/workflows/beam_PreCommit_Java.yml | 183 ++
...eCommit_Java_Amazon-Web-Services2_IO_Direct.yml | 7 +-
...reCommit_Java_Amazon-Web-Services_IO_Direct.yml | 7 +-
.../beam_PreCommit_Java_Amqp_IO_Direct.yml | 7 +-
.../beam_PreCommit_Java_Azure_IO_Direct.yml | 7 +-
.../beam_PreCommit_Java_Cassandra_IO_Direct.yml | 7 +-
.../beam_PreCommit_Java_Cdap_IO_Direct.yml | 7 +-
.../beam_PreCommit_Java_Clickhouse_IO_Direct.yml | 7 +-
.../beam_PreCommit_Java_Csv_IO_Direct.yml | 7 +-
... => beam_PreCommit_Java_Debezium_IO_Direct.yml} | 28 +-
...eam_PreCommit_Java_ElasticSearch_IO_Direct.yml} | 32 +-
.../beam_PreCommit_Java_Examples_Dataflow.yml | 7 +-
...eam_PreCommit_Java_Examples_Dataflow_Java11.yml | 9 +-
...am_PreCommit_Java_Examples_Dataflow_Java17.yml} | 59 +-
...ommit_Java_File-schema-transform_IO_Direct.yml} | 23 +-
.../beam_PreCommit_Java_Flink_Versions.yml | 7 +-
...t.yml => beam_PreCommit_Java_GCP_IO_Direct.yml} | 54 +-
.../beam_PreCommit_Java_HBase_IO_Direct.yml | 7 +-
.../beam_PreCommit_Java_HCatalog_IO_Direct.yml | 7 +-
.../beam_PreCommit_Java_Hadoop_IO_Direct.yml | 7 +-
...rect.yml => beam_PreCommit_Java_IOs_Direct.yml} | 46 +-
.../beam_PreCommit_Java_InfluxDb_IO_Direct.yml | 7 +-
.../beam_PreCommit_Java_JDBC_IO_Direct.yml | 7 +-
.../beam_PreCommit_Java_Jms_IO_Direct.yml | 7 +-
...yml => beam_PreCommit_Java_Kafka_IO_Direct.yml} | 34 +-
.../beam_PreCommit_Java_Kinesis_IO_Direct.yml | 7 +-
.../beam_PreCommit_Java_Kudu_IO_Direct.yml | 7 +-
.../beam_PreCommit_Java_MongoDb_IO_Direct.yml | 7 +-
.../beam_PreCommit_Java_Mqtt_IO_Direct.yml | 7 +-
...yml => beam_PreCommit_Java_Neo4j_IO_Direct.yml} | 33 +-
...yml => beam_PreCommit_Java_PVR_Flink_Batch.yml} | 43 +-
.../beam_PreCommit_Java_PVR_Flink_Docker.yml | 7 +-
.../beam_PreCommit_Java_Parquet_IO_Direct.yml | 7 +-
.../beam_PreCommit_Java_Pulsar_IO_Direct.yml | 7 +-
.../beam_PreCommit_Java_RabbitMq_IO_Direct.yml | 7 +-
.../beam_PreCommit_Java_Redis_IO_Direct.yml | 7 +-
.../beam_PreCommit_Java_SingleStore_IO_Direct.yml | 7 +-
.../beam_PreCommit_Java_Snowflake_IO_Direct.yml | 7 +-
.../beam_PreCommit_Java_Solr_IO_Direct.yml | 7 +-
.../beam_PreCommit_Java_Spark3_Versions.yml | 7 +-
.../beam_PreCommit_Java_Splunk_IO_Direct.yml | 7 +-
.../beam_PreCommit_Java_Thrift_IO_Direct.yml | 7 +-
.../beam_PreCommit_Java_Tika_IO_Direct.yml | 7 +-
.../workflows/beam_PreCommit_Kotlin_Examples.yml | 7 +-
.../workflows/beam_PreCommit_Portable_Python.yml | 7 +-
.github/workflows/beam_PreCommit_Python.yml | 7 +-
.github/workflows/beam_PreCommit_PythonDocker.yml | 7 +-
.github/workflows/beam_PreCommit_PythonDocs.yml | 7 +-
.../workflows/beam_PreCommit_PythonFormatter.yml | 7 +-
.github/workflows/beam_PreCommit_PythonLint.yml | 7 +-
.../workflows/beam_PreCommit_Python_Coverage.yml | 7 +-
.../workflows/beam_PreCommit_Python_Dataframes.yml | 7 +-
.../workflows/beam_PreCommit_Python_Examples.yml | 7 +-
.../beam_PreCommit_Python_Integration.yml | 7 +-
...hon.yml => beam_PreCommit_Python_PVR_Flink.yml} | 84 +-
.../workflows/beam_PreCommit_Python_Runners.yml | 7 +-
.../workflows/beam_PreCommit_Python_Transforms.yml | 7 +-
.github/workflows/beam_PreCommit_RAT.yml | 7 +-
...ommit_SQL_Java11.yml => beam_PreCommit_SQL.yml} | 40 +-
.github/workflows/beam_PreCommit_SQL_Java11.yml | 7 +-
.github/workflows/beam_PreCommit_SQL_Java17.yml | 20 +-
.github/workflows/beam_PreCommit_Spotless.yml | 97 +
.github/workflows/beam_PreCommit_Typescript.yml | 7 +-
.github/workflows/beam_PreCommit_Website.yml | 7 +-
.../workflows/beam_PreCommit_Website_Stage_GCS.yml | 7 +-
.github/workflows/beam_PreCommit_Whitespace.yml | 7 +-
...beam_Python_ValidatesContainer_Dataflow_ARM.yml | 7 +-
...it_RAT.yml => beam_Release_NightlySnapshot.yml} | 59 +-
...yml => beam_Release_Python_NightlySnapshot.yml} | 64 +-
.github/workflows/build_release_candidate.yml | 119 ++
.github/workflows/build_wheels.yml | 2 +-
.github/workflows/code_completion_plugin_tests.yml | 3 +
.github/workflows/dask_runner_tests.yml | 2 +-
.github/workflows/go_tests.yml | 2 +-
.github/workflows/java_tests.yml | 6 +-
.github/workflows/local_env_tests.yml | 7 +-
.github/workflows/playground_backend_precommit.yml | 6 +
.github/workflows/playground_frontend_test.yml | 7 +-
.github/workflows/python_dependency_tests.yml | 2 +-
.github/workflows/python_tests.yml | 2 +-
.github/workflows/tour_of_beam_backend.yml | 2 +-
.../workflows/tour_of_beam_backend_integration.yml | 7 +-
.github/workflows/tour_of_beam_frontend_test.yml | 7 +-
.github/workflows/typescript_tests.yml | 2 +-
.github/workflows/update_python_dependencies.yml | 5 +
.pre-commit-config.yaml | 2 +-
.../job_PostCommit_BeamMetrics_Publish.groovy | 41 -
.../job_Release_Python_NightlySnapshot.groovy | 60 -
.test-infra/jenkins/metrics_report/tox.ini | 4 +-
.test-infra/metrics/build.gradle | 6 +-
.../vertex-ai-featurestore/README.md | 74 +
.../vertex-ai-featurestore/featurestore.tf | 89 +
.../vertex-ai-featurestore/synthea.tfvars | 621 ++++++
.../vertex-ai-featurestore/variables.tf | 51 +
CHANGES.md | 22 +-
.../org/apache/beam/gradle/BeamModulePlugin.groovy | 88 +-
.../apache/beam/examples/snippets/Snippets.java | 11 +-
.../beam-ml/nlp_tensorflow_streaming.ipynb | 6 +-
.../beam-ml/run_inference_vertex_ai.ipynb | 396 ++++
.../beam-ml/speech_emotion_tensorflow.ipynb | 2159 ++++++++++++++++++++
release/src/main/scripts/github_actions_jobs.txt | 8 +
release/src/main/scripts/mass_comment.py | 20 +-
release/src/main/scripts/publish_docker_images.sh | 10 +-
.../ClasspathScanningResourcesDetectorTest.java | 2 +-
.../beam/runners/flink/FlinkPipelineOptions.java | 7 +
.../translation/wrappers/SourceInputFormat.java | 22 +-
.../dataflow/worker/WindmillStateInternals.java | 9 +-
.../dataflow/worker/util/BoundedQueueExecutor.java | 24 +-
.../grpcclient/GrpcDeadlineClientInterceptor.java | 58 +
.../windmill/grpcclient/GrpcWindmillServer.java | 154 +-
.../worker/StreamingDataflowWorkerTest.java | 49 +-
.../worker/WindmillStateInternalsTest.java | 65 +
.../grpcclient/GrpcWindmillServerTest.java | 71 +
.../apache/beam/runners/samza/TestSamzaRunner.java | 39 +-
sdks/go.mod | 30 +-
sdks/go.sum | 82 +-
sdks/go/container/tools/buffered_logging.go | 78 +
sdks/go/container/tools/buffered_logging_test.go | 169 ++
sdks/go/pkg/beam/core/graph/fn.go | 2 +-
sdks/go/pkg/beam/core/runtime/exec/sdf.go | 6 +-
sdks/go/pkg/beam/core/runtime/exec/to_string.go | 63 +
.../pkg/beam/core/runtime/exec/to_string_test.go | 57 +
sdks/go/pkg/beam/core/runtime/exec/translate.go | 3 +
sdks/go/pkg/beam/core/runtime/graphx/translate.go | 2 +
sdks/go/pkg/beam/core/runtime/harness/datamgr.go | 3 +-
sdks/go/pkg/beam/io/avroio/avroio.go | 41 +-
sdks/go/pkg/beam/io/bigqueryio/bigquery.go | 4 +-
sdks/go/pkg/beam/io/datastoreio/datastore.go | 10 +-
sdks/go/pkg/beam/io/parquetio/parquetio.go | 48 +-
sdks/go/pkg/beam/register/iter.go | 2 +-
sdks/go/pkg/beam/runners/dataflow/dataflow.go | 22 +
sdks/go/pkg/beam/runners/dataflow/dataflow_test.go | 77 +
sdks/go/pkg/beam/runners/prism/internal/coders.go | 32 +-
.../pkg/beam/runners/prism/internal/coders_test.go | 6 +-
.../beam/runners/prism/internal/config/config.go | 2 +-
.../prism/internal/engine/elementmanager.go | 31 +-
sdks/go/pkg/beam/runners/prism/internal/execute.go | 67 +-
.../beam/runners/prism/internal/handlerunner.go | 15 +-
.../beam/runners/prism/internal/jobservices/job.go | 13 +-
.../prism/internal/jobservices/management.go | 2 +-
.../runners/prism/internal/jobservices/server.go | 3 +-
.../pkg/beam/runners/prism/internal/preprocess.go | 2 +-
sdks/go/pkg/beam/runners/prism/internal/stage.go | 50 +-
.../runners/prism/internal/web/assets/style.css | 20 +
.../pkg/beam/runners/prism/internal/web/debugz.go | 150 ++
.../prism/internal/web/{index.html => debugz.html} | 15 +-
.../pkg/beam/runners/prism/internal/web/index.html | 3 +
sdks/go/pkg/beam/runners/prism/internal/web/web.go | 17 +-
.../beam/runners/prism/internal/worker/bundle.go | 24 +-
.../runners/prism/internal/worker/bundle_test.go | 2 +-
.../beam/runners/prism/internal/worker/worker.go | 111 +-
.../runners/prism/internal/worker/worker_test.go | 23 +-
.../pkg/beam/runners/universal/runnerlib/stage.go | 4 +-
sdks/go/pkg/beam/staticcheck.conf | 2 +-
sdks/go/pkg/beam/testing/passert/equals_test.go | 2 +-
sdks/go/pkg/beam/util/execx/exec.go | 17 +-
sdks/go/pkg/beam/x/debug/head_test.go | 4 +-
sdks/go/test/integration/integration.go | 8 +-
.../main/java/org/apache/beam/sdk/io/FileIO.java | 9 +
.../beam/sdk/schemas/FromRowUsingCreator.java | 320 +--
.../beam/sdk/schemas/utils/ConvertHelpers.java | 34 +-
.../utils/RowSchemaInformationProvider.java | 41 +
.../schemas/utils/SchemaInformationProvider.java} | 29 +-
sdks/java/expansion-service/container/build.gradle | 2 +-
.../sdk/extensions/avro/io/AvroDatumFactory.java | 25 +-
.../utils/AvroSchemaInformationProvider.java | 42 +
.../sdk/extensions/sql/BeamSqlSeekableTable.java | 12 +-
.../sql/impl/transform/BeamJoinTransforms.java | 15 +
.../beam/fn/harness/debug/OutputSampler.java | 12 +-
.../beam/fn/harness/debug/OutputSamplerTest.java | 8 +-
.../beam/sdk/io/gcp/bigquery/BigQueryHelpers.java | 2 +-
.../beam/sdk/io/gcp/bigquery/BigQueryIO.java | 39 +-
.../sdk/io/gcp/bigquery/CreateTableHelpers.java | 9 +
.../beam/sdk/io/gcp/bigquery/CreateTables.java | 5 +
.../sdk/io/gcp/bigquery/DynamicDestinations.java | 9 +
.../gcp/bigquery/DynamicDestinationsHelpers.java | 30 +
.../bigquery/StorageApiDynamicDestinations.java | 36 +-
.../bigquery/StorageApiWriteUnshardedRecords.java | 6 +-
.../bigquery/StorageApiWritesShardedRecords.java | 6 +-
.../beam/sdk/io/gcp/bigquery/WriteTables.java | 2 +
...ueryStorageWriteApiSchemaTransformProvider.java | 133 +-
.../io/gcp/bigtable/BigtableServiceFactory.java | 43 +-
.../apache/beam/sdk/io/gcp/spanner/SpannerIO.java | 10 +-
.../io/gcp/bigquery/StorageApiSinkRowUpdateIT.java | 42 +-
...StorageWriteApiSchemaTransformProviderTest.java | 28 +-
.../sdk/io/gcp/spanner/SpannerIOWriteTest.java | 29 +-
.../beam/sdk/io/jdbc/JdbcSchemaIOProvider.java | 5 +-
.../beam/sdk/io/jdbc/JdbcSchemaIOProviderTest.java | 44 +-
.../apache/beam/sdk/io/neo4j/Neo4jTestUtil.java | 3 +-
sdks/java/testing/tpcds/build.gradle | 1 +
.../controller-container/build.gradle | 2 +-
sdks/python/.pylintrc | 22 +-
sdks/python/apache_beam/dataframe/frames.py | 24 +-
sdks/python/apache_beam/dataframe/frames_test.py | 14 +-
.../apache_beam/examples/inference/README.md | 124 ++
.../inference/huggingface_language_modeling.py | 2 +-
.../pytorch_model_per_key_image_segmentation.py | 311 +++
.../inference/vertex_ai_llm_text_classification.py | 135 ++
.../apache_beam/examples/ml_transform}/__init__.py | 0
.../examples/ml_transform/ml_transform_basic.py | 1 -
.../examples/ml_transform/ml_transform_it_test.py | 106 +
.../ml_transform/vocab_tfidf_processing.py | 190 ++
sdks/python/apache_beam/internal/dill_pickler.py | 2 +-
sdks/python/apache_beam/io/fileio.py | 12 +
sdks/python/apache_beam/io/gcp/bigquery.py | 108 +-
.../apache_beam/io/gcp/bigquery_read_it_test.py | 13 +
.../apache_beam/io/gcp/bigquery_schema_tools.py | 20 +-
.../io/gcp/bigquery_schema_tools_test.py | 15 +
.../io/gcp/gcsfilesystem_integration_test.py | 139 ++
sdks/python/apache_beam/io/iobase.py | 3 +-
sdks/python/apache_beam/ml/inference/base.py | 433 +++-
sdks/python/apache_beam/ml/inference/base_test.py | 430 +++-
.../ml/inference/huggingface_inference.py | 4 +
.../apache_beam/ml/inference/pytorch_inference.py | 1 +
.../ml/inference/pytorch_inference_it_test.py | 48 +
.../ml/inference/vertex_ai_inference.py | 3 +
.../ml/inference/vertex_ai_inference_it_test.py | 22 +-
sdks/python/apache_beam/ml/transforms/tft.py | 84 +-
sdks/python/apache_beam/ml/transforms/tft_test.py | 175 ++
.../python/apache_beam/options/pipeline_options.py | 2 +-
.../apache_beam/options/pipeline_options_test.py | 383 ++--
.../options/pipeline_options_validator_test.py | 624 ++----
sdks/python/apache_beam/pipeline.py | 5 +-
sdks/python/apache_beam/runners/common.py | 8 +
.../runners/portability/fn_api_runner/execution.py | 9 +-
.../portability/fn_api_runner/translations.py | 31 +-
.../runners/portability/portable_runner.py | 33 +-
sdks/python/apache_beam/runners/render.py | 36 +-
sdks/python/apache_beam/runners/render_test.py | 13 +
sdks/python/apache_beam/runners/runner.py | 38 +-
sdks/python/apache_beam/runners/trivial_runner.py | 416 ++++
.../apache_beam/runners/trivial_runner_test.py | 74 +
.../testing/analyzers/perf_analysis_test.py | 21 +-
.../testing/analyzers/perf_analysis_utils.py | 51 +-
.../benchmarks/cloudml/cloudml_benchmark_test.py | 1 -
sdks/python/apache_beam/transforms/display.py | 2 +-
sdks/python/apache_beam/transforms/external.py | 108 +-
.../python/apache_beam/transforms/external_test.py | 10 +-
sdks/python/apache_beam/transforms/ptransform.py | 5 +-
sdks/python/apache_beam/transforms/util.py | 38 +-
sdks/python/apache_beam/transforms/util_test.py | 25 +
.../apache_beam/typehints/schema_registry.py | 7 +-
sdks/python/apache_beam/typehints/schemas.py | 20 +
sdks/python/apache_beam/typehints/schemas_test.py | 13 +
sdks/python/apache_beam/typehints/typehints.py | 9 +
sdks/python/apache_beam/yaml/main.py | 10 +-
sdks/python/apache_beam/yaml/pipeline.schema.yaml | 9 +-
sdks/python/apache_beam/yaml/readme_test.py | 20 +-
sdks/python/apache_beam/yaml/standard_io.yaml | 82 +
sdks/python/apache_beam/yaml/yaml_errors.md | 196 ++
sdks/python/apache_beam/yaml/yaml_io.py | 116 ++
sdks/python/apache_beam/yaml/yaml_mapping.py | 8 +-
sdks/python/apache_beam/yaml/yaml_provider.py | 136 +-
sdks/python/apache_beam/yaml/yaml_transform.py | 172 +-
.../apache_beam/yaml/yaml_transform_scope_test.py | 350 ++++
.../python/apache_beam/yaml/yaml_transform_test.py | 154 +-
.../apache_beam/yaml/yaml_transform_unit_test.py | 1034 ++++++++++
sdks/python/container/boot.go | 30 +-
sdks/python/container/piputil.go | 53 +-
.../container/py310/base_image_requirements.txt | 4 +-
.../container/py311/base_image_requirements.txt | 4 +-
.../container/py38/base_image_requirements.txt | 4 +-
.../container/py39/base_image_requirements.txt | 4 +-
.../expansion-service-container/build.gradle | 2 +-
sdks/python/scripts/run_snapshot_publish.sh | 4 +-
sdks/python/setup.py | 2 +-
sdks/python/tox.ini | 10 +-
sdks/typescript/package-lock.json | 14 +-
sdks/typescript/package.json | 2 +-
website/www/site/config.toml | 2 +-
website/www/site/content/en/blog/beam-2.47.0.md | 4 +-
website/www/site/content/en/blog/beam-2.48.0.md | 4 +-
website/www/site/content/en/blog/beam-2.49.0.md | 2 +
website/www/site/content/en/blog/beam-2.50.0.md | 244 +++
.../en/contribute/ptransform-style-guide.md | 1 +
.../site/content/en/contribute/release-guide.md | 361 ++--
.../content/en/documentation/ml/data-processing.md | 4 +-
.../site/content/en/documentation/ml/overview.md | 3 +-
.../content/en/documentation/ml/preprocess-data.md | 229 +++
.../site/content/en/documentation/runners/spark.md | 47 +-
.../en/documentation/runtime/environments.md | 8 +-
.../en/documentation/sdks/go-dependencies.md | 2 +-
.../en/documentation/sdks/java-dependencies.md | 2 +-
.../en/documentation/sdks/python-dependencies.md | 2 +-
.../sdks/python-pipeline-dependencies.md | 3 +-
.../www/site/content/en/get-started/downloads.md | 14 +-
.../partials/section-menu/en/documentation.html | 1 +
.../layouts/shortcodes/documentation/sdks.html | 11 +-
308 files changed, 13470 insertions(+), 2980 deletions(-)
copy .github/workflows/{beam_PostCommit_Go_Dataflow_ARM.yml =>
beam_PostCommit_BeamMetrics_Publish.yml} (64%)
copy .github/workflows/{beam_PreCommit_RAT.yml =>
beam_PostCommit_Go_VR_Samza.yml} (66%)
copy .github/workflows/{beam_PreCommit_ItFramework.yml =>
beam_PostCommit_Java_Avro_Versions.yml} (61%)
copy .github/workflows/{beam_PreCommit_Java_PVR_Flink_Docker.yml =>
beam_PostCommit_Java_Examples_Dataflow_Java11.yml} (60%)
copy .github/workflows/{beam_PreCommit_ItFramework.yml =>
beam_PostCommit_Java_Examples_Dataflow_Java17.yml} (60%)
copy .github/workflows/{beam_PreCommit_Python_Coverage.yml =>
beam_PostCommit_Python_Examples_Dataflow.yml} (68%)
copy .github/workflows/{beam_PreCommit_Python_Runners.yml =>
beam_PostCommit_Python_Examples_Direct.yml} (71%)
copy .github/workflows/{beam_PreCommit_Python_Runners.yml =>
beam_PostCommit_Python_Examples_Flink.yml} (69%)
copy .github/workflows/{beam_PreCommit_Python_Runners.yml =>
beam_PostCommit_Python_Examples_Spark.yml} (69%)
copy .github/workflows/{beam_PreCommit_Java_PVR_Flink_Docker.yml =>
beam_PostCommit_Sickbay_Python.yml} (54%)
copy .github/workflows/{beam_PreCommit_Java_Csv_IO_Direct.yml =>
beam_PostCommit_XVR_GoUsingJava_Dataflow.yml} (63%)
create mode 100644 .github/workflows/beam_PreCommit_Java.yml
copy .github/workflows/{beam_PreCommit_Java_JDBC_IO_Direct.yml =>
beam_PreCommit_Java_Debezium_IO_Direct.yml} (74%)
copy .github/workflows/{beam_PreCommit_Java_JDBC_IO_Direct.yml =>
beam_PreCommit_Java_ElasticSearch_IO_Direct.yml} (68%)
copy .github/workflows/{beam_PreCommit_SQL_Java17.yml =>
beam_PreCommit_Java_Examples_Dataflow_Java17.yml} (64%)
copy .github/workflows/{beam_PreCommit_Java_Cassandra_IO_Direct.yml =>
beam_PreCommit_Java_File-schema-transform_IO_Direct.yml} (73%)
copy .github/workflows/{beam_PreCommit_Java_Snowflake_IO_Direct.yml =>
beam_PreCommit_Java_GCP_IO_Direct.yml} (61%)
copy .github/workflows/{beam_PreCommit_Java_Azure_IO_Direct.yml =>
beam_PreCommit_Java_IOs_Direct.yml} (70%)
copy .github/workflows/{beam_PreCommit_Java_SingleStore_IO_Direct.yml =>
beam_PreCommit_Java_Kafka_IO_Direct.yml} (68%)
copy .github/workflows/{beam_PreCommit_Java_Snowflake_IO_Direct.yml =>
beam_PreCommit_Java_Neo4j_IO_Direct.yml} (73%)
copy .github/workflows/{beam_PreCommit_Java_PVR_Flink_Docker.yml =>
beam_PreCommit_Java_PVR_Flink_Batch.yml} (70%)
copy .github/workflows/{beam_PreCommit_Portable_Python.yml =>
beam_PreCommit_Python_PVR_Flink.yml} (61%)
copy .github/workflows/{beam_PreCommit_SQL_Java11.yml =>
beam_PreCommit_SQL.yml} (74%)
create mode 100644 .github/workflows/beam_PreCommit_Spotless.yml
copy .github/workflows/{beam_PreCommit_RAT.yml =>
beam_Release_NightlySnapshot.yml} (58%)
copy .github/workflows/{beam_PostCommit_Go_Dataflow_ARM.yml =>
beam_Release_Python_NightlySnapshot.yml} (58%)
delete mode 100644
.test-infra/jenkins/job_PostCommit_BeamMetrics_Publish.groovy
delete mode 100644
.test-infra/jenkins/job_Release_Python_NightlySnapshot.groovy
create mode 100644
.test-infra/terraform/google-cloud-platform/vertex-ai-featurestore/README.md
create mode 100644
.test-infra/terraform/google-cloud-platform/vertex-ai-featurestore/featurestore.tf
create mode 100644
.test-infra/terraform/google-cloud-platform/vertex-ai-featurestore/synthea.tfvars
create mode 100644
.test-infra/terraform/google-cloud-platform/vertex-ai-featurestore/variables.tf
create mode 100644 examples/notebooks/beam-ml/run_inference_vertex_ai.ipynb
create mode 100644 examples/notebooks/beam-ml/speech_emotion_tensorflow.ipynb
create mode 100644 release/src/main/scripts/github_actions_jobs.txt
create mode 100644
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/grpcclient/GrpcDeadlineClientInterceptor.java
create mode 100644 sdks/go/container/tools/buffered_logging.go
create mode 100644 sdks/go/container/tools/buffered_logging_test.go
create mode 100644 sdks/go/pkg/beam/core/runtime/exec/to_string.go
create mode 100644 sdks/go/pkg/beam/core/runtime/exec/to_string_test.go
create mode 100644 sdks/go/pkg/beam/runners/prism/internal/web/debugz.go
copy sdks/go/pkg/beam/runners/prism/internal/web/{index.html => debugz.html}
(75%)
create mode 100644
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/RowSchemaInformationProvider.java
copy
sdks/java/{extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSqlSeekableTable.java
=>
core/src/main/java/org/apache/beam/sdk/schemas/utils/SchemaInformationProvider.java}
(57%)
create mode 100644
sdks/java/extensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/schemas/utils/AvroSchemaInformationProvider.java
create mode 100644
sdks/python/apache_beam/examples/inference/pytorch_model_per_key_image_segmentation.py
create mode 100644
sdks/python/apache_beam/examples/inference/vertex_ai_llm_text_classification.py
copy {examples/notebooks/notebook_test_scripts =>
sdks/python/apache_beam/examples/ml_transform}/__init__.py (100%)
create mode 100644
sdks/python/apache_beam/examples/ml_transform/ml_transform_it_test.py
create mode 100644
sdks/python/apache_beam/examples/ml_transform/vocab_tfidf_processing.py
create mode 100644
sdks/python/apache_beam/io/gcp/gcsfilesystem_integration_test.py
create mode 100644 sdks/python/apache_beam/runners/trivial_runner.py
create mode 100644 sdks/python/apache_beam/runners/trivial_runner_test.py
create mode 100644 sdks/python/apache_beam/yaml/standard_io.yaml
create mode 100644 sdks/python/apache_beam/yaml/yaml_errors.md
create mode 100644 sdks/python/apache_beam/yaml/yaml_io.py
create mode 100644 sdks/python/apache_beam/yaml/yaml_transform_scope_test.py
create mode 100644 sdks/python/apache_beam/yaml/yaml_transform_unit_test.py
create mode 100644 website/www/site/content/en/blog/beam-2.50.0.md
create mode 100644
website/www/site/content/en/documentation/ml/preprocess-data.md
