tvalentyn commented on code in PR #40946:
URL: https://github.com/apache/beam/pull/40946#discussion_r4592040073


##########
website/www/site/content/en/blog/beam-2.78.0.md:
##########
@@ -0,0 +1,125 @@
+---
+title:  "Apache Beam 2.78.0"
+date:   2026-11-?? 14:00:00 -0500
+categories:
+  - blog
+  - release
+authors:
+  - vterentev
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+We are happy to present the new 2.78.0 release of Beam.
+This release includes both improvements and new functionality.
+See the [download page](/get-started/downloads/#2780-2026-11-??) for this 
release.
+
+<!--more-->
+
+For more information on changes in 2.78.0, check out the [detailed release 
notes](https://github.com/apache/beam/milestone/44).
+
+## Highlights
+
+### I/Os
+
+* Added a SchemaTransform for SplunkIO writes, available in Beam YAML as 
`WriteToSplunk` (Java/YAML) 
([#40470](https://github.com/apache/beam/issues/40470)).
+* (Java) FirestoreIO: added 
`RpcQosOptions.Builder.withRampupThrottlingDisabled()`, mirroring the existing 
DatastoreIO option, to opt out of the 500/50/5 write ramp-up 
([#27170](https://github.com/apache/beam/issues/27170)).
+* Added support for attributing BigQuery API quota and billing to a specific 
GCP project (quota project): `quota_project_id` parameter in `ReadFromBigQuery` 
or `--bigquery_quota_project_id` pipeline option, honored by the Python 
BigQuery IO transforms (`WriteToBigQuery` with `FILE_LOADS` or 
`STREAMING_INSERTS`; not yet `STORAGE_WRITE_API`, which is Java-backed), 
`--bigQueryQuotaProjectId` pipeline option (Java), and 
`bigqueryio.WithQuotaProject` read/query option (Go) 
([#37431](https://github.com/apache/beam/issues/37431)).
+* (Java/Python/Go) DebeziumIO: `readAsJson()` can now include `op` and the 
remaining Debezium `source` fields (e.g. MySQL `file`, `pos`, `row`, `ts_ms`; 
PostgreSQL `lsn`, `txId`, `ts_ms`) in `metadata` via `withFullMetadata(true)` / 
`full_metadata=True` / `debeziumio.FullMetadata()`. Off by default 
([#40701](https://github.com/apache/beam/issues/40701)).
+* [IcebergIO] AddFiles is now available through the Managed API with 
`Managed.write(Managed.ICEBERG_ADD_FILES)` (Java/Python) 
([#40827](https://github.com/apache/beam/issues/40827)).
+* (Java) IcebergIO: `AddFiles` can register files in several tables in one 
transform, naming each file's table from its path. Set `path_pattern` 
(`withPathPattern` in Java) to a glob with `{name}` placeholders, such as 
`gs://lake/export/{table}/**`, and make `table` a template over what it 
captures, such as `default.{table}`. The schema pre-pass then runs per table, 
so a table whose files conflict is refused and its files reported while every 
other table is still registered. Requires `schema_evolution_options` and 
`error_handling`; not supported with `dry_run`. Error rows gain a `table` field 
only when `path_pattern` is set, so an existing pipeline's error output is 
unchanged ([#40849](https://github.com/apache/beam/issues/40849)).
+
+### New Features / Improvements
+
+* (Java) Flink runner: sources now forward Beam metrics to Flink at most once 
per second instead of after every record, which was a significant share of the 
CPU of reading. Metrics in the pipeline result are not affected 
([#40607](https://github.com/apache/beam/issues/40607)).
+* (Go) Added `wait.On`, which delays each input window until the corresponding 
windows in its signal PCollections have closed 
([#39909](https://github.com/apache/beam/issues/39909)).
+* (Python) Expanded the SDK worker heap dump 
(`--experiments=enable_heap_dump`) with process RSS, CPython allocator/GC 
stats, and glibc `mallinfo2` native-heap/fragmentation stats to help 
distinguish native-heap from Python-object memory growth 
([#39244](https://github.com/apache/beam/issues/39244)).
+* (Python) Added typed decision questions and the `EvaluateDecisions` 
transform, with an optional Jev model adapter 
([#40220](https://github.com/apache/beam/pull/40220)).
+* The `disableCounterMetrics`, `disableStringSetMetrics` and 
`disableBoundedTrieMetrics` experiments are now honored by the Python SDK, as 
they already were in Java (Python) 
([#38746](https://github.com/apache/beam/issues/38746)).
+* (Go) Pipeline-level `--resource_hints` are now forwarded to expansion 
services, so they apply to cross-language transforms as they do in the Java and 
Python SDKs. The `max_active_bundles_per_worker` hint name is now also accepted 
by `--resource_hints` ([#23893](https://github.com/apache/beam/issues/23893)).
+* `QdrantWriteConfig` can now optionally store chunk content in the point 
payload via `content_payload_key`, so retrieved points include the chunk text 
(Python) ([#40476](https://github.com/apache/beam/issues/40476)).
+* ReadFromBigQuery now supports Lakehouse runtime catalog (BigLake metastore) 
tables with `method=DIRECT_READ`, using 4-part 
`project.catalog.namespace.table` identifiers. Previously 
`project:catalog.namespace.table` was silently mis-parsed and tables that 
report no `numBytes` failed to split (Python) 
([#39597](https://github.com/apache/beam/issues/39597)).
+* (Python) Added `AwsOptions` with `--aws_*` pipeline options (e.g. 
`--aws_access_key_id`, `--aws_region_name`, `--aws_endpoint_url`) that set 
defaults for all AWS service clients. The existing `--s3_*` options in 
`S3Options` override these defaults for the S3 client only 
([#40528](https://github.com/apache/beam/issues/40528)).
+* (Python) Added `--aws_credentials_secret_manager` so AWS credentials 
(`--aws_access_key_id`, `--aws_secret_access_key`, `--aws_session_token` and 
their `--s3_*` overrides) can be secret specs resolved through a secret manager 
such as `GoogleCloudSecretManager`, instead of plain-text values 
([#40528](https://github.com/apache/beam/issues/40528)).
+* (Python) Added `AwsSecret` (`AwsSecretsManager` / `type:awssecret`) to 
retrieve secrets from AWS Secrets Manager, configured from `AwsOptions` 
(`--aws_*`) with optional per-secret connection overrides 
([#40528](https://github.com/apache/beam/issues/40528)).
+* (Go) Added `schema.RegisterLogicalTypeConversion` and 
`schema.RegisterLogicalTypeCoder` to register portable logical types in one 
call, and Go types for the standard `beam:logical_type:date:v1` and 
`beam:logical_type:micros_instant:v1` logical types, so schemas from other SDKs 
with these types can be used in cross-language pipelines 
([#39684](https://github.com/apache/beam/issues/39684)).
+* (Python, Java, Go) Added `--dataflow_container_base_repository` (Python, Go) 
and `--dataflowContainerBaseRepository` (Java) to set the repository (e.g. 
`docker.example.com/my-prefix/cloud-dataflow/v1beta3`) from which the Dataflow 
runner pulls its default SDK container images, including the default images of 
cross-language Apache Beam environments. Defaults to 
`gcr.io/cloud-dataflow/v1beta3` 
([#40569](https://github.com/apache/beam/issues/40569)).
+* (Go) GCS and Dataflow clients now support non-default universes such as 
Trusted Partner Cloud (TPC), using the universe domain from 
`GOOGLE_CLOUD_UNIVERSE_DOMAIN` or the default credentials. Added a 
`--gcs_endpoint` pipeline option to override the GCS endpoint 
([#40569](https://github.com/apache/beam/issues/40569)).
+* (Java) `GcsUtil` now uses the java-storage client (GcsUtil V2) for GCS 
operations by default instead of gcsio (GcsUtil V1). To go back to V1, pass 
`--experiments=disable_gcsutil_v2`. Pipelines with the `use_grpc_for_gcs` 
experiment keep using V1. The `use_gcsutil_v2` experiment is no longer needed 
and is now ignored ([#40712](https://github.com/apache/beam/issues/40712)).
+
+### Breaking Changes

Review Comment:
   there was one CP'ed change that is missing from breaking changes: 
https://github.com/apache/beam/pull/41252/changes#r4592035466



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to