RocMarshal commented on code in PR #895:
URL: https://github.com/apache/flink-web/pull/895#discussion_r4645571582


##########
docs/content/posts/2026-12-25-release-2.4.0.md:
##########
@@ -0,0 +1,490 @@
+---
+title: "Apache Flink 2.4.0 Release Announcement"
+date: "2026-12-25T00:00:00.000Z"
+aliases:
+- /news/2026/12/25/release-2.4.0.html
+authors:
+- flink:
+  name: "Apache Flink PMC"
+
+---
+
+The Apache Flink PMC is pleased to announce the release of Apache Flink 2.4.0.
+
+This release expands Flink SQL with new join capabilities, more flexible 
changelog conversion,
+and richer Process Table Functions. It also introduces a Python DataFrame API, 
makes application
+state easier to explore through SQL, and improves the scalability of the 
HistoryServer.
+
+Key improvements include common subexpression elimination for SQL 
calculations, built-in UDF
+metrics, early firing for outer interval joins, and a new `LATERAL SNAPSHOT` 
join. In-place
+conversion to materialized tables simplifies adopting managed refresh 
pipelines, while state
+inspection and recovery improvements help users operate stateful applications.
+
+We extend our heartfelt thanks to all contributors for making this release 
possible!
+
+Let's dive into the details.
+
+# Flink SQL Improvements
+
+## Upgrade Calcite to 1.41.0
+
+Flink 2.4 upgrades Apache Calcite, the foundation of Flink's SQL parser and 
relational query
+planner, to version 1.41.0. This brings Flink's planner onto a more recent 
Calcite release and
+incorporates upstream fixes and improvements.
+
+The upgrade also provides the planner-level UUID type support used by the new 
native UUID
+capabilities described below.
+
+**More Information**
+
+* [FLINK-39859](https://issues.apache.org/jira/browse/FLINK-39859)
+
+## Common Subexpression Elimination in Calc Code Generation
+
+SQL queries often repeat the same calculation in several output columns or in 
both a filter and
+a projection. Previously, a Calc operator could evaluate each occurrence 
separately, even when
+the expression was deterministic and its inputs were identical.
+
+Flink 2.4 introduces common subexpression elimination (CSE) in Calc code 
generation for both
+batch and streaming execution. Reusable expressions are evaluated once and 
their results are
+shared, reducing redundant work for built-in functions and user-defined 
functions:
+
+```sql
+SELECT
+  expensive_udf(a, b) + 1 AS col1,
+  expensive_udf(a, b) + 2 AS col2
+FROM source_table
+WHERE expensive_udf(a, b) > 0;
+```
+
+For a deterministic `expensive_udf`, the Calc operator can reuse the result 
across these
+expressions. The optimization preserves short-circuit evaluation in 
expressions such as
+`AND`, `OR`, and `CASE`. Functions with side effects or non-deterministic 
results must declare
+`isDeterministic()` as `false` so that their calls are not incorrectly reused.
+
+**More Information**
+
+* [FLINK-39268](https://issues.apache.org/jira/browse/FLINK-39268)
+
+## Built-in Metrics for User-Defined Functions
+
+User-defined functions can be difficult to diagnose when they introduce 
latency or throw
+exceptions. Flink 2.4 adds opt-in metrics for synchronous and asynchronous 
scalar and table
+functions, making their behavior visible through the existing metrics 
reporters.
+
+Two metrics are registered per function on its executing operator:
+
+* **`udfProcessingTime`**: A histogram of sampled invocation durations. For 
asynchronous
+  functions, this measures the time from dispatch to completion.
+* **`udfExceptionCount`**: A counter of exceptions surfaced by the function, 
including
+  exceptional completion of asynchronous calls.
+
+```yaml
+# Enable UDF metrics (disabled by default)
+table.exec.udf-metric-enabled: true
+
+# Time one invocation out of every 100 (the default sampling interval)
+table.exec.udf-metric.sample-interval: 100
+```
+
+Processing-time sampling limits measurement overhead; exceptions are counted 
on every
+invocation. Aggregate functions and Process Table Functions are outside the 
scope of these
+metrics.
+
+**More Information**
+
+* 
[FLIP-485](https://cwiki.apache.org/confluence/spaces/FLINK/pages/373885706/FLIP-485+Add+UDF+Metrics)
+* [FLINK-38071](https://issues.apache.org/jira/browse/FLINK-38071)
+
+## Early Fire for Outer Interval Joins
+
+An outer interval join normally waits until an unmatched row's time interval 
closes before
+emitting it with a null-padded counterpart. For long intervals, this can delay 
useful results.
+Flink 2.4 introduces an `EARLY_FIRE` hint that emits the unmatched row after a 
configurable
+delay:
+
+```sql
+SELECT /*+ EARLY_FIRE('delay'='5s', 'time-mode'='proctime') */
+  o.id, s.ship_time
+FROM Orders o
+LEFT JOIN Shipments s
+  ON o.id = s.order_id
+  AND o.order_time BETWEEN s.ship_time - INTERVAL '1' HOUR AND s.ship_time;
+```
+
+If a matching shipment arrives later within the interval, Flink retracts the 
speculative result
+and emits the matched result. Downstream operators and sinks must therefore 
accept an updating
+changelog.
+
+The hint supports event-time and processing-time delays. Each unmatched outer 
row fires at
+most once; this is not a periodic trigger. It applies to `LEFT`, `RIGHT`, and 
`FULL` outer
+interval joins and is independent of the early-fire configuration for window 
aggregations.
+
+**More Information**
+
+* 
[FLIP-497](https://cwiki.apache.org/confluence/spaces/FLINK/pages/334760951/FLIP-497+Early+Fire+Support+for+Flink+SQL+Interval+Join)
+* [FLINK-36953](https://issues.apache.org/jira/browse/FLINK-36953)
+
+## FROM_CHANGELOG and TO_CHANGELOG: More Flexible Changelog Conversion
+
+Flink 2.4 extends `FROM_CHANGELOG` and `TO_CHANGELOG` with keyed conversion 
and additional
+control over invalid operation codes and delete records.
+
+* **Keyed upsert conversion**: `FROM_CHANGELOG` supports producing an upsert 
table from an
+  append-only CDC stream by combining `PARTITION BY` with an operation mapping 
that includes
+  `UPDATE_AFTER` but not `UPDATE_BEFORE`. The partition columns must identify 
the upsert key.
+* **Explicit error handling**: `error_handling` selects whether null or 
unknown input operation
+  codes fail the job (`FAIL`, the default) or are skipped (`SKIP`).
+* **Full or partial deletes**: `TO_CHANGELOG` supports keyed conversion and a
+  `produces_full_deletes` parameter, controlling whether delete records 
contain the full row
+  or only the identifying key columns.
+* **Partitioned Table API support**: `PartitionedTable` gains 
`fromChangelog()` and
+  `toChangelog()` convenience methods for keyed conversion.
+
+For example, a CDC stream containing insert, update-after, and delete events 
can be interpreted
+as an upsert table:
+
+```sql
+SELECT * FROM FROM_CHANGELOG(
+  input => TABLE cdc_events PARTITION BY id,
+  op_mapping => MAP[
+    'c', 'INSERT',
+    'u', 'UPDATE_AFTER',
+    'd', 'DELETE'
+  ]
+);
+```
+
+The partition columns define the output's upsert key, so they must match the 
unique key of the
+CDC data. Events for each key must arrive in the correct order.
+
+**More Information**
+
+* 
[FLIP-564](https://cwiki.apache.org/confluence/spaces/FLINK/pages/406620639/FLIP-564+Support+FROM_CHANGELOG+and+TO_CHANGELOG+built-in+PTFs)
+* [FLINK-39258](https://issues.apache.org/jira/browse/FLINK-39258)
+
+## Process Table Functions: Lazy State Access and Broadcast State
+
+Flink 2.4 adds lazy value-state access and broadcast capabilities to Process 
Table Functions
+for more advanced stateful SQL applications.
+
+The new `ValueView` provides explicit `value()`, `update()`, `isEmpty()`, and 
`clear()`
+operations. A function can access a value only when it needs it, avoiding the 
need to eagerly
+read and write every state entry for each invocation.
+
+Broadcast table arguments, declared with `BROADCAST_SEMANTIC_TABLE`, 
distribute control rows
+to all parallel instances. Functions can keep shared rules or configuration in
+`@StateHint(StateKind.BROADCAST)` state and consult it while processing their 
main inputs.
+The `NOTIFY_STATEFUL_SETS` trait additionally allows a broadcast update to 
notify existing
+stateful partitions.
+
+Together, these additions support patterns such as dynamically updated rule 
engines and
+configuration-driven processing directly in PTFs.
+
+**More Information**
+
+* 
[FLIP-565](https://cwiki.apache.org/confluence/spaces/FLINK/pages/406620840/FLIP-565+Improve+ProcessTableFunctions+for+late+data+handling+and+state+access)
+* [FLINK-39254](https://issues.apache.org/jira/browse/FLINK-39254)
+
+## A Test Harness for Process Table Functions
+
+Flink 2.4 introduces `ProcessTableFunctionTestHarness`, a lightweight 
framework for testing
+PTF business logic without starting a full Flink cluster.
+
+The harness lets developers configure scalar and table arguments, feed 
records, advance
+watermarks, and inspect emitted results. It also supports partitioned and 
multi-input
+functions, state inspection and initialization, timers, and state TTL.
+
+This makes it easier to test cases such as per-key state isolation, 
timer-driven output, and
+argument validation in ordinary unit tests. Integration tests remain the way 
to verify the
+complete SQL planner and runtime behavior.
+
+**More Information**
+
+* 
[FLIP-567](https://cwiki.apache.org/confluence/spaces/FLINK/pages/406622164/FLIP-567+Introduce+a+ProcessTableFunction+Test+Harness)
+* [FLINK-39357](https://issues.apache.org/jira/browse/FLINK-39357)
+
+## Strict UTF-8 Validation for BYTES-to-STRING Casts
+
+Previously, casting bytes to a string silently replaced invalid UTF-8 
sequences with the
+Unicode replacement character. Flink 2.4 makes this conversion strict: 
`CAST(bytes AS STRING)`
+now fails on malformed UTF-8, making data-quality problems visible.
+
+Two new functions let users choose how to handle these inputs:
+
+* `IS_VALID_UTF8(bytes)` checks whether a byte sequence is valid UTF-8.
+* `MAKE_VALID_UTF8(bytes)` explicitly decodes bytes while replacing malformed 
sequences.
+
+```sql
+-- Return NULL for invalid UTF-8
+SELECT TRY_CAST(raw_bytes AS STRING) FROM source_table;
+
+-- Preserve the previous replacement behavior explicitly
+SELECT MAKE_VALID_UTF8(raw_bytes) FROM source_table;
+```
+
+Applications that need time to migrate can temporarily restore the previous 
cast behavior
+with `table.exec.legacy-bytes-to-string-cast: true` (default: `false`). 
Connector authors can
+opt into strict validation with `StringData.fromUtf8Bytes()`; the existing
+`StringData.fromBytes()` API remains unchanged.
+
+**More Information**
+
+* 
[FLIP-568](https://cwiki.apache.org/confluence/spaces/FLINK/pages/406623034/FLIP-568+Strict+BYTES-to-STRING+CAST+with+UTF-8+Validation+Utilities)
+* [FLINK-39600](https://issues.apache.org/jira/browse/FLINK-39600)
+
+## Metadata Filter Push-Down for Table Sources
+
+Source metadata, such as a record's timestamp, partition, or offset, can be 
useful for limiting
+what a query reads. Flink 2.4 adds a dedicated push-down path for predicates 
on metadata
+columns, including columns declared with `METADATA VIRTUAL`.
+
+Connectors opt in through `SupportsReadingMetadata` and receive predicates 
expressed in
+metadata key names. They can use accepted predicates to reduce source reads, 
while filters
+that still need evaluation remain in the query. The metadata predicates are 
also preserved
+when compiling and restoring a query plan.
+
+This creates opportunities for metadata-aware pruning and seeking in 
supporting connectors.
+Existing connectors retain their behavior until they implement the new 
capability.
+
+**More Information**
+
+* [FLIP-574](https://cwiki.apache.org/confluence/x/54EmGQ)
+* [FLINK-39421](https://issues.apache.org/jira/browse/FLINK-39421)
+
+## In-place Conversion to Materialized Tables
+
+Adopting a materialized table previously required creating a separate object 
and changing its
+consumers. Flink 2.4 allows `CREATE OR ALTER MATERIALIZED TABLE` to convert an 
existing regular
+table in place, preserving its catalog identity and underlying storage while 
adding a managed
+refresh pipeline.
+
+Conversion is disabled by default. Enable it in the configuration used to 
initialize the
+`TableEnvironment` session:
+
+```yaml
+table.materialized-table.conversion-from-table.enabled: true
+```
+
+The conversion statement defines the materialized table's schema, options, 
query, and refresh
+properties as it would for a new materialized table. Existing primary keys and 
watermarks are
+not automatically carried over and must be declared again if needed. The 
catalog must also
+support the conversion operation.
+
+Conversion is one-way: there is no corresponding operation to turn the 
materialized table
+back into a regular table.
+
+**More Information**
+
+* [FLIP-578](https://cwiki.apache.org/confluence/x/rpImGQ)
+* [FLINK-39847](https://issues.apache.org/jira/browse/FLINK-39847)
+
+## LATERAL SNAPSHOT Join
+
+Event-time temporal joins provide precise historical matching, but their 
output depends on
+watermark progress. This can introduce latency when a dimension table changes 
only
+occasionally.
+
+Flink 2.4 introduces `LATERAL SNAPSHOT`, which joins each probe-side row 
against the build-side
+version currently held in state. An initial load phase buffers probe records 
until the
+configured loading condition is met; subsequent probe records can be joined 
immediately:
+
+```sql
+SELECT o.order_id, r.rate
+FROM Orders AS o
+LEFT JOIN LATERAL SNAPSHOT(
+  input => TABLE CurrencyRates,
+  on_time => DESCRIPTOR(update_time),
+  load_completed_idle_timeout => INTERVAL '30' SECOND
+) AS r
+ON o.currency = r.currency;
+```
+
+For streaming execution, `update_time` must be a watermarked time attribute. 
The optional idle
+timeout provides a way to finish loading when the build-side watermark stops 
advancing.
+
+The join supports inner and left outer joins without requiring a build-side 
primary key.
+Its streaming output is append-only, but matching follows processing-time 
state and can differ
+across executions. Use an event-time temporal join when exact historical 
correspondence is
+required.
+
+**More Information**
+
+* [FLIP-579](https://cwiki.apache.org/confluence/x/e5MmGQ)
+* [FLINK-39779](https://issues.apache.org/jira/browse/FLINK-39779)
+
+## Native UUID and Extended VARIANT Primitives
+
+Flink 2.4 introduces a native `UUID` logical type, allowing identifiers to 
retain their type
+instead of being represented only as strings or binary values. The type 
supports casts to
+and from strings and 16-byte binary values, comparison and ordering, and the 
new `UUID_V4()`
+and `UUID_V7()` generation functions.
+
+The release also extends `VARIANT` to cover the remaining primitives in the 
Parquet Variant
+encoding: time values, nanosecond timestamps with and without time zone, and 
UUIDs. This
+improves interoperability when reading typed variant values written by other 
engines.
+
+The existing `UUID()` function continues to return a string. For time values 
extracted into
+Flink SQL, the SQL runtime's `TIME(3)` precision limit still applies.
+
+**More Information**
+
+* 
[FLIP-604](https://cwiki.apache.org/confluence/spaces/FLINK/pages/446070993/FLIP-604+Complete+VARIANT+Primitive+Coverage+with+UUID+and+Timestamps)
+* [FLINK-40485](https://issues.apache.org/jira/browse/FLINK-40485)
+
+# PyFlink Improvements
+
+## A Python DataFrame API
+
+Flink 2.4 introduces `pyflink.dataframe`, a Python-oriented API built on the 
existing Table
+API and Flink runtime. Users can start with familiar Python records, 
dictionaries, pandas
+DataFrames, or Arrow tables and compose filtering, projection, joins, 
aggregations, and UDF
+calls through DataFrame operations:
+
+```python
+import pyflink.dataframe as pf
+
+people = pf.from_records(
+    [(1, "Alice", 30), (2, "Bob", 17)],
+    schema=["id", "name", "age"],
+)
+
+adults = (
+    people.filter(pf.col("age") >= 18)
+    .with_column("age_next_year", pf.col("age") + 1)
+    .select("id", "name", "age_next_year")
+)
+
+for row in adults.collect():
+    print(row)
+```
+
+The API includes connector-based reads and writes, windowing operations, and 
vectorized
+Python UDF support. Conversion to and from Table API tables allows it to fit 
into existing
+PyFlink applications. Operations use Flink's optimizer and execution 
semantics, with support
+for both batch and streaming workloads.
+
+**More Information**
+
+* 
[FLIP-591](https://cwiki.apache.org/confluence/spaces/FLINK/pages/430409184/FLIP-591+Introducing+Python+DataFrame+API+in+PyFlink)
+* [FLINK-40185](https://issues.apache.org/jira/browse/FLINK-40185)
+
+# Connectors
+
+## Job Information in Source Contexts
+
+Source implementations sometimes need the job ID or name for authentication, 
external offset
+stores, or log correlation. Flink 2.4 exposes `JobInfo` directly through
+`SourceReaderContext.getJobInfo()` and `SplitEnumeratorContext.getJobInfo()`, 
bringing the
+Source API in line with the job information already available to sinks.
+
+Connector authors can now obtain this information on both the reader and 
enumerator sides
+without inspecting metric variables or casting to a richer runtime context. 
The additions
+are default methods, preserving compatibility with existing context 
implementations.
+
+**More Information**
+
+* 
[FLIP-583](https://cwiki.apache.org/confluence/spaces/FLINK/pages/430407938/FLIP-583+Expose+JobInfo+on+Source+contexts)
+* [FLINK-39776](https://issues.apache.org/jira/browse/FLINK-39776)
+
+# Runtime Improvements
+
+## Checkpointing During Recovery: Filtering and Spilling
+
+Flink 2.4 extends checkpointing during recovery with a path that filters 
recovered channel-state
+records before processing and spills the recovered data to local disk.
+
+After rescaling, recovered buffers can contain records belonging to different 
subtasks.
+Filtering and reorganizing those records before they enter input channels 
ensures that a
+checkpoint taken during recovery does not capture duplicate copies of records 
assigned to
+other subtasks.
+
+The filtered data is written to local spill files and then fed into input 
channels as network
+buffers become available. This bounds recovery memory usage for large amounts 
of in-flight
+data. Recovery checkpoints account for both queued buffers and the remaining 
spilled data,
+allowing progress to be preserved while replay is still underway.
+
+This extends the opt-in checkpointing-during-recovery capability; it does not 
enable it by
+default.
+
+**More Information**
+
+* [FLIP-547](https://cwiki.apache.org/confluence/x/UQrxFg)
+* [FLINK-38544: Filtering record before processing with spilling 
strategy](https://issues.apache.org/jira/browse/FLINK-38544)
+
+## Pluggable Storage for the HistoryServer
+
+The HistoryServer traditionally unpacks archived job details into many local 
JSON files.
+For large deployments, this can consume substantial numbers of filesystem 
inodes.
+
+Flink 2.4 introduces a pluggable local archive storage abstraction and an 
embedded RocksDB
+backend. The RocksDB backend stores archived resources as key-value entries, 
reducing the
+number of small files:
+
+```yaml
+historyserver.archive.storage.type: ROCKSDB
+```
+
+File-based storage remains the default (`FILE`). This setting controls the 
HistoryServer's
+local archive cache; JobManager archive locations continue to be configured 
separately.
+
+**More Information**
+
+* 
[FLIP-584](https://cwiki.apache.org/confluence/spaces/FLINK/pages/430408093/FLIP-584+Support+Pluggable+Storage+Backend+for+HistoryServer)
+* [FLINK-39911](https://issues.apache.org/jira/browse/FLINK-39911)
+
+## Lazy Loading for the HistoryServer
+
+Loading every detail of every archived job can delay the HistoryServer's 
overview, especially
+when jobs have large execution graphs or extensive execution histories. Flink 
2.4 adds a
+`LAZY` archive loading mode that makes overview information available while 
detailed resources
+load asynchronously.
+
+```yaml
+historyserver.archive.load.mode: LAZY
+```
+
+If a user opens an archive whose details have not finished loading, the 
HistoryServer
+prioritizes that archive for retrieval. Separate thread pools handle 
background fetching and
+individual on-demand requests, so browsing a particular job does not have to 
wait behind all
+other archives.
+
+The existing `EAGER` loading mode remains the default.
+
+**More Information**
+
+* 
[FLIP-585](https://cwiki.apache.org/confluence/spaces/FLINK/pages/430408096/FLIP-585+Introduce+Lazy+Loading+for+HistoryServer)
+
+## State Catalog: Explore Checkpoints and Savepoints with SQL
+
+Inspecting application state has traditionally required custom State Processor 
API code or
+manually defined table schemas. Flink 2.4 introduces a read-only State Catalog 
that discovers
+checkpoints and savepoints under configured directories and exposes their 
contents through SQL.
+
+Each discovered snapshot becomes a database. State schemas are inferred from 
serializer
+snapshots, allowing supported keyed, non-keyed, and window state to be 
represented as tables.
+Large list and map states can also be exposed in flattened form, making 
individual entries
+easier to query.
+
+```sql
+CREATE CATALOG application_state WITH (
+  'type' = 'state',
+  'directory.prod' = 's3://my-bucket/flink/checkpoints'
+);
+
+USE CATALOG application_state;
+SHOW DATABASES;
+```
+
+Users can select a discovered database and explore its state tables without 
modifying the
+running application. For supported POJO and Avro serializers, state can be 
read without the
+original user classes on the classpath. Custom serializers without sufficient 
schema metadata
+are outside this automatic inference path.
+
+The catalog is read-only, making it suitable for investigating state growth, 
inspecting
+individual keys, and understanding the contents of a saved snapshot.
+
+**More Information**
+
+* 
[FLIP-599](https://cwiki.apache.org/confluence/spaces/FLINK/pages/438009922/FLIP-599+State+Catalog)
+* [FLINK-40175](https://issues.apache.org/jira/browse/FLINK-40175)

Review Comment:
   Missing `Upgrade Notes` and `List of Contributors` sections



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to