RocMarshal commented on code in PR #895: URL: https://github.com/apache/flink-web/pull/895#discussion_r4645571582
########## docs/content/posts/2026-12-25-release-2.4.0.md: ########## @@ -0,0 +1,490 @@ +--- +title: "Apache Flink 2.4.0 Release Announcement" +date: "2026-12-25T00:00:00.000Z" +aliases: +- /news/2026/12/25/release-2.4.0.html +authors: +- flink: + name: "Apache Flink PMC" + +--- + +The Apache Flink PMC is pleased to announce the release of Apache Flink 2.4.0. + +This release expands Flink SQL with new join capabilities, more flexible changelog conversion, +and richer Process Table Functions. It also introduces a Python DataFrame API, makes application +state easier to explore through SQL, and improves the scalability of the HistoryServer. + +Key improvements include common subexpression elimination for SQL calculations, built-in UDF +metrics, early firing for outer interval joins, and a new `LATERAL SNAPSHOT` join. In-place +conversion to materialized tables simplifies adopting managed refresh pipelines, while state +inspection and recovery improvements help users operate stateful applications. + +We extend our heartfelt thanks to all contributors for making this release possible! + +Let's dive into the details. + +# Flink SQL Improvements + +## Upgrade Calcite to 1.41.0 + +Flink 2.4 upgrades Apache Calcite, the foundation of Flink's SQL parser and relational query +planner, to version 1.41.0. This brings Flink's planner onto a more recent Calcite release and +incorporates upstream fixes and improvements. + +The upgrade also provides the planner-level UUID type support used by the new native UUID +capabilities described below. + +**More Information** + +* [FLINK-39859](https://issues.apache.org/jira/browse/FLINK-39859) + +## Common Subexpression Elimination in Calc Code Generation + +SQL queries often repeat the same calculation in several output columns or in both a filter and +a projection. Previously, a Calc operator could evaluate each occurrence separately, even when +the expression was deterministic and its inputs were identical. + +Flink 2.4 introduces common subexpression elimination (CSE) in Calc code generation for both +batch and streaming execution. Reusable expressions are evaluated once and their results are +shared, reducing redundant work for built-in functions and user-defined functions: + +```sql +SELECT + expensive_udf(a, b) + 1 AS col1, + expensive_udf(a, b) + 2 AS col2 +FROM source_table +WHERE expensive_udf(a, b) > 0; +``` + +For a deterministic `expensive_udf`, the Calc operator can reuse the result across these +expressions. The optimization preserves short-circuit evaluation in expressions such as +`AND`, `OR`, and `CASE`. Functions with side effects or non-deterministic results must declare +`isDeterministic()` as `false` so that their calls are not incorrectly reused. + +**More Information** + +* [FLINK-39268](https://issues.apache.org/jira/browse/FLINK-39268) + +## Built-in Metrics for User-Defined Functions + +User-defined functions can be difficult to diagnose when they introduce latency or throw +exceptions. Flink 2.4 adds opt-in metrics for synchronous and asynchronous scalar and table +functions, making their behavior visible through the existing metrics reporters. + +Two metrics are registered per function on its executing operator: + +* **`udfProcessingTime`**: A histogram of sampled invocation durations. For asynchronous + functions, this measures the time from dispatch to completion. +* **`udfExceptionCount`**: A counter of exceptions surfaced by the function, including + exceptional completion of asynchronous calls. + +```yaml +# Enable UDF metrics (disabled by default) +table.exec.udf-metric-enabled: true + +# Time one invocation out of every 100 (the default sampling interval) +table.exec.udf-metric.sample-interval: 100 +``` + +Processing-time sampling limits measurement overhead; exceptions are counted on every +invocation. Aggregate functions and Process Table Functions are outside the scope of these +metrics. + +**More Information** + +* [FLIP-485](https://cwiki.apache.org/confluence/spaces/FLINK/pages/373885706/FLIP-485+Add+UDF+Metrics) +* [FLINK-38071](https://issues.apache.org/jira/browse/FLINK-38071) + +## Early Fire for Outer Interval Joins + +An outer interval join normally waits until an unmatched row's time interval closes before +emitting it with a null-padded counterpart. For long intervals, this can delay useful results. +Flink 2.4 introduces an `EARLY_FIRE` hint that emits the unmatched row after a configurable +delay: + +```sql +SELECT /*+ EARLY_FIRE('delay'='5s', 'time-mode'='proctime') */ + o.id, s.ship_time +FROM Orders o +LEFT JOIN Shipments s + ON o.id = s.order_id + AND o.order_time BETWEEN s.ship_time - INTERVAL '1' HOUR AND s.ship_time; +``` + +If a matching shipment arrives later within the interval, Flink retracts the speculative result +and emits the matched result. Downstream operators and sinks must therefore accept an updating +changelog. + +The hint supports event-time and processing-time delays. Each unmatched outer row fires at +most once; this is not a periodic trigger. It applies to `LEFT`, `RIGHT`, and `FULL` outer +interval joins and is independent of the early-fire configuration for window aggregations. + +**More Information** + +* [FLIP-497](https://cwiki.apache.org/confluence/spaces/FLINK/pages/334760951/FLIP-497+Early+Fire+Support+for+Flink+SQL+Interval+Join) +* [FLINK-36953](https://issues.apache.org/jira/browse/FLINK-36953) + +## FROM_CHANGELOG and TO_CHANGELOG: More Flexible Changelog Conversion + +Flink 2.4 extends `FROM_CHANGELOG` and `TO_CHANGELOG` with keyed conversion and additional +control over invalid operation codes and delete records. + +* **Keyed upsert conversion**: `FROM_CHANGELOG` supports producing an upsert table from an + append-only CDC stream by combining `PARTITION BY` with an operation mapping that includes + `UPDATE_AFTER` but not `UPDATE_BEFORE`. The partition columns must identify the upsert key. +* **Explicit error handling**: `error_handling` selects whether null or unknown input operation + codes fail the job (`FAIL`, the default) or are skipped (`SKIP`). +* **Full or partial deletes**: `TO_CHANGELOG` supports keyed conversion and a + `produces_full_deletes` parameter, controlling whether delete records contain the full row + or only the identifying key columns. +* **Partitioned Table API support**: `PartitionedTable` gains `fromChangelog()` and + `toChangelog()` convenience methods for keyed conversion. + +For example, a CDC stream containing insert, update-after, and delete events can be interpreted +as an upsert table: + +```sql +SELECT * FROM FROM_CHANGELOG( + input => TABLE cdc_events PARTITION BY id, + op_mapping => MAP[ + 'c', 'INSERT', + 'u', 'UPDATE_AFTER', + 'd', 'DELETE' + ] +); +``` + +The partition columns define the output's upsert key, so they must match the unique key of the +CDC data. Events for each key must arrive in the correct order. + +**More Information** + +* [FLIP-564](https://cwiki.apache.org/confluence/spaces/FLINK/pages/406620639/FLIP-564+Support+FROM_CHANGELOG+and+TO_CHANGELOG+built-in+PTFs) +* [FLINK-39258](https://issues.apache.org/jira/browse/FLINK-39258) + +## Process Table Functions: Lazy State Access and Broadcast State + +Flink 2.4 adds lazy value-state access and broadcast capabilities to Process Table Functions +for more advanced stateful SQL applications. + +The new `ValueView` provides explicit `value()`, `update()`, `isEmpty()`, and `clear()` +operations. A function can access a value only when it needs it, avoiding the need to eagerly +read and write every state entry for each invocation. + +Broadcast table arguments, declared with `BROADCAST_SEMANTIC_TABLE`, distribute control rows +to all parallel instances. Functions can keep shared rules or configuration in +`@StateHint(StateKind.BROADCAST)` state and consult it while processing their main inputs. +The `NOTIFY_STATEFUL_SETS` trait additionally allows a broadcast update to notify existing +stateful partitions. + +Together, these additions support patterns such as dynamically updated rule engines and +configuration-driven processing directly in PTFs. + +**More Information** + +* [FLIP-565](https://cwiki.apache.org/confluence/spaces/FLINK/pages/406620840/FLIP-565+Improve+ProcessTableFunctions+for+late+data+handling+and+state+access) +* [FLINK-39254](https://issues.apache.org/jira/browse/FLINK-39254) + +## A Test Harness for Process Table Functions + +Flink 2.4 introduces `ProcessTableFunctionTestHarness`, a lightweight framework for testing +PTF business logic without starting a full Flink cluster. + +The harness lets developers configure scalar and table arguments, feed records, advance +watermarks, and inspect emitted results. It also supports partitioned and multi-input +functions, state inspection and initialization, timers, and state TTL. + +This makes it easier to test cases such as per-key state isolation, timer-driven output, and +argument validation in ordinary unit tests. Integration tests remain the way to verify the +complete SQL planner and runtime behavior. + +**More Information** + +* [FLIP-567](https://cwiki.apache.org/confluence/spaces/FLINK/pages/406622164/FLIP-567+Introduce+a+ProcessTableFunction+Test+Harness) +* [FLINK-39357](https://issues.apache.org/jira/browse/FLINK-39357) + +## Strict UTF-8 Validation for BYTES-to-STRING Casts + +Previously, casting bytes to a string silently replaced invalid UTF-8 sequences with the +Unicode replacement character. Flink 2.4 makes this conversion strict: `CAST(bytes AS STRING)` +now fails on malformed UTF-8, making data-quality problems visible. + +Two new functions let users choose how to handle these inputs: + +* `IS_VALID_UTF8(bytes)` checks whether a byte sequence is valid UTF-8. +* `MAKE_VALID_UTF8(bytes)` explicitly decodes bytes while replacing malformed sequences. + +```sql +-- Return NULL for invalid UTF-8 +SELECT TRY_CAST(raw_bytes AS STRING) FROM source_table; + +-- Preserve the previous replacement behavior explicitly +SELECT MAKE_VALID_UTF8(raw_bytes) FROM source_table; +``` + +Applications that need time to migrate can temporarily restore the previous cast behavior +with `table.exec.legacy-bytes-to-string-cast: true` (default: `false`). Connector authors can +opt into strict validation with `StringData.fromUtf8Bytes()`; the existing +`StringData.fromBytes()` API remains unchanged. + +**More Information** + +* [FLIP-568](https://cwiki.apache.org/confluence/spaces/FLINK/pages/406623034/FLIP-568+Strict+BYTES-to-STRING+CAST+with+UTF-8+Validation+Utilities) +* [FLINK-39600](https://issues.apache.org/jira/browse/FLINK-39600) + +## Metadata Filter Push-Down for Table Sources + +Source metadata, such as a record's timestamp, partition, or offset, can be useful for limiting +what a query reads. Flink 2.4 adds a dedicated push-down path for predicates on metadata +columns, including columns declared with `METADATA VIRTUAL`. + +Connectors opt in through `SupportsReadingMetadata` and receive predicates expressed in +metadata key names. They can use accepted predicates to reduce source reads, while filters +that still need evaluation remain in the query. The metadata predicates are also preserved +when compiling and restoring a query plan. + +This creates opportunities for metadata-aware pruning and seeking in supporting connectors. +Existing connectors retain their behavior until they implement the new capability. + +**More Information** + +* [FLIP-574](https://cwiki.apache.org/confluence/x/54EmGQ) +* [FLINK-39421](https://issues.apache.org/jira/browse/FLINK-39421) + +## In-place Conversion to Materialized Tables + +Adopting a materialized table previously required creating a separate object and changing its +consumers. Flink 2.4 allows `CREATE OR ALTER MATERIALIZED TABLE` to convert an existing regular +table in place, preserving its catalog identity and underlying storage while adding a managed +refresh pipeline. + +Conversion is disabled by default. Enable it in the configuration used to initialize the +`TableEnvironment` session: + +```yaml +table.materialized-table.conversion-from-table.enabled: true +``` + +The conversion statement defines the materialized table's schema, options, query, and refresh +properties as it would for a new materialized table. Existing primary keys and watermarks are +not automatically carried over and must be declared again if needed. The catalog must also +support the conversion operation. + +Conversion is one-way: there is no corresponding operation to turn the materialized table +back into a regular table. + +**More Information** + +* [FLIP-578](https://cwiki.apache.org/confluence/x/rpImGQ) +* [FLINK-39847](https://issues.apache.org/jira/browse/FLINK-39847) + +## LATERAL SNAPSHOT Join + +Event-time temporal joins provide precise historical matching, but their output depends on +watermark progress. This can introduce latency when a dimension table changes only +occasionally. + +Flink 2.4 introduces `LATERAL SNAPSHOT`, which joins each probe-side row against the build-side +version currently held in state. An initial load phase buffers probe records until the +configured loading condition is met; subsequent probe records can be joined immediately: + +```sql +SELECT o.order_id, r.rate +FROM Orders AS o +LEFT JOIN LATERAL SNAPSHOT( + input => TABLE CurrencyRates, + on_time => DESCRIPTOR(update_time), + load_completed_idle_timeout => INTERVAL '30' SECOND +) AS r +ON o.currency = r.currency; +``` + +For streaming execution, `update_time` must be a watermarked time attribute. The optional idle +timeout provides a way to finish loading when the build-side watermark stops advancing. + +The join supports inner and left outer joins without requiring a build-side primary key. +Its streaming output is append-only, but matching follows processing-time state and can differ +across executions. Use an event-time temporal join when exact historical correspondence is +required. + +**More Information** + +* [FLIP-579](https://cwiki.apache.org/confluence/x/e5MmGQ) +* [FLINK-39779](https://issues.apache.org/jira/browse/FLINK-39779) + +## Native UUID and Extended VARIANT Primitives + +Flink 2.4 introduces a native `UUID` logical type, allowing identifiers to retain their type +instead of being represented only as strings or binary values. The type supports casts to +and from strings and 16-byte binary values, comparison and ordering, and the new `UUID_V4()` +and `UUID_V7()` generation functions. + +The release also extends `VARIANT` to cover the remaining primitives in the Parquet Variant +encoding: time values, nanosecond timestamps with and without time zone, and UUIDs. This +improves interoperability when reading typed variant values written by other engines. + +The existing `UUID()` function continues to return a string. For time values extracted into +Flink SQL, the SQL runtime's `TIME(3)` precision limit still applies. + +**More Information** + +* [FLIP-604](https://cwiki.apache.org/confluence/spaces/FLINK/pages/446070993/FLIP-604+Complete+VARIANT+Primitive+Coverage+with+UUID+and+Timestamps) +* [FLINK-40485](https://issues.apache.org/jira/browse/FLINK-40485) + +# PyFlink Improvements + +## A Python DataFrame API + +Flink 2.4 introduces `pyflink.dataframe`, a Python-oriented API built on the existing Table +API and Flink runtime. Users can start with familiar Python records, dictionaries, pandas +DataFrames, or Arrow tables and compose filtering, projection, joins, aggregations, and UDF +calls through DataFrame operations: + +```python +import pyflink.dataframe as pf + +people = pf.from_records( + [(1, "Alice", 30), (2, "Bob", 17)], + schema=["id", "name", "age"], +) + +adults = ( + people.filter(pf.col("age") >= 18) + .with_column("age_next_year", pf.col("age") + 1) + .select("id", "name", "age_next_year") +) + +for row in adults.collect(): + print(row) +``` + +The API includes connector-based reads and writes, windowing operations, and vectorized +Python UDF support. Conversion to and from Table API tables allows it to fit into existing +PyFlink applications. Operations use Flink's optimizer and execution semantics, with support +for both batch and streaming workloads. + +**More Information** + +* [FLIP-591](https://cwiki.apache.org/confluence/spaces/FLINK/pages/430409184/FLIP-591+Introducing+Python+DataFrame+API+in+PyFlink) +* [FLINK-40185](https://issues.apache.org/jira/browse/FLINK-40185) + +# Connectors + +## Job Information in Source Contexts + +Source implementations sometimes need the job ID or name for authentication, external offset +stores, or log correlation. Flink 2.4 exposes `JobInfo` directly through +`SourceReaderContext.getJobInfo()` and `SplitEnumeratorContext.getJobInfo()`, bringing the +Source API in line with the job information already available to sinks. + +Connector authors can now obtain this information on both the reader and enumerator sides +without inspecting metric variables or casting to a richer runtime context. The additions +are default methods, preserving compatibility with existing context implementations. + +**More Information** + +* [FLIP-583](https://cwiki.apache.org/confluence/spaces/FLINK/pages/430407938/FLIP-583+Expose+JobInfo+on+Source+contexts) +* [FLINK-39776](https://issues.apache.org/jira/browse/FLINK-39776) + +# Runtime Improvements + +## Checkpointing During Recovery: Filtering and Spilling + +Flink 2.4 extends checkpointing during recovery with a path that filters recovered channel-state +records before processing and spills the recovered data to local disk. + +After rescaling, recovered buffers can contain records belonging to different subtasks. +Filtering and reorganizing those records before they enter input channels ensures that a +checkpoint taken during recovery does not capture duplicate copies of records assigned to +other subtasks. + +The filtered data is written to local spill files and then fed into input channels as network +buffers become available. This bounds recovery memory usage for large amounts of in-flight +data. Recovery checkpoints account for both queued buffers and the remaining spilled data, +allowing progress to be preserved while replay is still underway. + +This extends the opt-in checkpointing-during-recovery capability; it does not enable it by +default. + +**More Information** + +* [FLIP-547](https://cwiki.apache.org/confluence/x/UQrxFg) +* [FLINK-38544: Filtering record before processing with spilling strategy](https://issues.apache.org/jira/browse/FLINK-38544) + +## Pluggable Storage for the HistoryServer + +The HistoryServer traditionally unpacks archived job details into many local JSON files. +For large deployments, this can consume substantial numbers of filesystem inodes. + +Flink 2.4 introduces a pluggable local archive storage abstraction and an embedded RocksDB +backend. The RocksDB backend stores archived resources as key-value entries, reducing the +number of small files: + +```yaml +historyserver.archive.storage.type: ROCKSDB +``` + +File-based storage remains the default (`FILE`). This setting controls the HistoryServer's +local archive cache; JobManager archive locations continue to be configured separately. + +**More Information** + +* [FLIP-584](https://cwiki.apache.org/confluence/spaces/FLINK/pages/430408093/FLIP-584+Support+Pluggable+Storage+Backend+for+HistoryServer) +* [FLINK-39911](https://issues.apache.org/jira/browse/FLINK-39911) + +## Lazy Loading for the HistoryServer + +Loading every detail of every archived job can delay the HistoryServer's overview, especially +when jobs have large execution graphs or extensive execution histories. Flink 2.4 adds a +`LAZY` archive loading mode that makes overview information available while detailed resources +load asynchronously. + +```yaml +historyserver.archive.load.mode: LAZY +``` + +If a user opens an archive whose details have not finished loading, the HistoryServer +prioritizes that archive for retrieval. Separate thread pools handle background fetching and +individual on-demand requests, so browsing a particular job does not have to wait behind all +other archives. + +The existing `EAGER` loading mode remains the default. + +**More Information** + +* [FLIP-585](https://cwiki.apache.org/confluence/spaces/FLINK/pages/430408096/FLIP-585+Introduce+Lazy+Loading+for+HistoryServer) + +## State Catalog: Explore Checkpoints and Savepoints with SQL + +Inspecting application state has traditionally required custom State Processor API code or +manually defined table schemas. Flink 2.4 introduces a read-only State Catalog that discovers +checkpoints and savepoints under configured directories and exposes their contents through SQL. + +Each discovered snapshot becomes a database. State schemas are inferred from serializer +snapshots, allowing supported keyed, non-keyed, and window state to be represented as tables. +Large list and map states can also be exposed in flattened form, making individual entries +easier to query. + +```sql +CREATE CATALOG application_state WITH ( + 'type' = 'state', + 'directory.prod' = 's3://my-bucket/flink/checkpoints' +); + +USE CATALOG application_state; +SHOW DATABASES; +``` + +Users can select a discovered database and explore its state tables without modifying the +running application. For supported POJO and Avro serializers, state can be read without the +original user classes on the classpath. Custom serializers without sufficient schema metadata +are outside this automatic inference path. + +The catalog is read-only, making it suitable for investigating state growth, inspecting +individual keys, and understanding the contents of a saved snapshot. + +**More Information** + +* [FLIP-599](https://cwiki.apache.org/confluence/spaces/FLINK/pages/438009922/FLIP-599+State+Catalog) +* [FLINK-40175](https://issues.apache.org/jira/browse/FLINK-40175) Review Comment: Missing `Upgrade Notes` and `List of Contributors` sections -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
