autophagy commented on code in PR #895: URL: https://github.com/apache/flink-web/pull/895#discussion_r4642554729
########## docs/content/posts/2026-12-25-release-2.4.0.md: ########## @@ -0,0 +1,490 @@ +--- +title: "Apache Flink 2.4.0 Release Announcement" +date: "2026-12-25T00:00:00.000Z" +aliases: +- /news/2026/12/25/release-2.4.0.html +authors: +- flink: + name: "Apache Flink PMC" + +--- + +The Apache Flink PMC is pleased to announce the release of Apache Flink 2.4.0. + +This release expands Flink SQL with new join capabilities, more flexible changelog conversion, +and richer Process Table Functions. It also introduces a Python DataFrame API, makes application +state easier to explore through SQL, and improves the scalability of the HistoryServer. + +Key improvements include common subexpression elimination for SQL calculations, built-in UDF +metrics, early firing for outer interval joins, and a new `LATERAL SNAPSHOT` join. In-place +conversion to materialized tables simplifies adopting managed refresh pipelines, while state +inspection and recovery improvements help users operate stateful applications. + +We extend our heartfelt thanks to all contributors for making this release possible! + +Let's dive into the details. + +# Flink SQL Improvements + +## Upgrade Calcite to 1.41.0 + +Flink 2.4 upgrades Apache Calcite, the foundation of Flink's SQL parser and relational query +planner, to version 1.41.0. This brings Flink's planner onto a more recent Calcite release and +incorporates upstream fixes and improvements. + +The upgrade also provides the planner-level UUID type support used by the new native UUID +capabilities described below. + +**More Information** + +* [FLINK-39859](https://issues.apache.org/jira/browse/FLINK-39859) + +## Common Subexpression Elimination in Calc Code Generation + +SQL queries often repeat the same calculation in several output columns or in both a filter and +a projection. Previously, a Calc operator could evaluate each occurrence separately, even when +the expression was deterministic and its inputs were identical. + +Flink 2.4 introduces common subexpression elimination (CSE) in Calc code generation for both +batch and streaming execution. Reusable expressions are evaluated once and their results are +shared, reducing redundant work for built-in functions and user-defined functions: + +```sql +SELECT + expensive_udf(a, b) + 1 AS col1, + expensive_udf(a, b) + 2 AS col2 +FROM source_table +WHERE expensive_udf(a, b) > 0; +``` + +For a deterministic `expensive_udf`, the Calc operator can reuse the result across these +expressions. The optimization preserves short-circuit evaluation in expressions such as +`AND`, `OR`, and `CASE`. Functions with side effects or non-deterministic results must declare +`isDeterministic()` as `false` so that their calls are not incorrectly reused. + +**More Information** + +* [FLINK-39268](https://issues.apache.org/jira/browse/FLINK-39268) + +## Built-in Metrics for User-Defined Functions + +User-defined functions can be difficult to diagnose when they introduce latency or throw +exceptions. Flink 2.4 adds opt-in metrics for synchronous and asynchronous scalar and table +functions, making their behavior visible through the existing metrics reporters. + +Two metrics are registered per function on its executing operator: + +* **`udfProcessingTime`**: A histogram of sampled invocation durations. For asynchronous + functions, this measures the time from dispatch to completion. +* **`udfExceptionCount`**: A counter of exceptions surfaced by the function, including + exceptional completion of asynchronous calls. + +```yaml +# Enable UDF metrics (disabled by default) +table.exec.udf-metric-enabled: true + +# Time one invocation out of every 100 (the default sampling interval) +table.exec.udf-metric.sample-interval: 100 +``` + +Processing-time sampling limits measurement overhead; exceptions are counted on every +invocation. Aggregate functions and Process Table Functions are outside the scope of these +metrics. + +**More Information** + +* [FLIP-485](https://cwiki.apache.org/confluence/spaces/FLINK/pages/373885706/FLIP-485+Add+UDF+Metrics) +* [FLINK-38071](https://issues.apache.org/jira/browse/FLINK-38071) + +## Early Fire for Outer Interval Joins + +An outer interval join normally waits until an unmatched row's time interval closes before +emitting it with a null-padded counterpart. For long intervals, this can delay useful results. +Flink 2.4 introduces an `EARLY_FIRE` hint that emits the unmatched row after a configurable +delay: + +```sql +SELECT /*+ EARLY_FIRE('delay'='5s', 'time-mode'='proctime') */ + o.id, s.ship_time +FROM Orders o +LEFT JOIN Shipments s + ON o.id = s.order_id + AND o.order_time BETWEEN s.ship_time - INTERVAL '1' HOUR AND s.ship_time; +``` + +If a matching shipment arrives later within the interval, Flink retracts the speculative result +and emits the matched result. Downstream operators and sinks must therefore accept an updating +changelog. + +The hint supports event-time and processing-time delays. Each unmatched outer row fires at +most once; this is not a periodic trigger. It applies to `LEFT`, `RIGHT`, and `FULL` outer +interval joins and is independent of the early-fire configuration for window aggregations. + +**More Information** + +* [FLIP-497](https://cwiki.apache.org/confluence/spaces/FLINK/pages/334760951/FLIP-497+Early+Fire+Support+for+Flink+SQL+Interval+Join) +* [FLINK-36953](https://issues.apache.org/jira/browse/FLINK-36953) + +## FROM_CHANGELOG and TO_CHANGELOG: More Flexible Changelog Conversion + +Flink 2.4 extends `FROM_CHANGELOG` and `TO_CHANGELOG` with keyed conversion and additional +control over invalid operation codes and delete records. + +* **Keyed upsert conversion**: `FROM_CHANGELOG` supports producing an upsert table from an + append-only CDC stream by combining `PARTITION BY` with an operation mapping that includes + `UPDATE_AFTER` but not `UPDATE_BEFORE`. The partition columns must identify the upsert key. +* **Explicit error handling**: `error_handling` selects whether null or unknown input operation + codes fail the job (`FAIL`, the default) or are skipped (`SKIP`). +* **Full or partial deletes**: `TO_CHANGELOG` supports keyed conversion and a + `produces_full_deletes` parameter, controlling whether delete records contain the full row + or only the identifying key columns. +* **Partitioned Table API support**: `PartitionedTable` gains `fromChangelog()` and + `toChangelog()` convenience methods for keyed conversion. + +For example, a CDC stream containing insert, update-after, and delete events can be interpreted +as an upsert table: + +```sql +SELECT * FROM FROM_CHANGELOG( + input => TABLE cdc_events PARTITION BY id, + op_mapping => MAP[ + 'c', 'INSERT', + 'u', 'UPDATE_AFTER', + 'd', 'DELETE' + ] +); +``` + +The partition columns define the output's upsert key, so they must match the unique key of the +CDC data. Events for each key must arrive in the correct order. + +**More Information** + +* [FLIP-564](https://cwiki.apache.org/confluence/spaces/FLINK/pages/406620639/FLIP-564+Support+FROM_CHANGELOG+and+TO_CHANGELOG+built-in+PTFs) +* [FLINK-39258](https://issues.apache.org/jira/browse/FLINK-39258) + +## Process Table Functions: Lazy State Access and Broadcast State + +Flink 2.4 adds lazy value-state access and broadcast capabilities to Process Table Functions +for more advanced stateful SQL applications. + +The new `ValueView` provides explicit `value()`, `update()`, `isEmpty()`, and `clear()` +operations. A function can access a value only when it needs it, avoiding the need to eagerly +read and write every state entry for each invocation. + +Broadcast table arguments, declared with `BROADCAST_SEMANTIC_TABLE`, distribute control rows +to all parallel instances. Functions can keep shared rules or configuration in +`@StateHint(StateKind.BROADCAST)` state and consult it while processing their main inputs. +The `NOTIFY_STATEFUL_SETS` trait additionally allows a broadcast update to notify existing +stateful partitions. + +Together, these additions support patterns such as dynamically updated rule engines and +configuration-driven processing directly in PTFs. + +**More Information** + +* [FLIP-565](https://cwiki.apache.org/confluence/spaces/FLINK/pages/406620840/FLIP-565+Improve+ProcessTableFunctions+for+late+data+handling+and+state+access) +* [FLINK-39254](https://issues.apache.org/jira/browse/FLINK-39254) + +## A Test Harness for Process Table Functions + +Flink 2.4 introduces `ProcessTableFunctionTestHarness`, a lightweight framework for testing +PTF business logic without starting a full Flink cluster. + +The harness lets developers configure scalar and table arguments, feed records, advance +watermarks, and inspect emitted results. It also supports partitioned and multi-input +functions, state inspection and initialization, timers, and state TTL. Review Comment: Yes, @weiqingy is right, we ultimately decided against including state TTL mechanisms in the PTF test harness. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
