autophagy commented on code in PR #895:
URL: https://github.com/apache/flink-web/pull/895#discussion_r4642554729


##########
docs/content/posts/2026-12-25-release-2.4.0.md:
##########
@@ -0,0 +1,490 @@
+---
+title: "Apache Flink 2.4.0 Release Announcement"
+date: "2026-12-25T00:00:00.000Z"
+aliases:
+- /news/2026/12/25/release-2.4.0.html
+authors:
+- flink:
+  name: "Apache Flink PMC"
+
+---
+
+The Apache Flink PMC is pleased to announce the release of Apache Flink 2.4.0.
+
+This release expands Flink SQL with new join capabilities, more flexible 
changelog conversion,
+and richer Process Table Functions. It also introduces a Python DataFrame API, 
makes application
+state easier to explore through SQL, and improves the scalability of the 
HistoryServer.
+
+Key improvements include common subexpression elimination for SQL 
calculations, built-in UDF
+metrics, early firing for outer interval joins, and a new `LATERAL SNAPSHOT` 
join. In-place
+conversion to materialized tables simplifies adopting managed refresh 
pipelines, while state
+inspection and recovery improvements help users operate stateful applications.
+
+We extend our heartfelt thanks to all contributors for making this release 
possible!
+
+Let's dive into the details.
+
+# Flink SQL Improvements
+
+## Upgrade Calcite to 1.41.0
+
+Flink 2.4 upgrades Apache Calcite, the foundation of Flink's SQL parser and 
relational query
+planner, to version 1.41.0. This brings Flink's planner onto a more recent 
Calcite release and
+incorporates upstream fixes and improvements.
+
+The upgrade also provides the planner-level UUID type support used by the new 
native UUID
+capabilities described below.
+
+**More Information**
+
+* [FLINK-39859](https://issues.apache.org/jira/browse/FLINK-39859)
+
+## Common Subexpression Elimination in Calc Code Generation
+
+SQL queries often repeat the same calculation in several output columns or in 
both a filter and
+a projection. Previously, a Calc operator could evaluate each occurrence 
separately, even when
+the expression was deterministic and its inputs were identical.
+
+Flink 2.4 introduces common subexpression elimination (CSE) in Calc code 
generation for both
+batch and streaming execution. Reusable expressions are evaluated once and 
their results are
+shared, reducing redundant work for built-in functions and user-defined 
functions:
+
+```sql
+SELECT
+  expensive_udf(a, b) + 1 AS col1,
+  expensive_udf(a, b) + 2 AS col2
+FROM source_table
+WHERE expensive_udf(a, b) > 0;
+```
+
+For a deterministic `expensive_udf`, the Calc operator can reuse the result 
across these
+expressions. The optimization preserves short-circuit evaluation in 
expressions such as
+`AND`, `OR`, and `CASE`. Functions with side effects or non-deterministic 
results must declare
+`isDeterministic()` as `false` so that their calls are not incorrectly reused.
+
+**More Information**
+
+* [FLINK-39268](https://issues.apache.org/jira/browse/FLINK-39268)
+
+## Built-in Metrics for User-Defined Functions
+
+User-defined functions can be difficult to diagnose when they introduce 
latency or throw
+exceptions. Flink 2.4 adds opt-in metrics for synchronous and asynchronous 
scalar and table
+functions, making their behavior visible through the existing metrics 
reporters.
+
+Two metrics are registered per function on its executing operator:
+
+* **`udfProcessingTime`**: A histogram of sampled invocation durations. For 
asynchronous
+  functions, this measures the time from dispatch to completion.
+* **`udfExceptionCount`**: A counter of exceptions surfaced by the function, 
including
+  exceptional completion of asynchronous calls.
+
+```yaml
+# Enable UDF metrics (disabled by default)
+table.exec.udf-metric-enabled: true
+
+# Time one invocation out of every 100 (the default sampling interval)
+table.exec.udf-metric.sample-interval: 100
+```
+
+Processing-time sampling limits measurement overhead; exceptions are counted 
on every
+invocation. Aggregate functions and Process Table Functions are outside the 
scope of these
+metrics.
+
+**More Information**
+
+* 
[FLIP-485](https://cwiki.apache.org/confluence/spaces/FLINK/pages/373885706/FLIP-485+Add+UDF+Metrics)
+* [FLINK-38071](https://issues.apache.org/jira/browse/FLINK-38071)
+
+## Early Fire for Outer Interval Joins
+
+An outer interval join normally waits until an unmatched row's time interval 
closes before
+emitting it with a null-padded counterpart. For long intervals, this can delay 
useful results.
+Flink 2.4 introduces an `EARLY_FIRE` hint that emits the unmatched row after a 
configurable
+delay:
+
+```sql
+SELECT /*+ EARLY_FIRE('delay'='5s', 'time-mode'='proctime') */
+  o.id, s.ship_time
+FROM Orders o
+LEFT JOIN Shipments s
+  ON o.id = s.order_id
+  AND o.order_time BETWEEN s.ship_time - INTERVAL '1' HOUR AND s.ship_time;
+```
+
+If a matching shipment arrives later within the interval, Flink retracts the 
speculative result
+and emits the matched result. Downstream operators and sinks must therefore 
accept an updating
+changelog.
+
+The hint supports event-time and processing-time delays. Each unmatched outer 
row fires at
+most once; this is not a periodic trigger. It applies to `LEFT`, `RIGHT`, and 
`FULL` outer
+interval joins and is independent of the early-fire configuration for window 
aggregations.
+
+**More Information**
+
+* 
[FLIP-497](https://cwiki.apache.org/confluence/spaces/FLINK/pages/334760951/FLIP-497+Early+Fire+Support+for+Flink+SQL+Interval+Join)
+* [FLINK-36953](https://issues.apache.org/jira/browse/FLINK-36953)
+
+## FROM_CHANGELOG and TO_CHANGELOG: More Flexible Changelog Conversion
+
+Flink 2.4 extends `FROM_CHANGELOG` and `TO_CHANGELOG` with keyed conversion 
and additional
+control over invalid operation codes and delete records.
+
+* **Keyed upsert conversion**: `FROM_CHANGELOG` supports producing an upsert 
table from an
+  append-only CDC stream by combining `PARTITION BY` with an operation mapping 
that includes
+  `UPDATE_AFTER` but not `UPDATE_BEFORE`. The partition columns must identify 
the upsert key.
+* **Explicit error handling**: `error_handling` selects whether null or 
unknown input operation
+  codes fail the job (`FAIL`, the default) or are skipped (`SKIP`).
+* **Full or partial deletes**: `TO_CHANGELOG` supports keyed conversion and a
+  `produces_full_deletes` parameter, controlling whether delete records 
contain the full row
+  or only the identifying key columns.
+* **Partitioned Table API support**: `PartitionedTable` gains 
`fromChangelog()` and
+  `toChangelog()` convenience methods for keyed conversion.
+
+For example, a CDC stream containing insert, update-after, and delete events 
can be interpreted
+as an upsert table:
+
+```sql
+SELECT * FROM FROM_CHANGELOG(
+  input => TABLE cdc_events PARTITION BY id,
+  op_mapping => MAP[
+    'c', 'INSERT',
+    'u', 'UPDATE_AFTER',
+    'd', 'DELETE'
+  ]
+);
+```
+
+The partition columns define the output's upsert key, so they must match the 
unique key of the
+CDC data. Events for each key must arrive in the correct order.
+
+**More Information**
+
+* 
[FLIP-564](https://cwiki.apache.org/confluence/spaces/FLINK/pages/406620639/FLIP-564+Support+FROM_CHANGELOG+and+TO_CHANGELOG+built-in+PTFs)
+* [FLINK-39258](https://issues.apache.org/jira/browse/FLINK-39258)
+
+## Process Table Functions: Lazy State Access and Broadcast State
+
+Flink 2.4 adds lazy value-state access and broadcast capabilities to Process 
Table Functions
+for more advanced stateful SQL applications.
+
+The new `ValueView` provides explicit `value()`, `update()`, `isEmpty()`, and 
`clear()`
+operations. A function can access a value only when it needs it, avoiding the 
need to eagerly
+read and write every state entry for each invocation.
+
+Broadcast table arguments, declared with `BROADCAST_SEMANTIC_TABLE`, 
distribute control rows
+to all parallel instances. Functions can keep shared rules or configuration in
+`@StateHint(StateKind.BROADCAST)` state and consult it while processing their 
main inputs.
+The `NOTIFY_STATEFUL_SETS` trait additionally allows a broadcast update to 
notify existing
+stateful partitions.
+
+Together, these additions support patterns such as dynamically updated rule 
engines and
+configuration-driven processing directly in PTFs.
+
+**More Information**
+
+* 
[FLIP-565](https://cwiki.apache.org/confluence/spaces/FLINK/pages/406620840/FLIP-565+Improve+ProcessTableFunctions+for+late+data+handling+and+state+access)
+* [FLINK-39254](https://issues.apache.org/jira/browse/FLINK-39254)
+
+## A Test Harness for Process Table Functions
+
+Flink 2.4 introduces `ProcessTableFunctionTestHarness`, a lightweight 
framework for testing
+PTF business logic without starting a full Flink cluster.
+
+The harness lets developers configure scalar and table arguments, feed 
records, advance
+watermarks, and inspect emitted results. It also supports partitioned and 
multi-input
+functions, state inspection and initialization, timers, and state TTL.

Review Comment:
   Yes, @weiqingy is right, we ultimately decided against including state TTL 
mechanisms in the PTF test harness.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to