weiqingy commented on code in PR #29957:
URL: https://github.com/apache/flink/pull/29957#discussion_r4638436019


##########
docs/content/release-notes/flink-2.4.md:
##########
@@ -0,0 +1,359 @@
+---
+title: "Release Notes - Flink 2.4"
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Release notes - Flink 2.4
+
+These release notes discuss important aspects, such as configuration, behavior 
or dependencies,

Review Comment:
   Two changes can silently break an existing setup after the upgrade and 
aren't covered yet:
   - SLF4J moved from 1.7.36 to 2.0.20 with the `log4j-slf4j2-impl` binding 
([FLINK-40574](https://issues.apache.org/jira/browse/FLINK-40574)). Its JIRA 
release note says Logback 1.2 or `slf4j-log4j12` setups, and jars bundling 
`slf4j-api` 1.x in `lib/` or on YARN, end up with no logs.
   - Deprecated config keys are no longer read (FLINK-40813, FLINK-40815, 
FLINK-40816, FLINK-40817): `kubernetes.container.image`, 
`kubernetes.jobmanager.cpu`, `kubernetes.taskmanager.cpu`, 
`kubernetes.pod-template-file`, `yarn.ship-directories`, 
`yarn.per-job-cluster.include-user-jar`, `yarn.heartbeat-delay`, 
`akka.client.timeout`, and the `dstl.dfs.*` keys (now 
`state.changelog.dstl.dfs.*`).
   
   `flink-2.1.md` and `flink-2.2.md` had a "Dependency upgrades" section for 
this kind of change. Could we add one for SLF4J and for PyFlink's `apache-beam` 
range moving from `>=2.54.0,<=2.61.0` to `>=2.69.0,<=2.75.0` 
([FLINK-38469](https://issues.apache.org/jira/browse/FLINK-38469)), plus a 
short old-to-new table for the removed keys?



##########
docs/content/release-notes/flink-2.4.md:
##########
@@ -0,0 +1,359 @@
+---
+title: "Release Notes - Flink 2.4"
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Release notes - Flink 2.4
+
+These release notes discuss important aspects, such as configuration, behavior 
or dependencies,
+that changed between Flink 2.3 and Flink 2.4. Please read these notes 
carefully if you are
+planning to upgrade your Flink version to 2.4.
+
+### Table SQL / API

Review Comment:
   `table.exec.sink.require-on-conflict` now defaults to `false`, reversing the 
2.3 default ([FLINK-41478](https://issues.apache.org/jira/browse/FLINK-41478)), 
and the ticket carries a release note. Worth a short entry in this section?



##########
docs/content/release-notes/flink-2.4.md:
##########
@@ -0,0 +1,359 @@
+---
+title: "Release Notes - Flink 2.4"
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Release notes - Flink 2.4
+
+These release notes discuss important aspects, such as configuration, behavior 
or dependencies,
+that changed between Flink 2.3 and Flink 2.4. Please read these notes 
carefully if you are
+planning to upgrade your Flink version to 2.4.
+
+### Table SQL / API
+
+#### Upgrade Calcite to 1.41.0
+
+##### [FLINK-39859](https://issues.apache.org/jira/browse/FLINK-39859)
+
+Flink 2.4 upgrades Apache Calcite, the foundation of the SQL parser and 
optimizer, to version
+1.41.0. This incorporates the fixes and improvements from the intervening 
Calcite releases.
+Applications that depend directly on Calcite classes or extend Flink's planner 
should check
+their dependencies and custom planner code against the upgraded version.
+
+#### Common subexpression elimination in Calc code generation
+
+##### [FLINK-39268](https://issues.apache.org/jira/browse/FLINK-39268)
+
+Flink 2.4 avoids redundant evaluation of common subexpressions within a Calc 
operator, which
+combines filtering and projection. Repeated deterministic expressions can 
reuse an already
+computed result instead of being evaluated again for the same input row. This 
applies to both
+batch and streaming queries and can reduce the cost of repeated built-in or 
user-defined
+function calls.
+
+Reuse respects conditional evaluation, such as short-circuiting in `AND`, 
`OR`, and `CASE`.
+UDF authors should ensure that `isDeterministic()` correctly describes their 
function:
+functions with side effects or results that vary for identical inputs must 
return `false`.
+
+#### Built-in UDF metrics
+
+##### [FLINK-38071](https://issues.apache.org/jira/browse/FLINK-38071) 
(FLIP-485)
+
+SQL and Table API users can now monitor UDF execution through two built-in 
metrics:
+`udfProcessingTime`, a histogram of sampled invocation durations, and 
`udfExceptionCount`,
+a counter of exceptions from the function. Metrics are registered on the 
executing operator
+under `udf.<udf_name>`. For asynchronous functions, processing time includes 
the time from
+dispatch until completion.
+
+The feature is opt-in:
+
+- `table.exec.udf-metric-enabled` (default: `false`) enables metric collection.
+- `table.exec.udf-metric.sample-interval` (default: `100`) controls how often 
invocation
+  durations are measured. Exceptions are counted on every invocation.
+
+The metrics cover directly invoked synchronous and asynchronous scalar and 
table functions.
+Python functions, aggregate functions, and Process Table Functions are not 
covered. Enabling
+the metrics adds overhead; use sampling to control the cost.
+
+#### Early firing for interval joins
+
+##### [FLINK-36953](https://issues.apache.org/jira/browse/FLINK-36953) 
(FLIP-497)
+
+Outer interval joins can emit unmatched rows before their join interval 
closes. The new
+`EARLY_FIRE` hint emits a speculative, null-padded result after a configured 
delay. If a
+matching row arrives later within the interval, Flink corrects the speculative 
result with
+update-before and update-after records.
+
+```sql
+SELECT /*+ EARLY_FIRE('delay'='5s') */ o.id, s.ship_time
+FROM Orders o
+LEFT OUTER JOIN Shipments s
+ON o.id = s.order_id
+AND o.order_time BETWEEN s.ship_time - INTERVAL '1' HOUR AND s.ship_time;
+```
+
+The optional `time-mode` selects `rowtime` or `proctime`; it defaults to the 
time mode of the
+interval join. Each unmatched outer row fires at most once. The feature is 
disabled unless
+the hint is present and requires downstream operators and sinks that accept 
updates.
+Existing interval joins without the hint retain their behavior.

Review Comment:
   On `release-2.4` today, a batch query with `EARLY_FIRE` fails planning with 
`Unknown join strategy : EARLY_FIRE` 
([FLINK-41446](https://issues.apache.org/jira/browse/FLINK-41446)). My fix, 
#29908, makes batch ignore the hint; it's waiting for review and I'm aiming to 
get it into 2.4.0. If it lands, "In batch mode the hint is ignored." would fit 
after this line; if it misses, this is the place for a known-issue note. I'll 
post the exact sentence here once it's settled.



##########
docs/content/release-notes/flink-2.4.md:
##########
@@ -0,0 +1,359 @@
+---
+title: "Release Notes - Flink 2.4"
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Release notes - Flink 2.4
+
+These release notes discuss important aspects, such as configuration, behavior 
or dependencies,
+that changed between Flink 2.3 and Flink 2.4. Please read these notes 
carefully if you are
+planning to upgrade your Flink version to 2.4.
+
+### Table SQL / API
+
+#### Upgrade Calcite to 1.41.0
+
+##### [FLINK-39859](https://issues.apache.org/jira/browse/FLINK-39859)
+
+Flink 2.4 upgrades Apache Calcite, the foundation of the SQL parser and 
optimizer, to version
+1.41.0. This incorporates the fixes and improvements from the intervening 
Calcite releases.
+Applications that depend directly on Calcite classes or extend Flink's planner 
should check
+their dependencies and custom planner code against the upgraded version.
+
+#### Common subexpression elimination in Calc code generation
+
+##### [FLINK-39268](https://issues.apache.org/jira/browse/FLINK-39268)
+
+Flink 2.4 avoids redundant evaluation of common subexpressions within a Calc 
operator, which
+combines filtering and projection. Repeated deterministic expressions can 
reuse an already
+computed result instead of being evaluated again for the same input row. This 
applies to both
+batch and streaming queries and can reduce the cost of repeated built-in or 
user-defined
+function calls.
+
+Reuse respects conditional evaluation, such as short-circuiting in `AND`, 
`OR`, and `CASE`.
+UDF authors should ensure that `isDeterministic()` correctly describes their 
function:
+functions with side effects or results that vary for identical inputs must 
return `false`.
+
+#### Built-in UDF metrics
+
+##### [FLINK-38071](https://issues.apache.org/jira/browse/FLINK-38071) 
(FLIP-485)
+
+SQL and Table API users can now monitor UDF execution through two built-in 
metrics:
+`udfProcessingTime`, a histogram of sampled invocation durations, and 
`udfExceptionCount`,
+a counter of exceptions from the function. Metrics are registered on the 
executing operator
+under `udf.<udf_name>`. For asynchronous functions, processing time includes 
the time from
+dispatch until completion.
+
+The feature is opt-in:
+
+- `table.exec.udf-metric-enabled` (default: `false`) enables metric collection.
+- `table.exec.udf-metric.sample-interval` (default: `100`) controls how often 
invocation
+  durations are measured. Exceptions are counted on every invocation.
+
+The metrics cover directly invoked synchronous and asynchronous scalar and 
table functions.
+Python functions, aggregate functions, and Process Table Functions are not 
covered. Enabling
+the metrics adds overhead; use sampling to control the cost.
+
+#### Early firing for interval joins
+
+##### [FLINK-36953](https://issues.apache.org/jira/browse/FLINK-36953) 
(FLIP-497)
+
+Outer interval joins can emit unmatched rows before their join interval 
closes. The new
+`EARLY_FIRE` hint emits a speculative, null-padded result after a configured 
delay. If a
+matching row arrives later within the interval, Flink corrects the speculative 
result with
+update-before and update-after records.
+
+```sql
+SELECT /*+ EARLY_FIRE('delay'='5s') */ o.id, s.ship_time
+FROM Orders o
+LEFT OUTER JOIN Shipments s
+ON o.id = s.order_id
+AND o.order_time BETWEEN s.ship_time - INTERVAL '1' HOUR AND s.ship_time;
+```
+
+The optional `time-mode` selects `rowtime` or `proctime`; it defaults to the 
time mode of the
+interval join. Each unmatched outer row fires at most once. The feature is 
disabled unless
+the hint is present and requires downstream operators and sinks that accept 
updates.
+Existing interval joins without the hint retain their behavior.
+
+#### FROM_CHANGELOG and TO_CHANGELOG enhancements
+
+##### [FLINK-39258](https://issues.apache.org/jira/browse/FLINK-39258) 
(FLIP-564)
+
+Flink 2.4 adds partitioned processing and additional controls to the changelog 
conversion
+PTFs introduced in Flink 2.3:
+
+- `PARTITION BY` support to co-locate related changes, with corresponding 
convenience methods
+  on partitioned tables in the Table API.
+- Partitioned `FROM_CHANGELOG` calls can use the existing PTF `ORDER BY` 
support to reorder
+  CDC events per key using a watermarked time attribute. Late events are 
dropped by the
+  sorting operation.
+- `error_handling => 'FAIL' | 'SKIP'` for `FROM_CHANGELOG` to control how NULL 
or unmapped
+  operation codes are handled. The default is `FAIL`.
+- `produces_full_deletes` for `TO_CHANGELOG` to choose between full-row 
deletes and deletes
+  containing only identifying keys. The default is `true`.
+
+These additions make it easier to consume custom CDC streams and write 
changelog events to
+append-only sinks. Users must still provide correctly ordered, complete change 
events for
+the selected changelog mode; the functions do not reconstruct arbitrary 
missing before images.
+See the [Changelog Conversion 
documentation](https://nightlies.apache.org/flink/flink-docs-release-2.4/docs/sql/reference/queries/changelog/)
+for supported mappings and input requirements.
+
+#### Process Table Function state access and broadcast support
+
+##### [FLINK-39254](https://issues.apache.org/jira/browse/FLINK-39254) 
(FLIP-565)
+
+Flink 2.4 adds lazy value-state access and broadcast inputs to Process Table 
Functions:
+
+- `ValueView` provides lazy value-state access alongside `ListView` and 
`MapView`. A function
+  can read or update state only when needed, avoiding unnecessary 
serialization work.
+- `BROADCAST_SEMANTIC_TABLE` lets a PTF receive a rule or control table on 
every parallel
+  instance. State declared with `@StateHint(StateKind.BROADCAST)` is writable 
from the
+  broadcast input and readable while processing the regular inputs.
+- `NOTIFY_STATEFUL_SETS` lets a broadcast update trigger re-evaluation of 
existing stateful
+  partitions. This is opt-in because visiting all stateful partitions can be 
expensive.
+
+#### Process Table Function test harness
+
+##### [FLINK-39357](https://issues.apache.org/jira/browse/FLINK-39357) 
(FLIP-567)
+
+The new `ProcessTableFunctionTestHarness` lets developers test PTF business 
logic without
+starting a Flink cluster. Tests can configure scalar and table arguments, 
partitioning, and
+initial state, feed rows into individual inputs, and inspect the resulting 
output.
+
+The harness also supports advancing watermarks and the system clock, 
inspecting timers,

Review Comment:
   Three of these aren't in the 2.4 harness. The shipped `ptfs.md` lists state 
TTL and the update traits under "PTF Features Unsupported by the TestHarness", 
the harness context's `getChangelogMode()` always returns `insertOnly()` 
(changelog support is FLINK-40577, fixVersion 2.5.0), and there's no clock API: 
time only moves through `setWatermark()` / `setWatermarkForTable()`. Something 
like "The harness also supports advancing watermarks, inspecting timers, and 
reading or setting state per key." instead?



##########
docs/content/release-notes/flink-2.4.md:
##########
@@ -0,0 +1,359 @@
+---
+title: "Release Notes - Flink 2.4"
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Release notes - Flink 2.4
+
+These release notes discuss important aspects, such as configuration, behavior 
or dependencies,
+that changed between Flink 2.3 and Flink 2.4. Please read these notes 
carefully if you are
+planning to upgrade your Flink version to 2.4.
+
+### Table SQL / API
+
+#### Upgrade Calcite to 1.41.0
+
+##### [FLINK-39859](https://issues.apache.org/jira/browse/FLINK-39859)
+
+Flink 2.4 upgrades Apache Calcite, the foundation of the SQL parser and 
optimizer, to version
+1.41.0. This incorporates the fixes and improvements from the intervening 
Calcite releases.
+Applications that depend directly on Calcite classes or extend Flink's planner 
should check
+their dependencies and custom planner code against the upgraded version.
+
+#### Common subexpression elimination in Calc code generation
+
+##### [FLINK-39268](https://issues.apache.org/jira/browse/FLINK-39268)
+
+Flink 2.4 avoids redundant evaluation of common subexpressions within a Calc 
operator, which
+combines filtering and projection. Repeated deterministic expressions can 
reuse an already
+computed result instead of being evaluated again for the same input row. This 
applies to both
+batch and streaming queries and can reduce the cost of repeated built-in or 
user-defined
+function calls.
+
+Reuse respects conditional evaluation, such as short-circuiting in `AND`, 
`OR`, and `CASE`.
+UDF authors should ensure that `isDeterministic()` correctly describes their 
function:
+functions with side effects or results that vary for identical inputs must 
return `false`.
+
+#### Built-in UDF metrics
+
+##### [FLINK-38071](https://issues.apache.org/jira/browse/FLINK-38071) 
(FLIP-485)
+
+SQL and Table API users can now monitor UDF execution through two built-in 
metrics:
+`udfProcessingTime`, a histogram of sampled invocation durations, and 
`udfExceptionCount`,
+a counter of exceptions from the function. Metrics are registered on the 
executing operator
+under `udf.<udf_name>`. For asynchronous functions, processing time includes 
the time from
+dispatch until completion.
+
+The feature is opt-in:
+
+- `table.exec.udf-metric-enabled` (default: `false`) enables metric collection.
+- `table.exec.udf-metric.sample-interval` (default: `100`) controls how often 
invocation
+  durations are measured. Exceptions are counted on every invocation.
+
+The metrics cover directly invoked synchronous and asynchronous scalar and 
table functions.
+Python functions, aggregate functions, and Process Table Functions are not 
covered. Enabling
+the metrics adds overhead; use sampling to control the cost.
+
+#### Early firing for interval joins
+
+##### [FLINK-36953](https://issues.apache.org/jira/browse/FLINK-36953) 
(FLIP-497)
+
+Outer interval joins can emit unmatched rows before their join interval 
closes. The new
+`EARLY_FIRE` hint emits a speculative, null-padded result after a configured 
delay. If a
+matching row arrives later within the interval, Flink corrects the speculative 
result with
+update-before and update-after records.
+
+```sql
+SELECT /*+ EARLY_FIRE('delay'='5s') */ o.id, s.ship_time
+FROM Orders o
+LEFT OUTER JOIN Shipments s
+ON o.id = s.order_id
+AND o.order_time BETWEEN s.ship_time - INTERVAL '1' HOUR AND s.ship_time;
+```
+
+The optional `time-mode` selects `rowtime` or `proctime`; it defaults to the 
time mode of the
+interval join. Each unmatched outer row fires at most once. The feature is 
disabled unless
+the hint is present and requires downstream operators and sinks that accept 
updates.
+Existing interval joins without the hint retain their behavior.
+
+#### FROM_CHANGELOG and TO_CHANGELOG enhancements
+
+##### [FLINK-39258](https://issues.apache.org/jira/browse/FLINK-39258) 
(FLIP-564)
+
+Flink 2.4 adds partitioned processing and additional controls to the changelog 
conversion
+PTFs introduced in Flink 2.3:
+
+- `PARTITION BY` support to co-locate related changes, with corresponding 
convenience methods
+  on partitioned tables in the Table API.
+- Partitioned `FROM_CHANGELOG` calls can use the existing PTF `ORDER BY` 
support to reorder
+  CDC events per key using a watermarked time attribute. Late events are 
dropped by the
+  sorting operation.
+- `error_handling => 'FAIL' | 'SKIP'` for `FROM_CHANGELOG` to control how NULL 
or unmapped
+  operation codes are handled. The default is `FAIL`.
+- `produces_full_deletes` for `TO_CHANGELOG` to choose between full-row 
deletes and deletes
+  containing only identifying keys. The default is `true`.
+
+These additions make it easier to consume custom CDC streams and write 
changelog events to
+append-only sinks. Users must still provide correctly ordered, complete change 
events for
+the selected changelog mode; the functions do not reconstruct arbitrary 
missing before images.
+See the [Changelog Conversion 
documentation](https://nightlies.apache.org/flink/flink-docs-release-2.4/docs/sql/reference/queries/changelog/)
+for supported mappings and input requirements.
+
+#### Process Table Function state access and broadcast support
+
+##### [FLINK-39254](https://issues.apache.org/jira/browse/FLINK-39254) 
(FLIP-565)
+
+Flink 2.4 adds lazy value-state access and broadcast inputs to Process Table 
Functions:
+
+- `ValueView` provides lazy value-state access alongside `ListView` and 
`MapView`. A function
+  can read or update state only when needed, avoiding unnecessary 
serialization work.
+- `BROADCAST_SEMANTIC_TABLE` lets a PTF receive a rule or control table on 
every parallel
+  instance. State declared with `@StateHint(StateKind.BROADCAST)` is writable 
from the
+  broadcast input and readable while processing the regular inputs.
+- `NOTIFY_STATEFUL_SETS` lets a broadcast update trigger re-evaluation of 
existing stateful
+  partitions. This is opt-in because visiting all stateful partitions can be 
expensive.
+
+#### Process Table Function test harness
+
+##### [FLINK-39357](https://issues.apache.org/jira/browse/FLINK-39357) 
(FLIP-567)
+
+The new `ProcessTableFunctionTestHarness` lets developers test PTF business 
logic without
+starting a Flink cluster. Tests can configure scalar and table arguments, 
partitioning, and
+initial state, feed rows into individual inputs, and inspect the resulting 
output.
+
+The harness also supports advancing watermarks and the system clock, 
inspecting timers,
+accessing state per key, and testing state TTL and changelog handling. This 
makes temporal and
+stateful behavior easier to test deterministically. Full planner and runtime 
integration
+should still be covered by integration tests.
+
+#### Strict BYTES-to-STRING casts and UTF-8 utilities
+
+##### [FLINK-39600](https://issues.apache.org/jira/browse/FLINK-39600) 
(FLIP-568)
+
+Casting binary data to a string now validates UTF-8. Previously, malformed 
byte sequences
+were silently replaced with the Unicode replacement character `U+FFFD`; in 
Flink 2.4,
+`CAST(bytes AS STRING)` fails on invalid UTF-8 and `TRY_CAST(bytes AS STRING)` 
returns NULL.
+This is a behavior change for pipelines that previously accepted malformed 
input.
+
+Two new functions provide explicit handling:
+
+- `IS_VALID_UTF8(bytes)` returns whether the input is valid UTF-8, for 
filtering or routing
+  malformed records.
+- `MAKE_VALID_UTF8(bytes)` decodes the input and replaces invalid sequences 
with `U+FFFD`,
+  preserving the previous substitution behavior.
+
+Both functions return NULL for NULL input. As a migration option,
+`table.exec.legacy-bytes-to-string-cast` (default: `false`) can be set to 
`true` to restore
+the previous cast behavior.
+
+Connector authors can use the new `StringData.fromUtf8Bytes(byte[])` method 
for strict
+validation at ingestion. The existing `StringData.fromBytes(byte[])` remains 
unchanged.
+
+#### Metadata filter push-down for table sources
+
+##### [FLINK-39421](https://issues.apache.org/jira/browse/FLINK-39421) 
(FLIP-574)
+
+Source connectors can now accept filter predicates on metadata columns, such 
as offsets,
+partitions, or timestamps. This lets a supporting connector reduce the data it 
reads instead
+of applying all such filters after ingestion.
+
+Connectors opt in through the new `supportsMetadataFilterPushDown()` and
+`applyMetadataFilters()` methods on `SupportsReadingMetadata`. Predicates use 
metadata keys
+rather than SQL column aliases, and filters that the connector cannot fully 
apply remain in
+the query. Metadata predicates are also preserved when compiled plans are 
restored.
+Existing connectors retain their behavior unless they implement the new 
capability.
+
+#### In-place conversion of tables to materialized tables
+
+##### [FLINK-39847](https://issues.apache.org/jira/browse/FLINK-39847) 
(FLIP-578)
+
+`CREATE OR ALTER MATERIALIZED TABLE` can convert an existing regular table 
into a materialized
+table without dropping and recreating the catalog object. The conversion 
preserves the table's
+identity and underlying storage and starts a refresh job for the defining 
query.
+
+Conversion is disabled by default. Enable it when initializing the session 
with:

Review Comment:
   This option is read only from the root configuration 
(`SqlCreateOrAlterMaterializedTableConverter` calls `getRootConfiguration()`), 
and its own description says "a session-level SET statement has no effect". 
"When initializing the session" reads like a SQL Client `-i` init script, which 
is made of `SET` statements, and the user then only sees `Regular table does 
not support create or alter operation.` Could we say where to set it, e.g. 
"Enable it in `config.yaml`. A session-level `SET` statement has no effect:"?



##########
docs/content/release-notes/flink-2.4.md:
##########
@@ -0,0 +1,359 @@
+---
+title: "Release Notes - Flink 2.4"
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Release notes - Flink 2.4
+
+These release notes discuss important aspects, such as configuration, behavior 
or dependencies,
+that changed between Flink 2.3 and Flink 2.4. Please read these notes 
carefully if you are
+planning to upgrade your Flink version to 2.4.
+
+### Table SQL / API
+
+#### Upgrade Calcite to 1.41.0
+
+##### [FLINK-39859](https://issues.apache.org/jira/browse/FLINK-39859)
+
+Flink 2.4 upgrades Apache Calcite, the foundation of the SQL parser and 
optimizer, to version
+1.41.0. This incorporates the fixes and improvements from the intervening 
Calcite releases.
+Applications that depend directly on Calcite classes or extend Flink's planner 
should check
+their dependencies and custom planner code against the upgraded version.
+
+#### Common subexpression elimination in Calc code generation
+
+##### [FLINK-39268](https://issues.apache.org/jira/browse/FLINK-39268)
+
+Flink 2.4 avoids redundant evaluation of common subexpressions within a Calc 
operator, which
+combines filtering and projection. Repeated deterministic expressions can 
reuse an already
+computed result instead of being evaluated again for the same input row. This 
applies to both
+batch and streaming queries and can reduce the cost of repeated built-in or 
user-defined
+function calls.
+
+Reuse respects conditional evaluation, such as short-circuiting in `AND`, 
`OR`, and `CASE`.
+UDF authors should ensure that `isDeterministic()` correctly describes their 
function:
+functions with side effects or results that vary for identical inputs must 
return `false`.
+
+#### Built-in UDF metrics
+
+##### [FLINK-38071](https://issues.apache.org/jira/browse/FLINK-38071) 
(FLIP-485)
+
+SQL and Table API users can now monitor UDF execution through two built-in 
metrics:
+`udfProcessingTime`, a histogram of sampled invocation durations, and 
`udfExceptionCount`,
+a counter of exceptions from the function. Metrics are registered on the 
executing operator
+under `udf.<udf_name>`. For asynchronous functions, processing time includes 
the time from
+dispatch until completion.
+
+The feature is opt-in:
+
+- `table.exec.udf-metric-enabled` (default: `false`) enables metric collection.
+- `table.exec.udf-metric.sample-interval` (default: `100`) controls how often 
invocation
+  durations are measured. Exceptions are counted on every invocation.
+
+The metrics cover directly invoked synchronous and asynchronous scalar and 
table functions.
+Python functions, aggregate functions, and Process Table Functions are not 
covered. Enabling
+the metrics adds overhead; use sampling to control the cost.
+
+#### Early firing for interval joins
+
+##### [FLINK-36953](https://issues.apache.org/jira/browse/FLINK-36953) 
(FLIP-497)
+
+Outer interval joins can emit unmatched rows before their join interval 
closes. The new
+`EARLY_FIRE` hint emits a speculative, null-padded result after a configured 
delay. If a
+matching row arrives later within the interval, Flink corrects the speculative 
result with
+update-before and update-after records.
+
+```sql
+SELECT /*+ EARLY_FIRE('delay'='5s') */ o.id, s.ship_time
+FROM Orders o
+LEFT OUTER JOIN Shipments s
+ON o.id = s.order_id
+AND o.order_time BETWEEN s.ship_time - INTERVAL '1' HOUR AND s.ship_time;
+```
+
+The optional `time-mode` selects `rowtime` or `proctime`; it defaults to the 
time mode of the
+interval join. Each unmatched outer row fires at most once. The feature is 
disabled unless
+the hint is present and requires downstream operators and sinks that accept 
updates.
+Existing interval joins without the hint retain their behavior.
+
+#### FROM_CHANGELOG and TO_CHANGELOG enhancements
+
+##### [FLINK-39258](https://issues.apache.org/jira/browse/FLINK-39258) 
(FLIP-564)
+
+Flink 2.4 adds partitioned processing and additional controls to the changelog 
conversion
+PTFs introduced in Flink 2.3:
+
+- `PARTITION BY` support to co-locate related changes, with corresponding 
convenience methods
+  on partitioned tables in the Table API.
+- Partitioned `FROM_CHANGELOG` calls can use the existing PTF `ORDER BY` 
support to reorder
+  CDC events per key using a watermarked time attribute. Late events are 
dropped by the
+  sorting operation.
+- `error_handling => 'FAIL' | 'SKIP'` for `FROM_CHANGELOG` to control how NULL 
or unmapped
+  operation codes are handled. The default is `FAIL`.
+- `produces_full_deletes` for `TO_CHANGELOG` to choose between full-row 
deletes and deletes
+  containing only identifying keys. The default is `true`.
+
+These additions make it easier to consume custom CDC streams and write 
changelog events to
+append-only sinks. Users must still provide correctly ordered, complete change 
events for
+the selected changelog mode; the functions do not reconstruct arbitrary 
missing before images.
+See the [Changelog Conversion 
documentation](https://nightlies.apache.org/flink/flink-docs-release-2.4/docs/sql/reference/queries/changelog/)
+for supported mappings and input requirements.
+
+#### Process Table Function state access and broadcast support
+
+##### [FLINK-39254](https://issues.apache.org/jira/browse/FLINK-39254) 
(FLIP-565)
+
+Flink 2.4 adds lazy value-state access and broadcast inputs to Process Table 
Functions:
+
+- `ValueView` provides lazy value-state access alongside `ListView` and 
`MapView`. A function
+  can read or update state only when needed, avoiding unnecessary 
serialization work.
+- `BROADCAST_SEMANTIC_TABLE` lets a PTF receive a rule or control table on 
every parallel
+  instance. State declared with `@StateHint(StateKind.BROADCAST)` is writable 
from the
+  broadcast input and readable while processing the regular inputs.
+- `NOTIFY_STATEFUL_SETS` lets a broadcast update trigger re-evaluation of 
existing stateful
+  partitions. This is opt-in because visiting all stateful partitions can be 
expensive.
+
+#### Process Table Function test harness
+
+##### [FLINK-39357](https://issues.apache.org/jira/browse/FLINK-39357) 
(FLIP-567)
+
+The new `ProcessTableFunctionTestHarness` lets developers test PTF business 
logic without
+starting a Flink cluster. Tests can configure scalar and table arguments, 
partitioning, and
+initial state, feed rows into individual inputs, and inspect the resulting 
output.
+
+The harness also supports advancing watermarks and the system clock, 
inspecting timers,
+accessing state per key, and testing state TTL and changelog handling. This 
makes temporal and
+stateful behavior easier to test deterministically. Full planner and runtime 
integration
+should still be covered by integration tests.
+
+#### Strict BYTES-to-STRING casts and UTF-8 utilities
+
+##### [FLINK-39600](https://issues.apache.org/jira/browse/FLINK-39600) 
(FLIP-568)
+
+Casting binary data to a string now validates UTF-8. Previously, malformed 
byte sequences
+were silently replaced with the Unicode replacement character `U+FFFD`; in 
Flink 2.4,
+`CAST(bytes AS STRING)` fails on invalid UTF-8 and `TRY_CAST(bytes AS STRING)` 
returns NULL.
+This is a behavior change for pipelines that previously accepted malformed 
input.
+
+Two new functions provide explicit handling:
+
+- `IS_VALID_UTF8(bytes)` returns whether the input is valid UTF-8, for 
filtering or routing
+  malformed records.
+- `MAKE_VALID_UTF8(bytes)` decodes the input and replaces invalid sequences 
with `U+FFFD`,
+  preserving the previous substitution behavior.
+
+Both functions return NULL for NULL input. As a migration option,
+`table.exec.legacy-bytes-to-string-cast` (default: `false`) can be set to 
`true` to restore
+the previous cast behavior.
+
+Connector authors can use the new `StringData.fromUtf8Bytes(byte[])` method 
for strict
+validation at ingestion. The existing `StringData.fromBytes(byte[])` remains 
unchanged.
+
+#### Metadata filter push-down for table sources
+
+##### [FLINK-39421](https://issues.apache.org/jira/browse/FLINK-39421) 
(FLIP-574)
+
+Source connectors can now accept filter predicates on metadata columns, such 
as offsets,
+partitions, or timestamps. This lets a supporting connector reduce the data it 
reads instead
+of applying all such filters after ingestion.
+
+Connectors opt in through the new `supportsMetadataFilterPushDown()` and
+`applyMetadataFilters()` methods on `SupportsReadingMetadata`. Predicates use 
metadata keys
+rather than SQL column aliases, and filters that the connector cannot fully 
apply remain in
+the query. Metadata predicates are also preserved when compiled plans are 
restored.
+Existing connectors retain their behavior unless they implement the new 
capability.
+
+#### In-place conversion of tables to materialized tables
+
+##### [FLINK-39847](https://issues.apache.org/jira/browse/FLINK-39847) 
(FLIP-578)
+
+`CREATE OR ALTER MATERIALIZED TABLE` can convert an existing regular table 
into a materialized
+table without dropping and recreating the catalog object. The conversion 
preserves the table's
+identity and underlying storage and starts a refresh job for the defining 
query.
+
+Conversion is disabled by default. Enable it when initializing the session 
with:
+
+```yaml
+table.materialized-table.conversion-from-table.enabled: true
+```
+
+The catalog must support the new `convertTableToMaterializedTable` operation. 
The conversion
+statement defines the resulting schema and options; existing primary keys, 
watermarks, and
+other definitions must be declared again where needed. This is a one-way 
conversion: there is
+no corresponding operation to turn a materialized table back into a regular 
table.
+
+#### LATERAL SNAPSHOT joins
+
+##### [FLINK-39779](https://issues.apache.org/jira/browse/FLINK-39779) 
(FLIP-579)
+
+Flink SQL gains `LATERAL SNAPSHOT` joins for enriching an append-only stream 
with the current
+contents of an updating table. Unlike an event-time temporal join, the join 
does not wait for
+both inputs' watermarks for every result. It first buffers probe-side rows 
while initializing
+the build side, then joins arriving rows immediately against the current 
build-side state.
+
+`INNER` and `LEFT OUTER` joins are supported, and the build side does not need 
a primary key.
+Streaming queries specify a watermarked build-side column through `on_time`.
+`load_completed_time` controls when initialization completes; an optional
+`load_completed_idle_timeout` provides a fallback when build-side watermarks 
stop advancing.
+For static or infrequently updated build sides, configure this timeout or use 
a bounded source
+so the join does not remain in its initialization phase.
+
+Results follow processing-time semantics and can differ after recovery. Users 
requiring an
+exact event-time version should continue to use event-time temporal joins.
+See the [LATERAL SNAPSHOT join 
documentation](https://nightlies.apache.org/flink/flink-docs-release-2.4/docs/sql/reference/queries/joins/#lateral-snapshot-join)
+for the full syntax and initialization behavior.
+
+#### UUID type and extended VARIANT primitives
+
+##### [FLINK-40485](https://issues.apache.org/jira/browse/FLINK-40485) 
(FLIP-604)
+
+Flink 2.4 introduces a native `UUID` type backed by a 16-byte value, with 
explicit casts to
+and from strings and binary data, comparison and ordering support, and typed 
`UUID_V4()` and
+`UUID_V7()` generators. The existing `UUID()` function continues to return a 
string.
+
+The `VARIANT` type also supports time-of-day values, nanosecond-precision 
timestamps with and
+without local time zone, and UUID values. This extends compatibility with the 
corresponding
+Parquet Variant primitive encodings. Nanosecond timestamps can be read as 
`TIMESTAMP(9)` or
+`TIMESTAMP_LTZ(9)`; converting a VARIANT time-of-day value to SQL `TIME(3)` 
truncates any
+sub-millisecond precision.
+
+Existing VARIANT values remain readable. Older Flink versions cannot read 
values containing

Review Comment:
   `VARIANT` also has a breaking change that isn't mentioned: it, or a type 
containing it, can no longer be a grouping, `DISTINCT`, `ORDER BY` or join key, 
an operand of `=`, `<` or `IN`, or the compared argument of functions such as 
`ARRAY_CONTAINS` and `GREATEST`, so queries that planned on 2.3 can fail 
([FLINK-40910](https://issues.apache.org/jira/browse/FLINK-40910), 
[FLINK-40936](https://issues.apache.org/jira/browse/FLINK-40936)). Both 
tickets' release notes give the workaround (cast to a concrete type first). 
Would it fit next to this compatibility paragraph?



##########
docs/content/release-notes/flink-2.4.md:
##########
@@ -0,0 +1,359 @@
+---
+title: "Release Notes - Flink 2.4"
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Release notes - Flink 2.4
+
+These release notes discuss important aspects, such as configuration, behavior 
or dependencies,
+that changed between Flink 2.3 and Flink 2.4. Please read these notes 
carefully if you are
+planning to upgrade your Flink version to 2.4.
+
+### Table SQL / API
+
+#### Upgrade Calcite to 1.41.0
+
+##### [FLINK-39859](https://issues.apache.org/jira/browse/FLINK-39859)
+
+Flink 2.4 upgrades Apache Calcite, the foundation of the SQL parser and 
optimizer, to version
+1.41.0. This incorporates the fixes and improvements from the intervening 
Calcite releases.
+Applications that depend directly on Calcite classes or extend Flink's planner 
should check
+their dependencies and custom planner code against the upgraded version.
+
+#### Common subexpression elimination in Calc code generation
+
+##### [FLINK-39268](https://issues.apache.org/jira/browse/FLINK-39268)
+
+Flink 2.4 avoids redundant evaluation of common subexpressions within a Calc 
operator, which
+combines filtering and projection. Repeated deterministic expressions can 
reuse an already
+computed result instead of being evaluated again for the same input row. This 
applies to both
+batch and streaming queries and can reduce the cost of repeated built-in or 
user-defined
+function calls.
+
+Reuse respects conditional evaluation, such as short-circuiting in `AND`, 
`OR`, and `CASE`.
+UDF authors should ensure that `isDeterministic()` correctly describes their 
function:
+functions with side effects or results that vary for identical inputs must 
return `false`.
+
+#### Built-in UDF metrics
+
+##### [FLINK-38071](https://issues.apache.org/jira/browse/FLINK-38071) 
(FLIP-485)
+
+SQL and Table API users can now monitor UDF execution through two built-in 
metrics:
+`udfProcessingTime`, a histogram of sampled invocation durations, and 
`udfExceptionCount`,
+a counter of exceptions from the function. Metrics are registered on the 
executing operator
+under `udf.<udf_name>`. For asynchronous functions, processing time includes 
the time from
+dispatch until completion.
+
+The feature is opt-in:
+
+- `table.exec.udf-metric-enabled` (default: `false`) enables metric collection.
+- `table.exec.udf-metric.sample-interval` (default: `100`) controls how often 
invocation
+  durations are measured. Exceptions are counted on every invocation.
+
+The metrics cover directly invoked synchronous and asynchronous scalar and 
table functions.
+Python functions, aggregate functions, and Process Table Functions are not 
covered. Enabling
+the metrics adds overhead; use sampling to control the cost.
+
+#### Early firing for interval joins
+
+##### [FLINK-36953](https://issues.apache.org/jira/browse/FLINK-36953) 
(FLIP-497)
+
+Outer interval joins can emit unmatched rows before their join interval 
closes. The new
+`EARLY_FIRE` hint emits a speculative, null-padded result after a configured 
delay. If a
+matching row arrives later within the interval, Flink corrects the speculative 
result with
+update-before and update-after records.
+
+```sql
+SELECT /*+ EARLY_FIRE('delay'='5s') */ o.id, s.ship_time
+FROM Orders o
+LEFT OUTER JOIN Shipments s
+ON o.id = s.order_id
+AND o.order_time BETWEEN s.ship_time - INTERVAL '1' HOUR AND s.ship_time;
+```
+
+The optional `time-mode` selects `rowtime` or `proctime`; it defaults to the 
time mode of the
+interval join. Each unmatched outer row fires at most once. The feature is 
disabled unless
+the hint is present and requires downstream operators and sinks that accept 
updates.
+Existing interval joins without the hint retain their behavior.
+
+#### FROM_CHANGELOG and TO_CHANGELOG enhancements
+
+##### [FLINK-39258](https://issues.apache.org/jira/browse/FLINK-39258) 
(FLIP-564)
+
+Flink 2.4 adds partitioned processing and additional controls to the changelog 
conversion
+PTFs introduced in Flink 2.3:
+
+- `PARTITION BY` support to co-locate related changes, with corresponding 
convenience methods
+  on partitioned tables in the Table API.
+- Partitioned `FROM_CHANGELOG` calls can use the existing PTF `ORDER BY` 
support to reorder
+  CDC events per key using a watermarked time attribute. Late events are 
dropped by the
+  sorting operation.
+- `error_handling => 'FAIL' | 'SKIP'` for `FROM_CHANGELOG` to control how NULL 
or unmapped
+  operation codes are handled. The default is `FAIL`.
+- `produces_full_deletes` for `TO_CHANGELOG` to choose between full-row 
deletes and deletes
+  containing only identifying keys. The default is `true`.
+
+These additions make it easier to consume custom CDC streams and write 
changelog events to
+append-only sinks. Users must still provide correctly ordered, complete change 
events for
+the selected changelog mode; the functions do not reconstruct arbitrary 
missing before images.
+See the [Changelog Conversion 
documentation](https://nightlies.apache.org/flink/flink-docs-release-2.4/docs/sql/reference/queries/changelog/)
+for supported mappings and input requirements.
+
+#### Process Table Function state access and broadcast support
+
+##### [FLINK-39254](https://issues.apache.org/jira/browse/FLINK-39254) 
(FLIP-565)
+
+Flink 2.4 adds lazy value-state access and broadcast inputs to Process Table 
Functions:
+
+- `ValueView` provides lazy value-state access alongside `ListView` and 
`MapView`. A function
+  can read or update state only when needed, avoiding unnecessary 
serialization work.
+- `BROADCAST_SEMANTIC_TABLE` lets a PTF receive a rule or control table on 
every parallel
+  instance. State declared with `@StateHint(StateKind.BROADCAST)` is writable 
from the
+  broadcast input and readable while processing the regular inputs.
+- `NOTIFY_STATEFUL_SETS` lets a broadcast update trigger re-evaluation of 
existing stateful
+  partitions. This is opt-in because visiting all stateful partitions can be 
expensive.
+
+#### Process Table Function test harness
+
+##### [FLINK-39357](https://issues.apache.org/jira/browse/FLINK-39357) 
(FLIP-567)
+
+The new `ProcessTableFunctionTestHarness` lets developers test PTF business 
logic without
+starting a Flink cluster. Tests can configure scalar and table arguments, 
partitioning, and
+initial state, feed rows into individual inputs, and inspect the resulting 
output.
+
+The harness also supports advancing watermarks and the system clock, 
inspecting timers,
+accessing state per key, and testing state TTL and changelog handling. This 
makes temporal and
+stateful behavior easier to test deterministically. Full planner and runtime 
integration
+should still be covered by integration tests.
+
+#### Strict BYTES-to-STRING casts and UTF-8 utilities
+
+##### [FLINK-39600](https://issues.apache.org/jira/browse/FLINK-39600) 
(FLIP-568)
+
+Casting binary data to a string now validates UTF-8. Previously, malformed 
byte sequences
+were silently replaced with the Unicode replacement character `U+FFFD`; in 
Flink 2.4,
+`CAST(bytes AS STRING)` fails on invalid UTF-8 and `TRY_CAST(bytes AS STRING)` 
returns NULL.
+This is a behavior change for pipelines that previously accepted malformed 
input.
+
+Two new functions provide explicit handling:
+
+- `IS_VALID_UTF8(bytes)` returns whether the input is valid UTF-8, for 
filtering or routing
+  malformed records.
+- `MAKE_VALID_UTF8(bytes)` decodes the input and replaces invalid sequences 
with `U+FFFD`,
+  preserving the previous substitution behavior.
+
+Both functions return NULL for NULL input. As a migration option,
+`table.exec.legacy-bytes-to-string-cast` (default: `false`) can be set to 
`true` to restore
+the previous cast behavior.
+
+Connector authors can use the new `StringData.fromUtf8Bytes(byte[])` method 
for strict
+validation at ingestion. The existing `StringData.fromBytes(byte[])` remains 
unchanged.
+
+#### Metadata filter push-down for table sources
+
+##### [FLINK-39421](https://issues.apache.org/jira/browse/FLINK-39421) 
(FLIP-574)
+
+Source connectors can now accept filter predicates on metadata columns, such 
as offsets,
+partitions, or timestamps. This lets a supporting connector reduce the data it 
reads instead
+of applying all such filters after ingestion.
+
+Connectors opt in through the new `supportsMetadataFilterPushDown()` and
+`applyMetadataFilters()` methods on `SupportsReadingMetadata`. Predicates use 
metadata keys
+rather than SQL column aliases, and filters that the connector cannot fully 
apply remain in
+the query. Metadata predicates are also preserved when compiled plans are 
restored.
+Existing connectors retain their behavior unless they implement the new 
capability.
+
+#### In-place conversion of tables to materialized tables
+
+##### [FLINK-39847](https://issues.apache.org/jira/browse/FLINK-39847) 
(FLIP-578)
+
+`CREATE OR ALTER MATERIALIZED TABLE` can convert an existing regular table 
into a materialized
+table without dropping and recreating the catalog object. The conversion 
preserves the table's
+identity and underlying storage and starts a refresh job for the defining 
query.
+
+Conversion is disabled by default. Enable it when initializing the session 
with:
+
+```yaml
+table.materialized-table.conversion-from-table.enabled: true
+```
+
+The catalog must support the new `convertTableToMaterializedTable` operation. 
The conversion
+statement defines the resulting schema and options; existing primary keys, 
watermarks, and
+other definitions must be declared again where needed. This is a one-way 
conversion: there is
+no corresponding operation to turn a materialized table back into a regular 
table.
+
+#### LATERAL SNAPSHOT joins
+
+##### [FLINK-39779](https://issues.apache.org/jira/browse/FLINK-39779) 
(FLIP-579)
+
+Flink SQL gains `LATERAL SNAPSHOT` joins for enriching an append-only stream 
with the current
+contents of an updating table. Unlike an event-time temporal join, the join 
does not wait for
+both inputs' watermarks for every result. It first buffers probe-side rows 
while initializing
+the build side, then joins arriving rows immediately against the current 
build-side state.
+
+`INNER` and `LEFT OUTER` joins are supported, and the build side does not need 
a primary key.
+Streaming queries specify a watermarked build-side column through `on_time`.
+`load_completed_time` controls when initialization completes; an optional
+`load_completed_idle_timeout` provides a fallback when build-side watermarks 
stop advancing.
+For static or infrequently updated build sides, configure this timeout or use 
a bounded source
+so the join does not remain in its initialization phase.
+
+Results follow processing-time semantics and can differ after recovery. Users 
requiring an
+exact event-time version should continue to use event-time temporal joins.
+See the [LATERAL SNAPSHOT join 
documentation](https://nightlies.apache.org/flink/flink-docs-release-2.4/docs/sql/reference/queries/joins/#lateral-snapshot-join)
+for the full syntax and initialization behavior.
+
+#### UUID type and extended VARIANT primitives
+
+##### [FLINK-40485](https://issues.apache.org/jira/browse/FLINK-40485) 
(FLIP-604)
+
+Flink 2.4 introduces a native `UUID` type backed by a 16-byte value, with 
explicit casts to

Review Comment:
   Two `UUID` gaps aren't mentioned. Formats don't read or write `UUID` columns 
yet (the shipped `data-types.md` says to cast to `STRING` or `BINARY(16)` 
before a sink). And three sub-tasks targeting 2.4.0 are still open with no fix 
on `release-2.4`: 
[FLINK-41332](https://issues.apache.org/jira/browse/FLINK-41332) (`COMPILE 
PLAN` fails on a `UUID` literal), 
[FLINK-41333](https://issues.apache.org/jira/browse/FLINK-41333) (`MIN`/`MAX` 
and streaming `COUNT(DISTINCT)` reject `UUID`) and 
[FLINK-41334](https://issues.apache.org/jira/browse/FLINK-41334) (SQL Gateway 
JSON results and the JDBC driver can't return a `UUID` column). How about 
noting the formats gap here, and listing the three as known limitations if they 
don't land?



##########
docs/content/release-notes/flink-2.4.md:
##########
@@ -0,0 +1,359 @@
+---
+title: "Release Notes - Flink 2.4"
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Release notes - Flink 2.4
+
+These release notes discuss important aspects, such as configuration, behavior 
or dependencies,
+that changed between Flink 2.3 and Flink 2.4. Please read these notes 
carefully if you are
+planning to upgrade your Flink version to 2.4.
+
+### Table SQL / API
+
+#### Upgrade Calcite to 1.41.0
+
+##### [FLINK-39859](https://issues.apache.org/jira/browse/FLINK-39859)
+
+Flink 2.4 upgrades Apache Calcite, the foundation of the SQL parser and 
optimizer, to version
+1.41.0. This incorporates the fixes and improvements from the intervening 
Calcite releases.
+Applications that depend directly on Calcite classes or extend Flink's planner 
should check
+their dependencies and custom planner code against the upgraded version.
+
+#### Common subexpression elimination in Calc code generation
+
+##### [FLINK-39268](https://issues.apache.org/jira/browse/FLINK-39268)
+
+Flink 2.4 avoids redundant evaluation of common subexpressions within a Calc 
operator, which
+combines filtering and projection. Repeated deterministic expressions can 
reuse an already
+computed result instead of being evaluated again for the same input row. This 
applies to both
+batch and streaming queries and can reduce the cost of repeated built-in or 
user-defined
+function calls.
+
+Reuse respects conditional evaluation, such as short-circuiting in `AND`, 
`OR`, and `CASE`.
+UDF authors should ensure that `isDeterministic()` correctly describes their 
function:
+functions with side effects or results that vary for identical inputs must 
return `false`.
+
+#### Built-in UDF metrics
+
+##### [FLINK-38071](https://issues.apache.org/jira/browse/FLINK-38071) 
(FLIP-485)
+
+SQL and Table API users can now monitor UDF execution through two built-in 
metrics:
+`udfProcessingTime`, a histogram of sampled invocation durations, and 
`udfExceptionCount`,
+a counter of exceptions from the function. Metrics are registered on the 
executing operator
+under `udf.<udf_name>`. For asynchronous functions, processing time includes 
the time from
+dispatch until completion.
+
+The feature is opt-in:
+
+- `table.exec.udf-metric-enabled` (default: `false`) enables metric collection.
+- `table.exec.udf-metric.sample-interval` (default: `100`) controls how often 
invocation
+  durations are measured. Exceptions are counted on every invocation.
+
+The metrics cover directly invoked synchronous and asynchronous scalar and 
table functions.
+Python functions, aggregate functions, and Process Table Functions are not 
covered. Enabling
+the metrics adds overhead; use sampling to control the cost.
+
+#### Early firing for interval joins
+
+##### [FLINK-36953](https://issues.apache.org/jira/browse/FLINK-36953) 
(FLIP-497)
+
+Outer interval joins can emit unmatched rows before their join interval 
closes. The new
+`EARLY_FIRE` hint emits a speculative, null-padded result after a configured 
delay. If a
+matching row arrives later within the interval, Flink corrects the speculative 
result with
+update-before and update-after records.
+
+```sql
+SELECT /*+ EARLY_FIRE('delay'='5s') */ o.id, s.ship_time
+FROM Orders o
+LEFT OUTER JOIN Shipments s
+ON o.id = s.order_id
+AND o.order_time BETWEEN s.ship_time - INTERVAL '1' HOUR AND s.ship_time;
+```
+
+The optional `time-mode` selects `rowtime` or `proctime`; it defaults to the 
time mode of the
+interval join. Each unmatched outer row fires at most once. The feature is 
disabled unless
+the hint is present and requires downstream operators and sinks that accept 
updates.
+Existing interval joins without the hint retain their behavior.
+
+#### FROM_CHANGELOG and TO_CHANGELOG enhancements
+
+##### [FLINK-39258](https://issues.apache.org/jira/browse/FLINK-39258) 
(FLIP-564)
+
+Flink 2.4 adds partitioned processing and additional controls to the changelog 
conversion
+PTFs introduced in Flink 2.3:
+
+- `PARTITION BY` support to co-locate related changes, with corresponding 
convenience methods
+  on partitioned tables in the Table API.
+- Partitioned `FROM_CHANGELOG` calls can use the existing PTF `ORDER BY` 
support to reorder
+  CDC events per key using a watermarked time attribute. Late events are 
dropped by the
+  sorting operation.
+- `error_handling => 'FAIL' | 'SKIP'` for `FROM_CHANGELOG` to control how NULL 
or unmapped
+  operation codes are handled. The default is `FAIL`.
+- `produces_full_deletes` for `TO_CHANGELOG` to choose between full-row 
deletes and deletes
+  containing only identifying keys. The default is `true`.
+
+These additions make it easier to consume custom CDC streams and write 
changelog events to
+append-only sinks. Users must still provide correctly ordered, complete change 
events for
+the selected changelog mode; the functions do not reconstruct arbitrary 
missing before images.
+See the [Changelog Conversion 
documentation](https://nightlies.apache.org/flink/flink-docs-release-2.4/docs/sql/reference/queries/changelog/)
+for supported mappings and input requirements.
+
+#### Process Table Function state access and broadcast support
+
+##### [FLINK-39254](https://issues.apache.org/jira/browse/FLINK-39254) 
(FLIP-565)
+
+Flink 2.4 adds lazy value-state access and broadcast inputs to Process Table 
Functions:
+
+- `ValueView` provides lazy value-state access alongside `ListView` and 
`MapView`. A function
+  can read or update state only when needed, avoiding unnecessary 
serialization work.
+- `BROADCAST_SEMANTIC_TABLE` lets a PTF receive a rule or control table on 
every parallel
+  instance. State declared with `@StateHint(StateKind.BROADCAST)` is writable 
from the
+  broadcast input and readable while processing the regular inputs.
+- `NOTIFY_STATEFUL_SETS` lets a broadcast update trigger re-evaluation of 
existing stateful
+  partitions. This is opt-in because visiting all stateful partitions can be 
expensive.
+
+#### Process Table Function test harness
+
+##### [FLINK-39357](https://issues.apache.org/jira/browse/FLINK-39357) 
(FLIP-567)
+
+The new `ProcessTableFunctionTestHarness` lets developers test PTF business 
logic without
+starting a Flink cluster. Tests can configure scalar and table arguments, 
partitioning, and
+initial state, feed rows into individual inputs, and inspect the resulting 
output.
+
+The harness also supports advancing watermarks and the system clock, 
inspecting timers,
+accessing state per key, and testing state TTL and changelog handling. This 
makes temporal and
+stateful behavior easier to test deterministically. Full planner and runtime 
integration
+should still be covered by integration tests.
+
+#### Strict BYTES-to-STRING casts and UTF-8 utilities
+
+##### [FLINK-39600](https://issues.apache.org/jira/browse/FLINK-39600) 
(FLIP-568)
+
+Casting binary data to a string now validates UTF-8. Previously, malformed 
byte sequences
+were silently replaced with the Unicode replacement character `U+FFFD`; in 
Flink 2.4,
+`CAST(bytes AS STRING)` fails on invalid UTF-8 and `TRY_CAST(bytes AS STRING)` 
returns NULL.
+This is a behavior change for pipelines that previously accepted malformed 
input.
+
+Two new functions provide explicit handling:
+
+- `IS_VALID_UTF8(bytes)` returns whether the input is valid UTF-8, for 
filtering or routing
+  malformed records.
+- `MAKE_VALID_UTF8(bytes)` decodes the input and replaces invalid sequences 
with `U+FFFD`,
+  preserving the previous substitution behavior.
+
+Both functions return NULL for NULL input. As a migration option,
+`table.exec.legacy-bytes-to-string-cast` (default: `false`) can be set to 
`true` to restore
+the previous cast behavior.
+
+Connector authors can use the new `StringData.fromUtf8Bytes(byte[])` method 
for strict
+validation at ingestion. The existing `StringData.fromBytes(byte[])` remains 
unchanged.
+
+#### Metadata filter push-down for table sources
+
+##### [FLINK-39421](https://issues.apache.org/jira/browse/FLINK-39421) 
(FLIP-574)
+
+Source connectors can now accept filter predicates on metadata columns, such 
as offsets,
+partitions, or timestamps. This lets a supporting connector reduce the data it 
reads instead
+of applying all such filters after ingestion.
+
+Connectors opt in through the new `supportsMetadataFilterPushDown()` and
+`applyMetadataFilters()` methods on `SupportsReadingMetadata`. Predicates use 
metadata keys
+rather than SQL column aliases, and filters that the connector cannot fully 
apply remain in
+the query. Metadata predicates are also preserved when compiled plans are 
restored.
+Existing connectors retain their behavior unless they implement the new 
capability.
+
+#### In-place conversion of tables to materialized tables
+
+##### [FLINK-39847](https://issues.apache.org/jira/browse/FLINK-39847) 
(FLIP-578)
+
+`CREATE OR ALTER MATERIALIZED TABLE` can convert an existing regular table 
into a materialized
+table without dropping and recreating the catalog object. The conversion 
preserves the table's
+identity and underlying storage and starts a refresh job for the defining 
query.
+
+Conversion is disabled by default. Enable it when initializing the session 
with:
+
+```yaml
+table.materialized-table.conversion-from-table.enabled: true
+```
+
+The catalog must support the new `convertTableToMaterializedTable` operation. 
The conversion
+statement defines the resulting schema and options; existing primary keys, 
watermarks, and
+other definitions must be declared again where needed. This is a one-way 
conversion: there is
+no corresponding operation to turn a materialized table back into a regular 
table.
+
+#### LATERAL SNAPSHOT joins
+
+##### [FLINK-39779](https://issues.apache.org/jira/browse/FLINK-39779) 
(FLIP-579)
+
+Flink SQL gains `LATERAL SNAPSHOT` joins for enriching an append-only stream 
with the current
+contents of an updating table. Unlike an event-time temporal join, the join 
does not wait for
+both inputs' watermarks for every result. It first buffers probe-side rows 
while initializing
+the build side, then joins arriving rows immediately against the current 
build-side state.
+
+`INNER` and `LEFT OUTER` joins are supported, and the build side does not need 
a primary key.
+Streaming queries specify a watermarked build-side column through `on_time`.
+`load_completed_time` controls when initialization completes; an optional
+`load_completed_idle_timeout` provides a fallback when build-side watermarks 
stop advancing.
+For static or infrequently updated build sides, configure this timeout or use 
a bounded source
+so the join does not remain in its initialization phase.
+
+Results follow processing-time semantics and can differ after recovery. Users 
requiring an
+exact event-time version should continue to use event-time temporal joins.
+See the [LATERAL SNAPSHOT join 
documentation](https://nightlies.apache.org/flink/flink-docs-release-2.4/docs/sql/reference/queries/joins/#lateral-snapshot-join)
+for the full syntax and initialization behavior.
+
+#### UUID type and extended VARIANT primitives
+
+##### [FLINK-40485](https://issues.apache.org/jira/browse/FLINK-40485) 
(FLIP-604)
+
+Flink 2.4 introduces a native `UUID` type backed by a 16-byte value, with 
explicit casts to
+and from strings and binary data, comparison and ordering support, and typed 
`UUID_V4()` and
+`UUID_V7()` generators. The existing `UUID()` function continues to return a 
string.
+
+The `VARIANT` type also supports time-of-day values, nanosecond-precision 
timestamps with and
+without local time zone, and UUID values. This extends compatibility with the 
corresponding
+Parquet Variant primitive encodings. Nanosecond timestamps can be read as 
`TIMESTAMP(9)` or
+`TIMESTAMP_LTZ(9)`; converting a VARIANT time-of-day value to SQL `TIME(3)` 
truncates any
+sub-millisecond precision.
+
+Existing VARIANT values remain readable. Older Flink versions cannot read 
values containing
+the newly supported primitive codes, so consider this limitation before 
downgrading a job or
+sharing persisted VARIANT data with older consumers.
+
+### Python
+
+#### Python DataFrame API
+
+##### [FLINK-40185](https://issues.apache.org/jira/browse/FLINK-40185) 
(FLIP-591)
+
+PyFlink gains a DataFrame API under `pyflink.dataframe`. It offers a 
Python-oriented entry
+point for creating Flink tables from records, dictionaries, pandas DataFrames, 
or Arrow tables,
+and for reading and writing through Flink connectors.
+
+The API supports familiar transformations such as selection, filtering, joins, 
aggregation,
+windowing, and missing-value handling, as well as Python UDFs and batch 
transformations.
+`from_table()` and `to_table()` allow applications to move between DataFrames 
and the existing
+Table API.
+
+DataFrame operations use Flink's existing planner and runtime for batch and 
streaming
+execution. The API complements the Table API; it does not replace it or 
promise full
+compatibility with pandas or other DataFrame libraries.
+
+### Connectors
+
+#### Job information on source contexts
+
+##### [FLINK-39776](https://issues.apache.org/jira/browse/FLINK-39776) 
(FLIP-583)
+
+`SourceReaderContext` and `SplitEnumeratorContext` now expose `getJobInfo()`, 
giving source
+implementations direct access to the job ID and job name. This aligns the 
Source API with
+Sink V2 and supports use cases such as per-job authentication, external offset 
tracking,
+and log correlation without parsing metric scope variables.
+
+The new methods have default implementations, so existing context 
implementations remain
+source- and binary-compatible. Flink's runtime contexts provide the job 
information;
+third-party contexts and test stubs must override the methods if callers use 
them, as the
+default implementation throws `UnsupportedOperationException`.
+
+### Runtime
+
+#### Filtering and spilling recovered channel state
+
+##### [FLINK-38544](https://issues.apache.org/jira/browse/FLINK-38544) 
(FLIP-547 V2)
+
+Flink 2.4 extends checkpointing during recovery with a spill-based recovery 
path. Recovered
+channel state is filtered for the receiving subtask before records enter 
normal processing,
+and the resulting data is written to local spill files before being replayed 
through network
+buffers. This bounds memory usage when recovering large amounts of in-flight 
data.
+
+Filtering before processing also prevents records that belong to other 
subtasks after
+rescaling from being included again in subsequent recovery checkpoints. 
Unaligned checkpoints
+capture both queued and unread spilled data, allowing recovery progress to 
survive another
+failure or rescaling without duplicating the recovered records.
+
+The spill path is used when checkpointing during recovery is enabled through 
the existing
+configuration options and requires sufficient local disk space. Flink 2.4 also 
disables
+checkpointing during recovery when 
`execution.checkpointing.aligned-checkpoint-timeout` is
+greater than zero.
+
+#### Pluggable HistoryServer archive storage
+
+##### [FLINK-39911](https://issues.apache.org/jira/browse/FLINK-39911) 
(FLIP-584)
+
+The HistoryServer can store its local archive cache in an embedded RocksDB 
database instead
+of expanding every archive into many small JSON files. This reduces inode 
consumption when
+serving large numbers of jobs or jobs with extensive task-level details.
+
+Select the backend with `historyserver.archive.storage.type`:
+
+- `FILE` (default) keeps the existing file-based storage behavior.
+- `ROCKSDB` stores archive entries in an embedded database under
+  `historyserver.web.tmpdir/rocksdb`.
+
+This changes how the local cache is stored; completed-job archives continue to 
be read from
+the configured archive directories.
+
+#### Lazy loading in HistoryServer
+
+##### [FLINK-40097](https://issues.apache.org/jira/browse/FLINK-40097) 
(FLIP-585)
+
+The HistoryServer gains a `LAZY` archive load mode. Overview information is 
made available
+before all job details have been processed, while details are loaded 
asynchronously.
+Accessing an archive that has not yet finished loading triggers or waits for 
the relevant
+on-demand load, so users do not have to wait for every preceding archive to be 
processed.
+
+The mode is opt-in through `historyserver.archive.load.mode` (default: 
`EAGER`). Lazy loading
+uses two separately configurable thread pools:
+
+- `historyserver.lazy.fetch.executor.common.pool-size` (default: `4`) for 
background loads.
+- `historyserver.lazy.fetch.executor.individual.pool-size` (default: `4`) for 
individual
+  on-demand loads.
+
+The default `EAGER` mode retains synchronous archive loading. Lazy loading 
works with both
+the file and RocksDB local storage backends.
+
+#### State Catalog
+
+##### [FLINK-40175](https://issues.apache.org/jira/browse/FLINK-40175) 
(FLIP-599)
+
+The new read-only State Catalog discovers checkpoints and savepoints in 
configured
+directories and exposes their contents through Flink SQL. Each discovered 
snapshot becomes
+a database, with tables for operator state and views for inspecting snapshot 
metadata.
+This removes much of the manual schema definition previously needed to query 
saved state.
+
+The catalog infers schemas from supported serializer snapshots and exposes 
keyed state,
+flattened list and map state, non-keyed list/union/broadcast state, and window 
state.
+Supported POJO and Avro state can be read without the original user classes on 
the classpath;
+custom serialization formats still require compatible type information and 
reader support.
+
+Register the catalog using one or more labeled directories:
+
+```sql
+CREATE CATALOG saved_state WITH (
+  'type' = 'state',
+  'directory.prod' = 's3://my-bucket/flink/checkpoints'
+);
+```
+
+The catalog is provided by `flink-state-processing-api`. Snapshot metadata 
views additionally

Review Comment:
   The Maven artifact is `flink-state-processor-api` 
(`flink-libraries/flink-state-processing-api/pom.xml`); 
`flink-state-processing-api` is only the directory name. The binary 
distribution ships the jar in `opt/` 
(`flink-dist/src/main/assemblies/opt.xml`), so `CREATE CATALOG` with `'type' = 
'state'` fails until it's on the classpath, and there's no State Catalog docs 
page on `release-2.4` yet. Would something like this work: "The catalog is 
provided by `flink-state-processor-api`, which the distribution ships in 
`opt/`. Add it to the classpath before creating the catalog."?



##########
docs/content/release-notes/flink-2.4.md:
##########
@@ -0,0 +1,359 @@
+---
+title: "Release Notes - Flink 2.4"
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Release notes - Flink 2.4
+
+These release notes discuss important aspects, such as configuration, behavior 
or dependencies,
+that changed between Flink 2.3 and Flink 2.4. Please read these notes 
carefully if you are
+planning to upgrade your Flink version to 2.4.
+
+### Table SQL / API
+
+#### Upgrade Calcite to 1.41.0
+
+##### [FLINK-39859](https://issues.apache.org/jira/browse/FLINK-39859)
+
+Flink 2.4 upgrades Apache Calcite, the foundation of the SQL parser and 
optimizer, to version
+1.41.0. This incorporates the fixes and improvements from the intervening 
Calcite releases.
+Applications that depend directly on Calcite classes or extend Flink's planner 
should check
+their dependencies and custom planner code against the upgraded version.
+
+#### Common subexpression elimination in Calc code generation
+
+##### [FLINK-39268](https://issues.apache.org/jira/browse/FLINK-39268)
+
+Flink 2.4 avoids redundant evaluation of common subexpressions within a Calc 
operator, which
+combines filtering and projection. Repeated deterministic expressions can 
reuse an already
+computed result instead of being evaluated again for the same input row. This 
applies to both
+batch and streaming queries and can reduce the cost of repeated built-in or 
user-defined
+function calls.
+
+Reuse respects conditional evaluation, such as short-circuiting in `AND`, 
`OR`, and `CASE`.
+UDF authors should ensure that `isDeterministic()` correctly describes their 
function:
+functions with side effects or results that vary for identical inputs must 
return `false`.
+
+#### Built-in UDF metrics
+
+##### [FLINK-38071](https://issues.apache.org/jira/browse/FLINK-38071) 
(FLIP-485)
+
+SQL and Table API users can now monitor UDF execution through two built-in 
metrics:
+`udfProcessingTime`, a histogram of sampled invocation durations, and 
`udfExceptionCount`,
+a counter of exceptions from the function. Metrics are registered on the 
executing operator
+under `udf.<udf_name>`. For asynchronous functions, processing time includes 
the time from
+dispatch until completion.
+
+The feature is opt-in:
+
+- `table.exec.udf-metric-enabled` (default: `false`) enables metric collection.
+- `table.exec.udf-metric.sample-interval` (default: `100`) controls how often 
invocation
+  durations are measured. Exceptions are counted on every invocation.
+
+The metrics cover directly invoked synchronous and asynchronous scalar and 
table functions.
+Python functions, aggregate functions, and Process Table Functions are not 
covered. Enabling
+the metrics adds overhead; use sampling to control the cost.
+
+#### Early firing for interval joins
+
+##### [FLINK-36953](https://issues.apache.org/jira/browse/FLINK-36953) 
(FLIP-497)
+
+Outer interval joins can emit unmatched rows before their join interval 
closes. The new
+`EARLY_FIRE` hint emits a speculative, null-padded result after a configured 
delay. If a
+matching row arrives later within the interval, Flink corrects the speculative 
result with
+update-before and update-after records.
+
+```sql
+SELECT /*+ EARLY_FIRE('delay'='5s') */ o.id, s.ship_time
+FROM Orders o
+LEFT OUTER JOIN Shipments s
+ON o.id = s.order_id
+AND o.order_time BETWEEN s.ship_time - INTERVAL '1' HOUR AND s.ship_time;
+```
+
+The optional `time-mode` selects `rowtime` or `proctime`; it defaults to the 
time mode of the
+interval join. Each unmatched outer row fires at most once. The feature is 
disabled unless
+the hint is present and requires downstream operators and sinks that accept 
updates.
+Existing interval joins without the hint retain their behavior.
+
+#### FROM_CHANGELOG and TO_CHANGELOG enhancements
+
+##### [FLINK-39258](https://issues.apache.org/jira/browse/FLINK-39258) 
(FLIP-564)
+
+Flink 2.4 adds partitioned processing and additional controls to the changelog 
conversion
+PTFs introduced in Flink 2.3:
+
+- `PARTITION BY` support to co-locate related changes, with corresponding 
convenience methods
+  on partitioned tables in the Table API.
+- Partitioned `FROM_CHANGELOG` calls can use the existing PTF `ORDER BY` 
support to reorder
+  CDC events per key using a watermarked time attribute. Late events are 
dropped by the
+  sorting operation.
+- `error_handling => 'FAIL' | 'SKIP'` for `FROM_CHANGELOG` to control how NULL 
or unmapped
+  operation codes are handled. The default is `FAIL`.
+- `produces_full_deletes` for `TO_CHANGELOG` to choose between full-row 
deletes and deletes
+  containing only identifying keys. The default is `true`.
+
+These additions make it easier to consume custom CDC streams and write 
changelog events to
+append-only sinks. Users must still provide correctly ordered, complete change 
events for
+the selected changelog mode; the functions do not reconstruct arbitrary 
missing before images.
+See the [Changelog Conversion 
documentation](https://nightlies.apache.org/flink/flink-docs-release-2.4/docs/sql/reference/queries/changelog/)
+for supported mappings and input requirements.
+
+#### Process Table Function state access and broadcast support
+
+##### [FLINK-39254](https://issues.apache.org/jira/browse/FLINK-39254) 
(FLIP-565)
+
+Flink 2.4 adds lazy value-state access and broadcast inputs to Process Table 
Functions:
+
+- `ValueView` provides lazy value-state access alongside `ListView` and 
`MapView`. A function
+  can read or update state only when needed, avoiding unnecessary 
serialization work.
+- `BROADCAST_SEMANTIC_TABLE` lets a PTF receive a rule or control table on 
every parallel
+  instance. State declared with `@StateHint(StateKind.BROADCAST)` is writable 
from the
+  broadcast input and readable while processing the regular inputs.
+- `NOTIFY_STATEFUL_SETS` lets a broadcast update trigger re-evaluation of 
existing stateful
+  partitions. This is opt-in because visiting all stateful partitions can be 
expensive.
+
+#### Process Table Function test harness
+
+##### [FLINK-39357](https://issues.apache.org/jira/browse/FLINK-39357) 
(FLIP-567)
+
+The new `ProcessTableFunctionTestHarness` lets developers test PTF business 
logic without
+starting a Flink cluster. Tests can configure scalar and table arguments, 
partitioning, and
+initial state, feed rows into individual inputs, and inspect the resulting 
output.
+
+The harness also supports advancing watermarks and the system clock, 
inspecting timers,
+accessing state per key, and testing state TTL and changelog handling. This 
makes temporal and
+stateful behavior easier to test deterministically. Full planner and runtime 
integration
+should still be covered by integration tests.
+
+#### Strict BYTES-to-STRING casts and UTF-8 utilities
+
+##### [FLINK-39600](https://issues.apache.org/jira/browse/FLINK-39600) 
(FLIP-568)
+
+Casting binary data to a string now validates UTF-8. Previously, malformed 
byte sequences
+were silently replaced with the Unicode replacement character `U+FFFD`; in 
Flink 2.4,
+`CAST(bytes AS STRING)` fails on invalid UTF-8 and `TRY_CAST(bytes AS STRING)` 
returns NULL.
+This is a behavior change for pipelines that previously accepted malformed 
input.
+
+Two new functions provide explicit handling:
+
+- `IS_VALID_UTF8(bytes)` returns whether the input is valid UTF-8, for 
filtering or routing
+  malformed records.
+- `MAKE_VALID_UTF8(bytes)` decodes the input and replaces invalid sequences 
with `U+FFFD`,
+  preserving the previous substitution behavior.
+
+Both functions return NULL for NULL input. As a migration option,
+`table.exec.legacy-bytes-to-string-cast` (default: `false`) can be set to 
`true` to restore
+the previous cast behavior.
+
+Connector authors can use the new `StringData.fromUtf8Bytes(byte[])` method 
for strict
+validation at ingestion. The existing `StringData.fromBytes(byte[])` remains 
unchanged.
+
+#### Metadata filter push-down for table sources
+
+##### [FLINK-39421](https://issues.apache.org/jira/browse/FLINK-39421) 
(FLIP-574)
+
+Source connectors can now accept filter predicates on metadata columns, such 
as offsets,
+partitions, or timestamps. This lets a supporting connector reduce the data it 
reads instead
+of applying all such filters after ingestion.
+
+Connectors opt in through the new `supportsMetadataFilterPushDown()` and
+`applyMetadataFilters()` methods on `SupportsReadingMetadata`. Predicates use 
metadata keys
+rather than SQL column aliases, and filters that the connector cannot fully 
apply remain in
+the query. Metadata predicates are also preserved when compiled plans are 
restored.
+Existing connectors retain their behavior unless they implement the new 
capability.
+
+#### In-place conversion of tables to materialized tables
+
+##### [FLINK-39847](https://issues.apache.org/jira/browse/FLINK-39847) 
(FLIP-578)
+
+`CREATE OR ALTER MATERIALIZED TABLE` can convert an existing regular table 
into a materialized
+table without dropping and recreating the catalog object. The conversion 
preserves the table's
+identity and underlying storage and starts a refresh job for the defining 
query.
+
+Conversion is disabled by default. Enable it when initializing the session 
with:
+
+```yaml
+table.materialized-table.conversion-from-table.enabled: true
+```
+
+The catalog must support the new `convertTableToMaterializedTable` operation. 
The conversion
+statement defines the resulting schema and options; existing primary keys, 
watermarks, and
+other definitions must be declared again where needed. This is a one-way 
conversion: there is
+no corresponding operation to turn a materialized table back into a regular 
table.
+
+#### LATERAL SNAPSHOT joins
+
+##### [FLINK-39779](https://issues.apache.org/jira/browse/FLINK-39779) 
(FLIP-579)
+
+Flink SQL gains `LATERAL SNAPSHOT` joins for enriching an append-only stream 
with the current
+contents of an updating table. Unlike an event-time temporal join, the join 
does not wait for
+both inputs' watermarks for every result. It first buffers probe-side rows 
while initializing
+the build side, then joins arriving rows immediately against the current 
build-side state.
+
+`INNER` and `LEFT OUTER` joins are supported, and the build side does not need 
a primary key.
+Streaming queries specify a watermarked build-side column through `on_time`.
+`load_completed_time` controls when initialization completes; an optional
+`load_completed_idle_timeout` provides a fallback when build-side watermarks 
stop advancing.
+For static or infrequently updated build sides, configure this timeout or use 
a bounded source
+so the join does not remain in its initialization phase.
+
+Results follow processing-time semantics and can differ after recovery. Users 
requiring an
+exact event-time version should continue to use event-time temporal joins.
+See the [LATERAL SNAPSHOT join 
documentation](https://nightlies.apache.org/flink/flink-docs-release-2.4/docs/sql/reference/queries/joins/#lateral-snapshot-join)
+for the full syntax and initialization behavior.
+
+#### UUID type and extended VARIANT primitives
+
+##### [FLINK-40485](https://issues.apache.org/jira/browse/FLINK-40485) 
(FLIP-604)
+
+Flink 2.4 introduces a native `UUID` type backed by a 16-byte value, with 
explicit casts to
+and from strings and binary data, comparison and ordering support, and typed 
`UUID_V4()` and
+`UUID_V7()` generators. The existing `UUID()` function continues to return a 
string.
+
+The `VARIANT` type also supports time-of-day values, nanosecond-precision 
timestamps with and
+without local time zone, and UUID values. This extends compatibility with the 
corresponding
+Parquet Variant primitive encodings. Nanosecond timestamps can be read as 
`TIMESTAMP(9)` or
+`TIMESTAMP_LTZ(9)`; converting a VARIANT time-of-day value to SQL `TIME(3)` 
truncates any
+sub-millisecond precision.
+
+Existing VARIANT values remain readable. Older Flink versions cannot read 
values containing
+the newly supported primitive codes, so consider this limitation before 
downgrading a job or
+sharing persisted VARIANT data with older consumers.
+
+### Python
+
+#### Python DataFrame API
+
+##### [FLINK-40185](https://issues.apache.org/jira/browse/FLINK-40185) 
(FLIP-591)
+
+PyFlink gains a DataFrame API under `pyflink.dataframe`. It offers a 
Python-oriented entry
+point for creating Flink tables from records, dictionaries, pandas DataFrames, 
or Arrow tables,
+and for reading and writing through Flink connectors.
+
+The API supports familiar transformations such as selection, filtering, joins, 
aggregation,
+windowing, and missing-value handling, as well as Python UDFs and batch 
transformations.
+`from_table()` and `to_table()` allow applications to move between DataFrames 
and the existing
+Table API.
+
+DataFrame operations use Flink's existing planner and runtime for batch and 
streaming
+execution. The API complements the Table API; it does not replace it or 
promise full
+compatibility with pandas or other DataFrame libraries.
+
+### Connectors
+
+#### Job information on source contexts
+
+##### [FLINK-39776](https://issues.apache.org/jira/browse/FLINK-39776) 
(FLIP-583)
+
+`SourceReaderContext` and `SplitEnumeratorContext` now expose `getJobInfo()`, 
giving source
+implementations direct access to the job ID and job name. This aligns the 
Source API with
+Sink V2 and supports use cases such as per-job authentication, external offset 
tracking,
+and log correlation without parsing metric scope variables.
+
+The new methods have default implementations, so existing context 
implementations remain
+source- and binary-compatible. Flink's runtime contexts provide the job 
information;
+third-party contexts and test stubs must override the methods if callers use 
them, as the
+default implementation throws `UnsupportedOperationException`.
+
+### Runtime

Review Comment:
   Two runtime behavior changes with JIRA release notes aren't here yet:
   - The Adaptive Scheduler now actively triggers a checkpoint when it wants to 
rescale, through the new 
`jobmanager.adaptive-scheduler.rescale-trigger.active-checkpoint.enabled` 
(default `true`) 
([FLINK-36753](https://issues.apache.org/jira/browse/FLINK-36753)). Its JIRA 
note spells the key `obmanager.`, in case it gets copied.
   - `GET /jobs/overview`, `/jobs` and `/overview` now return an error instead 
of a partial job list when a JobMaster can't be queried 
([FLINK-40602](https://issues.apache.org/jira/browse/FLINK-40602)).
   
   Should these go in this section?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to