weiqingy commented on code in PR #29957:
URL: https://github.com/apache/flink/pull/29957#discussion_r4638436032


##########
docs/content/release-notes/flink-2.4.md:
##########
@@ -0,0 +1,359 @@
+---
+title: "Release Notes - Flink 2.4"
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Release notes - Flink 2.4
+
+These release notes discuss important aspects, such as configuration, behavior 
or dependencies,
+that changed between Flink 2.3 and Flink 2.4. Please read these notes 
carefully if you are
+planning to upgrade your Flink version to 2.4.
+
+### Table SQL / API
+
+#### Upgrade Calcite to 1.41.0
+
+##### [FLINK-39859](https://issues.apache.org/jira/browse/FLINK-39859)
+
+Flink 2.4 upgrades Apache Calcite, the foundation of the SQL parser and 
optimizer, to version
+1.41.0. This incorporates the fixes and improvements from the intervening 
Calcite releases.
+Applications that depend directly on Calcite classes or extend Flink's planner 
should check
+their dependencies and custom planner code against the upgraded version.
+
+#### Common subexpression elimination in Calc code generation
+
+##### [FLINK-39268](https://issues.apache.org/jira/browse/FLINK-39268)
+
+Flink 2.4 avoids redundant evaluation of common subexpressions within a Calc 
operator, which
+combines filtering and projection. Repeated deterministic expressions can 
reuse an already
+computed result instead of being evaluated again for the same input row. This 
applies to both
+batch and streaming queries and can reduce the cost of repeated built-in or 
user-defined
+function calls.
+
+Reuse respects conditional evaluation, such as short-circuiting in `AND`, 
`OR`, and `CASE`.
+UDF authors should ensure that `isDeterministic()` correctly describes their 
function:
+functions with side effects or results that vary for identical inputs must 
return `false`.
+
+#### Built-in UDF metrics
+
+##### [FLINK-38071](https://issues.apache.org/jira/browse/FLINK-38071) 
(FLIP-485)
+
+SQL and Table API users can now monitor UDF execution through two built-in 
metrics:
+`udfProcessingTime`, a histogram of sampled invocation durations, and 
`udfExceptionCount`,
+a counter of exceptions from the function. Metrics are registered on the 
executing operator
+under `udf.<udf_name>`. For asynchronous functions, processing time includes 
the time from
+dispatch until completion.
+
+The feature is opt-in:
+
+- `table.exec.udf-metric-enabled` (default: `false`) enables metric collection.
+- `table.exec.udf-metric.sample-interval` (default: `100`) controls how often 
invocation
+  durations are measured. Exceptions are counted on every invocation.
+
+The metrics cover directly invoked synchronous and asynchronous scalar and 
table functions.
+Python functions, aggregate functions, and Process Table Functions are not 
covered. Enabling
+the metrics adds overhead; use sampling to control the cost.
+
+#### Early firing for interval joins
+
+##### [FLINK-36953](https://issues.apache.org/jira/browse/FLINK-36953) 
(FLIP-497)
+
+Outer interval joins can emit unmatched rows before their join interval 
closes. The new
+`EARLY_FIRE` hint emits a speculative, null-padded result after a configured 
delay. If a
+matching row arrives later within the interval, Flink corrects the speculative 
result with
+update-before and update-after records.
+
+```sql
+SELECT /*+ EARLY_FIRE('delay'='5s') */ o.id, s.ship_time
+FROM Orders o
+LEFT OUTER JOIN Shipments s
+ON o.id = s.order_id
+AND o.order_time BETWEEN s.ship_time - INTERVAL '1' HOUR AND s.ship_time;
+```
+
+The optional `time-mode` selects `rowtime` or `proctime`; it defaults to the 
time mode of the
+interval join. Each unmatched outer row fires at most once. The feature is 
disabled unless
+the hint is present and requires downstream operators and sinks that accept 
updates.
+Existing interval joins without the hint retain their behavior.

Review Comment:
   On `release-2.4` today, a batch query with `EARLY_FIRE` fails planning with 
`Unknown join strategy : EARLY_FIRE` 
([FLINK-41446](https://issues.apache.org/jira/browse/FLINK-41446)). My fix, 
#29908, makes batch ignore the hint; it's waiting for review and I'm aiming to 
get it into 2.4.0. If it lands, "In batch mode the hint is ignored." would fit 
after this line; if it misses, this is the place for a known-issue note. I'll 
post the exact sentence here once it's settled.



##########
docs/content/release-notes/flink-2.4.md:
##########
@@ -0,0 +1,359 @@
+---
+title: "Release Notes - Flink 2.4"
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Release notes - Flink 2.4
+
+These release notes discuss important aspects, such as configuration, behavior 
or dependencies,

Review Comment:
   Flink 2.4 moves from SLF4J 1.7 to SLF4J 2.0 (FLINK-40574). Users who still 
use Logback 1.2, or who bundle slf4j-api 1.x in their jars, may get no logs 
after upgrading. Flink 2.4 also stops reading some old config keys, such as 
`kubernetes.container.image`, `yarn.ship-directories` and `akka.client.timeout` 
(FLINK-40734). Should we mention both here, so users know before they upgrade?



##########
docs/content/release-notes/flink-2.4.md:
##########
@@ -0,0 +1,359 @@
+---
+title: "Release Notes - Flink 2.4"
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Release notes - Flink 2.4
+
+These release notes discuss important aspects, such as configuration, behavior 
or dependencies,
+that changed between Flink 2.3 and Flink 2.4. Please read these notes 
carefully if you are
+planning to upgrade your Flink version to 2.4.
+
+### Table SQL / API
+
+#### Upgrade Calcite to 1.41.0
+
+##### [FLINK-39859](https://issues.apache.org/jira/browse/FLINK-39859)
+
+Flink 2.4 upgrades Apache Calcite, the foundation of the SQL parser and 
optimizer, to version
+1.41.0. This incorporates the fixes and improvements from the intervening 
Calcite releases.
+Applications that depend directly on Calcite classes or extend Flink's planner 
should check
+their dependencies and custom planner code against the upgraded version.
+
+#### Common subexpression elimination in Calc code generation
+
+##### [FLINK-39268](https://issues.apache.org/jira/browse/FLINK-39268)
+
+Flink 2.4 avoids redundant evaluation of common subexpressions within a Calc 
operator, which
+combines filtering and projection. Repeated deterministic expressions can 
reuse an already
+computed result instead of being evaluated again for the same input row. This 
applies to both
+batch and streaming queries and can reduce the cost of repeated built-in or 
user-defined
+function calls.
+
+Reuse respects conditional evaluation, such as short-circuiting in `AND`, 
`OR`, and `CASE`.
+UDF authors should ensure that `isDeterministic()` correctly describes their 
function:
+functions with side effects or results that vary for identical inputs must 
return `false`.
+
+#### Built-in UDF metrics
+
+##### [FLINK-38071](https://issues.apache.org/jira/browse/FLINK-38071) 
(FLIP-485)
+
+SQL and Table API users can now monitor UDF execution through two built-in 
metrics:
+`udfProcessingTime`, a histogram of sampled invocation durations, and 
`udfExceptionCount`,
+a counter of exceptions from the function. Metrics are registered on the 
executing operator
+under `udf.<udf_name>`. For asynchronous functions, processing time includes 
the time from
+dispatch until completion.
+
+The feature is opt-in:
+
+- `table.exec.udf-metric-enabled` (default: `false`) enables metric collection.
+- `table.exec.udf-metric.sample-interval` (default: `100`) controls how often 
invocation
+  durations are measured. Exceptions are counted on every invocation.
+
+The metrics cover directly invoked synchronous and asynchronous scalar and 
table functions.
+Python functions, aggregate functions, and Process Table Functions are not 
covered. Enabling
+the metrics adds overhead; use sampling to control the cost.
+
+#### Early firing for interval joins
+
+##### [FLINK-36953](https://issues.apache.org/jira/browse/FLINK-36953) 
(FLIP-497)
+
+Outer interval joins can emit unmatched rows before their join interval 
closes. The new
+`EARLY_FIRE` hint emits a speculative, null-padded result after a configured 
delay. If a
+matching row arrives later within the interval, Flink corrects the speculative 
result with
+update-before and update-after records.
+
+```sql
+SELECT /*+ EARLY_FIRE('delay'='5s') */ o.id, s.ship_time
+FROM Orders o
+LEFT OUTER JOIN Shipments s
+ON o.id = s.order_id
+AND o.order_time BETWEEN s.ship_time - INTERVAL '1' HOUR AND s.ship_time;
+```
+
+The optional `time-mode` selects `rowtime` or `proctime`; it defaults to the 
time mode of the
+interval join. Each unmatched outer row fires at most once. The feature is 
disabled unless
+the hint is present and requires downstream operators and sinks that accept 
updates.
+Existing interval joins without the hint retain their behavior.
+
+#### FROM_CHANGELOG and TO_CHANGELOG enhancements
+
+##### [FLINK-39258](https://issues.apache.org/jira/browse/FLINK-39258) 
(FLIP-564)
+
+Flink 2.4 adds partitioned processing and additional controls to the changelog 
conversion
+PTFs introduced in Flink 2.3:
+
+- `PARTITION BY` support to co-locate related changes, with corresponding 
convenience methods
+  on partitioned tables in the Table API.
+- Partitioned `FROM_CHANGELOG` calls can use the existing PTF `ORDER BY` 
support to reorder
+  CDC events per key using a watermarked time attribute. Late events are 
dropped by the
+  sorting operation.
+- `error_handling => 'FAIL' | 'SKIP'` for `FROM_CHANGELOG` to control how NULL 
or unmapped
+  operation codes are handled. The default is `FAIL`.
+- `produces_full_deletes` for `TO_CHANGELOG` to choose between full-row 
deletes and deletes
+  containing only identifying keys. The default is `true`.
+
+These additions make it easier to consume custom CDC streams and write 
changelog events to
+append-only sinks. Users must still provide correctly ordered, complete change 
events for
+the selected changelog mode; the functions do not reconstruct arbitrary 
missing before images.
+See the [Changelog Conversion 
documentation](https://nightlies.apache.org/flink/flink-docs-release-2.4/docs/sql/reference/queries/changelog/)
+for supported mappings and input requirements.
+
+#### Process Table Function state access and broadcast support
+
+##### [FLINK-39254](https://issues.apache.org/jira/browse/FLINK-39254) 
(FLIP-565)
+
+Flink 2.4 adds lazy value-state access and broadcast inputs to Process Table 
Functions:
+
+- `ValueView` provides lazy value-state access alongside `ListView` and 
`MapView`. A function
+  can read or update state only when needed, avoiding unnecessary 
serialization work.
+- `BROADCAST_SEMANTIC_TABLE` lets a PTF receive a rule or control table on 
every parallel
+  instance. State declared with `@StateHint(StateKind.BROADCAST)` is writable 
from the
+  broadcast input and readable while processing the regular inputs.
+- `NOTIFY_STATEFUL_SETS` lets a broadcast update trigger re-evaluation of 
existing stateful
+  partitions. This is opt-in because visiting all stateful partitions can be 
expensive.
+
+#### Process Table Function test harness
+
+##### [FLINK-39357](https://issues.apache.org/jira/browse/FLINK-39357) 
(FLIP-567)
+
+The new `ProcessTableFunctionTestHarness` lets developers test PTF business 
logic without
+starting a Flink cluster. Tests can configure scalar and table arguments, 
partitioning, and
+initial state, feed rows into individual inputs, and inspect the resulting 
output.
+
+The harness also supports advancing watermarks and the system clock, 
inspecting timers,

Review Comment:
   I think the 2.4 test harness does not support the system clock, state TTL, 
or changelog handling yet. The PTF docs (`ptfs.md`) list TTL as not supported, 
and changelog support is planned for 2.5 (FLINK-40577). Could we remove those 
three from this sentence?



##########
docs/content/release-notes/flink-2.4.md:
##########
@@ -0,0 +1,359 @@
+---
+title: "Release Notes - Flink 2.4"
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Release notes - Flink 2.4
+
+These release notes discuss important aspects, such as configuration, behavior 
or dependencies,
+that changed between Flink 2.3 and Flink 2.4. Please read these notes 
carefully if you are
+planning to upgrade your Flink version to 2.4.
+
+### Table SQL / API
+
+#### Upgrade Calcite to 1.41.0
+
+##### [FLINK-39859](https://issues.apache.org/jira/browse/FLINK-39859)
+
+Flink 2.4 upgrades Apache Calcite, the foundation of the SQL parser and 
optimizer, to version
+1.41.0. This incorporates the fixes and improvements from the intervening 
Calcite releases.
+Applications that depend directly on Calcite classes or extend Flink's planner 
should check
+their dependencies and custom planner code against the upgraded version.
+
+#### Common subexpression elimination in Calc code generation
+
+##### [FLINK-39268](https://issues.apache.org/jira/browse/FLINK-39268)
+
+Flink 2.4 avoids redundant evaluation of common subexpressions within a Calc 
operator, which
+combines filtering and projection. Repeated deterministic expressions can 
reuse an already
+computed result instead of being evaluated again for the same input row. This 
applies to both
+batch and streaming queries and can reduce the cost of repeated built-in or 
user-defined
+function calls.
+
+Reuse respects conditional evaluation, such as short-circuiting in `AND`, 
`OR`, and `CASE`.
+UDF authors should ensure that `isDeterministic()` correctly describes their 
function:
+functions with side effects or results that vary for identical inputs must 
return `false`.
+
+#### Built-in UDF metrics
+
+##### [FLINK-38071](https://issues.apache.org/jira/browse/FLINK-38071) 
(FLIP-485)
+
+SQL and Table API users can now monitor UDF execution through two built-in 
metrics:
+`udfProcessingTime`, a histogram of sampled invocation durations, and 
`udfExceptionCount`,
+a counter of exceptions from the function. Metrics are registered on the 
executing operator
+under `udf.<udf_name>`. For asynchronous functions, processing time includes 
the time from
+dispatch until completion.
+
+The feature is opt-in:
+
+- `table.exec.udf-metric-enabled` (default: `false`) enables metric collection.
+- `table.exec.udf-metric.sample-interval` (default: `100`) controls how often 
invocation
+  durations are measured. Exceptions are counted on every invocation.
+
+The metrics cover directly invoked synchronous and asynchronous scalar and 
table functions.
+Python functions, aggregate functions, and Process Table Functions are not 
covered. Enabling
+the metrics adds overhead; use sampling to control the cost.
+
+#### Early firing for interval joins
+
+##### [FLINK-36953](https://issues.apache.org/jira/browse/FLINK-36953) 
(FLIP-497)
+
+Outer interval joins can emit unmatched rows before their join interval 
closes. The new
+`EARLY_FIRE` hint emits a speculative, null-padded result after a configured 
delay. If a
+matching row arrives later within the interval, Flink corrects the speculative 
result with
+update-before and update-after records.
+
+```sql
+SELECT /*+ EARLY_FIRE('delay'='5s') */ o.id, s.ship_time
+FROM Orders o
+LEFT OUTER JOIN Shipments s
+ON o.id = s.order_id
+AND o.order_time BETWEEN s.ship_time - INTERVAL '1' HOUR AND s.ship_time;
+```
+
+The optional `time-mode` selects `rowtime` or `proctime`; it defaults to the 
time mode of the
+interval join. Each unmatched outer row fires at most once. The feature is 
disabled unless
+the hint is present and requires downstream operators and sinks that accept 
updates.
+Existing interval joins without the hint retain their behavior.
+
+#### FROM_CHANGELOG and TO_CHANGELOG enhancements
+
+##### [FLINK-39258](https://issues.apache.org/jira/browse/FLINK-39258) 
(FLIP-564)
+
+Flink 2.4 adds partitioned processing and additional controls to the changelog 
conversion
+PTFs introduced in Flink 2.3:
+
+- `PARTITION BY` support to co-locate related changes, with corresponding 
convenience methods
+  on partitioned tables in the Table API.
+- Partitioned `FROM_CHANGELOG` calls can use the existing PTF `ORDER BY` 
support to reorder
+  CDC events per key using a watermarked time attribute. Late events are 
dropped by the
+  sorting operation.
+- `error_handling => 'FAIL' | 'SKIP'` for `FROM_CHANGELOG` to control how NULL 
or unmapped
+  operation codes are handled. The default is `FAIL`.
+- `produces_full_deletes` for `TO_CHANGELOG` to choose between full-row 
deletes and deletes
+  containing only identifying keys. The default is `true`.
+
+These additions make it easier to consume custom CDC streams and write 
changelog events to
+append-only sinks. Users must still provide correctly ordered, complete change 
events for
+the selected changelog mode; the functions do not reconstruct arbitrary 
missing before images.
+See the [Changelog Conversion 
documentation](https://nightlies.apache.org/flink/flink-docs-release-2.4/docs/sql/reference/queries/changelog/)
+for supported mappings and input requirements.
+
+#### Process Table Function state access and broadcast support
+
+##### [FLINK-39254](https://issues.apache.org/jira/browse/FLINK-39254) 
(FLIP-565)
+
+Flink 2.4 adds lazy value-state access and broadcast inputs to Process Table 
Functions:
+
+- `ValueView` provides lazy value-state access alongside `ListView` and 
`MapView`. A function
+  can read or update state only when needed, avoiding unnecessary 
serialization work.
+- `BROADCAST_SEMANTIC_TABLE` lets a PTF receive a rule or control table on 
every parallel
+  instance. State declared with `@StateHint(StateKind.BROADCAST)` is writable 
from the
+  broadcast input and readable while processing the regular inputs.
+- `NOTIFY_STATEFUL_SETS` lets a broadcast update trigger re-evaluation of 
existing stateful
+  partitions. This is opt-in because visiting all stateful partitions can be 
expensive.
+
+#### Process Table Function test harness
+
+##### [FLINK-39357](https://issues.apache.org/jira/browse/FLINK-39357) 
(FLIP-567)
+
+The new `ProcessTableFunctionTestHarness` lets developers test PTF business 
logic without
+starting a Flink cluster. Tests can configure scalar and table arguments, 
partitioning, and
+initial state, feed rows into individual inputs, and inspect the resulting 
output.
+
+The harness also supports advancing watermarks and the system clock, 
inspecting timers,
+accessing state per key, and testing state TTL and changelog handling. This 
makes temporal and
+stateful behavior easier to test deterministically. Full planner and runtime 
integration
+should still be covered by integration tests.
+
+#### Strict BYTES-to-STRING casts and UTF-8 utilities
+
+##### [FLINK-39600](https://issues.apache.org/jira/browse/FLINK-39600) 
(FLIP-568)
+
+Casting binary data to a string now validates UTF-8. Previously, malformed 
byte sequences
+were silently replaced with the Unicode replacement character `U+FFFD`; in 
Flink 2.4,
+`CAST(bytes AS STRING)` fails on invalid UTF-8 and `TRY_CAST(bytes AS STRING)` 
returns NULL.
+This is a behavior change for pipelines that previously accepted malformed 
input.
+
+Two new functions provide explicit handling:
+
+- `IS_VALID_UTF8(bytes)` returns whether the input is valid UTF-8, for 
filtering or routing
+  malformed records.
+- `MAKE_VALID_UTF8(bytes)` decodes the input and replaces invalid sequences 
with `U+FFFD`,
+  preserving the previous substitution behavior.
+
+Both functions return NULL for NULL input. As a migration option,
+`table.exec.legacy-bytes-to-string-cast` (default: `false`) can be set to 
`true` to restore
+the previous cast behavior.
+
+Connector authors can use the new `StringData.fromUtf8Bytes(byte[])` method 
for strict
+validation at ingestion. The existing `StringData.fromBytes(byte[])` remains 
unchanged.
+
+#### Metadata filter push-down for table sources
+
+##### [FLINK-39421](https://issues.apache.org/jira/browse/FLINK-39421) 
(FLIP-574)
+
+Source connectors can now accept filter predicates on metadata columns, such 
as offsets,
+partitions, or timestamps. This lets a supporting connector reduce the data it 
reads instead
+of applying all such filters after ingestion.
+
+Connectors opt in through the new `supportsMetadataFilterPushDown()` and
+`applyMetadataFilters()` methods on `SupportsReadingMetadata`. Predicates use 
metadata keys
+rather than SQL column aliases, and filters that the connector cannot fully 
apply remain in
+the query. Metadata predicates are also preserved when compiled plans are 
restored.
+Existing connectors retain their behavior unless they implement the new 
capability.
+
+#### In-place conversion of tables to materialized tables
+
+##### [FLINK-39847](https://issues.apache.org/jira/browse/FLINK-39847) 
(FLIP-578)
+
+`CREATE OR ALTER MATERIALIZED TABLE` can convert an existing regular table 
into a materialized
+table without dropping and recreating the catalog object. The conversion 
preserves the table's
+identity and underlying storage and starts a refresh job for the defining 
query.
+
+Conversion is disabled by default. Enable it when initializing the session 
with:

Review Comment:
   I don't think `SET` works for this option. Flink reads it only when the 
session starts, for example from `config.yaml`, so a later `SET` statement has 
no effect. Could we tell users to put it in `config.yaml` instead?



##########
docs/content/release-notes/flink-2.4.md:
##########
@@ -0,0 +1,359 @@
+---
+title: "Release Notes - Flink 2.4"
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Release notes - Flink 2.4
+
+These release notes discuss important aspects, such as configuration, behavior 
or dependencies,
+that changed between Flink 2.3 and Flink 2.4. Please read these notes 
carefully if you are
+planning to upgrade your Flink version to 2.4.
+
+### Table SQL / API
+
+#### Upgrade Calcite to 1.41.0
+
+##### [FLINK-39859](https://issues.apache.org/jira/browse/FLINK-39859)
+
+Flink 2.4 upgrades Apache Calcite, the foundation of the SQL parser and 
optimizer, to version
+1.41.0. This incorporates the fixes and improvements from the intervening 
Calcite releases.
+Applications that depend directly on Calcite classes or extend Flink's planner 
should check
+their dependencies and custom planner code against the upgraded version.
+
+#### Common subexpression elimination in Calc code generation
+
+##### [FLINK-39268](https://issues.apache.org/jira/browse/FLINK-39268)
+
+Flink 2.4 avoids redundant evaluation of common subexpressions within a Calc 
operator, which
+combines filtering and projection. Repeated deterministic expressions can 
reuse an already
+computed result instead of being evaluated again for the same input row. This 
applies to both
+batch and streaming queries and can reduce the cost of repeated built-in or 
user-defined
+function calls.
+
+Reuse respects conditional evaluation, such as short-circuiting in `AND`, 
`OR`, and `CASE`.
+UDF authors should ensure that `isDeterministic()` correctly describes their 
function:
+functions with side effects or results that vary for identical inputs must 
return `false`.
+
+#### Built-in UDF metrics
+
+##### [FLINK-38071](https://issues.apache.org/jira/browse/FLINK-38071) 
(FLIP-485)
+
+SQL and Table API users can now monitor UDF execution through two built-in 
metrics:
+`udfProcessingTime`, a histogram of sampled invocation durations, and 
`udfExceptionCount`,
+a counter of exceptions from the function. Metrics are registered on the 
executing operator
+under `udf.<udf_name>`. For asynchronous functions, processing time includes 
the time from
+dispatch until completion.
+
+The feature is opt-in:
+
+- `table.exec.udf-metric-enabled` (default: `false`) enables metric collection.
+- `table.exec.udf-metric.sample-interval` (default: `100`) controls how often 
invocation
+  durations are measured. Exceptions are counted on every invocation.
+
+The metrics cover directly invoked synchronous and asynchronous scalar and 
table functions.
+Python functions, aggregate functions, and Process Table Functions are not 
covered. Enabling
+the metrics adds overhead; use sampling to control the cost.
+
+#### Early firing for interval joins
+
+##### [FLINK-36953](https://issues.apache.org/jira/browse/FLINK-36953) 
(FLIP-497)
+
+Outer interval joins can emit unmatched rows before their join interval 
closes. The new
+`EARLY_FIRE` hint emits a speculative, null-padded result after a configured 
delay. If a
+matching row arrives later within the interval, Flink corrects the speculative 
result with
+update-before and update-after records.
+
+```sql
+SELECT /*+ EARLY_FIRE('delay'='5s') */ o.id, s.ship_time
+FROM Orders o
+LEFT OUTER JOIN Shipments s
+ON o.id = s.order_id
+AND o.order_time BETWEEN s.ship_time - INTERVAL '1' HOUR AND s.ship_time;
+```
+
+The optional `time-mode` selects `rowtime` or `proctime`; it defaults to the 
time mode of the
+interval join. Each unmatched outer row fires at most once. The feature is 
disabled unless
+the hint is present and requires downstream operators and sinks that accept 
updates.
+Existing interval joins without the hint retain their behavior.
+
+#### FROM_CHANGELOG and TO_CHANGELOG enhancements
+
+##### [FLINK-39258](https://issues.apache.org/jira/browse/FLINK-39258) 
(FLIP-564)
+
+Flink 2.4 adds partitioned processing and additional controls to the changelog 
conversion
+PTFs introduced in Flink 2.3:
+
+- `PARTITION BY` support to co-locate related changes, with corresponding 
convenience methods
+  on partitioned tables in the Table API.
+- Partitioned `FROM_CHANGELOG` calls can use the existing PTF `ORDER BY` 
support to reorder
+  CDC events per key using a watermarked time attribute. Late events are 
dropped by the
+  sorting operation.
+- `error_handling => 'FAIL' | 'SKIP'` for `FROM_CHANGELOG` to control how NULL 
or unmapped
+  operation codes are handled. The default is `FAIL`.
+- `produces_full_deletes` for `TO_CHANGELOG` to choose between full-row 
deletes and deletes
+  containing only identifying keys. The default is `true`.
+
+These additions make it easier to consume custom CDC streams and write 
changelog events to
+append-only sinks. Users must still provide correctly ordered, complete change 
events for
+the selected changelog mode; the functions do not reconstruct arbitrary 
missing before images.
+See the [Changelog Conversion 
documentation](https://nightlies.apache.org/flink/flink-docs-release-2.4/docs/sql/reference/queries/changelog/)
+for supported mappings and input requirements.
+
+#### Process Table Function state access and broadcast support
+
+##### [FLINK-39254](https://issues.apache.org/jira/browse/FLINK-39254) 
(FLIP-565)
+
+Flink 2.4 adds lazy value-state access and broadcast inputs to Process Table 
Functions:
+
+- `ValueView` provides lazy value-state access alongside `ListView` and 
`MapView`. A function
+  can read or update state only when needed, avoiding unnecessary 
serialization work.
+- `BROADCAST_SEMANTIC_TABLE` lets a PTF receive a rule or control table on 
every parallel
+  instance. State declared with `@StateHint(StateKind.BROADCAST)` is writable 
from the
+  broadcast input and readable while processing the regular inputs.
+- `NOTIFY_STATEFUL_SETS` lets a broadcast update trigger re-evaluation of 
existing stateful
+  partitions. This is opt-in because visiting all stateful partitions can be 
expensive.
+
+#### Process Table Function test harness
+
+##### [FLINK-39357](https://issues.apache.org/jira/browse/FLINK-39357) 
(FLIP-567)
+
+The new `ProcessTableFunctionTestHarness` lets developers test PTF business 
logic without
+starting a Flink cluster. Tests can configure scalar and table arguments, 
partitioning, and
+initial state, feed rows into individual inputs, and inspect the resulting 
output.
+
+The harness also supports advancing watermarks and the system clock, 
inspecting timers,
+accessing state per key, and testing state TTL and changelog handling. This 
makes temporal and
+stateful behavior easier to test deterministically. Full planner and runtime 
integration
+should still be covered by integration tests.
+
+#### Strict BYTES-to-STRING casts and UTF-8 utilities
+
+##### [FLINK-39600](https://issues.apache.org/jira/browse/FLINK-39600) 
(FLIP-568)
+
+Casting binary data to a string now validates UTF-8. Previously, malformed 
byte sequences
+were silently replaced with the Unicode replacement character `U+FFFD`; in 
Flink 2.4,
+`CAST(bytes AS STRING)` fails on invalid UTF-8 and `TRY_CAST(bytes AS STRING)` 
returns NULL.
+This is a behavior change for pipelines that previously accepted malformed 
input.
+
+Two new functions provide explicit handling:
+
+- `IS_VALID_UTF8(bytes)` returns whether the input is valid UTF-8, for 
filtering or routing
+  malformed records.
+- `MAKE_VALID_UTF8(bytes)` decodes the input and replaces invalid sequences 
with `U+FFFD`,
+  preserving the previous substitution behavior.
+
+Both functions return NULL for NULL input. As a migration option,
+`table.exec.legacy-bytes-to-string-cast` (default: `false`) can be set to 
`true` to restore
+the previous cast behavior.
+
+Connector authors can use the new `StringData.fromUtf8Bytes(byte[])` method 
for strict
+validation at ingestion. The existing `StringData.fromBytes(byte[])` remains 
unchanged.
+
+#### Metadata filter push-down for table sources
+
+##### [FLINK-39421](https://issues.apache.org/jira/browse/FLINK-39421) 
(FLIP-574)
+
+Source connectors can now accept filter predicates on metadata columns, such 
as offsets,
+partitions, or timestamps. This lets a supporting connector reduce the data it 
reads instead
+of applying all such filters after ingestion.
+
+Connectors opt in through the new `supportsMetadataFilterPushDown()` and
+`applyMetadataFilters()` methods on `SupportsReadingMetadata`. Predicates use 
metadata keys
+rather than SQL column aliases, and filters that the connector cannot fully 
apply remain in
+the query. Metadata predicates are also preserved when compiled plans are 
restored.
+Existing connectors retain their behavior unless they implement the new 
capability.
+
+#### In-place conversion of tables to materialized tables
+
+##### [FLINK-39847](https://issues.apache.org/jira/browse/FLINK-39847) 
(FLIP-578)
+
+`CREATE OR ALTER MATERIALIZED TABLE` can convert an existing regular table 
into a materialized
+table without dropping and recreating the catalog object. The conversion 
preserves the table's
+identity and underlying storage and starts a refresh job for the defining 
query.
+
+Conversion is disabled by default. Enable it when initializing the session 
with:
+
+```yaml
+table.materialized-table.conversion-from-table.enabled: true
+```
+
+The catalog must support the new `convertTableToMaterializedTable` operation. 
The conversion
+statement defines the resulting schema and options; existing primary keys, 
watermarks, and
+other definitions must be declared again where needed. This is a one-way 
conversion: there is
+no corresponding operation to turn a materialized table back into a regular 
table.
+
+#### LATERAL SNAPSHOT joins
+
+##### [FLINK-39779](https://issues.apache.org/jira/browse/FLINK-39779) 
(FLIP-579)
+
+Flink SQL gains `LATERAL SNAPSHOT` joins for enriching an append-only stream 
with the current
+contents of an updating table. Unlike an event-time temporal join, the join 
does not wait for
+both inputs' watermarks for every result. It first buffers probe-side rows 
while initializing
+the build side, then joins arriving rows immediately against the current 
build-side state.
+
+`INNER` and `LEFT OUTER` joins are supported, and the build side does not need 
a primary key.
+Streaming queries specify a watermarked build-side column through `on_time`.
+`load_completed_time` controls when initialization completes; an optional
+`load_completed_idle_timeout` provides a fallback when build-side watermarks 
stop advancing.
+For static or infrequently updated build sides, configure this timeout or use 
a bounded source
+so the join does not remain in its initialization phase.
+
+Results follow processing-time semantics and can differ after recovery. Users 
requiring an
+exact event-time version should continue to use event-time temporal joins.
+See the [LATERAL SNAPSHOT join 
documentation](https://nightlies.apache.org/flink/flink-docs-release-2.4/docs/sql/reference/queries/joins/#lateral-snapshot-join)
+for the full syntax and initialization behavior.
+
+#### UUID type and extended VARIANT primitives
+
+##### [FLINK-40485](https://issues.apache.org/jira/browse/FLINK-40485) 
(FLIP-604)
+
+Flink 2.4 introduces a native `UUID` type backed by a 16-byte value, with 
explicit casts to
+and from strings and binary data, comparison and ordering support, and typed 
`UUID_V4()` and
+`UUID_V7()` generators. The existing `UUID()` function continues to return a 
string.
+
+The `VARIANT` type also supports time-of-day values, nanosecond-precision 
timestamps with and
+without local time zone, and UUID values. This extends compatibility with the 
corresponding
+Parquet Variant primitive encodings. Nanosecond timestamps can be read as 
`TIMESTAMP(9)` or
+`TIMESTAMP_LTZ(9)`; converting a VARIANT time-of-day value to SQL `TIME(3)` 
truncates any
+sub-millisecond precision.
+
+Existing VARIANT values remain readable. Older Flink versions cannot read 
values containing
+the newly supported primitive codes, so consider this limitation before 
downgrading a job or
+sharing persisted VARIANT data with older consumers.
+
+### Python
+
+#### Python DataFrame API
+
+##### [FLINK-40185](https://issues.apache.org/jira/browse/FLINK-40185) 
(FLIP-591)
+
+PyFlink gains a DataFrame API under `pyflink.dataframe`. It offers a 
Python-oriented entry
+point for creating Flink tables from records, dictionaries, pandas DataFrames, 
or Arrow tables,
+and for reading and writing through Flink connectors.
+
+The API supports familiar transformations such as selection, filtering, joins, 
aggregation,
+windowing, and missing-value handling, as well as Python UDFs and batch 
transformations.
+`from_table()` and `to_table()` allow applications to move between DataFrames 
and the existing
+Table API.
+
+DataFrame operations use Flink's existing planner and runtime for batch and 
streaming
+execution. The API complements the Table API; it does not replace it or 
promise full
+compatibility with pandas or other DataFrame libraries.
+
+### Connectors
+
+#### Job information on source contexts
+
+##### [FLINK-39776](https://issues.apache.org/jira/browse/FLINK-39776) 
(FLIP-583)
+
+`SourceReaderContext` and `SplitEnumeratorContext` now expose `getJobInfo()`, 
giving source
+implementations direct access to the job ID and job name. This aligns the 
Source API with
+Sink V2 and supports use cases such as per-job authentication, external offset 
tracking,
+and log correlation without parsing metric scope variables.
+
+The new methods have default implementations, so existing context 
implementations remain
+source- and binary-compatible. Flink's runtime contexts provide the job 
information;
+third-party contexts and test stubs must override the methods if callers use 
them, as the
+default implementation throws `UnsupportedOperationException`.
+
+### Runtime
+
+#### Filtering and spilling recovered channel state
+
+##### [FLINK-38544](https://issues.apache.org/jira/browse/FLINK-38544) 
(FLIP-547 V2)
+
+Flink 2.4 extends checkpointing during recovery with a spill-based recovery 
path. Recovered
+channel state is filtered for the receiving subtask before records enter 
normal processing,
+and the resulting data is written to local spill files before being replayed 
through network
+buffers. This bounds memory usage when recovering large amounts of in-flight 
data.
+
+Filtering before processing also prevents records that belong to other 
subtasks after
+rescaling from being included again in subsequent recovery checkpoints. 
Unaligned checkpoints
+capture both queued and unread spilled data, allowing recovery progress to 
survive another
+failure or rescaling without duplicating the recovered records.
+
+The spill path is used when checkpointing during recovery is enabled through 
the existing
+configuration options and requires sufficient local disk space. Flink 2.4 also 
disables
+checkpointing during recovery when 
`execution.checkpointing.aligned-checkpoint-timeout` is
+greater than zero.
+
+#### Pluggable HistoryServer archive storage
+
+##### [FLINK-39911](https://issues.apache.org/jira/browse/FLINK-39911) 
(FLIP-584)
+
+The HistoryServer can store its local archive cache in an embedded RocksDB 
database instead
+of expanding every archive into many small JSON files. This reduces inode 
consumption when
+serving large numbers of jobs or jobs with extensive task-level details.
+
+Select the backend with `historyserver.archive.storage.type`:
+
+- `FILE` (default) keeps the existing file-based storage behavior.
+- `ROCKSDB` stores archive entries in an embedded database under
+  `historyserver.web.tmpdir/rocksdb`.
+
+This changes how the local cache is stored; completed-job archives continue to 
be read from
+the configured archive directories.
+
+#### Lazy loading in HistoryServer
+
+##### [FLINK-40097](https://issues.apache.org/jira/browse/FLINK-40097) 
(FLIP-585)
+
+The HistoryServer gains a `LAZY` archive load mode. Overview information is 
made available
+before all job details have been processed, while details are loaded 
asynchronously.
+Accessing an archive that has not yet finished loading triggers or waits for 
the relevant
+on-demand load, so users do not have to wait for every preceding archive to be 
processed.
+
+The mode is opt-in through `historyserver.archive.load.mode` (default: 
`EAGER`). Lazy loading
+uses two separately configurable thread pools:
+
+- `historyserver.lazy.fetch.executor.common.pool-size` (default: `4`) for 
background loads.
+- `historyserver.lazy.fetch.executor.individual.pool-size` (default: `4`) for 
individual
+  on-demand loads.
+
+The default `EAGER` mode retains synchronous archive loading. Lazy loading 
works with both
+the file and RocksDB local storage backends.
+
+#### State Catalog
+
+##### [FLINK-40175](https://issues.apache.org/jira/browse/FLINK-40175) 
(FLIP-599)
+
+The new read-only State Catalog discovers checkpoints and savepoints in 
configured
+directories and exposes their contents through Flink SQL. Each discovered 
snapshot becomes
+a database, with tables for operator state and views for inspecting snapshot 
metadata.
+This removes much of the manual schema definition previously needed to query 
saved state.
+
+The catalog infers schemas from supported serializer snapshots and exposes 
keyed state,
+flattened list and map state, non-keyed list/union/broadcast state, and window 
state.
+Supported POJO and Avro state can be read without the original user classes on 
the classpath;
+custom serialization formats still require compatible type information and 
reader support.
+
+Register the catalog using one or more labeled directories:
+
+```sql
+CREATE CATALOG saved_state WITH (
+  'type' = 'state',
+  'directory.prod' = 's3://my-bucket/flink/checkpoints'
+);
+```
+
+The catalog is provided by `flink-state-processing-api`. Snapshot metadata 
views additionally

Review Comment:
   I think the Maven artifact is `flink-state-processor-api`. 
`flink-state-processing-api` is only the folder name in the repo. The jar is 
also in the `opt/` folder of the distribution, not `lib/`, so users need to add 
it to the classpath first. Should we say that here?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to