weiqingy commented on code in PR #29957: URL: https://github.com/apache/flink/pull/29957#discussion_r4638436019
########## docs/content/release-notes/flink-2.4.md: ########## @@ -0,0 +1,359 @@ +--- +title: "Release Notes - Flink 2.4" +--- + +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Release notes - Flink 2.4 + +These release notes discuss important aspects, such as configuration, behavior or dependencies, Review Comment: Two changes can silently break an existing setup after the upgrade and aren't covered yet: - SLF4J moved from 1.7.36 to 2.0.20 with the `log4j-slf4j2-impl` binding ([FLINK-40574](https://issues.apache.org/jira/browse/FLINK-40574)). Its JIRA release note says Logback 1.2 or `slf4j-log4j12` setups, and jars bundling `slf4j-api` 1.x in `lib/` or on YARN, end up with no logs. - Deprecated config keys are no longer read (FLINK-40813, FLINK-40815, FLINK-40816, FLINK-40817): `kubernetes.container.image`, `kubernetes.jobmanager.cpu`, `kubernetes.taskmanager.cpu`, `kubernetes.pod-template-file`, `yarn.ship-directories`, `yarn.per-job-cluster.include-user-jar`, `yarn.heartbeat-delay`, `akka.client.timeout`, and the `dstl.dfs.*` keys (now `state.changelog.dstl.dfs.*`). `flink-2.1.md` and `flink-2.2.md` had a "Dependency upgrades" section for this kind of change. Could we add one for SLF4J and for PyFlink's `apache-beam` range moving from `>=2.54.0,<=2.61.0` to `>=2.69.0,<=2.75.0` ([FLINK-38469](https://issues.apache.org/jira/browse/FLINK-38469)), plus a short old-to-new table for the removed keys? ########## docs/content/release-notes/flink-2.4.md: ########## @@ -0,0 +1,359 @@ +--- +title: "Release Notes - Flink 2.4" +--- + +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Release notes - Flink 2.4 + +These release notes discuss important aspects, such as configuration, behavior or dependencies, +that changed between Flink 2.3 and Flink 2.4. Please read these notes carefully if you are +planning to upgrade your Flink version to 2.4. + +### Table SQL / API Review Comment: `table.exec.sink.require-on-conflict` now defaults to `false`, reversing the 2.3 default ([FLINK-41478](https://issues.apache.org/jira/browse/FLINK-41478)), and the ticket carries a release note. Worth a short entry in this section? ########## docs/content/release-notes/flink-2.4.md: ########## @@ -0,0 +1,359 @@ +--- +title: "Release Notes - Flink 2.4" +--- + +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Release notes - Flink 2.4 + +These release notes discuss important aspects, such as configuration, behavior or dependencies, +that changed between Flink 2.3 and Flink 2.4. Please read these notes carefully if you are +planning to upgrade your Flink version to 2.4. + +### Table SQL / API + +#### Upgrade Calcite to 1.41.0 + +##### [FLINK-39859](https://issues.apache.org/jira/browse/FLINK-39859) + +Flink 2.4 upgrades Apache Calcite, the foundation of the SQL parser and optimizer, to version +1.41.0. This incorporates the fixes and improvements from the intervening Calcite releases. +Applications that depend directly on Calcite classes or extend Flink's planner should check +their dependencies and custom planner code against the upgraded version. + +#### Common subexpression elimination in Calc code generation + +##### [FLINK-39268](https://issues.apache.org/jira/browse/FLINK-39268) + +Flink 2.4 avoids redundant evaluation of common subexpressions within a Calc operator, which +combines filtering and projection. Repeated deterministic expressions can reuse an already +computed result instead of being evaluated again for the same input row. This applies to both +batch and streaming queries and can reduce the cost of repeated built-in or user-defined +function calls. + +Reuse respects conditional evaluation, such as short-circuiting in `AND`, `OR`, and `CASE`. +UDF authors should ensure that `isDeterministic()` correctly describes their function: +functions with side effects or results that vary for identical inputs must return `false`. + +#### Built-in UDF metrics + +##### [FLINK-38071](https://issues.apache.org/jira/browse/FLINK-38071) (FLIP-485) + +SQL and Table API users can now monitor UDF execution through two built-in metrics: +`udfProcessingTime`, a histogram of sampled invocation durations, and `udfExceptionCount`, +a counter of exceptions from the function. Metrics are registered on the executing operator +under `udf.<udf_name>`. For asynchronous functions, processing time includes the time from +dispatch until completion. + +The feature is opt-in: + +- `table.exec.udf-metric-enabled` (default: `false`) enables metric collection. +- `table.exec.udf-metric.sample-interval` (default: `100`) controls how often invocation + durations are measured. Exceptions are counted on every invocation. + +The metrics cover directly invoked synchronous and asynchronous scalar and table functions. +Python functions, aggregate functions, and Process Table Functions are not covered. Enabling +the metrics adds overhead; use sampling to control the cost. + +#### Early firing for interval joins + +##### [FLINK-36953](https://issues.apache.org/jira/browse/FLINK-36953) (FLIP-497) + +Outer interval joins can emit unmatched rows before their join interval closes. The new +`EARLY_FIRE` hint emits a speculative, null-padded result after a configured delay. If a +matching row arrives later within the interval, Flink corrects the speculative result with +update-before and update-after records. + +```sql +SELECT /*+ EARLY_FIRE('delay'='5s') */ o.id, s.ship_time +FROM Orders o +LEFT OUTER JOIN Shipments s +ON o.id = s.order_id +AND o.order_time BETWEEN s.ship_time - INTERVAL '1' HOUR AND s.ship_time; +``` + +The optional `time-mode` selects `rowtime` or `proctime`; it defaults to the time mode of the +interval join. Each unmatched outer row fires at most once. The feature is disabled unless +the hint is present and requires downstream operators and sinks that accept updates. +Existing interval joins without the hint retain their behavior. Review Comment: On `release-2.4` today, a batch query with `EARLY_FIRE` fails planning with `Unknown join strategy : EARLY_FIRE` ([FLINK-41446](https://issues.apache.org/jira/browse/FLINK-41446)). My fix, #29908, makes batch ignore the hint; it's waiting for review and I'm aiming to get it into 2.4.0. If it lands, "In batch mode the hint is ignored." would fit after this line; if it misses, this is the place for a known-issue note. I'll post the exact sentence here once it's settled. ########## docs/content/release-notes/flink-2.4.md: ########## @@ -0,0 +1,359 @@ +--- +title: "Release Notes - Flink 2.4" +--- + +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Release notes - Flink 2.4 + +These release notes discuss important aspects, such as configuration, behavior or dependencies, +that changed between Flink 2.3 and Flink 2.4. Please read these notes carefully if you are +planning to upgrade your Flink version to 2.4. + +### Table SQL / API + +#### Upgrade Calcite to 1.41.0 + +##### [FLINK-39859](https://issues.apache.org/jira/browse/FLINK-39859) + +Flink 2.4 upgrades Apache Calcite, the foundation of the SQL parser and optimizer, to version +1.41.0. This incorporates the fixes and improvements from the intervening Calcite releases. +Applications that depend directly on Calcite classes or extend Flink's planner should check +their dependencies and custom planner code against the upgraded version. + +#### Common subexpression elimination in Calc code generation + +##### [FLINK-39268](https://issues.apache.org/jira/browse/FLINK-39268) + +Flink 2.4 avoids redundant evaluation of common subexpressions within a Calc operator, which +combines filtering and projection. Repeated deterministic expressions can reuse an already +computed result instead of being evaluated again for the same input row. This applies to both +batch and streaming queries and can reduce the cost of repeated built-in or user-defined +function calls. + +Reuse respects conditional evaluation, such as short-circuiting in `AND`, `OR`, and `CASE`. +UDF authors should ensure that `isDeterministic()` correctly describes their function: +functions with side effects or results that vary for identical inputs must return `false`. + +#### Built-in UDF metrics + +##### [FLINK-38071](https://issues.apache.org/jira/browse/FLINK-38071) (FLIP-485) + +SQL and Table API users can now monitor UDF execution through two built-in metrics: +`udfProcessingTime`, a histogram of sampled invocation durations, and `udfExceptionCount`, +a counter of exceptions from the function. Metrics are registered on the executing operator +under `udf.<udf_name>`. For asynchronous functions, processing time includes the time from +dispatch until completion. + +The feature is opt-in: + +- `table.exec.udf-metric-enabled` (default: `false`) enables metric collection. +- `table.exec.udf-metric.sample-interval` (default: `100`) controls how often invocation + durations are measured. Exceptions are counted on every invocation. + +The metrics cover directly invoked synchronous and asynchronous scalar and table functions. +Python functions, aggregate functions, and Process Table Functions are not covered. Enabling +the metrics adds overhead; use sampling to control the cost. + +#### Early firing for interval joins + +##### [FLINK-36953](https://issues.apache.org/jira/browse/FLINK-36953) (FLIP-497) + +Outer interval joins can emit unmatched rows before their join interval closes. The new +`EARLY_FIRE` hint emits a speculative, null-padded result after a configured delay. If a +matching row arrives later within the interval, Flink corrects the speculative result with +update-before and update-after records. + +```sql +SELECT /*+ EARLY_FIRE('delay'='5s') */ o.id, s.ship_time +FROM Orders o +LEFT OUTER JOIN Shipments s +ON o.id = s.order_id +AND o.order_time BETWEEN s.ship_time - INTERVAL '1' HOUR AND s.ship_time; +``` + +The optional `time-mode` selects `rowtime` or `proctime`; it defaults to the time mode of the +interval join. Each unmatched outer row fires at most once. The feature is disabled unless +the hint is present and requires downstream operators and sinks that accept updates. +Existing interval joins without the hint retain their behavior. + +#### FROM_CHANGELOG and TO_CHANGELOG enhancements + +##### [FLINK-39258](https://issues.apache.org/jira/browse/FLINK-39258) (FLIP-564) + +Flink 2.4 adds partitioned processing and additional controls to the changelog conversion +PTFs introduced in Flink 2.3: + +- `PARTITION BY` support to co-locate related changes, with corresponding convenience methods + on partitioned tables in the Table API. +- Partitioned `FROM_CHANGELOG` calls can use the existing PTF `ORDER BY` support to reorder + CDC events per key using a watermarked time attribute. Late events are dropped by the + sorting operation. +- `error_handling => 'FAIL' | 'SKIP'` for `FROM_CHANGELOG` to control how NULL or unmapped + operation codes are handled. The default is `FAIL`. +- `produces_full_deletes` for `TO_CHANGELOG` to choose between full-row deletes and deletes + containing only identifying keys. The default is `true`. + +These additions make it easier to consume custom CDC streams and write changelog events to +append-only sinks. Users must still provide correctly ordered, complete change events for +the selected changelog mode; the functions do not reconstruct arbitrary missing before images. +See the [Changelog Conversion documentation](https://nightlies.apache.org/flink/flink-docs-release-2.4/docs/sql/reference/queries/changelog/) +for supported mappings and input requirements. + +#### Process Table Function state access and broadcast support + +##### [FLINK-39254](https://issues.apache.org/jira/browse/FLINK-39254) (FLIP-565) + +Flink 2.4 adds lazy value-state access and broadcast inputs to Process Table Functions: + +- `ValueView` provides lazy value-state access alongside `ListView` and `MapView`. A function + can read or update state only when needed, avoiding unnecessary serialization work. +- `BROADCAST_SEMANTIC_TABLE` lets a PTF receive a rule or control table on every parallel + instance. State declared with `@StateHint(StateKind.BROADCAST)` is writable from the + broadcast input and readable while processing the regular inputs. +- `NOTIFY_STATEFUL_SETS` lets a broadcast update trigger re-evaluation of existing stateful + partitions. This is opt-in because visiting all stateful partitions can be expensive. + +#### Process Table Function test harness + +##### [FLINK-39357](https://issues.apache.org/jira/browse/FLINK-39357) (FLIP-567) + +The new `ProcessTableFunctionTestHarness` lets developers test PTF business logic without +starting a Flink cluster. Tests can configure scalar and table arguments, partitioning, and +initial state, feed rows into individual inputs, and inspect the resulting output. + +The harness also supports advancing watermarks and the system clock, inspecting timers, Review Comment: Three of these aren't in the 2.4 harness. The shipped `ptfs.md` lists state TTL and the update traits under "PTF Features Unsupported by the TestHarness", the harness context's `getChangelogMode()` always returns `insertOnly()` (changelog support is FLINK-40577, fixVersion 2.5.0), and there's no clock API: time only moves through `setWatermark()` / `setWatermarkForTable()`. Something like "The harness also supports advancing watermarks, inspecting timers, and reading or setting state per key." instead? ########## docs/content/release-notes/flink-2.4.md: ########## @@ -0,0 +1,359 @@ +--- +title: "Release Notes - Flink 2.4" +--- + +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Release notes - Flink 2.4 + +These release notes discuss important aspects, such as configuration, behavior or dependencies, +that changed between Flink 2.3 and Flink 2.4. Please read these notes carefully if you are +planning to upgrade your Flink version to 2.4. + +### Table SQL / API + +#### Upgrade Calcite to 1.41.0 + +##### [FLINK-39859](https://issues.apache.org/jira/browse/FLINK-39859) + +Flink 2.4 upgrades Apache Calcite, the foundation of the SQL parser and optimizer, to version +1.41.0. This incorporates the fixes and improvements from the intervening Calcite releases. +Applications that depend directly on Calcite classes or extend Flink's planner should check +their dependencies and custom planner code against the upgraded version. + +#### Common subexpression elimination in Calc code generation + +##### [FLINK-39268](https://issues.apache.org/jira/browse/FLINK-39268) + +Flink 2.4 avoids redundant evaluation of common subexpressions within a Calc operator, which +combines filtering and projection. Repeated deterministic expressions can reuse an already +computed result instead of being evaluated again for the same input row. This applies to both +batch and streaming queries and can reduce the cost of repeated built-in or user-defined +function calls. + +Reuse respects conditional evaluation, such as short-circuiting in `AND`, `OR`, and `CASE`. +UDF authors should ensure that `isDeterministic()` correctly describes their function: +functions with side effects or results that vary for identical inputs must return `false`. + +#### Built-in UDF metrics + +##### [FLINK-38071](https://issues.apache.org/jira/browse/FLINK-38071) (FLIP-485) + +SQL and Table API users can now monitor UDF execution through two built-in metrics: +`udfProcessingTime`, a histogram of sampled invocation durations, and `udfExceptionCount`, +a counter of exceptions from the function. Metrics are registered on the executing operator +under `udf.<udf_name>`. For asynchronous functions, processing time includes the time from +dispatch until completion. + +The feature is opt-in: + +- `table.exec.udf-metric-enabled` (default: `false`) enables metric collection. +- `table.exec.udf-metric.sample-interval` (default: `100`) controls how often invocation + durations are measured. Exceptions are counted on every invocation. + +The metrics cover directly invoked synchronous and asynchronous scalar and table functions. +Python functions, aggregate functions, and Process Table Functions are not covered. Enabling +the metrics adds overhead; use sampling to control the cost. + +#### Early firing for interval joins + +##### [FLINK-36953](https://issues.apache.org/jira/browse/FLINK-36953) (FLIP-497) + +Outer interval joins can emit unmatched rows before their join interval closes. The new +`EARLY_FIRE` hint emits a speculative, null-padded result after a configured delay. If a +matching row arrives later within the interval, Flink corrects the speculative result with +update-before and update-after records. + +```sql +SELECT /*+ EARLY_FIRE('delay'='5s') */ o.id, s.ship_time +FROM Orders o +LEFT OUTER JOIN Shipments s +ON o.id = s.order_id +AND o.order_time BETWEEN s.ship_time - INTERVAL '1' HOUR AND s.ship_time; +``` + +The optional `time-mode` selects `rowtime` or `proctime`; it defaults to the time mode of the +interval join. Each unmatched outer row fires at most once. The feature is disabled unless +the hint is present and requires downstream operators and sinks that accept updates. +Existing interval joins without the hint retain their behavior. + +#### FROM_CHANGELOG and TO_CHANGELOG enhancements + +##### [FLINK-39258](https://issues.apache.org/jira/browse/FLINK-39258) (FLIP-564) + +Flink 2.4 adds partitioned processing and additional controls to the changelog conversion +PTFs introduced in Flink 2.3: + +- `PARTITION BY` support to co-locate related changes, with corresponding convenience methods + on partitioned tables in the Table API. +- Partitioned `FROM_CHANGELOG` calls can use the existing PTF `ORDER BY` support to reorder + CDC events per key using a watermarked time attribute. Late events are dropped by the + sorting operation. +- `error_handling => 'FAIL' | 'SKIP'` for `FROM_CHANGELOG` to control how NULL or unmapped + operation codes are handled. The default is `FAIL`. +- `produces_full_deletes` for `TO_CHANGELOG` to choose between full-row deletes and deletes + containing only identifying keys. The default is `true`. + +These additions make it easier to consume custom CDC streams and write changelog events to +append-only sinks. Users must still provide correctly ordered, complete change events for +the selected changelog mode; the functions do not reconstruct arbitrary missing before images. +See the [Changelog Conversion documentation](https://nightlies.apache.org/flink/flink-docs-release-2.4/docs/sql/reference/queries/changelog/) +for supported mappings and input requirements. + +#### Process Table Function state access and broadcast support + +##### [FLINK-39254](https://issues.apache.org/jira/browse/FLINK-39254) (FLIP-565) + +Flink 2.4 adds lazy value-state access and broadcast inputs to Process Table Functions: + +- `ValueView` provides lazy value-state access alongside `ListView` and `MapView`. A function + can read or update state only when needed, avoiding unnecessary serialization work. +- `BROADCAST_SEMANTIC_TABLE` lets a PTF receive a rule or control table on every parallel + instance. State declared with `@StateHint(StateKind.BROADCAST)` is writable from the + broadcast input and readable while processing the regular inputs. +- `NOTIFY_STATEFUL_SETS` lets a broadcast update trigger re-evaluation of existing stateful + partitions. This is opt-in because visiting all stateful partitions can be expensive. + +#### Process Table Function test harness + +##### [FLINK-39357](https://issues.apache.org/jira/browse/FLINK-39357) (FLIP-567) + +The new `ProcessTableFunctionTestHarness` lets developers test PTF business logic without +starting a Flink cluster. Tests can configure scalar and table arguments, partitioning, and +initial state, feed rows into individual inputs, and inspect the resulting output. + +The harness also supports advancing watermarks and the system clock, inspecting timers, +accessing state per key, and testing state TTL and changelog handling. This makes temporal and +stateful behavior easier to test deterministically. Full planner and runtime integration +should still be covered by integration tests. + +#### Strict BYTES-to-STRING casts and UTF-8 utilities + +##### [FLINK-39600](https://issues.apache.org/jira/browse/FLINK-39600) (FLIP-568) + +Casting binary data to a string now validates UTF-8. Previously, malformed byte sequences +were silently replaced with the Unicode replacement character `U+FFFD`; in Flink 2.4, +`CAST(bytes AS STRING)` fails on invalid UTF-8 and `TRY_CAST(bytes AS STRING)` returns NULL. +This is a behavior change for pipelines that previously accepted malformed input. + +Two new functions provide explicit handling: + +- `IS_VALID_UTF8(bytes)` returns whether the input is valid UTF-8, for filtering or routing + malformed records. +- `MAKE_VALID_UTF8(bytes)` decodes the input and replaces invalid sequences with `U+FFFD`, + preserving the previous substitution behavior. + +Both functions return NULL for NULL input. As a migration option, +`table.exec.legacy-bytes-to-string-cast` (default: `false`) can be set to `true` to restore +the previous cast behavior. + +Connector authors can use the new `StringData.fromUtf8Bytes(byte[])` method for strict +validation at ingestion. The existing `StringData.fromBytes(byte[])` remains unchanged. + +#### Metadata filter push-down for table sources + +##### [FLINK-39421](https://issues.apache.org/jira/browse/FLINK-39421) (FLIP-574) + +Source connectors can now accept filter predicates on metadata columns, such as offsets, +partitions, or timestamps. This lets a supporting connector reduce the data it reads instead +of applying all such filters after ingestion. + +Connectors opt in through the new `supportsMetadataFilterPushDown()` and +`applyMetadataFilters()` methods on `SupportsReadingMetadata`. Predicates use metadata keys +rather than SQL column aliases, and filters that the connector cannot fully apply remain in +the query. Metadata predicates are also preserved when compiled plans are restored. +Existing connectors retain their behavior unless they implement the new capability. + +#### In-place conversion of tables to materialized tables + +##### [FLINK-39847](https://issues.apache.org/jira/browse/FLINK-39847) (FLIP-578) + +`CREATE OR ALTER MATERIALIZED TABLE` can convert an existing regular table into a materialized +table without dropping and recreating the catalog object. The conversion preserves the table's +identity and underlying storage and starts a refresh job for the defining query. + +Conversion is disabled by default. Enable it when initializing the session with: Review Comment: This option is read only from the root configuration (`SqlCreateOrAlterMaterializedTableConverter` calls `getRootConfiguration()`), and its own description says "a session-level SET statement has no effect". "When initializing the session" reads like a SQL Client `-i` init script, which is made of `SET` statements, and the user then only sees `Regular table does not support create or alter operation.` Could we say where to set it, e.g. "Enable it in `config.yaml`. A session-level `SET` statement has no effect:"? ########## docs/content/release-notes/flink-2.4.md: ########## @@ -0,0 +1,359 @@ +--- +title: "Release Notes - Flink 2.4" +--- + +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Release notes - Flink 2.4 + +These release notes discuss important aspects, such as configuration, behavior or dependencies, +that changed between Flink 2.3 and Flink 2.4. Please read these notes carefully if you are +planning to upgrade your Flink version to 2.4. + +### Table SQL / API + +#### Upgrade Calcite to 1.41.0 + +##### [FLINK-39859](https://issues.apache.org/jira/browse/FLINK-39859) + +Flink 2.4 upgrades Apache Calcite, the foundation of the SQL parser and optimizer, to version +1.41.0. This incorporates the fixes and improvements from the intervening Calcite releases. +Applications that depend directly on Calcite classes or extend Flink's planner should check +their dependencies and custom planner code against the upgraded version. + +#### Common subexpression elimination in Calc code generation + +##### [FLINK-39268](https://issues.apache.org/jira/browse/FLINK-39268) + +Flink 2.4 avoids redundant evaluation of common subexpressions within a Calc operator, which +combines filtering and projection. Repeated deterministic expressions can reuse an already +computed result instead of being evaluated again for the same input row. This applies to both +batch and streaming queries and can reduce the cost of repeated built-in or user-defined +function calls. + +Reuse respects conditional evaluation, such as short-circuiting in `AND`, `OR`, and `CASE`. +UDF authors should ensure that `isDeterministic()` correctly describes their function: +functions with side effects or results that vary for identical inputs must return `false`. + +#### Built-in UDF metrics + +##### [FLINK-38071](https://issues.apache.org/jira/browse/FLINK-38071) (FLIP-485) + +SQL and Table API users can now monitor UDF execution through two built-in metrics: +`udfProcessingTime`, a histogram of sampled invocation durations, and `udfExceptionCount`, +a counter of exceptions from the function. Metrics are registered on the executing operator +under `udf.<udf_name>`. For asynchronous functions, processing time includes the time from +dispatch until completion. + +The feature is opt-in: + +- `table.exec.udf-metric-enabled` (default: `false`) enables metric collection. +- `table.exec.udf-metric.sample-interval` (default: `100`) controls how often invocation + durations are measured. Exceptions are counted on every invocation. + +The metrics cover directly invoked synchronous and asynchronous scalar and table functions. +Python functions, aggregate functions, and Process Table Functions are not covered. Enabling +the metrics adds overhead; use sampling to control the cost. + +#### Early firing for interval joins + +##### [FLINK-36953](https://issues.apache.org/jira/browse/FLINK-36953) (FLIP-497) + +Outer interval joins can emit unmatched rows before their join interval closes. The new +`EARLY_FIRE` hint emits a speculative, null-padded result after a configured delay. If a +matching row arrives later within the interval, Flink corrects the speculative result with +update-before and update-after records. + +```sql +SELECT /*+ EARLY_FIRE('delay'='5s') */ o.id, s.ship_time +FROM Orders o +LEFT OUTER JOIN Shipments s +ON o.id = s.order_id +AND o.order_time BETWEEN s.ship_time - INTERVAL '1' HOUR AND s.ship_time; +``` + +The optional `time-mode` selects `rowtime` or `proctime`; it defaults to the time mode of the +interval join. Each unmatched outer row fires at most once. The feature is disabled unless +the hint is present and requires downstream operators and sinks that accept updates. +Existing interval joins without the hint retain their behavior. + +#### FROM_CHANGELOG and TO_CHANGELOG enhancements + +##### [FLINK-39258](https://issues.apache.org/jira/browse/FLINK-39258) (FLIP-564) + +Flink 2.4 adds partitioned processing and additional controls to the changelog conversion +PTFs introduced in Flink 2.3: + +- `PARTITION BY` support to co-locate related changes, with corresponding convenience methods + on partitioned tables in the Table API. +- Partitioned `FROM_CHANGELOG` calls can use the existing PTF `ORDER BY` support to reorder + CDC events per key using a watermarked time attribute. Late events are dropped by the + sorting operation. +- `error_handling => 'FAIL' | 'SKIP'` for `FROM_CHANGELOG` to control how NULL or unmapped + operation codes are handled. The default is `FAIL`. +- `produces_full_deletes` for `TO_CHANGELOG` to choose between full-row deletes and deletes + containing only identifying keys. The default is `true`. + +These additions make it easier to consume custom CDC streams and write changelog events to +append-only sinks. Users must still provide correctly ordered, complete change events for +the selected changelog mode; the functions do not reconstruct arbitrary missing before images. +See the [Changelog Conversion documentation](https://nightlies.apache.org/flink/flink-docs-release-2.4/docs/sql/reference/queries/changelog/) +for supported mappings and input requirements. + +#### Process Table Function state access and broadcast support + +##### [FLINK-39254](https://issues.apache.org/jira/browse/FLINK-39254) (FLIP-565) + +Flink 2.4 adds lazy value-state access and broadcast inputs to Process Table Functions: + +- `ValueView` provides lazy value-state access alongside `ListView` and `MapView`. A function + can read or update state only when needed, avoiding unnecessary serialization work. +- `BROADCAST_SEMANTIC_TABLE` lets a PTF receive a rule or control table on every parallel + instance. State declared with `@StateHint(StateKind.BROADCAST)` is writable from the + broadcast input and readable while processing the regular inputs. +- `NOTIFY_STATEFUL_SETS` lets a broadcast update trigger re-evaluation of existing stateful + partitions. This is opt-in because visiting all stateful partitions can be expensive. + +#### Process Table Function test harness + +##### [FLINK-39357](https://issues.apache.org/jira/browse/FLINK-39357) (FLIP-567) + +The new `ProcessTableFunctionTestHarness` lets developers test PTF business logic without +starting a Flink cluster. Tests can configure scalar and table arguments, partitioning, and +initial state, feed rows into individual inputs, and inspect the resulting output. + +The harness also supports advancing watermarks and the system clock, inspecting timers, +accessing state per key, and testing state TTL and changelog handling. This makes temporal and +stateful behavior easier to test deterministically. Full planner and runtime integration +should still be covered by integration tests. + +#### Strict BYTES-to-STRING casts and UTF-8 utilities + +##### [FLINK-39600](https://issues.apache.org/jira/browse/FLINK-39600) (FLIP-568) + +Casting binary data to a string now validates UTF-8. Previously, malformed byte sequences +were silently replaced with the Unicode replacement character `U+FFFD`; in Flink 2.4, +`CAST(bytes AS STRING)` fails on invalid UTF-8 and `TRY_CAST(bytes AS STRING)` returns NULL. +This is a behavior change for pipelines that previously accepted malformed input. + +Two new functions provide explicit handling: + +- `IS_VALID_UTF8(bytes)` returns whether the input is valid UTF-8, for filtering or routing + malformed records. +- `MAKE_VALID_UTF8(bytes)` decodes the input and replaces invalid sequences with `U+FFFD`, + preserving the previous substitution behavior. + +Both functions return NULL for NULL input. As a migration option, +`table.exec.legacy-bytes-to-string-cast` (default: `false`) can be set to `true` to restore +the previous cast behavior. + +Connector authors can use the new `StringData.fromUtf8Bytes(byte[])` method for strict +validation at ingestion. The existing `StringData.fromBytes(byte[])` remains unchanged. + +#### Metadata filter push-down for table sources + +##### [FLINK-39421](https://issues.apache.org/jira/browse/FLINK-39421) (FLIP-574) + +Source connectors can now accept filter predicates on metadata columns, such as offsets, +partitions, or timestamps. This lets a supporting connector reduce the data it reads instead +of applying all such filters after ingestion. + +Connectors opt in through the new `supportsMetadataFilterPushDown()` and +`applyMetadataFilters()` methods on `SupportsReadingMetadata`. Predicates use metadata keys +rather than SQL column aliases, and filters that the connector cannot fully apply remain in +the query. Metadata predicates are also preserved when compiled plans are restored. +Existing connectors retain their behavior unless they implement the new capability. + +#### In-place conversion of tables to materialized tables + +##### [FLINK-39847](https://issues.apache.org/jira/browse/FLINK-39847) (FLIP-578) + +`CREATE OR ALTER MATERIALIZED TABLE` can convert an existing regular table into a materialized +table without dropping and recreating the catalog object. The conversion preserves the table's +identity and underlying storage and starts a refresh job for the defining query. + +Conversion is disabled by default. Enable it when initializing the session with: + +```yaml +table.materialized-table.conversion-from-table.enabled: true +``` + +The catalog must support the new `convertTableToMaterializedTable` operation. The conversion +statement defines the resulting schema and options; existing primary keys, watermarks, and +other definitions must be declared again where needed. This is a one-way conversion: there is +no corresponding operation to turn a materialized table back into a regular table. + +#### LATERAL SNAPSHOT joins + +##### [FLINK-39779](https://issues.apache.org/jira/browse/FLINK-39779) (FLIP-579) + +Flink SQL gains `LATERAL SNAPSHOT` joins for enriching an append-only stream with the current +contents of an updating table. Unlike an event-time temporal join, the join does not wait for +both inputs' watermarks for every result. It first buffers probe-side rows while initializing +the build side, then joins arriving rows immediately against the current build-side state. + +`INNER` and `LEFT OUTER` joins are supported, and the build side does not need a primary key. +Streaming queries specify a watermarked build-side column through `on_time`. +`load_completed_time` controls when initialization completes; an optional +`load_completed_idle_timeout` provides a fallback when build-side watermarks stop advancing. +For static or infrequently updated build sides, configure this timeout or use a bounded source +so the join does not remain in its initialization phase. + +Results follow processing-time semantics and can differ after recovery. Users requiring an +exact event-time version should continue to use event-time temporal joins. +See the [LATERAL SNAPSHOT join documentation](https://nightlies.apache.org/flink/flink-docs-release-2.4/docs/sql/reference/queries/joins/#lateral-snapshot-join) +for the full syntax and initialization behavior. + +#### UUID type and extended VARIANT primitives + +##### [FLINK-40485](https://issues.apache.org/jira/browse/FLINK-40485) (FLIP-604) + +Flink 2.4 introduces a native `UUID` type backed by a 16-byte value, with explicit casts to +and from strings and binary data, comparison and ordering support, and typed `UUID_V4()` and +`UUID_V7()` generators. The existing `UUID()` function continues to return a string. + +The `VARIANT` type also supports time-of-day values, nanosecond-precision timestamps with and +without local time zone, and UUID values. This extends compatibility with the corresponding +Parquet Variant primitive encodings. Nanosecond timestamps can be read as `TIMESTAMP(9)` or +`TIMESTAMP_LTZ(9)`; converting a VARIANT time-of-day value to SQL `TIME(3)` truncates any +sub-millisecond precision. + +Existing VARIANT values remain readable. Older Flink versions cannot read values containing Review Comment: `VARIANT` also has a breaking change that isn't mentioned: it, or a type containing it, can no longer be a grouping, `DISTINCT`, `ORDER BY` or join key, an operand of `=`, `<` or `IN`, or the compared argument of functions such as `ARRAY_CONTAINS` and `GREATEST`, so queries that planned on 2.3 can fail ([FLINK-40910](https://issues.apache.org/jira/browse/FLINK-40910), [FLINK-40936](https://issues.apache.org/jira/browse/FLINK-40936)). Both tickets' release notes give the workaround (cast to a concrete type first). Would it fit next to this compatibility paragraph? ########## docs/content/release-notes/flink-2.4.md: ########## @@ -0,0 +1,359 @@ +--- +title: "Release Notes - Flink 2.4" +--- + +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Release notes - Flink 2.4 + +These release notes discuss important aspects, such as configuration, behavior or dependencies, +that changed between Flink 2.3 and Flink 2.4. Please read these notes carefully if you are +planning to upgrade your Flink version to 2.4. + +### Table SQL / API + +#### Upgrade Calcite to 1.41.0 + +##### [FLINK-39859](https://issues.apache.org/jira/browse/FLINK-39859) + +Flink 2.4 upgrades Apache Calcite, the foundation of the SQL parser and optimizer, to version +1.41.0. This incorporates the fixes and improvements from the intervening Calcite releases. +Applications that depend directly on Calcite classes or extend Flink's planner should check +their dependencies and custom planner code against the upgraded version. + +#### Common subexpression elimination in Calc code generation + +##### [FLINK-39268](https://issues.apache.org/jira/browse/FLINK-39268) + +Flink 2.4 avoids redundant evaluation of common subexpressions within a Calc operator, which +combines filtering and projection. Repeated deterministic expressions can reuse an already +computed result instead of being evaluated again for the same input row. This applies to both +batch and streaming queries and can reduce the cost of repeated built-in or user-defined +function calls. + +Reuse respects conditional evaluation, such as short-circuiting in `AND`, `OR`, and `CASE`. +UDF authors should ensure that `isDeterministic()` correctly describes their function: +functions with side effects or results that vary for identical inputs must return `false`. + +#### Built-in UDF metrics + +##### [FLINK-38071](https://issues.apache.org/jira/browse/FLINK-38071) (FLIP-485) + +SQL and Table API users can now monitor UDF execution through two built-in metrics: +`udfProcessingTime`, a histogram of sampled invocation durations, and `udfExceptionCount`, +a counter of exceptions from the function. Metrics are registered on the executing operator +under `udf.<udf_name>`. For asynchronous functions, processing time includes the time from +dispatch until completion. + +The feature is opt-in: + +- `table.exec.udf-metric-enabled` (default: `false`) enables metric collection. +- `table.exec.udf-metric.sample-interval` (default: `100`) controls how often invocation + durations are measured. Exceptions are counted on every invocation. + +The metrics cover directly invoked synchronous and asynchronous scalar and table functions. +Python functions, aggregate functions, and Process Table Functions are not covered. Enabling +the metrics adds overhead; use sampling to control the cost. + +#### Early firing for interval joins + +##### [FLINK-36953](https://issues.apache.org/jira/browse/FLINK-36953) (FLIP-497) + +Outer interval joins can emit unmatched rows before their join interval closes. The new +`EARLY_FIRE` hint emits a speculative, null-padded result after a configured delay. If a +matching row arrives later within the interval, Flink corrects the speculative result with +update-before and update-after records. + +```sql +SELECT /*+ EARLY_FIRE('delay'='5s') */ o.id, s.ship_time +FROM Orders o +LEFT OUTER JOIN Shipments s +ON o.id = s.order_id +AND o.order_time BETWEEN s.ship_time - INTERVAL '1' HOUR AND s.ship_time; +``` + +The optional `time-mode` selects `rowtime` or `proctime`; it defaults to the time mode of the +interval join. Each unmatched outer row fires at most once. The feature is disabled unless +the hint is present and requires downstream operators and sinks that accept updates. +Existing interval joins without the hint retain their behavior. + +#### FROM_CHANGELOG and TO_CHANGELOG enhancements + +##### [FLINK-39258](https://issues.apache.org/jira/browse/FLINK-39258) (FLIP-564) + +Flink 2.4 adds partitioned processing and additional controls to the changelog conversion +PTFs introduced in Flink 2.3: + +- `PARTITION BY` support to co-locate related changes, with corresponding convenience methods + on partitioned tables in the Table API. +- Partitioned `FROM_CHANGELOG` calls can use the existing PTF `ORDER BY` support to reorder + CDC events per key using a watermarked time attribute. Late events are dropped by the + sorting operation. +- `error_handling => 'FAIL' | 'SKIP'` for `FROM_CHANGELOG` to control how NULL or unmapped + operation codes are handled. The default is `FAIL`. +- `produces_full_deletes` for `TO_CHANGELOG` to choose between full-row deletes and deletes + containing only identifying keys. The default is `true`. + +These additions make it easier to consume custom CDC streams and write changelog events to +append-only sinks. Users must still provide correctly ordered, complete change events for +the selected changelog mode; the functions do not reconstruct arbitrary missing before images. +See the [Changelog Conversion documentation](https://nightlies.apache.org/flink/flink-docs-release-2.4/docs/sql/reference/queries/changelog/) +for supported mappings and input requirements. + +#### Process Table Function state access and broadcast support + +##### [FLINK-39254](https://issues.apache.org/jira/browse/FLINK-39254) (FLIP-565) + +Flink 2.4 adds lazy value-state access and broadcast inputs to Process Table Functions: + +- `ValueView` provides lazy value-state access alongside `ListView` and `MapView`. A function + can read or update state only when needed, avoiding unnecessary serialization work. +- `BROADCAST_SEMANTIC_TABLE` lets a PTF receive a rule or control table on every parallel + instance. State declared with `@StateHint(StateKind.BROADCAST)` is writable from the + broadcast input and readable while processing the regular inputs. +- `NOTIFY_STATEFUL_SETS` lets a broadcast update trigger re-evaluation of existing stateful + partitions. This is opt-in because visiting all stateful partitions can be expensive. + +#### Process Table Function test harness + +##### [FLINK-39357](https://issues.apache.org/jira/browse/FLINK-39357) (FLIP-567) + +The new `ProcessTableFunctionTestHarness` lets developers test PTF business logic without +starting a Flink cluster. Tests can configure scalar and table arguments, partitioning, and +initial state, feed rows into individual inputs, and inspect the resulting output. + +The harness also supports advancing watermarks and the system clock, inspecting timers, +accessing state per key, and testing state TTL and changelog handling. This makes temporal and +stateful behavior easier to test deterministically. Full planner and runtime integration +should still be covered by integration tests. + +#### Strict BYTES-to-STRING casts and UTF-8 utilities + +##### [FLINK-39600](https://issues.apache.org/jira/browse/FLINK-39600) (FLIP-568) + +Casting binary data to a string now validates UTF-8. Previously, malformed byte sequences +were silently replaced with the Unicode replacement character `U+FFFD`; in Flink 2.4, +`CAST(bytes AS STRING)` fails on invalid UTF-8 and `TRY_CAST(bytes AS STRING)` returns NULL. +This is a behavior change for pipelines that previously accepted malformed input. + +Two new functions provide explicit handling: + +- `IS_VALID_UTF8(bytes)` returns whether the input is valid UTF-8, for filtering or routing + malformed records. +- `MAKE_VALID_UTF8(bytes)` decodes the input and replaces invalid sequences with `U+FFFD`, + preserving the previous substitution behavior. + +Both functions return NULL for NULL input. As a migration option, +`table.exec.legacy-bytes-to-string-cast` (default: `false`) can be set to `true` to restore +the previous cast behavior. + +Connector authors can use the new `StringData.fromUtf8Bytes(byte[])` method for strict +validation at ingestion. The existing `StringData.fromBytes(byte[])` remains unchanged. + +#### Metadata filter push-down for table sources + +##### [FLINK-39421](https://issues.apache.org/jira/browse/FLINK-39421) (FLIP-574) + +Source connectors can now accept filter predicates on metadata columns, such as offsets, +partitions, or timestamps. This lets a supporting connector reduce the data it reads instead +of applying all such filters after ingestion. + +Connectors opt in through the new `supportsMetadataFilterPushDown()` and +`applyMetadataFilters()` methods on `SupportsReadingMetadata`. Predicates use metadata keys +rather than SQL column aliases, and filters that the connector cannot fully apply remain in +the query. Metadata predicates are also preserved when compiled plans are restored. +Existing connectors retain their behavior unless they implement the new capability. + +#### In-place conversion of tables to materialized tables + +##### [FLINK-39847](https://issues.apache.org/jira/browse/FLINK-39847) (FLIP-578) + +`CREATE OR ALTER MATERIALIZED TABLE` can convert an existing regular table into a materialized +table without dropping and recreating the catalog object. The conversion preserves the table's +identity and underlying storage and starts a refresh job for the defining query. + +Conversion is disabled by default. Enable it when initializing the session with: + +```yaml +table.materialized-table.conversion-from-table.enabled: true +``` + +The catalog must support the new `convertTableToMaterializedTable` operation. The conversion +statement defines the resulting schema and options; existing primary keys, watermarks, and +other definitions must be declared again where needed. This is a one-way conversion: there is +no corresponding operation to turn a materialized table back into a regular table. + +#### LATERAL SNAPSHOT joins + +##### [FLINK-39779](https://issues.apache.org/jira/browse/FLINK-39779) (FLIP-579) + +Flink SQL gains `LATERAL SNAPSHOT` joins for enriching an append-only stream with the current +contents of an updating table. Unlike an event-time temporal join, the join does not wait for +both inputs' watermarks for every result. It first buffers probe-side rows while initializing +the build side, then joins arriving rows immediately against the current build-side state. + +`INNER` and `LEFT OUTER` joins are supported, and the build side does not need a primary key. +Streaming queries specify a watermarked build-side column through `on_time`. +`load_completed_time` controls when initialization completes; an optional +`load_completed_idle_timeout` provides a fallback when build-side watermarks stop advancing. +For static or infrequently updated build sides, configure this timeout or use a bounded source +so the join does not remain in its initialization phase. + +Results follow processing-time semantics and can differ after recovery. Users requiring an +exact event-time version should continue to use event-time temporal joins. +See the [LATERAL SNAPSHOT join documentation](https://nightlies.apache.org/flink/flink-docs-release-2.4/docs/sql/reference/queries/joins/#lateral-snapshot-join) +for the full syntax and initialization behavior. + +#### UUID type and extended VARIANT primitives + +##### [FLINK-40485](https://issues.apache.org/jira/browse/FLINK-40485) (FLIP-604) + +Flink 2.4 introduces a native `UUID` type backed by a 16-byte value, with explicit casts to Review Comment: Two `UUID` gaps aren't mentioned. Formats don't read or write `UUID` columns yet (the shipped `data-types.md` says to cast to `STRING` or `BINARY(16)` before a sink). And three sub-tasks targeting 2.4.0 are still open with no fix on `release-2.4`: [FLINK-41332](https://issues.apache.org/jira/browse/FLINK-41332) (`COMPILE PLAN` fails on a `UUID` literal), [FLINK-41333](https://issues.apache.org/jira/browse/FLINK-41333) (`MIN`/`MAX` and streaming `COUNT(DISTINCT)` reject `UUID`) and [FLINK-41334](https://issues.apache.org/jira/browse/FLINK-41334) (SQL Gateway JSON results and the JDBC driver can't return a `UUID` column). How about noting the formats gap here, and listing the three as known limitations if they don't land? ########## docs/content/release-notes/flink-2.4.md: ########## @@ -0,0 +1,359 @@ +--- +title: "Release Notes - Flink 2.4" +--- + +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Release notes - Flink 2.4 + +These release notes discuss important aspects, such as configuration, behavior or dependencies, +that changed between Flink 2.3 and Flink 2.4. Please read these notes carefully if you are +planning to upgrade your Flink version to 2.4. + +### Table SQL / API + +#### Upgrade Calcite to 1.41.0 + +##### [FLINK-39859](https://issues.apache.org/jira/browse/FLINK-39859) + +Flink 2.4 upgrades Apache Calcite, the foundation of the SQL parser and optimizer, to version +1.41.0. This incorporates the fixes and improvements from the intervening Calcite releases. +Applications that depend directly on Calcite classes or extend Flink's planner should check +their dependencies and custom planner code against the upgraded version. + +#### Common subexpression elimination in Calc code generation + +##### [FLINK-39268](https://issues.apache.org/jira/browse/FLINK-39268) + +Flink 2.4 avoids redundant evaluation of common subexpressions within a Calc operator, which +combines filtering and projection. Repeated deterministic expressions can reuse an already +computed result instead of being evaluated again for the same input row. This applies to both +batch and streaming queries and can reduce the cost of repeated built-in or user-defined +function calls. + +Reuse respects conditional evaluation, such as short-circuiting in `AND`, `OR`, and `CASE`. +UDF authors should ensure that `isDeterministic()` correctly describes their function: +functions with side effects or results that vary for identical inputs must return `false`. + +#### Built-in UDF metrics + +##### [FLINK-38071](https://issues.apache.org/jira/browse/FLINK-38071) (FLIP-485) + +SQL and Table API users can now monitor UDF execution through two built-in metrics: +`udfProcessingTime`, a histogram of sampled invocation durations, and `udfExceptionCount`, +a counter of exceptions from the function. Metrics are registered on the executing operator +under `udf.<udf_name>`. For asynchronous functions, processing time includes the time from +dispatch until completion. + +The feature is opt-in: + +- `table.exec.udf-metric-enabled` (default: `false`) enables metric collection. +- `table.exec.udf-metric.sample-interval` (default: `100`) controls how often invocation + durations are measured. Exceptions are counted on every invocation. + +The metrics cover directly invoked synchronous and asynchronous scalar and table functions. +Python functions, aggregate functions, and Process Table Functions are not covered. Enabling +the metrics adds overhead; use sampling to control the cost. + +#### Early firing for interval joins + +##### [FLINK-36953](https://issues.apache.org/jira/browse/FLINK-36953) (FLIP-497) + +Outer interval joins can emit unmatched rows before their join interval closes. The new +`EARLY_FIRE` hint emits a speculative, null-padded result after a configured delay. If a +matching row arrives later within the interval, Flink corrects the speculative result with +update-before and update-after records. + +```sql +SELECT /*+ EARLY_FIRE('delay'='5s') */ o.id, s.ship_time +FROM Orders o +LEFT OUTER JOIN Shipments s +ON o.id = s.order_id +AND o.order_time BETWEEN s.ship_time - INTERVAL '1' HOUR AND s.ship_time; +``` + +The optional `time-mode` selects `rowtime` or `proctime`; it defaults to the time mode of the +interval join. Each unmatched outer row fires at most once. The feature is disabled unless +the hint is present and requires downstream operators and sinks that accept updates. +Existing interval joins without the hint retain their behavior. + +#### FROM_CHANGELOG and TO_CHANGELOG enhancements + +##### [FLINK-39258](https://issues.apache.org/jira/browse/FLINK-39258) (FLIP-564) + +Flink 2.4 adds partitioned processing and additional controls to the changelog conversion +PTFs introduced in Flink 2.3: + +- `PARTITION BY` support to co-locate related changes, with corresponding convenience methods + on partitioned tables in the Table API. +- Partitioned `FROM_CHANGELOG` calls can use the existing PTF `ORDER BY` support to reorder + CDC events per key using a watermarked time attribute. Late events are dropped by the + sorting operation. +- `error_handling => 'FAIL' | 'SKIP'` for `FROM_CHANGELOG` to control how NULL or unmapped + operation codes are handled. The default is `FAIL`. +- `produces_full_deletes` for `TO_CHANGELOG` to choose between full-row deletes and deletes + containing only identifying keys. The default is `true`. + +These additions make it easier to consume custom CDC streams and write changelog events to +append-only sinks. Users must still provide correctly ordered, complete change events for +the selected changelog mode; the functions do not reconstruct arbitrary missing before images. +See the [Changelog Conversion documentation](https://nightlies.apache.org/flink/flink-docs-release-2.4/docs/sql/reference/queries/changelog/) +for supported mappings and input requirements. + +#### Process Table Function state access and broadcast support + +##### [FLINK-39254](https://issues.apache.org/jira/browse/FLINK-39254) (FLIP-565) + +Flink 2.4 adds lazy value-state access and broadcast inputs to Process Table Functions: + +- `ValueView` provides lazy value-state access alongside `ListView` and `MapView`. A function + can read or update state only when needed, avoiding unnecessary serialization work. +- `BROADCAST_SEMANTIC_TABLE` lets a PTF receive a rule or control table on every parallel + instance. State declared with `@StateHint(StateKind.BROADCAST)` is writable from the + broadcast input and readable while processing the regular inputs. +- `NOTIFY_STATEFUL_SETS` lets a broadcast update trigger re-evaluation of existing stateful + partitions. This is opt-in because visiting all stateful partitions can be expensive. + +#### Process Table Function test harness + +##### [FLINK-39357](https://issues.apache.org/jira/browse/FLINK-39357) (FLIP-567) + +The new `ProcessTableFunctionTestHarness` lets developers test PTF business logic without +starting a Flink cluster. Tests can configure scalar and table arguments, partitioning, and +initial state, feed rows into individual inputs, and inspect the resulting output. + +The harness also supports advancing watermarks and the system clock, inspecting timers, +accessing state per key, and testing state TTL and changelog handling. This makes temporal and +stateful behavior easier to test deterministically. Full planner and runtime integration +should still be covered by integration tests. + +#### Strict BYTES-to-STRING casts and UTF-8 utilities + +##### [FLINK-39600](https://issues.apache.org/jira/browse/FLINK-39600) (FLIP-568) + +Casting binary data to a string now validates UTF-8. Previously, malformed byte sequences +were silently replaced with the Unicode replacement character `U+FFFD`; in Flink 2.4, +`CAST(bytes AS STRING)` fails on invalid UTF-8 and `TRY_CAST(bytes AS STRING)` returns NULL. +This is a behavior change for pipelines that previously accepted malformed input. + +Two new functions provide explicit handling: + +- `IS_VALID_UTF8(bytes)` returns whether the input is valid UTF-8, for filtering or routing + malformed records. +- `MAKE_VALID_UTF8(bytes)` decodes the input and replaces invalid sequences with `U+FFFD`, + preserving the previous substitution behavior. + +Both functions return NULL for NULL input. As a migration option, +`table.exec.legacy-bytes-to-string-cast` (default: `false`) can be set to `true` to restore +the previous cast behavior. + +Connector authors can use the new `StringData.fromUtf8Bytes(byte[])` method for strict +validation at ingestion. The existing `StringData.fromBytes(byte[])` remains unchanged. + +#### Metadata filter push-down for table sources + +##### [FLINK-39421](https://issues.apache.org/jira/browse/FLINK-39421) (FLIP-574) + +Source connectors can now accept filter predicates on metadata columns, such as offsets, +partitions, or timestamps. This lets a supporting connector reduce the data it reads instead +of applying all such filters after ingestion. + +Connectors opt in through the new `supportsMetadataFilterPushDown()` and +`applyMetadataFilters()` methods on `SupportsReadingMetadata`. Predicates use metadata keys +rather than SQL column aliases, and filters that the connector cannot fully apply remain in +the query. Metadata predicates are also preserved when compiled plans are restored. +Existing connectors retain their behavior unless they implement the new capability. + +#### In-place conversion of tables to materialized tables + +##### [FLINK-39847](https://issues.apache.org/jira/browse/FLINK-39847) (FLIP-578) + +`CREATE OR ALTER MATERIALIZED TABLE` can convert an existing regular table into a materialized +table without dropping and recreating the catalog object. The conversion preserves the table's +identity and underlying storage and starts a refresh job for the defining query. + +Conversion is disabled by default. Enable it when initializing the session with: + +```yaml +table.materialized-table.conversion-from-table.enabled: true +``` + +The catalog must support the new `convertTableToMaterializedTable` operation. The conversion +statement defines the resulting schema and options; existing primary keys, watermarks, and +other definitions must be declared again where needed. This is a one-way conversion: there is +no corresponding operation to turn a materialized table back into a regular table. + +#### LATERAL SNAPSHOT joins + +##### [FLINK-39779](https://issues.apache.org/jira/browse/FLINK-39779) (FLIP-579) + +Flink SQL gains `LATERAL SNAPSHOT` joins for enriching an append-only stream with the current +contents of an updating table. Unlike an event-time temporal join, the join does not wait for +both inputs' watermarks for every result. It first buffers probe-side rows while initializing +the build side, then joins arriving rows immediately against the current build-side state. + +`INNER` and `LEFT OUTER` joins are supported, and the build side does not need a primary key. +Streaming queries specify a watermarked build-side column through `on_time`. +`load_completed_time` controls when initialization completes; an optional +`load_completed_idle_timeout` provides a fallback when build-side watermarks stop advancing. +For static or infrequently updated build sides, configure this timeout or use a bounded source +so the join does not remain in its initialization phase. + +Results follow processing-time semantics and can differ after recovery. Users requiring an +exact event-time version should continue to use event-time temporal joins. +See the [LATERAL SNAPSHOT join documentation](https://nightlies.apache.org/flink/flink-docs-release-2.4/docs/sql/reference/queries/joins/#lateral-snapshot-join) +for the full syntax and initialization behavior. + +#### UUID type and extended VARIANT primitives + +##### [FLINK-40485](https://issues.apache.org/jira/browse/FLINK-40485) (FLIP-604) + +Flink 2.4 introduces a native `UUID` type backed by a 16-byte value, with explicit casts to +and from strings and binary data, comparison and ordering support, and typed `UUID_V4()` and +`UUID_V7()` generators. The existing `UUID()` function continues to return a string. + +The `VARIANT` type also supports time-of-day values, nanosecond-precision timestamps with and +without local time zone, and UUID values. This extends compatibility with the corresponding +Parquet Variant primitive encodings. Nanosecond timestamps can be read as `TIMESTAMP(9)` or +`TIMESTAMP_LTZ(9)`; converting a VARIANT time-of-day value to SQL `TIME(3)` truncates any +sub-millisecond precision. + +Existing VARIANT values remain readable. Older Flink versions cannot read values containing +the newly supported primitive codes, so consider this limitation before downgrading a job or +sharing persisted VARIANT data with older consumers. + +### Python + +#### Python DataFrame API + +##### [FLINK-40185](https://issues.apache.org/jira/browse/FLINK-40185) (FLIP-591) + +PyFlink gains a DataFrame API under `pyflink.dataframe`. It offers a Python-oriented entry +point for creating Flink tables from records, dictionaries, pandas DataFrames, or Arrow tables, +and for reading and writing through Flink connectors. + +The API supports familiar transformations such as selection, filtering, joins, aggregation, +windowing, and missing-value handling, as well as Python UDFs and batch transformations. +`from_table()` and `to_table()` allow applications to move between DataFrames and the existing +Table API. + +DataFrame operations use Flink's existing planner and runtime for batch and streaming +execution. The API complements the Table API; it does not replace it or promise full +compatibility with pandas or other DataFrame libraries. + +### Connectors + +#### Job information on source contexts + +##### [FLINK-39776](https://issues.apache.org/jira/browse/FLINK-39776) (FLIP-583) + +`SourceReaderContext` and `SplitEnumeratorContext` now expose `getJobInfo()`, giving source +implementations direct access to the job ID and job name. This aligns the Source API with +Sink V2 and supports use cases such as per-job authentication, external offset tracking, +and log correlation without parsing metric scope variables. + +The new methods have default implementations, so existing context implementations remain +source- and binary-compatible. Flink's runtime contexts provide the job information; +third-party contexts and test stubs must override the methods if callers use them, as the +default implementation throws `UnsupportedOperationException`. + +### Runtime + +#### Filtering and spilling recovered channel state + +##### [FLINK-38544](https://issues.apache.org/jira/browse/FLINK-38544) (FLIP-547 V2) + +Flink 2.4 extends checkpointing during recovery with a spill-based recovery path. Recovered +channel state is filtered for the receiving subtask before records enter normal processing, +and the resulting data is written to local spill files before being replayed through network +buffers. This bounds memory usage when recovering large amounts of in-flight data. + +Filtering before processing also prevents records that belong to other subtasks after +rescaling from being included again in subsequent recovery checkpoints. Unaligned checkpoints +capture both queued and unread spilled data, allowing recovery progress to survive another +failure or rescaling without duplicating the recovered records. + +The spill path is used when checkpointing during recovery is enabled through the existing +configuration options and requires sufficient local disk space. Flink 2.4 also disables +checkpointing during recovery when `execution.checkpointing.aligned-checkpoint-timeout` is +greater than zero. + +#### Pluggable HistoryServer archive storage + +##### [FLINK-39911](https://issues.apache.org/jira/browse/FLINK-39911) (FLIP-584) + +The HistoryServer can store its local archive cache in an embedded RocksDB database instead +of expanding every archive into many small JSON files. This reduces inode consumption when +serving large numbers of jobs or jobs with extensive task-level details. + +Select the backend with `historyserver.archive.storage.type`: + +- `FILE` (default) keeps the existing file-based storage behavior. +- `ROCKSDB` stores archive entries in an embedded database under + `historyserver.web.tmpdir/rocksdb`. + +This changes how the local cache is stored; completed-job archives continue to be read from +the configured archive directories. + +#### Lazy loading in HistoryServer + +##### [FLINK-40097](https://issues.apache.org/jira/browse/FLINK-40097) (FLIP-585) + +The HistoryServer gains a `LAZY` archive load mode. Overview information is made available +before all job details have been processed, while details are loaded asynchronously. +Accessing an archive that has not yet finished loading triggers or waits for the relevant +on-demand load, so users do not have to wait for every preceding archive to be processed. + +The mode is opt-in through `historyserver.archive.load.mode` (default: `EAGER`). Lazy loading +uses two separately configurable thread pools: + +- `historyserver.lazy.fetch.executor.common.pool-size` (default: `4`) for background loads. +- `historyserver.lazy.fetch.executor.individual.pool-size` (default: `4`) for individual + on-demand loads. + +The default `EAGER` mode retains synchronous archive loading. Lazy loading works with both +the file and RocksDB local storage backends. + +#### State Catalog + +##### [FLINK-40175](https://issues.apache.org/jira/browse/FLINK-40175) (FLIP-599) + +The new read-only State Catalog discovers checkpoints and savepoints in configured +directories and exposes their contents through Flink SQL. Each discovered snapshot becomes +a database, with tables for operator state and views for inspecting snapshot metadata. +This removes much of the manual schema definition previously needed to query saved state. + +The catalog infers schemas from supported serializer snapshots and exposes keyed state, +flattened list and map state, non-keyed list/union/broadcast state, and window state. +Supported POJO and Avro state can be read without the original user classes on the classpath; +custom serialization formats still require compatible type information and reader support. + +Register the catalog using one or more labeled directories: + +```sql +CREATE CATALOG saved_state WITH ( + 'type' = 'state', + 'directory.prod' = 's3://my-bucket/flink/checkpoints' +); +``` + +The catalog is provided by `flink-state-processing-api`. Snapshot metadata views additionally Review Comment: The Maven artifact is `flink-state-processor-api` (`flink-libraries/flink-state-processing-api/pom.xml`); `flink-state-processing-api` is only the directory name. The binary distribution ships the jar in `opt/` (`flink-dist/src/main/assemblies/opt.xml`), so `CREATE CATALOG` with `'type' = 'state'` fails until it's on the classpath, and there's no State Catalog docs page on `release-2.4` yet. Would something like this work: "The catalog is provided by `flink-state-processor-api`, which the distribution ships in `opt/`. Add it to the classpath before creating the catalog."? ########## docs/content/release-notes/flink-2.4.md: ########## @@ -0,0 +1,359 @@ +--- +title: "Release Notes - Flink 2.4" +--- + +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Release notes - Flink 2.4 + +These release notes discuss important aspects, such as configuration, behavior or dependencies, +that changed between Flink 2.3 and Flink 2.4. Please read these notes carefully if you are +planning to upgrade your Flink version to 2.4. + +### Table SQL / API + +#### Upgrade Calcite to 1.41.0 + +##### [FLINK-39859](https://issues.apache.org/jira/browse/FLINK-39859) + +Flink 2.4 upgrades Apache Calcite, the foundation of the SQL parser and optimizer, to version +1.41.0. This incorporates the fixes and improvements from the intervening Calcite releases. +Applications that depend directly on Calcite classes or extend Flink's planner should check +their dependencies and custom planner code against the upgraded version. + +#### Common subexpression elimination in Calc code generation + +##### [FLINK-39268](https://issues.apache.org/jira/browse/FLINK-39268) + +Flink 2.4 avoids redundant evaluation of common subexpressions within a Calc operator, which +combines filtering and projection. Repeated deterministic expressions can reuse an already +computed result instead of being evaluated again for the same input row. This applies to both +batch and streaming queries and can reduce the cost of repeated built-in or user-defined +function calls. + +Reuse respects conditional evaluation, such as short-circuiting in `AND`, `OR`, and `CASE`. +UDF authors should ensure that `isDeterministic()` correctly describes their function: +functions with side effects or results that vary for identical inputs must return `false`. + +#### Built-in UDF metrics + +##### [FLINK-38071](https://issues.apache.org/jira/browse/FLINK-38071) (FLIP-485) + +SQL and Table API users can now monitor UDF execution through two built-in metrics: +`udfProcessingTime`, a histogram of sampled invocation durations, and `udfExceptionCount`, +a counter of exceptions from the function. Metrics are registered on the executing operator +under `udf.<udf_name>`. For asynchronous functions, processing time includes the time from +dispatch until completion. + +The feature is opt-in: + +- `table.exec.udf-metric-enabled` (default: `false`) enables metric collection. +- `table.exec.udf-metric.sample-interval` (default: `100`) controls how often invocation + durations are measured. Exceptions are counted on every invocation. + +The metrics cover directly invoked synchronous and asynchronous scalar and table functions. +Python functions, aggregate functions, and Process Table Functions are not covered. Enabling +the metrics adds overhead; use sampling to control the cost. + +#### Early firing for interval joins + +##### [FLINK-36953](https://issues.apache.org/jira/browse/FLINK-36953) (FLIP-497) + +Outer interval joins can emit unmatched rows before their join interval closes. The new +`EARLY_FIRE` hint emits a speculative, null-padded result after a configured delay. If a +matching row arrives later within the interval, Flink corrects the speculative result with +update-before and update-after records. + +```sql +SELECT /*+ EARLY_FIRE('delay'='5s') */ o.id, s.ship_time +FROM Orders o +LEFT OUTER JOIN Shipments s +ON o.id = s.order_id +AND o.order_time BETWEEN s.ship_time - INTERVAL '1' HOUR AND s.ship_time; +``` + +The optional `time-mode` selects `rowtime` or `proctime`; it defaults to the time mode of the +interval join. Each unmatched outer row fires at most once. The feature is disabled unless +the hint is present and requires downstream operators and sinks that accept updates. +Existing interval joins without the hint retain their behavior. + +#### FROM_CHANGELOG and TO_CHANGELOG enhancements + +##### [FLINK-39258](https://issues.apache.org/jira/browse/FLINK-39258) (FLIP-564) + +Flink 2.4 adds partitioned processing and additional controls to the changelog conversion +PTFs introduced in Flink 2.3: + +- `PARTITION BY` support to co-locate related changes, with corresponding convenience methods + on partitioned tables in the Table API. +- Partitioned `FROM_CHANGELOG` calls can use the existing PTF `ORDER BY` support to reorder + CDC events per key using a watermarked time attribute. Late events are dropped by the + sorting operation. +- `error_handling => 'FAIL' | 'SKIP'` for `FROM_CHANGELOG` to control how NULL or unmapped + operation codes are handled. The default is `FAIL`. +- `produces_full_deletes` for `TO_CHANGELOG` to choose between full-row deletes and deletes + containing only identifying keys. The default is `true`. + +These additions make it easier to consume custom CDC streams and write changelog events to +append-only sinks. Users must still provide correctly ordered, complete change events for +the selected changelog mode; the functions do not reconstruct arbitrary missing before images. +See the [Changelog Conversion documentation](https://nightlies.apache.org/flink/flink-docs-release-2.4/docs/sql/reference/queries/changelog/) +for supported mappings and input requirements. + +#### Process Table Function state access and broadcast support + +##### [FLINK-39254](https://issues.apache.org/jira/browse/FLINK-39254) (FLIP-565) + +Flink 2.4 adds lazy value-state access and broadcast inputs to Process Table Functions: + +- `ValueView` provides lazy value-state access alongside `ListView` and `MapView`. A function + can read or update state only when needed, avoiding unnecessary serialization work. +- `BROADCAST_SEMANTIC_TABLE` lets a PTF receive a rule or control table on every parallel + instance. State declared with `@StateHint(StateKind.BROADCAST)` is writable from the + broadcast input and readable while processing the regular inputs. +- `NOTIFY_STATEFUL_SETS` lets a broadcast update trigger re-evaluation of existing stateful + partitions. This is opt-in because visiting all stateful partitions can be expensive. + +#### Process Table Function test harness + +##### [FLINK-39357](https://issues.apache.org/jira/browse/FLINK-39357) (FLIP-567) + +The new `ProcessTableFunctionTestHarness` lets developers test PTF business logic without +starting a Flink cluster. Tests can configure scalar and table arguments, partitioning, and +initial state, feed rows into individual inputs, and inspect the resulting output. + +The harness also supports advancing watermarks and the system clock, inspecting timers, +accessing state per key, and testing state TTL and changelog handling. This makes temporal and +stateful behavior easier to test deterministically. Full planner and runtime integration +should still be covered by integration tests. + +#### Strict BYTES-to-STRING casts and UTF-8 utilities + +##### [FLINK-39600](https://issues.apache.org/jira/browse/FLINK-39600) (FLIP-568) + +Casting binary data to a string now validates UTF-8. Previously, malformed byte sequences +were silently replaced with the Unicode replacement character `U+FFFD`; in Flink 2.4, +`CAST(bytes AS STRING)` fails on invalid UTF-8 and `TRY_CAST(bytes AS STRING)` returns NULL. +This is a behavior change for pipelines that previously accepted malformed input. + +Two new functions provide explicit handling: + +- `IS_VALID_UTF8(bytes)` returns whether the input is valid UTF-8, for filtering or routing + malformed records. +- `MAKE_VALID_UTF8(bytes)` decodes the input and replaces invalid sequences with `U+FFFD`, + preserving the previous substitution behavior. + +Both functions return NULL for NULL input. As a migration option, +`table.exec.legacy-bytes-to-string-cast` (default: `false`) can be set to `true` to restore +the previous cast behavior. + +Connector authors can use the new `StringData.fromUtf8Bytes(byte[])` method for strict +validation at ingestion. The existing `StringData.fromBytes(byte[])` remains unchanged. + +#### Metadata filter push-down for table sources + +##### [FLINK-39421](https://issues.apache.org/jira/browse/FLINK-39421) (FLIP-574) + +Source connectors can now accept filter predicates on metadata columns, such as offsets, +partitions, or timestamps. This lets a supporting connector reduce the data it reads instead +of applying all such filters after ingestion. + +Connectors opt in through the new `supportsMetadataFilterPushDown()` and +`applyMetadataFilters()` methods on `SupportsReadingMetadata`. Predicates use metadata keys +rather than SQL column aliases, and filters that the connector cannot fully apply remain in +the query. Metadata predicates are also preserved when compiled plans are restored. +Existing connectors retain their behavior unless they implement the new capability. + +#### In-place conversion of tables to materialized tables + +##### [FLINK-39847](https://issues.apache.org/jira/browse/FLINK-39847) (FLIP-578) + +`CREATE OR ALTER MATERIALIZED TABLE` can convert an existing regular table into a materialized +table without dropping and recreating the catalog object. The conversion preserves the table's +identity and underlying storage and starts a refresh job for the defining query. + +Conversion is disabled by default. Enable it when initializing the session with: + +```yaml +table.materialized-table.conversion-from-table.enabled: true +``` + +The catalog must support the new `convertTableToMaterializedTable` operation. The conversion +statement defines the resulting schema and options; existing primary keys, watermarks, and +other definitions must be declared again where needed. This is a one-way conversion: there is +no corresponding operation to turn a materialized table back into a regular table. + +#### LATERAL SNAPSHOT joins + +##### [FLINK-39779](https://issues.apache.org/jira/browse/FLINK-39779) (FLIP-579) + +Flink SQL gains `LATERAL SNAPSHOT` joins for enriching an append-only stream with the current +contents of an updating table. Unlike an event-time temporal join, the join does not wait for +both inputs' watermarks for every result. It first buffers probe-side rows while initializing +the build side, then joins arriving rows immediately against the current build-side state. + +`INNER` and `LEFT OUTER` joins are supported, and the build side does not need a primary key. +Streaming queries specify a watermarked build-side column through `on_time`. +`load_completed_time` controls when initialization completes; an optional +`load_completed_idle_timeout` provides a fallback when build-side watermarks stop advancing. +For static or infrequently updated build sides, configure this timeout or use a bounded source +so the join does not remain in its initialization phase. + +Results follow processing-time semantics and can differ after recovery. Users requiring an +exact event-time version should continue to use event-time temporal joins. +See the [LATERAL SNAPSHOT join documentation](https://nightlies.apache.org/flink/flink-docs-release-2.4/docs/sql/reference/queries/joins/#lateral-snapshot-join) +for the full syntax and initialization behavior. + +#### UUID type and extended VARIANT primitives + +##### [FLINK-40485](https://issues.apache.org/jira/browse/FLINK-40485) (FLIP-604) + +Flink 2.4 introduces a native `UUID` type backed by a 16-byte value, with explicit casts to +and from strings and binary data, comparison and ordering support, and typed `UUID_V4()` and +`UUID_V7()` generators. The existing `UUID()` function continues to return a string. + +The `VARIANT` type also supports time-of-day values, nanosecond-precision timestamps with and +without local time zone, and UUID values. This extends compatibility with the corresponding +Parquet Variant primitive encodings. Nanosecond timestamps can be read as `TIMESTAMP(9)` or +`TIMESTAMP_LTZ(9)`; converting a VARIANT time-of-day value to SQL `TIME(3)` truncates any +sub-millisecond precision. + +Existing VARIANT values remain readable. Older Flink versions cannot read values containing +the newly supported primitive codes, so consider this limitation before downgrading a job or +sharing persisted VARIANT data with older consumers. + +### Python + +#### Python DataFrame API + +##### [FLINK-40185](https://issues.apache.org/jira/browse/FLINK-40185) (FLIP-591) + +PyFlink gains a DataFrame API under `pyflink.dataframe`. It offers a Python-oriented entry +point for creating Flink tables from records, dictionaries, pandas DataFrames, or Arrow tables, +and for reading and writing through Flink connectors. + +The API supports familiar transformations such as selection, filtering, joins, aggregation, +windowing, and missing-value handling, as well as Python UDFs and batch transformations. +`from_table()` and `to_table()` allow applications to move between DataFrames and the existing +Table API. + +DataFrame operations use Flink's existing planner and runtime for batch and streaming +execution. The API complements the Table API; it does not replace it or promise full +compatibility with pandas or other DataFrame libraries. + +### Connectors + +#### Job information on source contexts + +##### [FLINK-39776](https://issues.apache.org/jira/browse/FLINK-39776) (FLIP-583) + +`SourceReaderContext` and `SplitEnumeratorContext` now expose `getJobInfo()`, giving source +implementations direct access to the job ID and job name. This aligns the Source API with +Sink V2 and supports use cases such as per-job authentication, external offset tracking, +and log correlation without parsing metric scope variables. + +The new methods have default implementations, so existing context implementations remain +source- and binary-compatible. Flink's runtime contexts provide the job information; +third-party contexts and test stubs must override the methods if callers use them, as the +default implementation throws `UnsupportedOperationException`. + +### Runtime Review Comment: Two runtime behavior changes with JIRA release notes aren't here yet: - The Adaptive Scheduler now actively triggers a checkpoint when it wants to rescale, through the new `jobmanager.adaptive-scheduler.rescale-trigger.active-checkpoint.enabled` (default `true`) ([FLINK-36753](https://issues.apache.org/jira/browse/FLINK-36753)). Its JIRA note spells the key `obmanager.`, in case it gets copied. - `GET /jobs/overview`, `/jobs` and `/overview` now return an error instead of a partial job list when a JobMaster can't be queried ([FLINK-40602](https://issues.apache.org/jira/browse/FLINK-40602)). Should these go in this section? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
