weiqingy commented on code in PR #895: URL: https://github.com/apache/flink-web/pull/895#discussion_r4638436350
########## docs/content/posts/2026-12-25-release-2.4.0.md: ########## @@ -0,0 +1,490 @@ +--- +title: "Apache Flink 2.4.0 Release Announcement" +date: "2026-12-25T00:00:00.000Z" +aliases: +- /news/2026/12/25/release-2.4.0.html +authors: +- flink: + name: "Apache Flink PMC" + +--- + +The Apache Flink PMC is pleased to announce the release of Apache Flink 2.4.0. + +This release expands Flink SQL with new join capabilities, more flexible changelog conversion, +and richer Process Table Functions. It also introduces a Python DataFrame API, makes application +state easier to explore through SQL, and improves the scalability of the HistoryServer. + +Key improvements include common subexpression elimination for SQL calculations, built-in UDF +metrics, early firing for outer interval joins, and a new `LATERAL SNAPSHOT` join. In-place +conversion to materialized tables simplifies adopting managed refresh pipelines, while state +inspection and recovery improvements help users operate stateful applications. + +We extend our heartfelt thanks to all contributors for making this release possible! + +Let's dive into the details. + +# Flink SQL Improvements + +## Upgrade Calcite to 1.41.0 + +Flink 2.4 upgrades Apache Calcite, the foundation of Flink's SQL parser and relational query +planner, to version 1.41.0. This brings Flink's planner onto a more recent Calcite release and +incorporates upstream fixes and improvements. + +The upgrade also provides the planner-level UUID type support used by the new native UUID +capabilities described below. + +**More Information** + +* [FLINK-39859](https://issues.apache.org/jira/browse/FLINK-39859) + +## Common Subexpression Elimination in Calc Code Generation + +SQL queries often repeat the same calculation in several output columns or in both a filter and +a projection. Previously, a Calc operator could evaluate each occurrence separately, even when +the expression was deterministic and its inputs were identical. + +Flink 2.4 introduces common subexpression elimination (CSE) in Calc code generation for both +batch and streaming execution. Reusable expressions are evaluated once and their results are +shared, reducing redundant work for built-in functions and user-defined functions: + +```sql +SELECT + expensive_udf(a, b) + 1 AS col1, + expensive_udf(a, b) + 2 AS col2 +FROM source_table +WHERE expensive_udf(a, b) > 0; +``` + +For a deterministic `expensive_udf`, the Calc operator can reuse the result across these +expressions. The optimization preserves short-circuit evaluation in expressions such as +`AND`, `OR`, and `CASE`. Functions with side effects or non-deterministic results must declare +`isDeterministic()` as `false` so that their calls are not incorrectly reused. + +**More Information** + +* [FLINK-39268](https://issues.apache.org/jira/browse/FLINK-39268) + +## Built-in Metrics for User-Defined Functions + +User-defined functions can be difficult to diagnose when they introduce latency or throw +exceptions. Flink 2.4 adds opt-in metrics for synchronous and asynchronous scalar and table +functions, making their behavior visible through the existing metrics reporters. + +Two metrics are registered per function on its executing operator: + +* **`udfProcessingTime`**: A histogram of sampled invocation durations. For asynchronous + functions, this measures the time from dispatch to completion. +* **`udfExceptionCount`**: A counter of exceptions surfaced by the function, including + exceptional completion of asynchronous calls. + +```yaml +# Enable UDF metrics (disabled by default) +table.exec.udf-metric-enabled: true + +# Time one invocation out of every 100 (the default sampling interval) +table.exec.udf-metric.sample-interval: 100 +``` + +Processing-time sampling limits measurement overhead; exceptions are counted on every +invocation. Aggregate functions and Process Table Functions are outside the scope of these +metrics. + +**More Information** + +* [FLIP-485](https://cwiki.apache.org/confluence/spaces/FLINK/pages/373885706/FLIP-485+Add+UDF+Metrics) +* [FLINK-38071](https://issues.apache.org/jira/browse/FLINK-38071) + +## Early Fire for Outer Interval Joins + +An outer interval join normally waits until an unmatched row's time interval closes before +emitting it with a null-padded counterpart. For long intervals, this can delay useful results. +Flink 2.4 introduces an `EARLY_FIRE` hint that emits the unmatched row after a configurable +delay: + +```sql +SELECT /*+ EARLY_FIRE('delay'='5s', 'time-mode'='proctime') */ + o.id, s.ship_time +FROM Orders o +LEFT JOIN Shipments s + ON o.id = s.order_id + AND o.order_time BETWEEN s.ship_time - INTERVAL '1' HOUR AND s.ship_time; +``` + +If a matching shipment arrives later within the interval, Flink retracts the speculative result +and emits the matched result. Downstream operators and sinks must therefore accept an updating +changelog. + +The hint supports event-time and processing-time delays. Each unmatched outer row fires at +most once; this is not a periodic trigger. It applies to `LEFT`, `RIGHT`, and `FULL` outer +interval joins and is independent of the early-fire configuration for window aggregations. + +**More Information** + +* [FLIP-497](https://cwiki.apache.org/confluence/spaces/FLINK/pages/334760951/FLIP-497+Early+Fire+Support+for+Flink+SQL+Interval+Join) +* [FLINK-36953](https://issues.apache.org/jira/browse/FLINK-36953) + +## FROM_CHANGELOG and TO_CHANGELOG: More Flexible Changelog Conversion + +Flink 2.4 extends `FROM_CHANGELOG` and `TO_CHANGELOG` with keyed conversion and additional +control over invalid operation codes and delete records. + +* **Keyed upsert conversion**: `FROM_CHANGELOG` supports producing an upsert table from an + append-only CDC stream by combining `PARTITION BY` with an operation mapping that includes + `UPDATE_AFTER` but not `UPDATE_BEFORE`. The partition columns must identify the upsert key. +* **Explicit error handling**: `error_handling` selects whether null or unknown input operation + codes fail the job (`FAIL`, the default) or are skipped (`SKIP`). +* **Full or partial deletes**: `TO_CHANGELOG` supports keyed conversion and a + `produces_full_deletes` parameter, controlling whether delete records contain the full row + or only the identifying key columns. +* **Partitioned Table API support**: `PartitionedTable` gains `fromChangelog()` and + `toChangelog()` convenience methods for keyed conversion. + +For example, a CDC stream containing insert, update-after, and delete events can be interpreted +as an upsert table: + +```sql +SELECT * FROM FROM_CHANGELOG( + input => TABLE cdc_events PARTITION BY id, + op_mapping => MAP[ + 'c', 'INSERT', + 'u', 'UPDATE_AFTER', + 'd', 'DELETE' + ] +); +``` + +The partition columns define the output's upsert key, so they must match the unique key of the +CDC data. Events for each key must arrive in the correct order. + +**More Information** + +* [FLIP-564](https://cwiki.apache.org/confluence/spaces/FLINK/pages/406620639/FLIP-564+Support+FROM_CHANGELOG+and+TO_CHANGELOG+built-in+PTFs) +* [FLINK-39258](https://issues.apache.org/jira/browse/FLINK-39258) + +## Process Table Functions: Lazy State Access and Broadcast State + +Flink 2.4 adds lazy value-state access and broadcast capabilities to Process Table Functions +for more advanced stateful SQL applications. + +The new `ValueView` provides explicit `value()`, `update()`, `isEmpty()`, and `clear()` Review Comment: I think the method names are `getValue()` and `setValue()`, not `value()` and `update()`. Those two belong to `ValueState`. Could we fix the names here? ########## docs/content/posts/2026-12-25-release-2.4.0.md: ########## @@ -0,0 +1,490 @@ +--- +title: "Apache Flink 2.4.0 Release Announcement" +date: "2026-12-25T00:00:00.000Z" +aliases: +- /news/2026/12/25/release-2.4.0.html +authors: +- flink: + name: "Apache Flink PMC" + +--- + +The Apache Flink PMC is pleased to announce the release of Apache Flink 2.4.0. + +This release expands Flink SQL with new join capabilities, more flexible changelog conversion, +and richer Process Table Functions. It also introduces a Python DataFrame API, makes application +state easier to explore through SQL, and improves the scalability of the HistoryServer. + +Key improvements include common subexpression elimination for SQL calculations, built-in UDF +metrics, early firing for outer interval joins, and a new `LATERAL SNAPSHOT` join. In-place +conversion to materialized tables simplifies adopting managed refresh pipelines, while state +inspection and recovery improvements help users operate stateful applications. + +We extend our heartfelt thanks to all contributors for making this release possible! + +Let's dive into the details. + +# Flink SQL Improvements + +## Upgrade Calcite to 1.41.0 + +Flink 2.4 upgrades Apache Calcite, the foundation of Flink's SQL parser and relational query +planner, to version 1.41.0. This brings Flink's planner onto a more recent Calcite release and +incorporates upstream fixes and improvements. + +The upgrade also provides the planner-level UUID type support used by the new native UUID +capabilities described below. + +**More Information** + +* [FLINK-39859](https://issues.apache.org/jira/browse/FLINK-39859) + +## Common Subexpression Elimination in Calc Code Generation + +SQL queries often repeat the same calculation in several output columns or in both a filter and +a projection. Previously, a Calc operator could evaluate each occurrence separately, even when +the expression was deterministic and its inputs were identical. + +Flink 2.4 introduces common subexpression elimination (CSE) in Calc code generation for both +batch and streaming execution. Reusable expressions are evaluated once and their results are +shared, reducing redundant work for built-in functions and user-defined functions: + +```sql +SELECT + expensive_udf(a, b) + 1 AS col1, + expensive_udf(a, b) + 2 AS col2 +FROM source_table +WHERE expensive_udf(a, b) > 0; +``` + +For a deterministic `expensive_udf`, the Calc operator can reuse the result across these +expressions. The optimization preserves short-circuit evaluation in expressions such as +`AND`, `OR`, and `CASE`. Functions with side effects or non-deterministic results must declare +`isDeterministic()` as `false` so that their calls are not incorrectly reused. + +**More Information** + +* [FLINK-39268](https://issues.apache.org/jira/browse/FLINK-39268) + +## Built-in Metrics for User-Defined Functions + +User-defined functions can be difficult to diagnose when they introduce latency or throw +exceptions. Flink 2.4 adds opt-in metrics for synchronous and asynchronous scalar and table +functions, making their behavior visible through the existing metrics reporters. + +Two metrics are registered per function on its executing operator: + +* **`udfProcessingTime`**: A histogram of sampled invocation durations. For asynchronous + functions, this measures the time from dispatch to completion. +* **`udfExceptionCount`**: A counter of exceptions surfaced by the function, including + exceptional completion of asynchronous calls. + +```yaml +# Enable UDF metrics (disabled by default) +table.exec.udf-metric-enabled: true + +# Time one invocation out of every 100 (the default sampling interval) +table.exec.udf-metric.sample-interval: 100 +``` + +Processing-time sampling limits measurement overhead; exceptions are counted on every +invocation. Aggregate functions and Process Table Functions are outside the scope of these +metrics. + +**More Information** + +* [FLIP-485](https://cwiki.apache.org/confluence/spaces/FLINK/pages/373885706/FLIP-485+Add+UDF+Metrics) +* [FLINK-38071](https://issues.apache.org/jira/browse/FLINK-38071) + +## Early Fire for Outer Interval Joins + +An outer interval join normally waits until an unmatched row's time interval closes before +emitting it with a null-padded counterpart. For long intervals, this can delay useful results. +Flink 2.4 introduces an `EARLY_FIRE` hint that emits the unmatched row after a configurable +delay: + +```sql +SELECT /*+ EARLY_FIRE('delay'='5s', 'time-mode'='proctime') */ + o.id, s.ship_time +FROM Orders o +LEFT JOIN Shipments s + ON o.id = s.order_id + AND o.order_time BETWEEN s.ship_time - INTERVAL '1' HOUR AND s.ship_time; +``` + +If a matching shipment arrives later within the interval, Flink retracts the speculative result +and emits the matched result. Downstream operators and sinks must therefore accept an updating +changelog. + +The hint supports event-time and processing-time delays. Each unmatched outer row fires at +most once; this is not a periodic trigger. It applies to `LEFT`, `RIGHT`, and `FULL` outer +interval joins and is independent of the early-fire configuration for window aggregations. + +**More Information** + +* [FLIP-497](https://cwiki.apache.org/confluence/spaces/FLINK/pages/334760951/FLIP-497+Early+Fire+Support+for+Flink+SQL+Interval+Join) +* [FLINK-36953](https://issues.apache.org/jira/browse/FLINK-36953) + +## FROM_CHANGELOG and TO_CHANGELOG: More Flexible Changelog Conversion + +Flink 2.4 extends `FROM_CHANGELOG` and `TO_CHANGELOG` with keyed conversion and additional +control over invalid operation codes and delete records. + +* **Keyed upsert conversion**: `FROM_CHANGELOG` supports producing an upsert table from an + append-only CDC stream by combining `PARTITION BY` with an operation mapping that includes + `UPDATE_AFTER` but not `UPDATE_BEFORE`. The partition columns must identify the upsert key. +* **Explicit error handling**: `error_handling` selects whether null or unknown input operation + codes fail the job (`FAIL`, the default) or are skipped (`SKIP`). +* **Full or partial deletes**: `TO_CHANGELOG` supports keyed conversion and a + `produces_full_deletes` parameter, controlling whether delete records contain the full row + or only the identifying key columns. +* **Partitioned Table API support**: `PartitionedTable` gains `fromChangelog()` and + `toChangelog()` convenience methods for keyed conversion. + +For example, a CDC stream containing insert, update-after, and delete events can be interpreted +as an upsert table: + +```sql +SELECT * FROM FROM_CHANGELOG( + input => TABLE cdc_events PARTITION BY id, + op_mapping => MAP[ + 'c', 'INSERT', + 'u', 'UPDATE_AFTER', + 'd', 'DELETE' + ] +); +``` + +The partition columns define the output's upsert key, so they must match the unique key of the +CDC data. Events for each key must arrive in the correct order. + +**More Information** + +* [FLIP-564](https://cwiki.apache.org/confluence/spaces/FLINK/pages/406620639/FLIP-564+Support+FROM_CHANGELOG+and+TO_CHANGELOG+built-in+PTFs) +* [FLINK-39258](https://issues.apache.org/jira/browse/FLINK-39258) + +## Process Table Functions: Lazy State Access and Broadcast State + +Flink 2.4 adds lazy value-state access and broadcast capabilities to Process Table Functions +for more advanced stateful SQL applications. + +The new `ValueView` provides explicit `value()`, `update()`, `isEmpty()`, and `clear()` +operations. A function can access a value only when it needs it, avoiding the need to eagerly +read and write every state entry for each invocation. + +Broadcast table arguments, declared with `BROADCAST_SEMANTIC_TABLE`, distribute control rows +to all parallel instances. Functions can keep shared rules or configuration in +`@StateHint(StateKind.BROADCAST)` state and consult it while processing their main inputs. +The `NOTIFY_STATEFUL_SETS` trait additionally allows a broadcast update to notify existing +stateful partitions. + +Together, these additions support patterns such as dynamically updated rule engines and +configuration-driven processing directly in PTFs. + +**More Information** + +* [FLIP-565](https://cwiki.apache.org/confluence/spaces/FLINK/pages/406620840/FLIP-565+Improve+ProcessTableFunctions+for+late+data+handling+and+state+access) +* [FLINK-39254](https://issues.apache.org/jira/browse/FLINK-39254) + +## A Test Harness for Process Table Functions + +Flink 2.4 introduces `ProcessTableFunctionTestHarness`, a lightweight framework for testing +PTF business logic without starting a full Flink cluster. + +The harness lets developers configure scalar and table arguments, feed records, advance +watermarks, and inspect emitted results. It also supports partitioned and multi-input +functions, state inspection and initialization, timers, and state TTL. Review Comment: I think the test harness does not support state TTL yet. The PTF docs (`ptfs.md`) list it as not supported. Could we remove "state TTL" from this sentence? ########## docs/content/posts/2026-12-25-release-2.4.0.md: ########## @@ -0,0 +1,490 @@ +--- +title: "Apache Flink 2.4.0 Release Announcement" +date: "2026-12-25T00:00:00.000Z" +aliases: +- /news/2026/12/25/release-2.4.0.html +authors: +- flink: + name: "Apache Flink PMC" + +--- + +The Apache Flink PMC is pleased to announce the release of Apache Flink 2.4.0. + +This release expands Flink SQL with new join capabilities, more flexible changelog conversion, +and richer Process Table Functions. It also introduces a Python DataFrame API, makes application +state easier to explore through SQL, and improves the scalability of the HistoryServer. + +Key improvements include common subexpression elimination for SQL calculations, built-in UDF +metrics, early firing for outer interval joins, and a new `LATERAL SNAPSHOT` join. In-place +conversion to materialized tables simplifies adopting managed refresh pipelines, while state +inspection and recovery improvements help users operate stateful applications. + +We extend our heartfelt thanks to all contributors for making this release possible! + +Let's dive into the details. + +# Flink SQL Improvements + +## Upgrade Calcite to 1.41.0 + +Flink 2.4 upgrades Apache Calcite, the foundation of Flink's SQL parser and relational query +planner, to version 1.41.0. This brings Flink's planner onto a more recent Calcite release and +incorporates upstream fixes and improvements. + +The upgrade also provides the planner-level UUID type support used by the new native UUID +capabilities described below. + +**More Information** + +* [FLINK-39859](https://issues.apache.org/jira/browse/FLINK-39859) + +## Common Subexpression Elimination in Calc Code Generation + +SQL queries often repeat the same calculation in several output columns or in both a filter and +a projection. Previously, a Calc operator could evaluate each occurrence separately, even when +the expression was deterministic and its inputs were identical. + +Flink 2.4 introduces common subexpression elimination (CSE) in Calc code generation for both +batch and streaming execution. Reusable expressions are evaluated once and their results are +shared, reducing redundant work for built-in functions and user-defined functions: + +```sql +SELECT + expensive_udf(a, b) + 1 AS col1, + expensive_udf(a, b) + 2 AS col2 +FROM source_table +WHERE expensive_udf(a, b) > 0; +``` + +For a deterministic `expensive_udf`, the Calc operator can reuse the result across these +expressions. The optimization preserves short-circuit evaluation in expressions such as +`AND`, `OR`, and `CASE`. Functions with side effects or non-deterministic results must declare +`isDeterministic()` as `false` so that their calls are not incorrectly reused. + +**More Information** + +* [FLINK-39268](https://issues.apache.org/jira/browse/FLINK-39268) + +## Built-in Metrics for User-Defined Functions + +User-defined functions can be difficult to diagnose when they introduce latency or throw +exceptions. Flink 2.4 adds opt-in metrics for synchronous and asynchronous scalar and table +functions, making their behavior visible through the existing metrics reporters. + +Two metrics are registered per function on its executing operator: + +* **`udfProcessingTime`**: A histogram of sampled invocation durations. For asynchronous + functions, this measures the time from dispatch to completion. +* **`udfExceptionCount`**: A counter of exceptions surfaced by the function, including + exceptional completion of asynchronous calls. + +```yaml +# Enable UDF metrics (disabled by default) +table.exec.udf-metric-enabled: true + +# Time one invocation out of every 100 (the default sampling interval) +table.exec.udf-metric.sample-interval: 100 +``` + +Processing-time sampling limits measurement overhead; exceptions are counted on every +invocation. Aggregate functions and Process Table Functions are outside the scope of these +metrics. + +**More Information** + +* [FLIP-485](https://cwiki.apache.org/confluence/spaces/FLINK/pages/373885706/FLIP-485+Add+UDF+Metrics) +* [FLINK-38071](https://issues.apache.org/jira/browse/FLINK-38071) + +## Early Fire for Outer Interval Joins + +An outer interval join normally waits until an unmatched row's time interval closes before +emitting it with a null-padded counterpart. For long intervals, this can delay useful results. +Flink 2.4 introduces an `EARLY_FIRE` hint that emits the unmatched row after a configurable +delay: + +```sql +SELECT /*+ EARLY_FIRE('delay'='5s', 'time-mode'='proctime') */ + o.id, s.ship_time +FROM Orders o +LEFT JOIN Shipments s + ON o.id = s.order_id + AND o.order_time BETWEEN s.ship_time - INTERVAL '1' HOUR AND s.ship_time; +``` + +If a matching shipment arrives later within the interval, Flink retracts the speculative result +and emits the matched result. Downstream operators and sinks must therefore accept an updating +changelog. + +The hint supports event-time and processing-time delays. Each unmatched outer row fires at +most once; this is not a periodic trigger. It applies to `LEFT`, `RIGHT`, and `FULL` outer +interval joins and is independent of the early-fire configuration for window aggregations. + +**More Information** + +* [FLIP-497](https://cwiki.apache.org/confluence/spaces/FLINK/pages/334760951/FLIP-497+Early+Fire+Support+for+Flink+SQL+Interval+Join) +* [FLINK-36953](https://issues.apache.org/jira/browse/FLINK-36953) + +## FROM_CHANGELOG and TO_CHANGELOG: More Flexible Changelog Conversion + +Flink 2.4 extends `FROM_CHANGELOG` and `TO_CHANGELOG` with keyed conversion and additional +control over invalid operation codes and delete records. + +* **Keyed upsert conversion**: `FROM_CHANGELOG` supports producing an upsert table from an + append-only CDC stream by combining `PARTITION BY` with an operation mapping that includes + `UPDATE_AFTER` but not `UPDATE_BEFORE`. The partition columns must identify the upsert key. +* **Explicit error handling**: `error_handling` selects whether null or unknown input operation + codes fail the job (`FAIL`, the default) or are skipped (`SKIP`). +* **Full or partial deletes**: `TO_CHANGELOG` supports keyed conversion and a + `produces_full_deletes` parameter, controlling whether delete records contain the full row + or only the identifying key columns. +* **Partitioned Table API support**: `PartitionedTable` gains `fromChangelog()` and + `toChangelog()` convenience methods for keyed conversion. + +For example, a CDC stream containing insert, update-after, and delete events can be interpreted +as an upsert table: + +```sql +SELECT * FROM FROM_CHANGELOG( + input => TABLE cdc_events PARTITION BY id, + op_mapping => MAP[ + 'c', 'INSERT', + 'u', 'UPDATE_AFTER', + 'd', 'DELETE' + ] +); +``` + +The partition columns define the output's upsert key, so they must match the unique key of the +CDC data. Events for each key must arrive in the correct order. + +**More Information** + +* [FLIP-564](https://cwiki.apache.org/confluence/spaces/FLINK/pages/406620639/FLIP-564+Support+FROM_CHANGELOG+and+TO_CHANGELOG+built-in+PTFs) +* [FLINK-39258](https://issues.apache.org/jira/browse/FLINK-39258) + +## Process Table Functions: Lazy State Access and Broadcast State + +Flink 2.4 adds lazy value-state access and broadcast capabilities to Process Table Functions +for more advanced stateful SQL applications. + +The new `ValueView` provides explicit `value()`, `update()`, `isEmpty()`, and `clear()` +operations. A function can access a value only when it needs it, avoiding the need to eagerly +read and write every state entry for each invocation. + +Broadcast table arguments, declared with `BROADCAST_SEMANTIC_TABLE`, distribute control rows +to all parallel instances. Functions can keep shared rules or configuration in +`@StateHint(StateKind.BROADCAST)` state and consult it while processing their main inputs. +The `NOTIFY_STATEFUL_SETS` trait additionally allows a broadcast update to notify existing +stateful partitions. + +Together, these additions support patterns such as dynamically updated rule engines and +configuration-driven processing directly in PTFs. + +**More Information** + +* [FLIP-565](https://cwiki.apache.org/confluence/spaces/FLINK/pages/406620840/FLIP-565+Improve+ProcessTableFunctions+for+late+data+handling+and+state+access) +* [FLINK-39254](https://issues.apache.org/jira/browse/FLINK-39254) + +## A Test Harness for Process Table Functions + +Flink 2.4 introduces `ProcessTableFunctionTestHarness`, a lightweight framework for testing +PTF business logic without starting a full Flink cluster. + +The harness lets developers configure scalar and table arguments, feed records, advance +watermarks, and inspect emitted results. It also supports partitioned and multi-input +functions, state inspection and initialization, timers, and state TTL. + +This makes it easier to test cases such as per-key state isolation, timer-driven output, and +argument validation in ordinary unit tests. Integration tests remain the way to verify the +complete SQL planner and runtime behavior. + +**More Information** + +* [FLIP-567](https://cwiki.apache.org/confluence/spaces/FLINK/pages/406622164/FLIP-567+Introduce+a+ProcessTableFunction+Test+Harness) +* [FLINK-39357](https://issues.apache.org/jira/browse/FLINK-39357) + +## Strict UTF-8 Validation for BYTES-to-STRING Casts + +Previously, casting bytes to a string silently replaced invalid UTF-8 sequences with the +Unicode replacement character. Flink 2.4 makes this conversion strict: `CAST(bytes AS STRING)` +now fails on malformed UTF-8, making data-quality problems visible. + +Two new functions let users choose how to handle these inputs: + +* `IS_VALID_UTF8(bytes)` checks whether a byte sequence is valid UTF-8. +* `MAKE_VALID_UTF8(bytes)` explicitly decodes bytes while replacing malformed sequences. + +```sql +-- Return NULL for invalid UTF-8 +SELECT TRY_CAST(raw_bytes AS STRING) FROM source_table; + +-- Preserve the previous replacement behavior explicitly +SELECT MAKE_VALID_UTF8(raw_bytes) FROM source_table; +``` + +Applications that need time to migrate can temporarily restore the previous cast behavior +with `table.exec.legacy-bytes-to-string-cast: true` (default: `false`). Connector authors can +opt into strict validation with `StringData.fromUtf8Bytes()`; the existing +`StringData.fromBytes()` API remains unchanged. + +**More Information** + +* [FLIP-568](https://cwiki.apache.org/confluence/spaces/FLINK/pages/406623034/FLIP-568+Strict+BYTES-to-STRING+CAST+with+UTF-8+Validation+Utilities) +* [FLINK-39600](https://issues.apache.org/jira/browse/FLINK-39600) + +## Metadata Filter Push-Down for Table Sources + +Source metadata, such as a record's timestamp, partition, or offset, can be useful for limiting +what a query reads. Flink 2.4 adds a dedicated push-down path for predicates on metadata +columns, including columns declared with `METADATA VIRTUAL`. + +Connectors opt in through `SupportsReadingMetadata` and receive predicates expressed in +metadata key names. They can use accepted predicates to reduce source reads, while filters +that still need evaluation remain in the query. The metadata predicates are also preserved +when compiling and restoring a query plan. + +This creates opportunities for metadata-aware pruning and seeking in supporting connectors. +Existing connectors retain their behavior until they implement the new capability. + +**More Information** + +* [FLIP-574](https://cwiki.apache.org/confluence/x/54EmGQ) +* [FLINK-39421](https://issues.apache.org/jira/browse/FLINK-39421) + +## In-place Conversion to Materialized Tables + +Adopting a materialized table previously required creating a separate object and changing its +consumers. Flink 2.4 allows `CREATE OR ALTER MATERIALIZED TABLE` to convert an existing regular +table in place, preserving its catalog identity and underlying storage while adding a managed +refresh pipeline. + +Conversion is disabled by default. Enable it in the configuration used to initialize the Review Comment: I don't think `SET` works for this option. Flink reads it only when the session starts, for example from `config.yaml`, so a later `SET` statement has no effect. Could we tell users to put it in `config.yaml` instead? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
