autophagy commented on code in PR #29957: URL: https://github.com/apache/flink/pull/29957#discussion_r4642565772
########## docs/content/release-notes/flink-2.4.md: ########## @@ -0,0 +1,359 @@ +--- +title: "Release Notes - Flink 2.4" +--- + +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Release notes - Flink 2.4 + +These release notes discuss important aspects, such as configuration, behavior or dependencies, +that changed between Flink 2.3 and Flink 2.4. Please read these notes carefully if you are +planning to upgrade your Flink version to 2.4. + +### Table SQL / API + +#### Upgrade Calcite to 1.41.0 + +##### [FLINK-39859](https://issues.apache.org/jira/browse/FLINK-39859) + +Flink 2.4 upgrades Apache Calcite, the foundation of the SQL parser and optimizer, to version +1.41.0. This incorporates the fixes and improvements from the intervening Calcite releases. +Applications that depend directly on Calcite classes or extend Flink's planner should check +their dependencies and custom planner code against the upgraded version. + +#### Common subexpression elimination in Calc code generation + +##### [FLINK-39268](https://issues.apache.org/jira/browse/FLINK-39268) + +Flink 2.4 avoids redundant evaluation of common subexpressions within a Calc operator, which +combines filtering and projection. Repeated deterministic expressions can reuse an already +computed result instead of being evaluated again for the same input row. This applies to both +batch and streaming queries and can reduce the cost of repeated built-in or user-defined +function calls. + +Reuse respects conditional evaluation, such as short-circuiting in `AND`, `OR`, and `CASE`. +UDF authors should ensure that `isDeterministic()` correctly describes their function: +functions with side effects or results that vary for identical inputs must return `false`. + +#### Built-in UDF metrics + +##### [FLINK-38071](https://issues.apache.org/jira/browse/FLINK-38071) (FLIP-485) + +SQL and Table API users can now monitor UDF execution through two built-in metrics: +`udfProcessingTime`, a histogram of sampled invocation durations, and `udfExceptionCount`, +a counter of exceptions from the function. Metrics are registered on the executing operator +under `udf.<udf_name>`. For asynchronous functions, processing time includes the time from +dispatch until completion. + +The feature is opt-in: + +- `table.exec.udf-metric-enabled` (default: `false`) enables metric collection. +- `table.exec.udf-metric.sample-interval` (default: `100`) controls how often invocation + durations are measured. Exceptions are counted on every invocation. + +The metrics cover directly invoked synchronous and asynchronous scalar and table functions. +Python functions, aggregate functions, and Process Table Functions are not covered. Enabling +the metrics adds overhead; use sampling to control the cost. + +#### Early firing for interval joins + +##### [FLINK-36953](https://issues.apache.org/jira/browse/FLINK-36953) (FLIP-497) + +Outer interval joins can emit unmatched rows before their join interval closes. The new +`EARLY_FIRE` hint emits a speculative, null-padded result after a configured delay. If a +matching row arrives later within the interval, Flink corrects the speculative result with +update-before and update-after records. + +```sql +SELECT /*+ EARLY_FIRE('delay'='5s') */ o.id, s.ship_time +FROM Orders o +LEFT OUTER JOIN Shipments s +ON o.id = s.order_id +AND o.order_time BETWEEN s.ship_time - INTERVAL '1' HOUR AND s.ship_time; +``` + +The optional `time-mode` selects `rowtime` or `proctime`; it defaults to the time mode of the +interval join. Each unmatched outer row fires at most once. The feature is disabled unless +the hint is present and requires downstream operators and sinks that accept updates. +Existing interval joins without the hint retain their behavior. + +#### FROM_CHANGELOG and TO_CHANGELOG enhancements + +##### [FLINK-39258](https://issues.apache.org/jira/browse/FLINK-39258) (FLIP-564) + +Flink 2.4 adds partitioned processing and additional controls to the changelog conversion +PTFs introduced in Flink 2.3: + +- `PARTITION BY` support to co-locate related changes, with corresponding convenience methods + on partitioned tables in the Table API. +- Partitioned `FROM_CHANGELOG` calls can use the existing PTF `ORDER BY` support to reorder + CDC events per key using a watermarked time attribute. Late events are dropped by the + sorting operation. +- `error_handling => 'FAIL' | 'SKIP'` for `FROM_CHANGELOG` to control how NULL or unmapped + operation codes are handled. The default is `FAIL`. +- `produces_full_deletes` for `TO_CHANGELOG` to choose between full-row deletes and deletes + containing only identifying keys. The default is `true`. + +These additions make it easier to consume custom CDC streams and write changelog events to +append-only sinks. Users must still provide correctly ordered, complete change events for +the selected changelog mode; the functions do not reconstruct arbitrary missing before images. +See the [Changelog Conversion documentation](https://nightlies.apache.org/flink/flink-docs-release-2.4/docs/sql/reference/queries/changelog/) +for supported mappings and input requirements. + +#### Process Table Function state access and broadcast support + +##### [FLINK-39254](https://issues.apache.org/jira/browse/FLINK-39254) (FLIP-565) + +Flink 2.4 adds lazy value-state access and broadcast inputs to Process Table Functions: + +- `ValueView` provides lazy value-state access alongside `ListView` and `MapView`. A function + can read or update state only when needed, avoiding unnecessary serialization work. +- `BROADCAST_SEMANTIC_TABLE` lets a PTF receive a rule or control table on every parallel + instance. State declared with `@StateHint(StateKind.BROADCAST)` is writable from the + broadcast input and readable while processing the regular inputs. +- `NOTIFY_STATEFUL_SETS` lets a broadcast update trigger re-evaluation of existing stateful + partitions. This is opt-in because visiting all stateful partitions can be expensive. + +#### Process Table Function test harness + +##### [FLINK-39357](https://issues.apache.org/jira/browse/FLINK-39357) (FLIP-567) + +The new `ProcessTableFunctionTestHarness` lets developers test PTF business logic without +starting a Flink cluster. Tests can configure scalar and table arguments, partitioning, and +initial state, feed rows into individual inputs, and inspect the resulting output. + +The harness also supports advancing watermarks and the system clock, inspecting timers, Review Comment: Yes, @weiqingy is right. State TTL and system clock stuff was deliberately removed from the final implementation after discussion in some PRs. Changelog support didn't make the cut for 2.4. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
