twalthr opened a new pull request, #29987:
URL: https://github.com/apache/flink/pull/29987

   ## What is the purpose of the change
   
   Stream physical nodes passed `InputProperty.DEFAULT` (`UNKNOWN` 
distribution) to their ExecNodes, even when their physical rule had requested a 
SINGLETON, HASH, or BROADCAST distribution. Since FLINK-25995, 
`ExecNodeBase#inputsContainSingleton()` relies on the node's own input 
properties. As a result, singleton operators implementing 
`MultipleTransformationTranslator` were not pinned to max parallelism 1. These 
are `StreamExecIntervalJoin` (without equi-join keys), `StreamExecMatch` 
(without `PARTITION BY`), and `StreamExecTemporalSort`. Other singleton 
operators are covered by the `GlobalPartitioner` fallback that FLINK-30814 
added for `SingleTransformationTranslator`s.
   
   When such an operator is rescaled (adaptive scheduler, 
`pipeline.jobvertex-parallelism-overrides`, ...), the `GlobalPartitioner` still 
sends all records to subtask 0. The key group of the empty state key is 
generally owned by another subtask, so the job fails with:
   
   ```
   java.lang.IllegalArgumentException: Key group X is not in KeyGroupRange{...}
   ```
   
   This PR makes stream physical nodes, and the remaining batch physical nodes, 
declare the distribution requested by their physical rule. The declaration is 
taken from the input's distribution trait. These rules request distributions 
strictly, and a strict HASH or SINGLETON request is only satisfied by an 
identical trait, so the input's trait is exactly the requested distribution. 
The batch sort, whose rule may request a range distribution, only declares 
SINGLETON.
   
   ## Brief change log
   
   - Add `InputProperty#fromInput(RelNode)` and 
`InputProperty#toRequiredDistribution(RelDistribution)`. 
`CommonPhysicalExchange` and `StreamPhysicalMultiJoin` now use them instead of 
their own copies of the mapping.
   - Declare the requested input distribution in all stream physical nodes 
whose rule requests one:
     - joins: regular, interval, temporal, window, lateral snapshot, delta;
     - aggregations: group, global, incremental, table, window, Python, over;
     - rank/deduplicate, sort, sort limit, limit, temporal sort, 
`MATCH_RECOGNIZE`, window rank/deduplicate, changelog normalize.
   - Conditional declarations, mirroring the rules: session window table 
functions, lookup joins with the hash shuffle of the `LOOKUP` hint (moved to 
`CommonPhysicalLookupJoin#deriveInputProperty`, shared with batch), and PTF 
table arguments with set or broadcast semantics.
   - Nodes without a distribution requirement (Calc, Correlate, Union, local 
aggregates, ...) keep `InputProperty.DEFAULT`, as in batch.
   - Declare the requested input distribution in the remaining batch physical 
nodes whose rule requests one:
     - `BatchPhysicalMatch` (SINGLETON or HASH, requested strictly) and 
`BatchPhysicalLookupJoin` with the hash shuffle of the `LOOKUP` hint;
     - `BatchPhysicalSort`: SINGLETON when its input is singleton (global 
`ORDER BY`, sort in front of a global rank). Its rule may request a range 
distribution, so only the singleton case is declared 
(`InputProperty#fromSingletonInput`).
   
     Previously only the `GlobalPartitioner` fallback (FLINK-30814) pinned 
these. With a declared HASH, batch `MATCH_RECOGNIZE` now gets the same 
treatment by `ForwardHashExchangeProcessor` as other hash-consuming batch 
nodes: when the hash exchange was pushed below a Calc, a `keep_input_as_is` 
exchange is added in front of the Match, so the hash distribution survives if 
the adaptive batch scheduler picks different parallelisms for Calc and Match 
(one batch plan baseline changes).
   - Extract the `GlobalPartitioner` fallback of `ExecNodeBase` into 
`hasSingletonExchangeInput` and document why it is kept. It is still needed for:
     - compiled plans from before this change, which contain UNKNOWN input 
properties;
     - nodes without a distribution requirement (e.g. Calc) that directly 
consume a singleton exchange.
   
   ## Verifying this change
   
   This change added tests and can be verified as follows:
   
   - Added `TransformationsTest#testSingletonInputPinsParallelism`. Before this 
change it fails for:
     - interval join, temporal sort, and `MATCH_RECOGNIZE`;
     - a `LIMIT` over a `LIMIT`. Here the SINGLETON input is satisfied by the 
inner limit without an exchange, so the `GlobalPartitioner` fallback does not 
apply.
   - Added `InputPropertyTest`. It checks the declared distributions of the 
nodes that request a distribution only conditionally:
     - session vs. other window table functions;
     - lookup joins with and without the shuffle hint;
     - a PTF with a set semantic table argument, with and without `PARTITION 
BY`.
   
     It also checks one node that always requests a distribution, and the batch 
nodes: `MATCH_RECOGNIZE` with and without `PARTITION BY`, the lookup join with 
the shuffle hint, and the sort (SINGLETON for a global `ORDER BY`, undeclared 
on a hash input).
   - Restore tests whose compiled plans declare SINGLETON inputs:
     - new `interval-join-event-time-without-equi-keys`;
     - the existing `MATCH_ORDER_BY_EVENT_TIME` program (`MATCH_RECOGNIZE` 
without `PARTITION BY`) is now also registered in the stream 
`MatchRecognizeRestoreTest`.
   - Existing restore tests (plans with `UNKNOWN` distribution) pass unchanged, 
so existing compiled plans keep their max parallelism and stay restorable.
   - Updated compiled plan test baselines. The only change is 
`requiredDistribution` going from `UNKNOWN` to `HASH`/`SINGLETON`.
   - Updated the batch plan baseline of `MatchRecognizeTest#testCascadeMatch` 
(additional `keep_input_as_is` exchange).
   
   ## Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency): no
     - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: no
     - The serializers: no
     - The runtime per-record code paths (performance sensitive): no
     - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
       - Newly compiled plans pin singleton operators to max parallelism 1. 
Existing compiled plans are not affected.
     - The S3 file system connector: no
   
   ## Documentation
   
     - Does this pull request introduce a new feature? no
     - If yes, how is the feature documented? not applicable
   
   ---
   
   ##### Was generative AI tooling used to co-author this PR?
   
   - [X] Yes (please specify the tool below)
   
   Generated-by: Claude Code (Claude Opus 5.5)
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to