twalthr opened a new pull request, #29987:
URL: https://github.com/apache/flink/pull/29987
## What is the purpose of the change
Stream physical nodes passed `InputProperty.DEFAULT` (`UNKNOWN`
distribution) to their ExecNodes, even when their physical rule had requested a
SINGLETON, HASH, or BROADCAST distribution. Since FLINK-25995,
`ExecNodeBase#inputsContainSingleton()` relies on the node's own input
properties. As a result, singleton operators implementing
`MultipleTransformationTranslator` were not pinned to max parallelism 1. These
are `StreamExecIntervalJoin` (without equi-join keys), `StreamExecMatch`
(without `PARTITION BY`), and `StreamExecTemporalSort`. Other singleton
operators are covered by the `GlobalPartitioner` fallback that FLINK-30814
added for `SingleTransformationTranslator`s.
When such an operator is rescaled (adaptive scheduler,
`pipeline.jobvertex-parallelism-overrides`, ...), the `GlobalPartitioner` still
sends all records to subtask 0. The key group of the empty state key is
generally owned by another subtask, so the job fails with:
```
java.lang.IllegalArgumentException: Key group X is not in KeyGroupRange{...}
```
This PR makes stream physical nodes, and the remaining batch physical nodes,
declare the distribution requested by their physical rule. The declaration is
taken from the input's distribution trait. These rules request distributions
strictly, and a strict HASH or SINGLETON request is only satisfied by an
identical trait, so the input's trait is exactly the requested distribution.
The batch sort, whose rule may request a range distribution, only declares
SINGLETON.
## Brief change log
- Add `InputProperty#fromInput(RelNode)` and
`InputProperty#toRequiredDistribution(RelDistribution)`.
`CommonPhysicalExchange` and `StreamPhysicalMultiJoin` now use them instead of
their own copies of the mapping.
- Declare the requested input distribution in all stream physical nodes
whose rule requests one:
- joins: regular, interval, temporal, window, lateral snapshot, delta;
- aggregations: group, global, incremental, table, window, Python, over;
- rank/deduplicate, sort, sort limit, limit, temporal sort,
`MATCH_RECOGNIZE`, window rank/deduplicate, changelog normalize.
- Conditional declarations, mirroring the rules: session window table
functions, lookup joins with the hash shuffle of the `LOOKUP` hint (moved to
`CommonPhysicalLookupJoin#deriveInputProperty`, shared with batch), and PTF
table arguments with set or broadcast semantics.
- Nodes without a distribution requirement (Calc, Correlate, Union, local
aggregates, ...) keep `InputProperty.DEFAULT`, as in batch.
- Declare the requested input distribution in the remaining batch physical
nodes whose rule requests one:
- `BatchPhysicalMatch` (SINGLETON or HASH, requested strictly) and
`BatchPhysicalLookupJoin` with the hash shuffle of the `LOOKUP` hint;
- `BatchPhysicalSort`: SINGLETON when its input is singleton (global
`ORDER BY`, sort in front of a global rank). Its rule may request a range
distribution, so only the singleton case is declared
(`InputProperty#fromSingletonInput`).
Previously only the `GlobalPartitioner` fallback (FLINK-30814) pinned
these. With a declared HASH, batch `MATCH_RECOGNIZE` now gets the same
treatment by `ForwardHashExchangeProcessor` as other hash-consuming batch
nodes: when the hash exchange was pushed below a Calc, a `keep_input_as_is`
exchange is added in front of the Match, so the hash distribution survives if
the adaptive batch scheduler picks different parallelisms for Calc and Match
(one batch plan baseline changes).
- Extract the `GlobalPartitioner` fallback of `ExecNodeBase` into
`hasSingletonExchangeInput` and document why it is kept. It is still needed for:
- compiled plans from before this change, which contain UNKNOWN input
properties;
- nodes without a distribution requirement (e.g. Calc) that directly
consume a singleton exchange.
## Verifying this change
This change added tests and can be verified as follows:
- Added `TransformationsTest#testSingletonInputPinsParallelism`. Before this
change it fails for:
- interval join, temporal sort, and `MATCH_RECOGNIZE`;
- a `LIMIT` over a `LIMIT`. Here the SINGLETON input is satisfied by the
inner limit without an exchange, so the `GlobalPartitioner` fallback does not
apply.
- Added `InputPropertyTest`. It checks the declared distributions of the
nodes that request a distribution only conditionally:
- session vs. other window table functions;
- lookup joins with and without the shuffle hint;
- a PTF with a set semantic table argument, with and without `PARTITION
BY`.
It also checks one node that always requests a distribution, and the batch
nodes: `MATCH_RECOGNIZE` with and without `PARTITION BY`, the lookup join with
the shuffle hint, and the sort (SINGLETON for a global `ORDER BY`, undeclared
on a hash input).
- Restore tests whose compiled plans declare SINGLETON inputs:
- new `interval-join-event-time-without-equi-keys`;
- the existing `MATCH_ORDER_BY_EVENT_TIME` program (`MATCH_RECOGNIZE`
without `PARTITION BY`) is now also registered in the stream
`MatchRecognizeRestoreTest`.
- Existing restore tests (plans with `UNKNOWN` distribution) pass unchanged,
so existing compiled plans keep their max parallelism and stay restorable.
- Updated compiled plan test baselines. The only change is
`requiredDistribution` going from `UNKNOWN` to `HASH`/`SINGLETON`.
- Updated the batch plan baseline of `MatchRecognizeTest#testCascadeMatch`
(additional `keep_input_as_is` exchange).
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
- Newly compiled plans pin singleton operators to max parallelism 1.
Existing compiled plans are not affected.
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? not applicable
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (please specify the tool below)
Generated-by: Claude Code (Claude Opus 5.5)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]