MartijnVisser opened a new pull request, #29003:
URL: https://github.com/apache/flink/pull/29003
What is the purpose of the change
A streaming query that sorts on a non-time attribute (e.g. `SELECT a FROM t
ORDER BY c`, where c is not an ascending time attribute) is not supported.
Today that rejection is thrown from `StreamExecSort.translateToPlanInternal`,
i.e. during execution-plan translation, so it only surfaces when the job is
(re)submitted rather than during optimization. As a result COMPILE PLAN FOR
'<such a query>' succeeds and the statement fails only later; in
SQL-gateway/REST flows this can lead to a restart loop before the deterministic
error reaches the user.
This change rejects the sort during optimization instead, so it is reported
by `optimize()` / `COMPILE PLAN` before job submission. It also replaces the
previous message (which was inaccurate for a descending time attribute, it
claimed "non-time-attribute" even when the column was a time attribute) with
two targeted messages.
Brief change log
- `StreamPhysicalSortRule.convert()` rejects a non-temporal streaming sort
during the physical optimization phase (consistent with e.g.
`StreamPhysicalOverAggregateRule`), reusing `SortUtil.getFirstSortField`.
- Two error messages live in `SortUtil` and are shared by the rule and the
exec node: one for a non-time-attribute primary key, one for a time attribute
sorted DESC.
- The `StreamExecSort.translateToPlanInternal` check is kept as a backstop
for compiled plans loaded via loadPlan, which bypasses `optimize()`.
Verifying this change
This change added tests and can be verified as follows:
- Added `SortValidationTest` (parameterized over the non-temporal sort
shapes, plus a `COMPILE PLAN` case) asserting the rejection now happens at
planning time.
- Adapted existing tests whose ORDER BY became invalid in streaming: those
that sorted only incidentally still assert their feature
(`TableSinkTest.testDistribution`, `GroupingSetsTest.testFromBlogspot`); the
rest assert the rejection
(`GroupingSetsTest.testRollupPlusOrderBy`/`testGroupingInOrderByClause`,
streaming `PartialInsertTest.testPartialInsertWithOrderBy`,
`SortITCase.testDisableSortNonTemporalField`).
- `mvn verify` passes for `flink-table-planner,` including `SortITCase`
end-to-end.
Note: because the exception is thrown from a rule during the Volcano phase,
the top-level message is wrapped by `FlinkVolcanoProgram` ("Sql optimization:
…") with the original text as the root cause, that's the same shape as other
rule-phase rejections (GROUPING SETS, OVER aggregates).
Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with
@Public(Evolving): no
- The serializers: no (CompiledPlan JSON schema for StreamExecSort is
unchanged)
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
- The S3 file system connector: no
Documentation
- Does this pull request introduce a new feature? no
---
Was generative AI tooling used to co-author this PR?
- [x] Yes (please specify the tool below)
Generated-by: Claude Code (Claude Opus 4.8)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
