xiangfu0 opened a new pull request, #19311:
URL: https://github.com/apache/pinot/pull/19311
Adds row pattern recognition to the multi-stage query engine, following the
shape of the UNNEST support added in #17168: a new plan node, an
exchange-insertion rule, and an intermediate-stage operator.
Pinot's parser already accepted the full `MATCH_RECOGNIZE` grammar —
`Parser.jj` carries Calcite's Babel production. What was missing was
operator-table registration, validation, a plan node, and all of execution.
### Front end
- Register
`PREV`/`NEXT`/`FIRST`/`LAST`/`CLASSIFIER`/`MATCH_NUMBER`/`RUNNING`/`FINAL` in
`PinotOperatorTable`, which is a strict allow-list.
- New `MatchRecognizeValidator` runs on the `SqlNode` tree before
conversion. It rewrites an **omitted** `AFTER MATCH` clause to `SKIP PAST LAST
ROW`. Calcite substitutes `SKIP TO NEXT ROW`, but SQL:2016, Trino, Snowflake
and Oracle all default to `SKIP PAST LAST ROW`; the two differ in whether
matches may overlap, so a query ported from another engine would otherwise
silently return different rows. Conversion erases the omitted-vs-explicit
distinction, so the rewrite has to happen here.
- Deferred constructs are rejected at planning time with actionable messages
rather than mis-executing: `ALL ROWS PER MATCH`, `SUBSET`, `PERMUTE`, pattern
exclusions, `WITHIN`, aggregates in `DEFINE`, `NULLS FIRST/LAST`, and `ORDER
BY`/`PARTITION BY` on expressions (the last of which otherwise fail inside
`SqlToRelConverter` with `AssertionError`/`ClassCastException`).
### Plan and wire format
- `MatchNode` at `plan.proto` tag 19, encoding the pattern as a
self-contained recursive `RowPattern` with a symbol table rather than a
`RexCall` tree, with field numbers reserved for `SUBSET` and `WITHIN`.
- `PatternFieldRef` in `expressions.proto`, so `RexPatternFieldRef` can no
longer degrade to a plain `InputRef` and produce wrong-but-type-correct results.
- `PinotMatchExchangeNodeInsertRule` hash-distributes on the `PARTITION BY`
keys and prepends them to the sort collation, so rows arrive clustered and the
operator matches and flushes one partition at a time. A missing `PARTITION BY`
is rejected by default (it collapses the table onto one worker);
`allowMatchRecognizeWithoutPartitionBy` overrides.
### Runtime
- `PatternToNfaCompiler` builds an NFA with prioritized transitions —
alternation in source order, greedy takes the loop edge first and reluctant the
exit edge, `{n,m}` via counter registers rather than state unrolling — so
depth-first traversal yields the SQL:2016 preferred match first.
- `MatchOperator` evaluates `DEFINE` predicates over a classifier tape
supporting `PREV`/`NEXT`/`FIRST`/`LAST`/`CLASSIFIER`/`MATCH_NUMBER`, emits
`MEASURES`, and advances per the `AFTER MATCH SKIP` mode.
- Guardrails throw rather than truncate, since a truncated pattern result is
a silently wrong one: `maxRowsInMatch`, `maxStepsPerMatchAttempt`, and an
empty-cycle guard.
### Scope
v1 covers `PARTITION BY`, mandatory `ORDER BY`, `MEASURES`, `ONE ROW PER
MATCH`, all four `AFTER MATCH SKIP` modes, the full pattern algebra including
reluctant quantifiers and anchors, and single-variable aggregates in `MEASURES`.
`MATCH_RECOGNIZE` is **not** yet supported under the v2 physical optimizer
or lite mode. Those paths are covered by ignore flags in the tests rather than
returning wrong results.
### Testing
- Unit tests for the validator, plan node, rel conversion, exchange rule,
NFA compiler, operator and limits.
- `PlanNodeSerDeTest` round-trip coverage for the new node.
- Resource-based query tests, plus `MatchRecognizeIntegrationTest` running
against a real cluster (11 cases), including a three-way assertion that an
omitted `AFTER MATCH` matches explicit `SKIP PAST LAST ROW` and that explicit
`SKIP TO NEXT ROW` returns the extra overlapping rows.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
