songwdfu opened a new pull request, #15983:
URL: https://github.com/apache/pinot/pull/15983
This PR involves several additions / changes to MQE's pinot-query-planner:
1. Changes to Rules.
These changes to rules are tested with customized queries on TPC-H, which
show major improvements.
- Introduced `CoreRules.JOIN_PUSH_TRANSITIVE_PREDICATES` and
`CoreRules.AGGREGATE_JOIN_REMOVE` rule that are mostly risk free and helpful to
simplify plan tree.
- Modified `PinotAggregateReduceFunctionsRule` to only reduce SUM and AVG to
SUM0 when the resulting SUM0 could possibly benefit from
`CoreRules.AGGREGATE_CASE_TO_FILTER`. Enabling of this rule is configurable.
- Introduced `CoreRules.AGGREGATE_JOIN_TRANSPOSE_EXTENDED` that pushes
aggregate functions below joins when possible. Enabling of this is configurable.
- Added `PruneEmptyRules.CORRELATE_LEFT_INSTANCE` and its right equivalence
to remove dummy correlate left / right joins, which aren't supported.
- Added a round of `PRUNE_RULES` before applying `BASIC_RULES` to simplify
the operator tree for better rule matching.
- Minor re-ordering of `BASIC_RULES`
- For more detail and examples see 2af8808cd4791ede8f5af9b67a9c4230e9d042ac
and 19ba1022c4cd3302bc58241717fee2af8e709dda, the correlate remove rules are
added in 8f3619cf9c7ed9f3cb7f1895f2c202044e463d1e.
2. Enabled dynamically turning on / off rules with queryOptions.
Currently supported knobs `useAggregateFunctionsPushdownRule` and
`useCaseToFilterReduceSumAndAvgRule` as a PoC. This introduced a change that
now `optProgram` is also rebuilt per query (same as `traitProgram` now), this
simply iterate through the rule List and checks whether it is enabled. Takes ~3
ms on my laptop and should be fine.
For detail see 8f3619cf9c7ed9f3cb7f1895f2c202044e463d1e
3. Introduced a logger into QueryEnvironment that logs the parsed,
decorrelated, and trimmed plan, this is helpful for debugging QO process along
with Calcite logs.
The above changes are mostly tested by building and running queries.
Instructions:
1. The PR has to be tagged with at least one of the following labels (*):
1. `feature`
7. `bugfix`
8. `performance`
9. `ui`
10. `backward-incompat`
11. `release-notes` (**)
2. Remove these instructions before publishing the PR.
(*) Other labels to consider:
- `testing`
- `dependencies`
- `docker`
- `kubernetes`
- `observability`
- `security`
- `code-style`
- `extension-point`
- `refactor`
- `cleanup`
(**) Use `release-notes` label for scenarios like:
- New configuration options
- Deprecation of configurations
- Signature changes to public methods/interfaces
- New plugins added or old plugins removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
