924060929 commented on PR #67725:
URL: https://github.com/apache/doris/pull/67725#issuecomment-5600249038
A suggested long-term architecture, for clarity only — I do **not** think
this full refactor should be required in this PR:
The generic planner should model connector partition pruning as an explicit
result-producing operation rather than requiring every engine to materialize
the same eager `Map<String, PartitionItem>` lifecycle.
```text
LogicalFilter + LogicalFileScan
|
| partition-relevant conjuncts, snapshot and scan parameters
v
Connector partition-pruning interface
|
|-- Hive: HMS get_partitions_by_filter, with full-list fallback
|-- Iceberg/Paimon: manifest or SDK expression pruning
|-- Hudi: timeline and partition-path pruning
|-- MaxCompute: remote partition-spec pruning
v
PartitionPruningResult
- explicit state: deferred / materialized / unsupported
- selected partition domain or lazy partition source
- updated ConnectorTableHandle / connector-native metadata
- consumed conjunct indices and remaining predicate
- exact vs approximate/superset result
- OptionalLong totalPartitionCount, where unknown stays unknown
v
PhysicalPlanTranslator -> PluginDrivenScanNode
```
The important properties would be:
1. `LogicalFileScan` should not eagerly enumerate every partition before the
filter is available. The pruning rule should invoke the connector with only the
partition-relevant conjuncts.
2. Each connector may use its native pruning model. A connector that does
not have stable Hive-style partition names should not be forced through a
Hive-shaped map merely to participate.
3. The result should carry the updated handle or connector-native partition
metadata into the physical scan, so the scan does not repeat the same
metastore/SDK pruning call.
4. Exact consumed conjuncts may be removed from the upper scan predicate;
unsupported or approximate pushdown must remain as residual predicates for BE
evaluation.
5. A remote failure or unsupported dialect should return an explicit
unsupported/fallback result, not be inferred from an empty map or singleton
identity.
6. Unknown total partition count should remain unknown. State and control
flow should not be encoded in `totalPartitionNum`.
7. Full scans that need lazy or batched split generation should use an
explicit lazy partition source/state rather than being repaired in
`doFinalize()` after carrying an empty sentinel through logical planning.
This would support different external engines without adding engine-specific
branches to Nereids or coupling logical pruning state to `PluginDrivenScanNode`
finalization. For the current PR, a smaller safe implementation is reasonable;
the full result type, handle propagation and residual unification can be
tracked as maintainer-owned follow-up work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
