zhaorongsheng opened a new pull request, #67709:
URL: https://github.com/apache/doris/pull/67709
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: Hive partition pruning loaded every partition name into FE
before applying predicates. On tables with very large partition counts,
planning could wait for a full HMS listing and time out. This change pushes
supported partition-only predicates to HMS get_partitions_by_filter, expands IN
predicates into HMS-compatible OR expressions, caches the returned partition
metadata, and falls back to the existing local pruning path when HMS rejects a
filter. It also keeps batch split sources reachable until explicit query
cleanup and treats a released source as end of input instead of closing the
Thrift connection.
### Release note
Improve planning latency and stability for selective Hive partition queries.
### Check List (For Author)
- Test: Manual test
- Java HMS probe against music_dwd.dwd_log_music_user_action_di:
equality, IN/OR, and range filters succeeded; IN/OR returned 116 partitions.
- Rolled the FE jar through a three-FE test cluster and verified a query
with dt plus pt_action/pt_os IN predicates returns rows without setting
num_partitions_in_batch_mode.
- build.sh --fe succeeded.
- Behavior changed: Yes (supported Hive partition predicates are pushed to
HMS; rejected HMS filters fall back to local pruning)
- Does this need documentation: No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
