wenzhenghu opened a new issue, #65750: URL: https://github.com/apache/doris/issues/65750
## Search before asking I searched existing Apache Doris issues and did not find an issue covering these HMS incremental-event cache consistency problems. ## Version Apache Doris `master` at `3a75387e61388bd886eeef38b794d5ed4eb298bf`. These problems were found while reviewing #65126, but they are already present on the current master baseline and were not introduced by that PR. ## What's Wrong? There are four independent correctness gaps in HMS incremental event handling. ### 1. ALTER TABLE/ALTER DATABASE rename can be incorrectly cancelled by a load-through "local existence" check `AlterTableEvent.processRename()` calls `externalTableExistInLocal()` before applying the rename: https://github.com/apache/doris/blob/3a75387e61388bd886eeef38b794d5ed4eb298bf/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/event/AlterTableEvent.java#L107-L123 However, `HMSExternalCatalog.tableExistInLocal()` eventually calls the normal load-through table lookup rather than a cache-only lookup: https://github.com/apache/doris/blob/3a75387e61388bd886eeef38b794d5ed4eb298bf/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HMSExternalCatalog.java#L190-L198 After HMS has renamed `old_table` to `new_table`, querying `new_table` from this check can load it from HMS and return `true`. The event is then cancelled, and cached state for `old_table` may remain. `AlterDatabaseEvent.processRename()` has the same structural problem through `catalog.getDbNullable(dbAfter.getName())`, although database rename is less commonly reachable in standard Hive. ### 2. ADD/DROP PARTITION events do not fence an in-flight partition-values load, and DROP may skip lower-level cache invalidation Both `addPartitionsCache()` and `dropPartitionsCache()` return immediately when the partition-values entry is absent: https://github.com/apache/doris/blob/3a75387e61388bd886eeef38b794d5ed4eb298bf/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HiveExternalMetaCache.java#L707-L746 https://github.com/apache/doris/blob/3a75387e61388bd886eeef38b794d5ed4eb298bf/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HiveExternalMetaCache.java#L748-L789 This causes two problems: * If a partition-values load started before the event, the event does not invalidate/bump the entry, so the pre-event snapshot may still be published afterward. * If the partition-values entry has been evicted while partition metadata or file listings remain cached, a DROP PARTITION event returns before invalidating those lower-level entries. The second case can expose stale data when a partition name is later recreated with a different path or files. It also prevents the cache-miss fallback added by #65334 from being reached through this DROP event path. ### 3. Incremental CREATE_DATABASE/CREATE_TABLE events can bypass include/exclude visibility filters Full names loading applies `include_database_list`, `exclude_database_list`, and `include_table_list`: https://github.com/apache/doris/blob/3a75387e61388bd886eeef38b794d5ed4eb298bf/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java#L551-L616 https://github.com/apache/doris/blob/3a75387e61388bd886eeef38b794d5ed4eb298bf/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java#L343-L356 The incremental register paths directly update the legacy metadata cache without applying the same visibility policy: https://github.com/apache/doris/blob/3a75387e61388bd886eeef38b794d5ed4eb298bf/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HMSExternalCatalog.java#L205-L216 https://github.com/apache/doris/blob/3a75387e61388bd886eeef38b794d5ed4eb298bf/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalDatabase.java#L649-L663 When a names cache is already hot, an HMS CREATE event can therefore add a database/table that should remain hidden. The object can subsequently become queryable through the cached name. ### 4. DROP_PARTITION records a DATABASE deletion in ExternalMetaIdMgr `DropPartitionEvent.transferToMetaIdMappings()` uses `META_OBJECT_TYPE_DATABASE` instead of `META_OBJECT_TYPE_PARTITION`: https://github.com/apache/doris/blob/3a75387e61388bd886eeef38b794d5ed4eb298bf/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/event/DropPartitionEvent.java#L142-L152 `ExternalMetaIdMgr` interprets this as a request to remove the entire database mapping subtree: https://github.com/apache/doris/blob/3a75387e61388bd886eeef38b794d5ed4eb298bf/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalMetaIdMgr.java#L145-L169 Current production use of the ExternalMetaIdMgr lookup APIs is limited, so the immediate query impact appears limited, but the persisted/replayed external metadata ID state is incorrect. ## What You Expected? * Rename events should inspect only locally published cache state and should always remove the old local name. * Partition events should prevent pre-event snapshots from being published and should invalidate partition/file caches even when partition-values cache state is cold. * Incremental create events should apply the same visibility rules as full names loading. * DROP_PARTITION should delete only the corresponding partition metadata-ID mapping. ## How to Reproduce? ### Rename 1. Enable HMS incremental event synchronization. 2. Ensure the target name is not locally cached. 3. Rename `old_table` to `new_table` in Hive. 4. Process the ALTER_TABLE event. 5. Observe that the existence check loads `new_table` from HMS and cancels the event; cached `old_table` state may remain. ### Partition cache 1. Cache partition metadata/file listings for `p=1`. 2. Evict or invalidate only the table's partition-values entry, or block an in-flight partition-values load. 3. Drop `p=1` through HMS and process the DROP_PARTITION event. 4. Recreate `p=1` with a different location/files. 5. The old partition metadata or file listing may still be reused. ### Visibility filter 1. Create an HMS catalog with an include/exclude database filter or `include_table_list`. 2. Warm the corresponding names cache. 3. Create a filtered-out database/table directly in Hive. 4. Process the HMS CREATE event. 5. Observe that the filtered object is inserted into the cached names and becomes visible/queryable. ### External metadata ID mapping 1. Add a database/table/partition mapping to `ExternalMetaIdMgr`. 2. Process the partition's DROP_PARTITION mapping. 3. Observe that the database mapping subtree is removed instead of only the partition mapping. ## Suggested Fixes 1. Introduce a cache-only local-existence check for rename handling. Always invalidate the old name; only skip publishing the new object when it is already locally present. 2. For ADD/DROP PARTITION, invalidate or bump the partition-values key when it is cold/loading. DROP should unconditionally invalidate partition metadata/file cache entries for each dropped partition before optionally updating a hot partition-values snapshot. 3. Reuse a shared database/table visibility predicate in both full names loading and incremental registration. 4. Change the DROP_PARTITION mapping type to `META_OBJECT_TYPE_PARTITION` and add an event mapping unit test. The four fixes can be implemented as separate PRs if preferred. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
