924060929 commented on code in PR #68196:
URL: https://github.com/apache/doris/pull/68196#discussion_r4131132566
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java:
##########
@@ -714,26 +711,38 @@ public void resetToUninitialized(boolean invalidCache) {
*/
public void onRefreshCache(boolean invalidCache) {
setLastUpdateTime(System.currentTimeMillis());
- refreshMetaCacheOnly(invalidCache);
- if (invalidCache) {
- Env.getCurrentEnv().getExtMetaCacheMgr().invalidateCatalog(id);
+ try {
+ refreshMetaCacheOnly(invalidCache);
+ } finally {
+ if (invalidCache) {
+ Env.getCurrentEnv().getExtMetaCacheMgr().invalidateCatalog(id);
+ }
}
}
/**
* Refresh meta cache only (database level cache), without invalidating
catalog level cache.
*/
- private void refreshMetaCacheOnly(boolean invalidCache) {
+ private synchronized void refreshMetaCacheOnly(boolean invalidCache) {
Review Comment:
Fixed in 32828a485c6. now swaps the names/object generations under the
catalog monitor, then runs retired-cache removal callbacks after releasing it.
The catalog-wide row-count opening fence remains before the swap and the
metadata-only closing fence remains after callbacks. A latch test verifies a
second thread can acquire the catalog monitor while the callback is running;
CatalogMgrTest and the FE build pass.
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/HiveInsertExecutor.java:
##########
@@ -83,17 +83,40 @@ protected void doBeforeCommit() throws UserException {
protected void doAfterCommit() throws DdlException {
HMSExternalTable hmsTable = (HMSExternalTable) table;
+ // The transaction is already committed. Fence the row-count cache by
the held table
+ // identity before any fallible cache work (including
isPartitionedTable reinitialization),
+ // so an evicted or partially reloaded table cannot retain the
pre-insert count.
+
Env.getCurrentEnv().getExtMetaCacheMgr().invalidateRowCountCache(hmsTable);
+
// For partitioned tables, do selective partition refresh
// For non-partitioned tables, do full table cache invalidation
List<String> modifiedPartNames = Lists.newArrayList();
List<String> newPartNames = Lists.newArrayList();
- if (hmsTable.isPartitionedTable() && partitionUpdates != null &&
!partitionUpdates.isEmpty()) {
- HiveExternalMetaCache cache =
Env.getCurrentEnv().getExtMetaCacheMgr()
- .hive(hmsTable.getCatalog().getId());
- cache.refreshAffectedPartitions(hmsTable, partitionUpdates,
modifiedPartNames, newPartNames);
- } else {
- // Non-partitioned table or no partition updates, do full table
refresh
-
Env.getCurrentEnv().getExtMetaCacheMgr().invalidateTableCache(hmsTable);
+ try {
+ if (hmsTable.isPartitionedTable() && partitionUpdates != null &&
!partitionUpdates.isEmpty()) {
+ HiveExternalMetaCache cache =
Env.getCurrentEnv().getExtMetaCacheMgr()
+ .hive(hmsTable.getCatalog().getId());
+ cache.refreshAffectedPartitions(hmsTable, partitionUpdates,
modifiedPartNames, newPartNames);
+ // Close the admission window opened by the fence above: a
load admitted after it can
+ // compute the pre-insert value from the still-resident file
list and publish it.
+
Env.getCurrentEnv().getExtMetaCacheMgr().invalidateRowCountCache(hmsTable);
Review Comment:
The held HMS table metadata is now retired on successful selective refresh,
leader full refresh, and follower/event paths. This follow-up fixes the
selective replay failure branch as well: even if the conservative full-cache
fallback throws, unsets the held table's state and closes the row-count
fence. injects failures in both cache operations and verifies both actions; 12
RefreshManagerTest cases pass on head 32828a485c6.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
