This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new f87373a2948 [fix](catalog) rebuild idToCatalog map after replay
(#43772)
f87373a2948 is described below
commit f87373a29489a4bb9df3f4ff304bb0cab54ed86d
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Thu Nov 14 16:23:43 2024 +0800
[fix](catalog) rebuild idToCatalog map after replay (#43772)
### What problem does this PR solve?
Problem Summary:
After deserialized from Gson, the ConcurrentHashMap may become normal
non-threadsafe HashMap,
This may causing following issue:
```
2024-11-12 16:43:53,101 ERROR (stateListener|13)
[Env.transferToNonMaster():1921] failed to transfer to non-master.
java.util.ConcurrentModificationException: null
at
java.util.LinkedHashMap$LinkedHashIterator.nextNode(LinkedHashMap.java:756)
~[?:?]
at
java.util.LinkedHashMap$LinkedValueIterator.next(LinkedHashMap.java:783) ~[?:?]
at
org.apache.doris.datasource.CatalogMgr.registerCatalogRefreshListener(CatalogMgr.java:802)
~[doris-fe.jar:1.2-SNAPSHOT]
at
org.apache.doris.catalog.Env.postProcessAfterMetadataReplayed(Env.java:1751)
~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.catalog.Env.transferToNonMaster(Env.java:1899)
~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.catalog.Env.access$1300(Env.java:347)
~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.catalog.Env$5.runOneCycle(Env.java:2850)
~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.common.util.Daemon.run(Daemon.java:119)
~[doris-fe.jar:1.2-SNAPSHOT]
```
This PR rebuild the `idToCatalog` and `nameToCatalog` map in
`gsonPostProcess()` of `CatalogMgr`,
to solve the issue.
---
.../main/java/org/apache/doris/datasource/CatalogMgr.java | 13 ++++++++++---
1 file changed, 10 insertions(+), 3 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogMgr.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogMgr.java
index 93a4c1b19e6..91273736ebf 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogMgr.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogMgr.java
@@ -93,9 +93,9 @@ public class CatalogMgr implements Writable,
GsonPostProcessable {
private final MonitoredReentrantReadWriteLock lock = new
MonitoredReentrantReadWriteLock(true);
@SerializedName(value = "idToCatalog")
- private final Map<Long, CatalogIf<? extends DatabaseIf<? extends
TableIf>>> idToCatalog = Maps.newConcurrentMap();
+ private Map<Long, CatalogIf<? extends DatabaseIf<? extends TableIf>>>
idToCatalog = Maps.newConcurrentMap();
// this map will be regenerated from idToCatalog, so not need to persist.
- private final Map<String, CatalogIf> nameToCatalog =
Maps.newConcurrentMap();
+ private Map<String, CatalogIf> nameToCatalog = Maps.newConcurrentMap();
// Use a separate instance to facilitate access.
// internalDataSource still exists in idToCatalog and nameToCatalog
@@ -817,10 +817,17 @@ public class CatalogMgr implements Writable,
GsonPostProcessable {
@Override
public void gsonPostProcess() throws IOException {
+ // After deserializing from Gson, the concurrent map may become a
normal map.
+ // So here we reconstruct the concurrent map.
+ Map<Long, CatalogIf<? extends DatabaseIf<? extends TableIf>>>
newIdToCatalog = Maps.newConcurrentMap();
+ Map<String, CatalogIf> newNameToCatalog = Maps.newConcurrentMap();
for (CatalogIf catalog : idToCatalog.values()) {
- nameToCatalog.put(catalog.getName(), catalog);
+ newNameToCatalog.put(catalog.getName(), catalog);
+ newIdToCatalog.put(catalog.getId(), catalog);
// ATTN: can not call catalog.getProperties() here, because
ResourceMgr is not replayed yet.
}
+ this.idToCatalog = newIdToCatalog;
+ this.nameToCatalog = newNameToCatalog;
internalCatalog = (InternalCatalog)
idToCatalog.get(InternalCatalog.INTERNAL_CATALOG_ID);
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
