Siyao Meng created HDDS-13583:
---------------------------------
Summary: Recon MetricsException: Metrics source userTableCache
already exists
Key: HDDS-13583
URL: https://issues.apache.org/jira/browse/HDDS-13583
Project: Apache Ozone
Issue Type: Bug
Components: Ozone Recon
Reporter: Siyao Meng
When running ozone dev cluster with docker compose, this shows up when I left
the cluster running for more than five minutes:
{code}
2025-08-16 13:36:27.862 | 2025-08-16 20:36:27,862 [Recon-SyncOM-0] INFO
impl.OzoneManagerServiceProviderImpl: Attempting to update Recon OM DB with new
snapshot located at: /data/metadata/om.snapshot.db_1755376585468
2025-08-16 13:36:27.904 | 2025-08-16 20:36:27,904 [Recon-SyncOM-0] INFO
recovery.ReconOmMetadataManagerImpl: Created OM DB handle from snapshot at
/data/metadata/om.snapshot.db_1755376585468.
2025-08-16 13:36:27.904 | 2025-08-16 20:36:27,904 [Recon-SyncOM-0] WARN
om.OmMetadataManagerImpl: Overwriting an existing table userTable:
TypedTable-userTable(String->PersistedUserVolumeInfo)
2025-08-16 13:36:27.911 | 2025-08-16 20:36:27,910 [Recon-SyncOM-0] ERROR
impl.OzoneManagerServiceProviderImpl: Unable to update Recon's metadata with
new OM DB.
2025-08-16 13:36:27.911 | org.apache.hadoop.metrics2.MetricsException: Metrics
source userTableCache already exists!
2025-08-16 13:36:27.911 | at
org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newSourceName(DefaultMetricsSystem.java:152)
2025-08-16 13:36:27.911 | at
org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.sourceName(DefaultMetricsSystem.java:125)
2025-08-16 13:36:27.911 | at
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:229)
2025-08-16 13:36:27.911 | at
org.apache.hadoop.hdds.utils.TableCacheMetrics.create(TableCacheMetrics.java:48)
2025-08-16 13:36:27.911 | at
org.apache.hadoop.hdds.utils.db.TypedTable.createCacheMetrics(TypedTable.java:436)
2025-08-16 13:36:27.911 | at
org.apache.hadoop.ozone.om.OmMetadataManagerImpl$TableInitializer.get(OmMetadataManagerImpl.java:1863)
2025-08-16 13:36:27.911 | at
org.apache.hadoop.ozone.om.OmMetadataManagerImpl$TableInitializer.get(OmMetadataManagerImpl.java:1846)
2025-08-16 13:36:27.911 | at
org.apache.hadoop.ozone.om.OmMetadataManagerImpl.initializeOmTables(OmMetadataManagerImpl.java:443)
2025-08-16 13:36:27.911 | at
org.apache.hadoop.ozone.recon.recovery.ReconOmMetadataManagerImpl.initializeNewRdbStore(ReconOmMetadataManagerImpl.java:103)
2025-08-16 13:36:27.911 | at
org.apache.hadoop.ozone.recon.recovery.ReconOmMetadataManagerImpl.updateOmDB(ReconOmMetadataManagerImpl.java:126)
2025-08-16 13:36:27.911 | at
org.apache.hadoop.ozone.recon.spi.impl.OzoneManagerServiceProviderImpl.updateReconOmDBWithNewSnapshot(OzoneManagerServiceProviderImpl.java:513)
2025-08-16 13:36:27.911 | at
org.apache.hadoop.ozone.recon.spi.impl.OzoneManagerServiceProviderImpl.syncDataFromOM(OzoneManagerServiceProviderImpl.java:709)
2025-08-16 13:36:27.911 | at
org.apache.hadoop.ozone.recon.spi.impl.OzoneManagerServiceProviderImpl.lambda$startSyncDataFromOM$2(OzoneManagerServiceProviderImpl.java:329)
2025-08-16 13:36:27.911 | at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
2025-08-16 13:36:27.911 | at
java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:358)
2025-08-16 13:36:27.911 | at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
2025-08-16 13:36:27.911 | at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
2025-08-16 13:36:27.911 | at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
2025-08-16 13:36:27.911 | at
java.base/java.lang.Thread.run(Thread.java:1583)
2025-08-16 13:36:27.911 | 2025-08-16 20:36:27,911 [Recon-SyncOM-0] INFO
recon.ReconContext: Update healthStatus of Recon from true to false.
2025-08-16 13:36:27.911 | 2025-08-16 20:36:27,911 [Recon-SyncOM-0] INFO
impl.OzoneManagerServiceProviderImpl: Sequence number after sync: 2386622
2025-08-16 13:37:27.912 | 2025-08-16 20:37:27,912 [Recon-SyncOM-0] INFO
impl.OzoneManagerServiceProviderImpl: Last known sequence number before sync:
2386622
2025-08-16 13:37:27.912 | 2025-08-16 20:37:27,912 [Recon-SyncOM-0] INFO
impl.OzoneManagerServiceProviderImpl: Seq number of Recon's OM DB : 2386622
2025-08-16 13:37:27.929 | 2025-08-16 20:37:27,928 [Recon-SyncOM-0] WARN
impl.OzoneManagerServiceProviderImpl: Unable to get and apply delta updates
from OM: Invalid transaction log iterator when getting updates since sequence
number 2386622, falling back to full snapshot
2025-08-16 13:37:27.929 | 2025-08-16 20:37:27,928 [Recon-SyncOM-0] INFO
impl.OzoneManagerServiceProviderImpl: Delta updates received from OM : 0 loops,
0 records
2025-08-16 13:37:27.929 | 2025-08-16 20:37:27,928 [Recon-SyncOM-0] INFO
impl.OzoneManagerServiceProviderImpl: Obtaining full snapshot from Ozone Manager
2025-08-16 13:37:27.929 | 2025-08-16 20:37:27,929 [Recon-SyncOM-0] INFO
impl.OzoneManagerServiceProviderImpl: Permissions for Recon DB directory
'/data/metadata/recon' meet the minimum required permissions '750'
2025-08-16 13:37:27.930 | 2025-08-16 20:37:27,930 [Recon-SyncOM-0] INFO
impl.OzoneManagerServiceProviderImpl: Successfully deleted existing OM DB
snapshot directory: /data/metadata/om.snapshot.db_1755376585468
{code}
To investigated whether this breaks things.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
