[
https://issues.apache.org/jira/browse/HDDS-13583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krishna Kumar Asawa reassigned HDDS-13583:
------------------------------------------
Assignee: Arafat Khan
> Recon MetricsException: Metrics source userTableCache already exists
> --------------------------------------------------------------------
>
> Key: HDDS-13583
> URL: https://issues.apache.org/jira/browse/HDDS-13583
> Project: Apache Ozone
> Issue Type: Bug
> Components: Ozone Recon
> Reporter: Siyao Meng
> Assignee: Arafat Khan
> Priority: Major
>
> Commit HEAD is at
> [a5cb560cb5a41f7bd9f9fe7c7f4ca3cd65aaf8e2|https://github.com/apache/ozone/commits/a5cb560cb5a41f7bd9f9fe7c7f4ca3cd65aaf8e2]
> When running ozone dev cluster with docker compose, this shows up when I left
> the cluster running for more than five minutes:
> {code}
> 2025-08-16 13:36:27.862 | 2025-08-16 20:36:27,862 [Recon-SyncOM-0] INFO
> impl.OzoneManagerServiceProviderImpl: Attempting to update Recon OM DB with
> new snapshot located at: /data/metadata/om.snapshot.db_1755376585468
> 2025-08-16 13:36:27.904 | 2025-08-16 20:36:27,904 [Recon-SyncOM-0] INFO
> recovery.ReconOmMetadataManagerImpl: Created OM DB handle from snapshot at
> /data/metadata/om.snapshot.db_1755376585468.
> 2025-08-16 13:36:27.904 | 2025-08-16 20:36:27,904 [Recon-SyncOM-0] WARN
> om.OmMetadataManagerImpl: Overwriting an existing table userTable:
> TypedTable-userTable(String->PersistedUserVolumeInfo)
> 2025-08-16 13:36:27.911 | 2025-08-16 20:36:27,910 [Recon-SyncOM-0] ERROR
> impl.OzoneManagerServiceProviderImpl: Unable to update Recon's metadata with
> new OM DB.
> 2025-08-16 13:36:27.911 | org.apache.hadoop.metrics2.MetricsException:
> Metrics source userTableCache already exists!
> 2025-08-16 13:36:27.911 | at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newSourceName(DefaultMetricsSystem.java:152)
> 2025-08-16 13:36:27.911 | at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.sourceName(DefaultMetricsSystem.java:125)
> 2025-08-16 13:36:27.911 | at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:229)
> 2025-08-16 13:36:27.911 | at
> org.apache.hadoop.hdds.utils.TableCacheMetrics.create(TableCacheMetrics.java:48)
> 2025-08-16 13:36:27.911 | at
> org.apache.hadoop.hdds.utils.db.TypedTable.createCacheMetrics(TypedTable.java:436)
> 2025-08-16 13:36:27.911 | at
> org.apache.hadoop.ozone.om.OmMetadataManagerImpl$TableInitializer.get(OmMetadataManagerImpl.java:1863)
> 2025-08-16 13:36:27.911 | at
> org.apache.hadoop.ozone.om.OmMetadataManagerImpl$TableInitializer.get(OmMetadataManagerImpl.java:1846)
> 2025-08-16 13:36:27.911 | at
> org.apache.hadoop.ozone.om.OmMetadataManagerImpl.initializeOmTables(OmMetadataManagerImpl.java:443)
> 2025-08-16 13:36:27.911 | at
> org.apache.hadoop.ozone.recon.recovery.ReconOmMetadataManagerImpl.initializeNewRdbStore(ReconOmMetadataManagerImpl.java:103)
> 2025-08-16 13:36:27.911 | at
> org.apache.hadoop.ozone.recon.recovery.ReconOmMetadataManagerImpl.updateOmDB(ReconOmMetadataManagerImpl.java:126)
> 2025-08-16 13:36:27.911 | at
> org.apache.hadoop.ozone.recon.spi.impl.OzoneManagerServiceProviderImpl.updateReconOmDBWithNewSnapshot(OzoneManagerServiceProviderImpl.java:513)
> 2025-08-16 13:36:27.911 | at
> org.apache.hadoop.ozone.recon.spi.impl.OzoneManagerServiceProviderImpl.syncDataFromOM(OzoneManagerServiceProviderImpl.java:709)
> 2025-08-16 13:36:27.911 | at
> org.apache.hadoop.ozone.recon.spi.impl.OzoneManagerServiceProviderImpl.lambda$startSyncDataFromOM$2(OzoneManagerServiceProviderImpl.java:329)
> 2025-08-16 13:36:27.911 | at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
> 2025-08-16 13:36:27.911 | at
> java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:358)
> 2025-08-16 13:36:27.911 | at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
> 2025-08-16 13:36:27.911 | at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
> 2025-08-16 13:36:27.911 | at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
> 2025-08-16 13:36:27.911 | at
> java.base/java.lang.Thread.run(Thread.java:1583)
> 2025-08-16 13:36:27.911 | 2025-08-16 20:36:27,911 [Recon-SyncOM-0] INFO
> recon.ReconContext: Update healthStatus of Recon from true to false.
> 2025-08-16 13:36:27.911 | 2025-08-16 20:36:27,911 [Recon-SyncOM-0] INFO
> impl.OzoneManagerServiceProviderImpl: Sequence number after sync: 2386622
> 2025-08-16 13:37:27.912 | 2025-08-16 20:37:27,912 [Recon-SyncOM-0] INFO
> impl.OzoneManagerServiceProviderImpl: Last known sequence number before sync:
> 2386622
> 2025-08-16 13:37:27.912 | 2025-08-16 20:37:27,912 [Recon-SyncOM-0] INFO
> impl.OzoneManagerServiceProviderImpl: Seq number of Recon's OM DB : 2386622
> 2025-08-16 13:37:27.929 | 2025-08-16 20:37:27,928 [Recon-SyncOM-0] WARN
> impl.OzoneManagerServiceProviderImpl: Unable to get and apply delta updates
> from OM: Invalid transaction log iterator when getting updates since sequence
> number 2386622, falling back to full snapshot
> 2025-08-16 13:37:27.929 | 2025-08-16 20:37:27,928 [Recon-SyncOM-0] INFO
> impl.OzoneManagerServiceProviderImpl: Delta updates received from OM : 0
> loops, 0 records
> 2025-08-16 13:37:27.929 | 2025-08-16 20:37:27,928 [Recon-SyncOM-0] INFO
> impl.OzoneManagerServiceProviderImpl: Obtaining full snapshot from Ozone
> Manager
> 2025-08-16 13:37:27.929 | 2025-08-16 20:37:27,929 [Recon-SyncOM-0] INFO
> impl.OzoneManagerServiceProviderImpl: Permissions for Recon DB directory
> '/data/metadata/recon' meet the minimum required permissions '750'
> 2025-08-16 13:37:27.930 | 2025-08-16 20:37:27,930 [Recon-SyncOM-0] INFO
> impl.OzoneManagerServiceProviderImpl: Successfully deleted existing OM DB
> snapshot directory: /data/metadata/om.snapshot.db_1755376585468
> {code}
> Note: A few minutes before that, there is a successful attempt:
> {code}
> 2025-08-16 13:34:25.301 | 2025-08-16 20:34:25,300 [Recon-SyncOM-0] INFO
> recon.ReconContext: Update healthStatus of Recon from true to true.
> 2025-08-16 13:34:25.301 | 2025-08-16 20:34:25,301 [Recon-SyncOM-0] INFO
> impl.OzoneManagerServiceProviderImpl: Attempting to update Recon OM DB with
> new snapshot located at: /data/metadata/om.snapshot.db_1755376465092
> 2025-08-16 13:34:25.330 | 2025-08-16 20:34:25,330 [Recon-SyncOM-0] INFO
> recovery.ReconOmMetadataManagerImpl: Created OM DB handle from snapshot at
> /data/metadata/om.snapshot.db_1755376465092.
> 2025-08-16 13:34:25.341 | 2025-08-16 20:34:25,341 [Recon-SyncOM-0] INFO
> impl.OzoneManagerServiceProviderImpl: Successfully updated Recon OM DB with
> new snapshot.
> {code}
> To investigated whether this breaks things.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
