[
https://issues.apache.org/jira/browse/HDDS-8940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17754810#comment-17754810
]
Hemant Kumar commented on HDDS-8940:
------------------------------------
There is another occurrence of this issue.
Important logs.
{code}
...
2023-08-14 19:29:45,520 INFO [OM StateMachine ApplyTransaction Thread -
0]-org.apache.hadoop.ozone.om.request.snapshot.OMSnapshotCreateRequest: Created
snapshot: 'cm-46-1692041157584-0' with snapshotId:
'c2ae0246-626b-4a09-b1c8-0e76997c26ee' under path 'vol23io/bucket870io'
...
...
2023-08-14 19:30:25,363 INFO
[SstFilteringService#0]-org.apache.hadoop.hdds.utils.db.RocksDatabase: Deleting
sst file /001319.sst corresponding to column family keyTable from db:
/var/lib/hadoop-ozone/om/data/db.snapshots/checkpointState/om.db-c2ae0246-626b-4a09-b1c8-0e76997c26ee
...
...
2023-08-15 17:07:18,444 INFO [OM StateMachine ApplyTransaction Thread -
0]-org.apache.hadoop.ozone.om.request.snapshot.OMSnapshotCreateRequest: Created
snapshot: 'cm-tmp-2651cc03-a0e7-4fba-ab6c-810a863f4073' with snapshotId:
'002cfe90-0684-4dd3-9239-487a92067fa6' under path 'vol23io/bucket870io'
...
...
2023-08-15 17:07:41,334 INFO
[snapshot-diff-job-thread-id-2]-org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager:
Started snap diff report generation for volume: 'vol23io', bucket:
'bucket870io', fromSnapshot: 'cm-46-1692041157584-0', toSnapshot:
'cm-tmp-2651cc03-a0e7-4fba-ab6c-810a863f4073'
2023-08-15 17:07:41,334 INFO
[snapshot-diff-job-thread-id-2]-org.apache.hadoop.ozone.om.snapshot.SnapshotCache:
Loading snapshot. Table key: /vol23io/bucket870io/cm-46-1692041157584-0
2023-08-15 17:07:41,335 INFO
[snapshot-diff-job-thread-id-2]-org.apache.hadoop.ozone.om.helpers.OmKeyInfo:
OmKeyInfo.getCodec ignorePipeline = true
2023-08-15 17:07:41,371 INFO
[snapshot-diff-job-thread-id-2]-org.apache.hadoop.ozone.om.snapshot.SnapshotCache:
Loading snapshot. Table key:
/vol23io/bucket870io/cm-tmp-2651cc03-a0e7-4fba-ab6c-810a863f4073
2023-08-15 17:07:41,371 INFO
[snapshot-diff-job-thread-id-2]-org.apache.hadoop.ozone.om.helpers.OmKeyInfo:
OmKeyInfo.getCodec ignorePipeline = true
2023-08-15 17:07:41,438 WARN
[snapshot-diff-job-thread-id-2]-org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager:
Failed to get SST diff file using RocksDBCheckpointDiffer. It will fallback to
full diff now.
java.io.FileNotFoundException: Can't find SST file: 001319.sst
at
org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer.getAbsoluteSstFilePath(RocksDBCheckpointDiffer.java:688)
at
org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer.filterRelevantSstFilesFullPath(RocksDBCheckpointDiffer.java:954)
at
org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer.getSSTDiffList(RocksDBCheckpointDiffer.java:938)
at
org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer.getSSTDiffListWithFullPath(RocksDBCheckpointDiffer.java:875)
at
org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager.getDeltaFiles(SnapshotDiffManager.java:1237)
at
org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager.getDeltaFilesAndDiffKeysToObjectIdToKeyMap(SnapshotDiffManager.java:1067)
at
org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager.lambda$3(SnapshotDiffManager.java:949)
at
org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager.generateSnapshotDiffReport(SnapshotDiffManager.java:1015)
at
org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager.lambda$2(SnapshotDiffManager.java:742)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
2023-08-15 17:07:41,439 WARN
[snapshot-diff-job-thread-id-2]-org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager:
RocksDBCheckpointDiffer is not available, falling back to slow path
2023-08-15 17:07:41,522 WARN
[snapshot-diff-job-thread-id-2]-org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager:
Failed to get SST diff file using RocksDBCheckpointDiffer. It will fallback to
full diff now.
java.io.FileNotFoundException: Can't find SST file: 001319.sst
at
org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer.getAbsoluteSstFilePath(RocksDBCheckpointDiffer.java:688)
at
org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer.filterRelevantSstFilesFullPath(RocksDBCheckpointDiffer.java:954)
at
org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer.getSSTDiffList(RocksDBCheckpointDiffer.java:938)
at
org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer.getSSTDiffListWithFullPath(RocksDBCheckpointDiffer.java:875)
at
org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager.getDeltaFiles(SnapshotDiffManager.java:1237)
at
org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager.getDeltaFilesAndDiffKeysToObjectIdToKeyMap(SnapshotDiffManager.java:1067)
at
org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager.lambda$4(SnapshotDiffManager.java:959)
at
org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager.generateSnapshotDiffReport(SnapshotDiffManager.java:1015)
at
org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager.lambda$2(SnapshotDiffManager.java:742)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
2023-08-15 17:07:41,522 WARN
[snapshot-diff-job-thread-id-2]-org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager:
RocksDBCheckpointDiffer is not available, falling back to slow path
2023-08-15 17:07:41,531 INFO
[snapshot-diff-job-thread-id-2]-org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager:
Starting diff report generation for jobId:
2239d0e4-86ff-48f9-9919-08a6b8e5ad7f.
...
{code}
> SST files are missing on optimized snapDiff path.
> -------------------------------------------------
>
> Key: HDDS-8940
> URL: https://issues.apache.org/jira/browse/HDDS-8940
> Project: Apache Ozone
> Issue Type: Sub-task
> Reporter: Hemant Kumar
> Assignee: Hemant Kumar
> Priority: Major
> Attachments: HDDS-8940_Compaction_Dag.png
>
>
> While running snapDiff, we are seeing SST files missing on optimized snapDiff
> path.
> {code}
> 2023-06-23 19:59:16,323 [snapshot-diff-job-thread-id-14] ERROR
> org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager: Caught checked
> exception during diff report generation for volume: volume1 bucket: bucket1,
> fromSnapshot: alma2 and toSnapshot:
> cm-tmp-0ae3d532-237d-4df2-83f9-4844d153521e
> java.io.FileNotFoundException: Can't find SST file: 010788.sst
> at
> org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer.getAbsoluteSstFilePath(RocksDBCheckpointDiffer.java:654)
> at
> org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer.filterRelevantSstFilesFullPath(RocksDBCheckpointDiffer.java:949)
> at
> org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer.getSSTDiffList(RocksDBCheckpointDiffer.java:933)
> at
> org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer.getSSTDiffListWithFullPath(RocksDBCheckpointDiffer.java:868)
> at
> org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager.getDeltaFiles(SnapshotDiffManager.java:929)
> at
> org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager.getDeltaFilesAndDiffKeysToObjectIdToKeyMap(SnapshotDiffManager.java:793)
> at
> org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager.generateSnapshotDiffReport(SnapshotDiffManager.java:721)
> at
> org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager.lambda$0(SnapshotDiffManager.java:565)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
