[
https://issues.apache.org/jira/browse/HDDS-8940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17738696#comment-17738696
]
Hemant Kumar commented on HDDS-8940:
------------------------------------
In all three cases, compaction files were created even before the snapshot was
taken. In theory, it doesn't matter any compaction happens before the snapshot
is taken.
Case 1:
Grepped the snapshotId from log.
{code}
2023-06-23 12:02:44,140 [OM StateMachine ApplyTransaction Thread - 0] INFO
org.apache.hadoop.ozone.om.request.snapshot.OMSnapshotCreateRequest: Created
snapshot 'alma' under path 'volume1/bucket1'
2023-06-23 12:02:44,303 [OMDoubleBufferFlushThread] INFO
org.apache.hadoop.ozone.om.OmSnapshotManager: Created checkpoint :
/var/lib/hadoop-ozone/om/data/db.snapshots/checkpointState/om.db-e865833e-9a54-41fd-a2cc-5f79f045b757
for snapshot alma
.....
2023-06-23 17:12:33,708 [OM StateMachine ApplyTransaction Thread - 0] INFO
org.apache.hadoop.ozone.om.request.snapshot.OMSnapshotCreateRequest: Created
snapshot 'cm-tmp-1f92bdf3-cbba-4a43-b266-224333ab471f' under path
'volume1/bucket1'
2023-06-23 17:12:33,839 [OMDoubleBufferFlushThread] INFO
org.apache.hadoop.ozone.om.OmSnapshotManager: Created checkpoint :
/var/lib/hadoop-ozone/om/data/db.snapshots/checkpointState/om.db-b230a175-c7f1-4ba5-8dc1-fa91bb4119ed
for snapshot cm-tmp-1f92bdf3-cbba-4a43-b266-224333ab471f
{code}
Grepped the corresponding compaction log file.
{code}
>From snapshot:
0000000000000163326.log:S 207161 e865833e-9a54-41fd-a2cc-5f79f045b757
1687521764122
To snapshot:
0000000000000207161.log:S 211720 b230a175-c7f1-4ba5-8dc1-fa91bb4119ed
1687540353703
{code}
Missing file: 010520 was taken before from snapshot: alma
{code}
[root@host compaction-log]# grep 010520 0000000000000*
0000000000000163326.log:C 010518,010512,010504,010496,010490:010520
0000000000000163326.log:C 010547,010540,010533,010526,010520:010549
{code}
Case 2:
Grepped the snapshotId from log.
{code}2023-06-23 17:41:06,608 [OM StateMachine ApplyTransaction Thread - 0]
INFO org.apache.hadoop.ozone.om.request.snapshot.OMSnapshotCreateRequest:
Created snapshot 'alma2' under path 'volume1/bucket1'
2023-06-23 17:41:06,716 [OMDoubleBufferFlushThread] INFO
org.apache.hadoop.ozone.om.OmSnapshotManager: Created checkpoint :
/var/lib/hadoop-ozone/om/data/db.snapshots/checkpointState/om.db-77982b43-6040-46d1-89c8-9a7c7c4d446c
for snapshot alma2
.....
2023-06-23 19:59:08,922 [OM StateMachine ApplyTransaction Thread - 0] INFO
org.apache.hadoop.ozone.om.request.snapshot.OMSnapshotCreateRequest: Created
snapshot 'cm-tmp-0ae3d532-237d-4df2-83f9-4844d153521e' under path
'volume1/bucket1'
2023-06-23 19:59:09,042 [OMDoubleBufferFlushThread] INFO
org.apache.hadoop.ozone.om.OmSnapshotManager: Created checkpoint :
/var/lib/hadoop-ozone/om/data/db.snapshots/checkpointState/om.db-63f64150-e78d-461d-90d9-104290141ee9
for snapshot cm-tmp-0ae3d532-237d-4df2-83f9-4844d153521e
{code}
Grepped the corresponding compaction log file.
{code}
>From snapshot:
0000000000000211822.log:S 212075 77982b43-6040-46d1-89c8-9a7c7c4d446c
1687542066607
To snapshot:
0000000000000213938.log:S 214181 63f64150-e78d-461d-90d9-104290141ee9
1687550348921
{code}
Missing file: 010788 was taken before from snapshot: alma2
{code}
[root@host compaction-log]# grep 010788 0000000000000*
0000000000000211720.log:C 010786,010779,010772,010764,010757:010788
0000000000000211822.log:C 010815,010809,010803,010795,010788:010817
{code}
Case 3:
Grepped the snapshotId from log.
{code}
2023-06-23 18:27:25,267 [OM StateMachine ApplyTransaction Thread - 0] INFO
org.apache.hadoop.ozone.om.request.snapshot.OMSnapshotCreateRequest: Created
snapshot 'alma4' under path 'volume1/bucket1'
2023-06-23 18:27:25,460 [OMDoubleBufferFlushThread] INFO
org.apache.hadoop.ozone.om.OmSnapshotManager: Created checkpoint :
/var/lib/hadoop-ozone/om/data/db.snapshots/checkpointState/om.db-7f41dad2-2510-4626-a697-84a56e8c7bcb
for snapshot alma4
.....
2023-06-23 20:02:15,918 [OM StateMachine ApplyTransaction Thread - 0] INFO
org.apache.hadoop.ozone.om.request.snapshot.OMSnapshotCreateRequest: Created
snapshot 'cm-tmp-30b1fd30-804e-4f69-a3ef-2dbfab363ded' under path
'volume1/bucket1'
2023-06-23 20:02:16,010 [OMDoubleBufferFlushThread] INFO
org.apache.hadoop.ozone.om.OmSnapshotManager: Created checkpoint :
/var/lib/hadoop-ozone/om/data/db.snapshots/checkpointState/om.db-454c0676-374f-4505-beaa-dff3684c1a83
for snapshot cm-tmp-30b1fd30-804e-4f69-a3ef-2dbfab363ded
{code}
Grepped the corresponding compaction log file.
{code}
>From snapshot:
0000000000000212819.log:S 212871 7f41dad2-2510-4626-a697-84a56e8c7bcb
1687544845266
To snapshot:
0000000000000214341.log:S 214418 454c0676-374f-4505-beaa-dff3684c1a83
1687550535917
{code}
Missing file: 010849 was taken before from snapshot: alma4
{code}
[root@host compaction-log]# grep 010849 0000000000000*
0000000000000212075.log:C 010847,010840,010832,010825,010817:010849
0000000000000212819.log:C 010886,010873,010865,010857,010849:010889
{code}
All the compaction logs:
{code}
0000000000000000000.log
0000000000000163279.log
0000000000000163288.log
0000000000000163326.log
0000000000000207161.log
0000000000000211720.log
0000000000000211822.log
0000000000000212075.log
0000000000000212682.log
0000000000000212764.log
0000000000000212819.log
0000000000000212871.log
0000000000000212921.log
0000000000000213743.log
0000000000000213795.log
0000000000000213829.log
0000000000000213893.log
0000000000000213938.log
0000000000000214181.log
0000000000000214341.log
0000000000000214418.log
0000000000000214472.log
0000000000000214534.log
0000000000000214589.log
0000000000000218071.log
0000000000000218174.log
0000000000000218265.log
0000000000000218344.log
0000000000000218387.log
0000000000000268654.log
0000000000000271883.log
{code}
> SST files are missing on optimized snapDiff path.
> -------------------------------------------------
>
> Key: HDDS-8940
> URL: https://issues.apache.org/jira/browse/HDDS-8940
> Project: Apache Ozone
> Issue Type: Sub-task
> Reporter: Hemant Kumar
> Assignee: Hemant Kumar
> Priority: Major
>
> While running snapDiff, we are seeing SST files missing on optimized snapDiff
> path.
> {code}
> 2023-06-23 19:59:16,323 [snapshot-diff-job-thread-id-14] ERROR
> org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager: Caught checked
> exception during diff report generation for volume: volume1 bucket: bucket1,
> fromSnapshot: alma2 and toSnapshot:
> cm-tmp-0ae3d532-237d-4df2-83f9-4844d153521e
> java.io.FileNotFoundException: Can't find SST file: 010788.sst
> at
> org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer.getAbsoluteSstFilePath(RocksDBCheckpointDiffer.java:654)
> at
> org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer.filterRelevantSstFilesFullPath(RocksDBCheckpointDiffer.java:949)
> at
> org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer.getSSTDiffList(RocksDBCheckpointDiffer.java:933)
> at
> org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer.getSSTDiffListWithFullPath(RocksDBCheckpointDiffer.java:868)
> at
> org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager.getDeltaFiles(SnapshotDiffManager.java:929)
> at
> org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager.getDeltaFilesAndDiffKeysToObjectIdToKeyMap(SnapshotDiffManager.java:793)
> at
> org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager.generateSnapshotDiffReport(SnapshotDiffManager.java:721)
> at
> org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager.lambda$0(SnapshotDiffManager.java:565)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
