[
https://issues.apache.org/jira/browse/HDDS-8940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17754810#comment-17754810
]
Hemant Kumar edited comment on HDDS-8940 at 8/16/23 1:30 AM:
-------------------------------------------------------------
There is another occurrence of this issue.
Important logs.
{code}
...
2023-08-14 19:29:45,520 INFO [OM StateMachine ApplyTransaction Thread -
0]-org.apache.hadoop.ozone.om.request.snapshot.OMSnapshotCreateRequest: Created
snapshot: 'cm-46-1692041157584-0' with snapshotId:
'c2ae0246-626b-4a09-b1c8-0e76997c26ee' under path 'vol23io/bucket870io'
...
...
2023-08-14 19:30:25,363 INFO
[SstFilteringService#0]-org.apache.hadoop.hdds.utils.db.RocksDatabase: Deleting
sst file /001319.sst corresponding to column family keyTable from db:
/var/lib/hadoop-ozone/om/data/db.snapshots/checkpointState/om.db-c2ae0246-626b-4a09-b1c8-0e76997c26ee
...
...
2023-08-14 19:43:19,498 INFO
[CompactionDagPruningService]-org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer:
Removing SST files: [000971, 000210, 000297, 001386, 000457, 000611, 000699,
000974, 000214, 000973, 001025, 001026, 001389, 000615, 000736, 000856, 000619,
000618, 001393, 001273, 001153, 001395, 000443, 001137, 000964, 001134, 000203,
000687, 001377, 000323, 000565, 001015, 000968, 000846, 000206, 000569, 000602,
000848, 001021, 000450, 000570, 000172, 000476, 001049, 000992, 000111, 000110,
000594, 000513, 001287, 001320, 001200, 001168, 000356, 000516, 000757, 000911,
001328, 000519, 000915, 001209, 000918, 001296, 001290, 000481, 001172, 001293,
000102, 000465, 000860, 000468, 000588, 000863, 000620, 001312, 000748, 000109,
000505, 000229, 001319, 000628, 000908, 001284, 000190, 000472, 000350, 000471,
000194, 001346, 000651, 000375, 000253, 000374, 000891, 000777, 001343, 000258,
000412, 000775, 000418, 000814, 000811, 000899, 000818, 001229, 000817, 000936,
000819, 001075, 000262, 000140, 001072, 000487, 000883, 000244, 000364, 000760,
000766, 000920, 001331, 000127, 000248, 000765, 000522, 000367, 000763, 000406,
000921, 001339, 000409, 000808, 000929, 001063, 001184, 001186, 001220, 001180,
000370, 000157, 001005, 001247, 000277, 000310, 000552, 001127, 000672, 001007,
001129, 000315, 000711, 000799, 000953, 000314, 000159, 000555, 001124, 000715,
000834, 000717, 000958, 001090, 001250, 001010, 001131, 000285, 000663, 000541,
000661, 001238, 000143, 000264, 000425, 000667, 001078, 001232, 001113, 000422,
001235, 000306, 000702, 000706, 001120, 001242, 000670, 000152, 001082, 000392]
as part of SST file pruning.
...
...
2023-08-15 17:07:18,444 INFO [OM StateMachine ApplyTransaction Thread -
0]-org.apache.hadoop.ozone.om.request.snapshot.OMSnapshotCreateRequest: Created
snapshot: 'cm-tmp-2651cc03-a0e7-4fba-ab6c-810a863f4073' with snapshotId:
'002cfe90-0684-4dd3-9239-487a92067fa6' under path 'vol23io/bucket870io'
...
...
2023-08-15 17:07:41,334 INFO
[snapshot-diff-job-thread-id-2]-org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager:
Started snap diff report generation for volume: 'vol23io', bucket:
'bucket870io', fromSnapshot: 'cm-46-1692041157584-0', toSnapshot:
'cm-tmp-2651cc03-a0e7-4fba-ab6c-810a863f4073'
2023-08-15 17:07:41,334 INFO
[snapshot-diff-job-thread-id-2]-org.apache.hadoop.ozone.om.snapshot.SnapshotCache:
Loading snapshot. Table key: /vol23io/bucket870io/cm-46-1692041157584-0
2023-08-15 17:07:41,335 INFO
[snapshot-diff-job-thread-id-2]-org.apache.hadoop.ozone.om.helpers.OmKeyInfo:
OmKeyInfo.getCodec ignorePipeline = true
2023-08-15 17:07:41,371 INFO
[snapshot-diff-job-thread-id-2]-org.apache.hadoop.ozone.om.snapshot.SnapshotCache:
Loading snapshot. Table key:
/vol23io/bucket870io/cm-tmp-2651cc03-a0e7-4fba-ab6c-810a863f4073
2023-08-15 17:07:41,371 INFO
[snapshot-diff-job-thread-id-2]-org.apache.hadoop.ozone.om.helpers.OmKeyInfo:
OmKeyInfo.getCodec ignorePipeline = true
2023-08-15 17:07:41,438 WARN
[snapshot-diff-job-thread-id-2]-org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager:
Failed to get SST diff file using RocksDBCheckpointDiffer. It will fallback to
full diff now.
java.io.FileNotFoundException: Can't find SST file: 001319.sst
at
org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer.getAbsoluteSstFilePath(RocksDBCheckpointDiffer.java:688)
at
org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer.filterRelevantSstFilesFullPath(RocksDBCheckpointDiffer.java:954)
at
org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer.getSSTDiffList(RocksDBCheckpointDiffer.java:938)
at
org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer.getSSTDiffListWithFullPath(RocksDBCheckpointDiffer.java:875)
at
org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager.getDeltaFiles(SnapshotDiffManager.java:1237)
at
org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager.getDeltaFilesAndDiffKeysToObjectIdToKeyMap(SnapshotDiffManager.java:1067)
at
org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager.lambda$3(SnapshotDiffManager.java:949)
at
org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager.generateSnapshotDiffReport(SnapshotDiffManager.java:1015)
at
org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager.lambda$2(SnapshotDiffManager.java:742)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
2023-08-15 17:07:41,439 WARN
[snapshot-diff-job-thread-id-2]-org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager:
RocksDBCheckpointDiffer is not available, falling back to slow path
2023-08-15 17:07:41,522 WARN
[snapshot-diff-job-thread-id-2]-org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager:
Failed to get SST diff file using RocksDBCheckpointDiffer. It will fallback to
full diff now.
java.io.FileNotFoundException: Can't find SST file: 001319.sst
at
org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer.getAbsoluteSstFilePath(RocksDBCheckpointDiffer.java:688)
at
org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer.filterRelevantSstFilesFullPath(RocksDBCheckpointDiffer.java:954)
at
org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer.getSSTDiffList(RocksDBCheckpointDiffer.java:938)
at
org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer.getSSTDiffListWithFullPath(RocksDBCheckpointDiffer.java:875)
at
org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager.getDeltaFiles(SnapshotDiffManager.java:1237)
at
org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager.getDeltaFilesAndDiffKeysToObjectIdToKeyMap(SnapshotDiffManager.java:1067)
at
org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager.lambda$4(SnapshotDiffManager.java:959)
at
org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager.generateSnapshotDiffReport(SnapshotDiffManager.java:1015)
at
org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager.lambda$2(SnapshotDiffManager.java:742)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
2023-08-15 17:07:41,522 WARN
[snapshot-diff-job-thread-id-2]-org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager:
RocksDBCheckpointDiffer is not available, falling back to slow path
2023-08-15 17:07:41,531 INFO
[snapshot-diff-job-thread-id-2]-org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager:
Starting diff report generation for jobId:
2239d0e4-86ff-48f9-9919-08a6b8e5ad7f.
...
{code}
was (Author: JIRAUSER297350):
There is another occurrence of this issue.
Important logs.
{code}
...
2023-08-14 19:29:45,520 INFO [OM StateMachine ApplyTransaction Thread -
0]-org.apache.hadoop.ozone.om.request.snapshot.OMSnapshotCreateRequest: Created
snapshot: 'cm-46-1692041157584-0' with snapshotId:
'c2ae0246-626b-4a09-b1c8-0e76997c26ee' under path 'vol23io/bucket870io'
...
...
2023-08-14 19:30:25,363 INFO
[SstFilteringService#0]-org.apache.hadoop.hdds.utils.db.RocksDatabase: Deleting
sst file /001319.sst corresponding to column family keyTable from db:
/var/lib/hadoop-ozone/om/data/db.snapshots/checkpointState/om.db-c2ae0246-626b-4a09-b1c8-0e76997c26ee
...
...
2023-08-15 17:07:18,444 INFO [OM StateMachine ApplyTransaction Thread -
0]-org.apache.hadoop.ozone.om.request.snapshot.OMSnapshotCreateRequest: Created
snapshot: 'cm-tmp-2651cc03-a0e7-4fba-ab6c-810a863f4073' with snapshotId:
'002cfe90-0684-4dd3-9239-487a92067fa6' under path 'vol23io/bucket870io'
...
...
2023-08-15 17:07:41,334 INFO
[snapshot-diff-job-thread-id-2]-org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager:
Started snap diff report generation for volume: 'vol23io', bucket:
'bucket870io', fromSnapshot: 'cm-46-1692041157584-0', toSnapshot:
'cm-tmp-2651cc03-a0e7-4fba-ab6c-810a863f4073'
2023-08-15 17:07:41,334 INFO
[snapshot-diff-job-thread-id-2]-org.apache.hadoop.ozone.om.snapshot.SnapshotCache:
Loading snapshot. Table key: /vol23io/bucket870io/cm-46-1692041157584-0
2023-08-15 17:07:41,335 INFO
[snapshot-diff-job-thread-id-2]-org.apache.hadoop.ozone.om.helpers.OmKeyInfo:
OmKeyInfo.getCodec ignorePipeline = true
2023-08-15 17:07:41,371 INFO
[snapshot-diff-job-thread-id-2]-org.apache.hadoop.ozone.om.snapshot.SnapshotCache:
Loading snapshot. Table key:
/vol23io/bucket870io/cm-tmp-2651cc03-a0e7-4fba-ab6c-810a863f4073
2023-08-15 17:07:41,371 INFO
[snapshot-diff-job-thread-id-2]-org.apache.hadoop.ozone.om.helpers.OmKeyInfo:
OmKeyInfo.getCodec ignorePipeline = true
2023-08-15 17:07:41,438 WARN
[snapshot-diff-job-thread-id-2]-org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager:
Failed to get SST diff file using RocksDBCheckpointDiffer. It will fallback to
full diff now.
java.io.FileNotFoundException: Can't find SST file: 001319.sst
at
org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer.getAbsoluteSstFilePath(RocksDBCheckpointDiffer.java:688)
at
org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer.filterRelevantSstFilesFullPath(RocksDBCheckpointDiffer.java:954)
at
org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer.getSSTDiffList(RocksDBCheckpointDiffer.java:938)
at
org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer.getSSTDiffListWithFullPath(RocksDBCheckpointDiffer.java:875)
at
org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager.getDeltaFiles(SnapshotDiffManager.java:1237)
at
org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager.getDeltaFilesAndDiffKeysToObjectIdToKeyMap(SnapshotDiffManager.java:1067)
at
org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager.lambda$3(SnapshotDiffManager.java:949)
at
org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager.generateSnapshotDiffReport(SnapshotDiffManager.java:1015)
at
org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager.lambda$2(SnapshotDiffManager.java:742)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
2023-08-15 17:07:41,439 WARN
[snapshot-diff-job-thread-id-2]-org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager:
RocksDBCheckpointDiffer is not available, falling back to slow path
2023-08-15 17:07:41,522 WARN
[snapshot-diff-job-thread-id-2]-org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager:
Failed to get SST diff file using RocksDBCheckpointDiffer. It will fallback to
full diff now.
java.io.FileNotFoundException: Can't find SST file: 001319.sst
at
org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer.getAbsoluteSstFilePath(RocksDBCheckpointDiffer.java:688)
at
org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer.filterRelevantSstFilesFullPath(RocksDBCheckpointDiffer.java:954)
at
org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer.getSSTDiffList(RocksDBCheckpointDiffer.java:938)
at
org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer.getSSTDiffListWithFullPath(RocksDBCheckpointDiffer.java:875)
at
org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager.getDeltaFiles(SnapshotDiffManager.java:1237)
at
org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager.getDeltaFilesAndDiffKeysToObjectIdToKeyMap(SnapshotDiffManager.java:1067)
at
org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager.lambda$4(SnapshotDiffManager.java:959)
at
org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager.generateSnapshotDiffReport(SnapshotDiffManager.java:1015)
at
org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager.lambda$2(SnapshotDiffManager.java:742)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
2023-08-15 17:07:41,522 WARN
[snapshot-diff-job-thread-id-2]-org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager:
RocksDBCheckpointDiffer is not available, falling back to slow path
2023-08-15 17:07:41,531 INFO
[snapshot-diff-job-thread-id-2]-org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager:
Starting diff report generation for jobId:
2239d0e4-86ff-48f9-9919-08a6b8e5ad7f.
...
{code}
> SST files are missing on optimized snapDiff path.
> -------------------------------------------------
>
> Key: HDDS-8940
> URL: https://issues.apache.org/jira/browse/HDDS-8940
> Project: Apache Ozone
> Issue Type: Sub-task
> Reporter: Hemant Kumar
> Assignee: Hemant Kumar
> Priority: Major
> Attachments: HDDS-8940_Compaction_Dag.png
>
>
> While running snapDiff, we are seeing SST files missing on optimized snapDiff
> path.
> {code}
> 2023-06-23 19:59:16,323 [snapshot-diff-job-thread-id-14] ERROR
> org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager: Caught checked
> exception during diff report generation for volume: volume1 bucket: bucket1,
> fromSnapshot: alma2 and toSnapshot:
> cm-tmp-0ae3d532-237d-4df2-83f9-4844d153521e
> java.io.FileNotFoundException: Can't find SST file: 010788.sst
> at
> org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer.getAbsoluteSstFilePath(RocksDBCheckpointDiffer.java:654)
> at
> org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer.filterRelevantSstFilesFullPath(RocksDBCheckpointDiffer.java:949)
> at
> org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer.getSSTDiffList(RocksDBCheckpointDiffer.java:933)
> at
> org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer.getSSTDiffListWithFullPath(RocksDBCheckpointDiffer.java:868)
> at
> org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager.getDeltaFiles(SnapshotDiffManager.java:929)
> at
> org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager.getDeltaFilesAndDiffKeysToObjectIdToKeyMap(SnapshotDiffManager.java:793)
> at
> org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager.generateSnapshotDiffReport(SnapshotDiffManager.java:721)
> at
> org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager.lambda$0(SnapshotDiffManager.java:565)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
