[
https://issues.apache.org/jira/browse/HDDS-8987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDDS-8987:
--------------------------------
Status: Patch Available (was: Open)
> SST Filtering service should be stopped while before copying incremental
> snapshot
> ---------------------------------------------------------------------------------
>
> Key: HDDS-8987
> URL: https://issues.apache.org/jira/browse/HDDS-8987
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: Snapshot
> Reporter: Swaminathan Balachandran
> Assignee: Swaminathan Balachandran
> Priority: Major
> Labels: pull-request-available
> Attachments: a.log
>
>
> Currently when only incremental sst files are copied. The sst files are
> listed in the candidate rocksdb directory & are excluded from copying from
> leader. But SSTFiltering service could interfere with this and go about
> deleting the sst files which have been excluded while the data is being
> copied. Thus when the leader state is copied and installed certain sst files
> may be missing which could be present in the leader's sst manifest file.
> Attached are some logs for this.
> {code:java}
> 2023-07-07 13:29:42,626 [qtp501491806-254] INFO utils.DBCheckpointServlet
> (DBCheckpointServlet.java:generateSnapshotCheckpoint(203)) - Excluded SST
> [/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.checkpoints/om.db_checkpoint_1688761782175/000076.sst,
>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.checkpoints/om.db_checkpoint_1688761782175/000064.sst,
>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.checkpoints/om.db_checkpoint_1688761782175/000059.sst,
>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.checkpoints/om.db_checkpoint_1688761782175/000067.sst,
>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.checkpoints/om.db_checkpoint_1688761782175/000072.sst,
>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.checkpoints/om.db_checkpoint_1688761782175/000080.sst,
>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.checkpoints/om.db_checkpoint_1688761782175/000057.sst,
>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.checkpoints/om.db_checkpoint_1688761782175/000081.sst,
>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.checkpoints/om.db_checkpoint_1688761782175/000083.sst,
>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.checkpoints/om.db_checkpoint_1688761782175/000051.sst,
>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.checkpoints/om.db_checkpoint_1688761782175/000078.sst,
>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.checkpoints/om.db_checkpoint_1688761782175/000053.sst,
>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/diffState/compaction-sst-backup/000062.sst,
>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/diffState/compaction-sst-backup/000061.sst,
>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/diffState/compaction-sst-backup/000075.sst,
>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/diffState/compaction-sst-backup/000074.sst,
>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/diffState/compaction-sst-backup/000060.sst,
>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/diffState/compaction-sst-backup/000070.sst,
>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/diffState/compaction-sst-backup/000059.sst,
>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/diffState/compaction-sst-backup/000065.sst,
>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/diffState/compaction-sst-backup/000067.sst,
>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/diffState/compaction-sst-backup/000072.sst,
>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/diffState/compaction-sst-backup/000056.sst,
>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/diffState/compaction-sst-backup/000068.sst,
>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/diffState/compaction-sst-backup/000054.sst,
>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/diffState/compaction-sst-backup/000055.sst,
>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/diffState/compaction-sst-backup/000069.sst,
>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/diffState/compaction-sst-backup/000051.sst,
>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/checkpointState/om.db-65b61a8e-412a-4d2d-bb82-fca03aa09fe8/000062.sst,
>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/checkpointState/om.db-65b61a8e-412a-4d2d-bb82-fca03aa09fe8/000061.sst,
>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/checkpointState/om.db-65b61a8e-412a-4d2d-bb82-fca03aa09fe8/000060.sst,
>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/checkpointState/om.db-65b61a8e-412a-4d2d-bb82-fca03aa09fe8/000059.sst,
>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/checkpointState/om.db-65b61a8e-412a-4d2d-bb82-fca03aa09fe8/000057.sst,
>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/checkpointState/om.db-65b61a8e-412a-4d2d-bb82-fca03aa09fe8/000056.sst,
>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/checkpointState/om.db-65b61a8e-412a-4d2d-bb82-fca03aa09fe8/000054.sst,
>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/checkpointState/om.db-65b61a8e-412a-4d2d-bb82-fca03aa09fe8/000055.sst,
>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/checkpointState/om.db-65b61a8e-412a-4d2d-bb82-fca03aa09fe8/000051.sst,
>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/checkpointState/om.db-65b61a8e-412a-4d2d-bb82-fca03aa09fe8/000053.sst,
>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/checkpointState/om.db-aa3c01d4-a80a-47b1-8cde-979b62f0390d/000076.sst,
>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/checkpointState/om.db-aa3c01d4-a80a-47b1-8cde-979b62f0390d/000060.sst,
>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/checkpointState/om.db-aa3c01d4-a80a-47b1-8cde-979b62f0390d/000064.sst,
>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/checkpointState/om.db-aa3c01d4-a80a-47b1-8cde-979b62f0390d/000059.sst,
>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/checkpointState/om.db-aa3c01d4-a80a-47b1-8cde-979b62f0390d/000065.sst,
>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/checkpointState/om.db-aa3c01d4-a80a-47b1-8cde-979b62f0390d/000073.sst,
>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/checkpointState/om.db-aa3c01d4-a80a-47b1-8cde-979b62f0390d/000067.sst,
>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/checkpointState/om.db-aa3c01d4-a80a-47b1-8cde-979b62f0390d/000072.sst,
>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/checkpointState/om.db-aa3c01d4-a80a-47b1-8cde-979b62f0390d/000080.sst,
>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/checkpointState/om.db-aa3c01d4-a80a-47b1-8cde-979b62f0390d/000057.sst,
>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/checkpointState/om.db-aa3c01d4-a80a-47b1-8cde-979b62f0390d/000081.sst,
>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/checkpointState/om.db-aa3c01d4-a80a-47b1-8cde-979b62f0390d/000068.sst,
>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/checkpointState/om.db-aa3c01d4-a80a-47b1-8cde-979b62f0390d/000054.sst,
>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/checkpointState/om.db-aa3c01d4-a80a-47b1-8cde-979b62f0390d/000051.sst,
>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/checkpointState/om.db-aa3c01d4-a80a-47b1-8cde-979b62f0390d/000079.sst,
>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/checkpointState/om.db-aa3c01d4-a80a-47b1-8cde-979b62f0390d/000078.sst,
>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/checkpointState/om.db-aa3c01d4-a80a-47b1-8cde-979b62f0390d/000053.sst]
> from the latest checkpoint.
> 2023-07-07 13:29:43,072 [pool-122-thread-1] INFO snapshot.OmSnapshotUtils
> (OmSnapshotUtils.java:createHardLinks(121)) - Swaminathan. Copying Files
> 2023-07-07 13:29:43,072 [pool-122-thread-1] INFO snapshot.OmSnapshotUtils
> (OmSnapshotUtils.java:createHardLinks(128)) - Swaminathan. Copying Files ->
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/000076.sst
> =>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/db.snapshots/checkpointState/om.db-780b4467-cf75-43ab-b027-2bd41051e6a6/000076.sst
> 2023-07-07 13:29:43,073 [pool-122-thread-1] INFO snapshot.OmSnapshotUtils
> (OmSnapshotUtils.java:createHardLinks(128)) - Swaminathan. Copying Files ->
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/000095.sst
> =>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/db.snapshots/checkpointState/om.db-780b4467-cf75-43ab-b027-2bd41051e6a6/000095.sst
> 2023-07-07 13:29:43,073 [pool-122-thread-1] INFO snapshot.OmSnapshotUtils
> (OmSnapshotUtils.java:createHardLinks(128)) - Swaminathan. Copying Files ->
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/000057.sst
> =>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/db.snapshots/checkpointState/om.db-780b4467-cf75-43ab-b027-2bd41051e6a6/000057.sst
> 2023-07-07 13:29:43,073 [pool-122-thread-1] INFO snapshot.OmSnapshotUtils
> (OmSnapshotUtils.java:createHardLinks(128)) - Swaminathan. Copying Files ->
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/000090.sst
> =>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/db.snapshots/checkpointState/om.db-780b4467-cf75-43ab-b027-2bd41051e6a6/000090.sst
> 2023-07-07 13:29:43,073 [pool-122-thread-1] INFO snapshot.OmSnapshotUtils
> (OmSnapshotUtils.java:createHardLinks(128)) - Swaminathan. Copying Files ->
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/000093.sst
> =>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/db.snapshots/checkpointState/om.db-780b4467-cf75-43ab-b027-2bd41051e6a6/000093.sst
> 2023-07-07 13:29:43,074 [pool-122-thread-1] INFO snapshot.OmSnapshotUtils
> (OmSnapshotUtils.java:createHardLinks(128)) - Swaminathan. Copying Files ->
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/000073.sst
> =>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/db.snapshots/diffState/compaction-sst-backup/000073.sst
> 2023-07-07 13:29:43,074 [pool-122-thread-1] INFO snapshot.OmSnapshotUtils
> (OmSnapshotUtils.java:createHardLinks(128)) - Swaminathan. Copying Files ->
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/000080.sst
> =>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/db.snapshots/checkpointState/om.db-780b4467-cf75-43ab-b027-2bd41051e6a6/000080.sst
> 2023-07-07 13:29:43,074 [pool-122-thread-1] INFO snapshot.OmSnapshotUtils
> (OmSnapshotUtils.java:createHardLinks(128)) - Swaminathan. Copying Files ->
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/000092.sst
> =>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/db.snapshots/checkpointState/om.db-780b4467-cf75-43ab-b027-2bd41051e6a6/000092.sst
> 2023-07-07 13:29:43,075 [pool-122-thread-1] INFO snapshot.OmSnapshotUtils
> (OmSnapshotUtils.java:createHardLinks(128)) - Swaminathan. Copying Files ->
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/000084.sst
> =>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/db.snapshots/diffState/compaction-sst-backup/000084.sst
> 2023-07-07 13:29:43,075 [pool-122-thread-1] INFO snapshot.OmSnapshotUtils
> (OmSnapshotUtils.java:createHardLinks(128)) - Swaminathan. Copying Files ->
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/000086.sst
> =>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/db.snapshots/checkpointState/om.db-780b4467-cf75-43ab-b027-2bd41051e6a6/000086.sst
> 2023-07-07 13:29:43,075 [pool-122-thread-1] INFO snapshot.OmSnapshotUtils
> (OmSnapshotUtils.java:createHardLinks(128)) - Swaminathan. Copying Files ->
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/000088.sst
> =>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/db.snapshots/checkpointState/om.db-780b4467-cf75-43ab-b027-2bd41051e6a6/000088.sst
> 2023-07-07 13:29:43,075 [pool-122-thread-1] INFO snapshot.OmSnapshotUtils
> (OmSnapshotUtils.java:createHardLinks(128)) - Swaminathan. Copying Files ->
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/000064.sst
> =>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/db.snapshots/checkpointState/om.db-780b4467-cf75-43ab-b027-2bd41051e6a6/000064.sst
> 2023-07-07 13:29:43,076 [pool-122-thread-1] INFO snapshot.OmSnapshotUtils
> (OmSnapshotUtils.java:createHardLinks(128)) - Swaminathan. Copying Files ->
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/000083.sst
> =>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/db.snapshots/checkpointState/om.db-780b4467-cf75-43ab-b027-2bd41051e6a6/000083.sst
> 2023-07-07 13:29:43,076 [pool-122-thread-1] INFO snapshot.OmSnapshotUtils
> (OmSnapshotUtils.java:createHardLinks(128)) - Swaminathan. Copying Files ->
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/000081.sst
> =>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/db.snapshots/checkpointState/om.db-780b4467-cf75-43ab-b027-2bd41051e6a6/000081.sst
> 2023-07-07 13:29:43,076 [pool-122-thread-1] INFO snapshot.OmSnapshotUtils
> (OmSnapshotUtils.java:createHardLinks(128)) - Swaminathan. Copying Files ->
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/db.snapshots/checkpointState/om.db-65b61a8e-412a-4d2d-bb82-fca03aa09fe8/000051.sst
> =>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/db.snapshots/checkpointState/om.db-780b4467-cf75-43ab-b027-2bd41051e6a6/000051.sst
> 2023-07-07 13:29:43,077 [pool-122-thread-1] INFO snapshot.OmSnapshotUtils
> (OmSnapshotUtils.java:createHardLinks(128)) - Swaminathan. Copying Files ->
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/000094.sst
> =>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/db.snapshots/checkpointState/om.db-780b4467-cf75-43ab-b027-2bd41051e6a6/000094.sst
> 2023-07-07 13:29:43,077 [pool-122-thread-1] INFO snapshot.OmSnapshotUtils
> (OmSnapshotUtils.java:createHardLinks(128)) - Swaminathan. Copying Files ->
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/db.snapshots/checkpointState/om.db-aa3c01d4-a80a-47b1-8cde-979b62f0390d/000053.sst
> =>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/db.snapshots/checkpointState/om.db-780b4467-cf75-43ab-b027-2bd41051e6a6/000053.sst
> 2023-07-07 13:29:43,077 [pool-122-thread-1] INFO snapshot.OmSnapshotUtils
> (OmSnapshotUtils.java:createHardLinks(128)) - Swaminathan. Copying Files ->
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/db.snapshots/diffState/compaction-sst-backup/000067.sst
> =>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/db.snapshots/checkpointState/om.db-780b4467-cf75-43ab-b027-2bd41051e6a6/000067.sst
> 2023-07-07 13:29:43,077 [pool-122-thread-1] INFO snapshot.OmSnapshotUtils
> (OmSnapshotUtils.java:createHardLinks(128)) - Swaminathan. Copying Files ->
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/db.snapshots/checkpointState/om.db-aa3c01d4-a80a-47b1-8cde-979b62f0390d/000079.sst
> =>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/db.snapshots/diffState/compaction-sst-backup/000079.sst
> 2023-07-07 13:29:43,078 [pool-122-thread-1] INFO snapshot.OmSnapshotUtils
> (OmSnapshotUtils.java:createHardLinks(128)) - Swaminathan. Copying Files ->
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/db.snapshots/checkpointState/om.db-aa3c01d4-a80a-47b1-8cde-979b62f0390d/000078.sst
> =>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/db.snapshots/checkpointState/om.db-780b4467-cf75-43ab-b027-2bd41051e6a6/000078.sst
> 2023-07-07 13:29:43,078 [pool-122-thread-1] INFO snapshot.OmSnapshotUtils
> (OmSnapshotUtils.java:createHardLinks(128)) - Swaminathan. Copying Files ->
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/db.snapshots/checkpointState/om.db-65b61a8e-412a-4d2d-bb82-fca03aa09fe8/000059.sst
> =>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/db.snapshots/checkpointState/om.db-780b4467-cf75-43ab-b027-2bd41051e6a6/000059.sst
> 2023-07-07 13:29:43,078 [pool-122-thread-1] INFO snapshot.OmSnapshotUtils
> (OmSnapshotUtils.java:createHardLinks(128)) - Swaminathan. Copying Files ->
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/000072.sst
> =>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/db.snapshots/checkpointState/om.db-780b4467-cf75-43ab-b027-2bd41051e6a6/000072.sst
> 2023-07-07 13:29:43,079 [pool-122-thread-1] INFO snapshot.OmSnapshotUtils
> (OmSnapshotUtils.java:createHardLinks(128)) - Swaminathan. Copying Files ->
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/000089.sst
> =>
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/db.snapshots/checkpointState/om.db-780b4467-cf75-43ab-b027-2bd41051e6a6/000089.sst
> 2023-07-07 13:29:43,079 [pool-122-thread-1] INFO snapshot.OmSnapshotUtils
> (OmSnapshotUtils.java:createHardLinks(133)) - Swaminathan. Copying Files
> 2023-07-07 13:29:43,485 [pool-122-thread-1] INFO rocksdb.RocksDB
> (DBStoreBuilder.java:log(397)) - [le/delete_scheduler.cc:77] Deleted file
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/db.snapshots/checkpointState/om.db-aa3c01d4-a80a-47b1-8cde-979b62f0390d/000076.sst
> immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio
> 0.250000
> 2023-07-07 13:29:43,485 [pool-122-thread-1] INFO rocksdb.RocksDB
> (DBStoreBuilder.java:log(397)) - [/db_impl/db_impl_files.cc:361] [JOB 3]
> Delete
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/db.snapshots/checkpointState/om.db-aa3c01d4-a80a-47b1-8cde-979b62f0390d/000076.sst
> type=2 #76 -- OK
> 2023-07-07 13:29:43,504 [pool-122-thread-1] INFO rocksdb.RocksDB
> (DBStoreBuilder.java:log(397)) - SST files in
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/om.db
> dir, Total Num: 22, files: 000051.sst 000053.sst 000057.sst 000059.sst
> 000064.sst 000067.sst 000072.sst 000076.sst 000078.sst
> 2023-07-07 13:29:45,019 [Thread-2839] INFO rocksdb.RocksDB
> (DBStoreBuilder.java:log(397)) - [le/delete_scheduler.cc:77] Deleted file
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/om.db/000076.sst
> immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio
> 0.250000
> 2023-07-07 13:29:45,019 [Thread-2840] INFO rocksdb.RocksDB
> (DBStoreBuilder.java:log(397)) - [/db_impl/db_impl_files.cc:361] [JOB 49]
> Delete
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/om.db/000076.sst
> type=2 #76 -- OK
> 2023-07-07 13:29:45,624 [Thread-3062] INFO rocksdb.RocksDB
> (DBStoreBuilder.java:log(397)) - [le/delete_scheduler.cc:77] Deleted file
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-1/om.db/000076.sst
> immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio
> 0.250000
> 2023-07-07 13:29:45,625 [Thread-3063] INFO rocksdb.RocksDB
> (DBStoreBuilder.java:log(397)) - [/db_impl/db_impl_files.cc:361] [JOB 43]
> Delete
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-1/om.db/000076.sst
> type=2 #76 -- OK
> 2023-07-07 13:29:46,531 [Thread-3400] INFO rocksdb.RocksDB
> (DBStoreBuilder.java:log(397)) - [le/delete_scheduler.cc:77] Deleted file
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/om.db/000076.sst
> immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio
> 0.250000
> 2023-07-07 13:29:46,531 [Thread-3401] INFO rocksdb.RocksDB
> (DBStoreBuilder.java:log(397)) - [/db_impl/db_impl_files.cc:361] [JOB 9]
> Delete
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/om.db/000076.sst
> type=2 #76 -- OK
> java.lang.IllegalStateException: java.io.IOException: Failed init RocksDB, db
> path :
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/db.snapshots/checkpointState/om.db-aa3c01d4-a80a-47b1-8cde-979b62f0390d,
> exception :org.rocksdb.RocksDBException Corruption: IO error: No such file
> or directory: While open a file for random read:
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/db.snapshots/checkpointState/om.db-aa3c01d4-a80a-47b1-8cde-979b62f0390d/000076.ldb:
> No such file or directory in file
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/db.snapshots/checkpointState/om.db-aa3c01d4-a80a-47b1-8cde-979b62f0390d/MANIFEST-000005
> at
> org.apache.hadoop.ozone.om.snapshot.SnapshotCache.lambda$get$1(SnapshotCache.java:179)
> at
> java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1892)
> at
> org.apache.hadoop.ozone.om.snapshot.SnapshotCache.get(SnapshotCache.java:167)
> at
> org.apache.hadoop.ozone.om.OmSnapshotManager.checkForSnapshot(OmSnapshotManager.java:699)
> at
> org.apache.hadoop.ozone.om.OzoneManager.getReader(OzoneManager.java:4580)
> at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:2810)
> at
> org.apache.hadoop.ozone.om.TestOMRatisSnapshots.checkSnapshot(TestOMRatisSnapshots.java:310)
> at
> org.apache.hadoop.ozone.om.TestOMRatisSnapshots.testInstallIncrementalSnapshot(TestOMRatisSnapshots.java:475)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:725)
> at
> org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
> at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
> at
> org.junit.jupiter.engine.extension.TimeoutInvocation.proceed(TimeoutInvocation.java:46)
> at
> org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
> at
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140)
> at
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:84)
> at
> org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
> at
> org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
> at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
> at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
> at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
> at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
> at
> org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
> at
> org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
> at
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:214)
> at
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
> at
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:210)
> at
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:135)
> at
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:66)
> at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
> at
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
> at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
> at
> org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
> at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
> at
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
> at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
> at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
> at java.util.ArrayList.forEach(ArrayList.java:1259) at
> org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
> at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
> at
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
> at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
> at
> org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
> at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
> at
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
> at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
> at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
> at java.util.ArrayList.forEach(ArrayList.java:1259) at
> org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
> at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
> at
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
> at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
> at
> org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
> at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
> at
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
> at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
> at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
> at
> org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:35)
> at
> org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
> at
> org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:54)
> at
> org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:107)
> at
> org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:88)
> at
> org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:54)
> at
> org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:67)
> at
> org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:52)
> at
> org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114)
> at
> org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86)
> at
> org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86)
> at
> org.junit.platform.launcher.core.SessionPerRequestLauncher.execute(SessionPerRequestLauncher.java:53)
> at
> com.intellij.junit5.JUnit5IdeaTestRunner.startRunnerWithArgs(JUnit5IdeaTestRunner.java:71)
> at
> com.intellij.rt.junit.IdeaTestRunner$Repeater$1.execute(IdeaTestRunner.java:38)
> at
> com.intellij.rt.execution.junit.TestsRepeater.repeat(TestsRepeater.java:11)
> at
> com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:35)
> at
> com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:235)
> at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:54)Caused
> by: java.io.IOException: Failed init RocksDB, db path :
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/db.snapshots/checkpointState/om.db-aa3c01d4-a80a-47b1-8cde-979b62f0390d,
> exception :org.rocksdb.RocksDBException Corruption: IO error: No such file
> or directory: While open a file for random read:
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/db.snapshots/checkpointState/om.db-aa3c01d4-a80a-47b1-8cde-979b62f0390d/000076.ldb:
> No such file or directory in file
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/db.snapshots/checkpointState/om.db-aa3c01d4-a80a-47b1-8cde-979b62f0390d/MANIFEST-000005
> at org.apache.hadoop.hdds.utils.db.RDBStore.<init>(RDBStore.java:172)
> at
> org.apache.hadoop.hdds.utils.db.DBStoreBuilder.build(DBStoreBuilder.java:220)
> at
> org.apache.hadoop.ozone.om.OmMetadataManagerImpl.loadDB(OmMetadataManagerImpl.java:583)
> at
> org.apache.hadoop.ozone.om.OmMetadataManagerImpl.<init>(OmMetadataManagerImpl.java:391)
> at
> org.apache.hadoop.ozone.om.OmSnapshotManager$1.load(OmSnapshotManager.java:360)
> at
> org.apache.hadoop.ozone.om.OmSnapshotManager$1.load(OmSnapshotManager.java:330)
> at
> org.apache.hadoop.ozone.om.snapshot.SnapshotCache.lambda$get$1(SnapshotCache.java:171)
> ... 77 moreCaused by: java.io.IOException: class
> org.apache.hadoop.hdds.utils.db.RocksDatabase: Failed to open
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/db.snapshots/checkpointState/om.db-aa3c01d4-a80a-47b1-8cde-979b62f0390d;
> status : Corruption; message : Corruption: IO error: No such file or
> directory: While open a file for random read:
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/db.snapshots/checkpointState/om.db-aa3c01d4-a80a-47b1-8cde-979b62f0390d/000076.ldb:
> No such file or directory in file
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/db.snapshots/checkpointState/om.db-aa3c01d4-a80a-47b1-8cde-979b62f0390d/MANIFEST-000005
> at
> org.apache.hadoop.hdds.utils.HddsServerUtil.toIOException(HddsServerUtil.java:628)
> at
> org.apache.hadoop.hdds.utils.db.RocksDatabase.toIOException(RocksDatabase.java:90)
> at
> org.apache.hadoop.hdds.utils.db.RocksDatabase.open(RocksDatabase.java:167)
> at org.apache.hadoop.hdds.utils.db.RDBStore.<init>(RDBStore.java:110) ...
> 83 moreCaused by: org.rocksdb.RocksDBException: Corruption: IO error: No such
> file or directory: While open a file for random read:
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/db.snapshots/checkpointState/om.db-aa3c01d4-a80a-47b1-8cde-979b62f0390d/000076.ldb:
> No such file or directory in file
> /Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/db.snapshots/checkpointState/om.db-aa3c01d4-a80a-47b1-8cde-979b62f0390d/MANIFEST-000005
> at org.rocksdb.RocksDB.open(Native Method) at
> org.rocksdb.RocksDB.open(RocksDB.java:307) at
> org.apache.hadoop.hdds.utils.db.managed.ManagedRocksDB.open(ManagedRocksDB.java:67)
> at
> org.apache.hadoop.hdds.utils.db.RocksDatabase.open(RocksDatabase.java:153)
> ... 84 more
> {code}
> In order to avoid such race condition, sstfiltering service should be stopped
> before copying the sst files.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
