Swaminathan Balachandran created HDDS-8987:
----------------------------------------------
Summary: SST Filtering service should be stopped while before
copying incremental snapshot
Key: HDDS-8987
URL: https://issues.apache.org/jira/browse/HDDS-8987
Project: Apache Ozone
Issue Type: Sub-task
Components: Snapshot
Reporter: Swaminathan Balachandran
Assignee: Swaminathan Balachandran
Attachments: a.log
Currently when only incremental sst files are copied. The sst files are listed
in the candidate rocksdb directory & are excluded from copying from leader. But
SSTFiltering service could interfere with this and go about deleting the sst
files which have been excluded while the data is being copied. Thus when the
leader state is copied and installed certain sst files may be missing which
could be present in the leader's sst manifest file.
Attached are some logs for this.
{code:java}
2023-07-07 13:29:42,626 [qtp501491806-254] INFO utils.DBCheckpointServlet
(DBCheckpointServlet.java:generateSnapshotCheckpoint(203)) - Excluded SST
[/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.checkpoints/om.db_checkpoint_1688761782175/000076.sst,
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.checkpoints/om.db_checkpoint_1688761782175/000064.sst,
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.checkpoints/om.db_checkpoint_1688761782175/000059.sst,
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.checkpoints/om.db_checkpoint_1688761782175/000067.sst,
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.checkpoints/om.db_checkpoint_1688761782175/000072.sst,
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.checkpoints/om.db_checkpoint_1688761782175/000080.sst,
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.checkpoints/om.db_checkpoint_1688761782175/000057.sst,
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.checkpoints/om.db_checkpoint_1688761782175/000081.sst,
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.checkpoints/om.db_checkpoint_1688761782175/000083.sst,
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.checkpoints/om.db_checkpoint_1688761782175/000051.sst,
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.checkpoints/om.db_checkpoint_1688761782175/000078.sst,
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.checkpoints/om.db_checkpoint_1688761782175/000053.sst,
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/diffState/compaction-sst-backup/000062.sst,
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/diffState/compaction-sst-backup/000061.sst,
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/diffState/compaction-sst-backup/000075.sst,
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/diffState/compaction-sst-backup/000074.sst,
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/diffState/compaction-sst-backup/000060.sst,
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/diffState/compaction-sst-backup/000070.sst,
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/diffState/compaction-sst-backup/000059.sst,
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/diffState/compaction-sst-backup/000065.sst,
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/diffState/compaction-sst-backup/000067.sst,
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/diffState/compaction-sst-backup/000072.sst,
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/diffState/compaction-sst-backup/000056.sst,
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/diffState/compaction-sst-backup/000068.sst,
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/diffState/compaction-sst-backup/000054.sst,
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/diffState/compaction-sst-backup/000055.sst,
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/diffState/compaction-sst-backup/000069.sst,
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/diffState/compaction-sst-backup/000051.sst,
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/checkpointState/om.db-65b61a8e-412a-4d2d-bb82-fca03aa09fe8/000062.sst,
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/checkpointState/om.db-65b61a8e-412a-4d2d-bb82-fca03aa09fe8/000061.sst,
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/checkpointState/om.db-65b61a8e-412a-4d2d-bb82-fca03aa09fe8/000060.sst,
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/checkpointState/om.db-65b61a8e-412a-4d2d-bb82-fca03aa09fe8/000059.sst,
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/checkpointState/om.db-65b61a8e-412a-4d2d-bb82-fca03aa09fe8/000057.sst,
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/checkpointState/om.db-65b61a8e-412a-4d2d-bb82-fca03aa09fe8/000056.sst,
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/checkpointState/om.db-65b61a8e-412a-4d2d-bb82-fca03aa09fe8/000054.sst,
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/checkpointState/om.db-65b61a8e-412a-4d2d-bb82-fca03aa09fe8/000055.sst,
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/checkpointState/om.db-65b61a8e-412a-4d2d-bb82-fca03aa09fe8/000051.sst,
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/checkpointState/om.db-65b61a8e-412a-4d2d-bb82-fca03aa09fe8/000053.sst,
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/checkpointState/om.db-aa3c01d4-a80a-47b1-8cde-979b62f0390d/000076.sst,
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/checkpointState/om.db-aa3c01d4-a80a-47b1-8cde-979b62f0390d/000060.sst,
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/checkpointState/om.db-aa3c01d4-a80a-47b1-8cde-979b62f0390d/000064.sst,
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/checkpointState/om.db-aa3c01d4-a80a-47b1-8cde-979b62f0390d/000059.sst,
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/checkpointState/om.db-aa3c01d4-a80a-47b1-8cde-979b62f0390d/000065.sst,
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/checkpointState/om.db-aa3c01d4-a80a-47b1-8cde-979b62f0390d/000073.sst,
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/checkpointState/om.db-aa3c01d4-a80a-47b1-8cde-979b62f0390d/000067.sst,
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/checkpointState/om.db-aa3c01d4-a80a-47b1-8cde-979b62f0390d/000072.sst,
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/checkpointState/om.db-aa3c01d4-a80a-47b1-8cde-979b62f0390d/000080.sst,
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/checkpointState/om.db-aa3c01d4-a80a-47b1-8cde-979b62f0390d/000057.sst,
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/checkpointState/om.db-aa3c01d4-a80a-47b1-8cde-979b62f0390d/000081.sst,
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/checkpointState/om.db-aa3c01d4-a80a-47b1-8cde-979b62f0390d/000068.sst,
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/checkpointState/om.db-aa3c01d4-a80a-47b1-8cde-979b62f0390d/000054.sst,
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/checkpointState/om.db-aa3c01d4-a80a-47b1-8cde-979b62f0390d/000051.sst,
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/checkpointState/om.db-aa3c01d4-a80a-47b1-8cde-979b62f0390d/000079.sst,
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/checkpointState/om.db-aa3c01d4-a80a-47b1-8cde-979b62f0390d/000078.sst,
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/db.snapshots/checkpointState/om.db-aa3c01d4-a80a-47b1-8cde-979b62f0390d/000053.sst]
from the latest checkpoint.
2023-07-07 13:29:43,072 [pool-122-thread-1] INFO snapshot.OmSnapshotUtils
(OmSnapshotUtils.java:createHardLinks(121)) - Swaminathan. Copying Files
2023-07-07 13:29:43,072 [pool-122-thread-1] INFO snapshot.OmSnapshotUtils
(OmSnapshotUtils.java:createHardLinks(128)) - Swaminathan. Copying Files ->
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/000076.sst
=>
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/db.snapshots/checkpointState/om.db-780b4467-cf75-43ab-b027-2bd41051e6a6/000076.sst
2023-07-07 13:29:43,073 [pool-122-thread-1] INFO snapshot.OmSnapshotUtils
(OmSnapshotUtils.java:createHardLinks(128)) - Swaminathan. Copying Files ->
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/000095.sst
=>
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/db.snapshots/checkpointState/om.db-780b4467-cf75-43ab-b027-2bd41051e6a6/000095.sst
2023-07-07 13:29:43,073 [pool-122-thread-1] INFO snapshot.OmSnapshotUtils
(OmSnapshotUtils.java:createHardLinks(128)) - Swaminathan. Copying Files ->
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/000057.sst
=>
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/db.snapshots/checkpointState/om.db-780b4467-cf75-43ab-b027-2bd41051e6a6/000057.sst
2023-07-07 13:29:43,073 [pool-122-thread-1] INFO snapshot.OmSnapshotUtils
(OmSnapshotUtils.java:createHardLinks(128)) - Swaminathan. Copying Files ->
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/000090.sst
=>
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/db.snapshots/checkpointState/om.db-780b4467-cf75-43ab-b027-2bd41051e6a6/000090.sst
2023-07-07 13:29:43,073 [pool-122-thread-1] INFO snapshot.OmSnapshotUtils
(OmSnapshotUtils.java:createHardLinks(128)) - Swaminathan. Copying Files ->
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/000093.sst
=>
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/db.snapshots/checkpointState/om.db-780b4467-cf75-43ab-b027-2bd41051e6a6/000093.sst
2023-07-07 13:29:43,074 [pool-122-thread-1] INFO snapshot.OmSnapshotUtils
(OmSnapshotUtils.java:createHardLinks(128)) - Swaminathan. Copying Files ->
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/000073.sst
=>
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/db.snapshots/diffState/compaction-sst-backup/000073.sst
2023-07-07 13:29:43,074 [pool-122-thread-1] INFO snapshot.OmSnapshotUtils
(OmSnapshotUtils.java:createHardLinks(128)) - Swaminathan. Copying Files ->
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/000080.sst
=>
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/db.snapshots/checkpointState/om.db-780b4467-cf75-43ab-b027-2bd41051e6a6/000080.sst
2023-07-07 13:29:43,074 [pool-122-thread-1] INFO snapshot.OmSnapshotUtils
(OmSnapshotUtils.java:createHardLinks(128)) - Swaminathan. Copying Files ->
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/000092.sst
=>
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/db.snapshots/checkpointState/om.db-780b4467-cf75-43ab-b027-2bd41051e6a6/000092.sst
2023-07-07 13:29:43,075 [pool-122-thread-1] INFO snapshot.OmSnapshotUtils
(OmSnapshotUtils.java:createHardLinks(128)) - Swaminathan. Copying Files ->
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/000084.sst
=>
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/db.snapshots/diffState/compaction-sst-backup/000084.sst
2023-07-07 13:29:43,075 [pool-122-thread-1] INFO snapshot.OmSnapshotUtils
(OmSnapshotUtils.java:createHardLinks(128)) - Swaminathan. Copying Files ->
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/000086.sst
=>
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/db.snapshots/checkpointState/om.db-780b4467-cf75-43ab-b027-2bd41051e6a6/000086.sst
2023-07-07 13:29:43,075 [pool-122-thread-1] INFO snapshot.OmSnapshotUtils
(OmSnapshotUtils.java:createHardLinks(128)) - Swaminathan. Copying Files ->
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/000088.sst
=>
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/db.snapshots/checkpointState/om.db-780b4467-cf75-43ab-b027-2bd41051e6a6/000088.sst
2023-07-07 13:29:43,075 [pool-122-thread-1] INFO snapshot.OmSnapshotUtils
(OmSnapshotUtils.java:createHardLinks(128)) - Swaminathan. Copying Files ->
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/000064.sst
=>
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/db.snapshots/checkpointState/om.db-780b4467-cf75-43ab-b027-2bd41051e6a6/000064.sst
2023-07-07 13:29:43,076 [pool-122-thread-1] INFO snapshot.OmSnapshotUtils
(OmSnapshotUtils.java:createHardLinks(128)) - Swaminathan. Copying Files ->
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/000083.sst
=>
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/db.snapshots/checkpointState/om.db-780b4467-cf75-43ab-b027-2bd41051e6a6/000083.sst
2023-07-07 13:29:43,076 [pool-122-thread-1] INFO snapshot.OmSnapshotUtils
(OmSnapshotUtils.java:createHardLinks(128)) - Swaminathan. Copying Files ->
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/000081.sst
=>
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/db.snapshots/checkpointState/om.db-780b4467-cf75-43ab-b027-2bd41051e6a6/000081.sst
2023-07-07 13:29:43,076 [pool-122-thread-1] INFO snapshot.OmSnapshotUtils
(OmSnapshotUtils.java:createHardLinks(128)) - Swaminathan. Copying Files ->
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/db.snapshots/checkpointState/om.db-65b61a8e-412a-4d2d-bb82-fca03aa09fe8/000051.sst
=>
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/db.snapshots/checkpointState/om.db-780b4467-cf75-43ab-b027-2bd41051e6a6/000051.sst
2023-07-07 13:29:43,077 [pool-122-thread-1] INFO snapshot.OmSnapshotUtils
(OmSnapshotUtils.java:createHardLinks(128)) - Swaminathan. Copying Files ->
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/000094.sst
=>
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/db.snapshots/checkpointState/om.db-780b4467-cf75-43ab-b027-2bd41051e6a6/000094.sst
2023-07-07 13:29:43,077 [pool-122-thread-1] INFO snapshot.OmSnapshotUtils
(OmSnapshotUtils.java:createHardLinks(128)) - Swaminathan. Copying Files ->
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/db.snapshots/checkpointState/om.db-aa3c01d4-a80a-47b1-8cde-979b62f0390d/000053.sst
=>
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/db.snapshots/checkpointState/om.db-780b4467-cf75-43ab-b027-2bd41051e6a6/000053.sst
2023-07-07 13:29:43,077 [pool-122-thread-1] INFO snapshot.OmSnapshotUtils
(OmSnapshotUtils.java:createHardLinks(128)) - Swaminathan. Copying Files ->
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/db.snapshots/diffState/compaction-sst-backup/000067.sst
=>
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/db.snapshots/checkpointState/om.db-780b4467-cf75-43ab-b027-2bd41051e6a6/000067.sst
2023-07-07 13:29:43,077 [pool-122-thread-1] INFO snapshot.OmSnapshotUtils
(OmSnapshotUtils.java:createHardLinks(128)) - Swaminathan. Copying Files ->
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/db.snapshots/checkpointState/om.db-aa3c01d4-a80a-47b1-8cde-979b62f0390d/000079.sst
=>
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/db.snapshots/diffState/compaction-sst-backup/000079.sst
2023-07-07 13:29:43,078 [pool-122-thread-1] INFO snapshot.OmSnapshotUtils
(OmSnapshotUtils.java:createHardLinks(128)) - Swaminathan. Copying Files ->
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/db.snapshots/checkpointState/om.db-aa3c01d4-a80a-47b1-8cde-979b62f0390d/000078.sst
=>
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/db.snapshots/checkpointState/om.db-780b4467-cf75-43ab-b027-2bd41051e6a6/000078.sst
2023-07-07 13:29:43,078 [pool-122-thread-1] INFO snapshot.OmSnapshotUtils
(OmSnapshotUtils.java:createHardLinks(128)) - Swaminathan. Copying Files ->
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/db.snapshots/checkpointState/om.db-65b61a8e-412a-4d2d-bb82-fca03aa09fe8/000059.sst
=>
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/db.snapshots/checkpointState/om.db-780b4467-cf75-43ab-b027-2bd41051e6a6/000059.sst
2023-07-07 13:29:43,078 [pool-122-thread-1] INFO snapshot.OmSnapshotUtils
(OmSnapshotUtils.java:createHardLinks(128)) - Swaminathan. Copying Files ->
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/000072.sst
=>
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/db.snapshots/checkpointState/om.db-780b4467-cf75-43ab-b027-2bd41051e6a6/000072.sst
2023-07-07 13:29:43,079 [pool-122-thread-1] INFO snapshot.OmSnapshotUtils
(OmSnapshotUtils.java:createHardLinks(128)) - Swaminathan. Copying Files ->
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/000089.sst
=>
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/snapshot/om.db.candidate/db.snapshots/checkpointState/om.db-780b4467-cf75-43ab-b027-2bd41051e6a6/000089.sst
2023-07-07 13:29:43,079 [pool-122-thread-1] INFO snapshot.OmSnapshotUtils
(OmSnapshotUtils.java:createHardLinks(133)) - Swaminathan. Copying Files
2023-07-07 13:29:43,485 [pool-122-thread-1] INFO rocksdb.RocksDB
(DBStoreBuilder.java:log(397)) - [le/delete_scheduler.cc:77] Deleted file
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/db.snapshots/checkpointState/om.db-aa3c01d4-a80a-47b1-8cde-979b62f0390d/000076.sst
immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio
0.250000
2023-07-07 13:29:43,485 [pool-122-thread-1] INFO rocksdb.RocksDB
(DBStoreBuilder.java:log(397)) - [/db_impl/db_impl_files.cc:361] [JOB 3] Delete
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/db.snapshots/checkpointState/om.db-aa3c01d4-a80a-47b1-8cde-979b62f0390d/000076.sst
type=2 #76 -- OK
2023-07-07 13:29:43,504 [pool-122-thread-1] INFO rocksdb.RocksDB
(DBStoreBuilder.java:log(397)) - SST files in
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/om.db
dir, Total Num: 22, files: 000051.sst 000053.sst 000057.sst 000059.sst
000064.sst 000067.sst 000072.sst 000076.sst 000078.sst
2023-07-07 13:29:45,019 [Thread-2839] INFO rocksdb.RocksDB
(DBStoreBuilder.java:log(397)) - [le/delete_scheduler.cc:77] Deleted file
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/om.db/000076.sst
immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio
0.250000
2023-07-07 13:29:45,019 [Thread-2840] INFO rocksdb.RocksDB
(DBStoreBuilder.java:log(397)) - [/db_impl/db_impl_files.cc:361] [JOB 49]
Delete
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-2/om.db/000076.sst
type=2 #76 -- OK
2023-07-07 13:29:45,624 [Thread-3062] INFO rocksdb.RocksDB
(DBStoreBuilder.java:log(397)) - [le/delete_scheduler.cc:77] Deleted file
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-1/om.db/000076.sst
immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio
0.250000
2023-07-07 13:29:45,625 [Thread-3063] INFO rocksdb.RocksDB
(DBStoreBuilder.java:log(397)) - [/db_impl/db_impl_files.cc:361] [JOB 43]
Delete
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-1/om.db/000076.sst
type=2 #76 -- OK
2023-07-07 13:29:46,531 [Thread-3400] INFO rocksdb.RocksDB
(DBStoreBuilder.java:log(397)) - [le/delete_scheduler.cc:77] Deleted file
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/om.db/000076.sst
immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio
0.250000
2023-07-07 13:29:46,531 [Thread-3401] INFO rocksdb.RocksDB
(DBStoreBuilder.java:log(397)) - [/db_impl/db_impl_files.cc:361] [JOB 9] Delete
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/om.db/000076.sst
type=2 #76 -- OK
java.lang.IllegalStateException: java.io.IOException: Failed init RocksDB, db
path :
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/db.snapshots/checkpointState/om.db-aa3c01d4-a80a-47b1-8cde-979b62f0390d,
exception :org.rocksdb.RocksDBException Corruption: IO error: No such file or
directory: While open a file for random read:
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/db.snapshots/checkpointState/om.db-aa3c01d4-a80a-47b1-8cde-979b62f0390d/000076.ldb:
No such file or directory in file
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/db.snapshots/checkpointState/om.db-aa3c01d4-a80a-47b1-8cde-979b62f0390d/MANIFEST-000005
at
org.apache.hadoop.ozone.om.snapshot.SnapshotCache.lambda$get$1(SnapshotCache.java:179)
at
java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1892) at
org.apache.hadoop.ozone.om.snapshot.SnapshotCache.get(SnapshotCache.java:167)
at
org.apache.hadoop.ozone.om.OmSnapshotManager.checkForSnapshot(OmSnapshotManager.java:699)
at
org.apache.hadoop.ozone.om.OzoneManager.getReader(OzoneManager.java:4580) at
org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:2810) at
org.apache.hadoop.ozone.om.TestOMRatisSnapshots.checkSnapshot(TestOMRatisSnapshots.java:310)
at
org.apache.hadoop.ozone.om.TestOMRatisSnapshots.testInstallIncrementalSnapshot(TestOMRatisSnapshots.java:475)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:725)
at
org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
at
org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
at
org.junit.jupiter.engine.extension.TimeoutInvocation.proceed(TimeoutInvocation.java:46)
at
org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
at
org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140)
at
org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:84)
at
org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
at
org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
at
org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
at
org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
at
org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
at
org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
at
org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
at
org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
at
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:214)
at
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:210)
at
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:135)
at
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:66)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
at
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
at
org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137) at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
at
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
at java.util.ArrayList.forEach(ArrayList.java:1259) at
org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
at
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
at
org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137) at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
at
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
at java.util.ArrayList.forEach(ArrayList.java:1259) at
org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
at
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
at
org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137) at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
at
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
at
org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:35)
at
org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
at
org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:54)
at
org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:107)
at
org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:88)
at
org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:54)
at
org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:67)
at
org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:52)
at
org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114)
at
org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86)
at
org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86)
at
org.junit.platform.launcher.core.SessionPerRequestLauncher.execute(SessionPerRequestLauncher.java:53)
at
com.intellij.junit5.JUnit5IdeaTestRunner.startRunnerWithArgs(JUnit5IdeaTestRunner.java:71)
at
com.intellij.rt.junit.IdeaTestRunner$Repeater$1.execute(IdeaTestRunner.java:38)
at
com.intellij.rt.execution.junit.TestsRepeater.repeat(TestsRepeater.java:11) at
com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:35)
at
com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:235)
at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:54)Caused by:
java.io.IOException: Failed init RocksDB, db path :
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/db.snapshots/checkpointState/om.db-aa3c01d4-a80a-47b1-8cde-979b62f0390d,
exception :org.rocksdb.RocksDBException Corruption: IO error: No such file or
directory: While open a file for random read:
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/db.snapshots/checkpointState/om.db-aa3c01d4-a80a-47b1-8cde-979b62f0390d/000076.ldb:
No such file or directory in file
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/db.snapshots/checkpointState/om.db-aa3c01d4-a80a-47b1-8cde-979b62f0390d/MANIFEST-000005
at org.apache.hadoop.hdds.utils.db.RDBStore.<init>(RDBStore.java:172) at
org.apache.hadoop.hdds.utils.db.DBStoreBuilder.build(DBStoreBuilder.java:220)
at
org.apache.hadoop.ozone.om.OmMetadataManagerImpl.loadDB(OmMetadataManagerImpl.java:583)
at
org.apache.hadoop.ozone.om.OmMetadataManagerImpl.<init>(OmMetadataManagerImpl.java:391)
at
org.apache.hadoop.ozone.om.OmSnapshotManager$1.load(OmSnapshotManager.java:360)
at
org.apache.hadoop.ozone.om.OmSnapshotManager$1.load(OmSnapshotManager.java:330)
at
org.apache.hadoop.ozone.om.snapshot.SnapshotCache.lambda$get$1(SnapshotCache.java:171)
... 77 moreCaused by: java.io.IOException: class
org.apache.hadoop.hdds.utils.db.RocksDatabase: Failed to open
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/db.snapshots/checkpointState/om.db-aa3c01d4-a80a-47b1-8cde-979b62f0390d;
status : Corruption; message : Corruption: IO error: No such file or
directory: While open a file for random read:
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/db.snapshots/checkpointState/om.db-aa3c01d4-a80a-47b1-8cde-979b62f0390d/000076.ldb:
No such file or directory in file
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/db.snapshots/checkpointState/om.db-aa3c01d4-a80a-47b1-8cde-979b62f0390d/MANIFEST-000005
at
org.apache.hadoop.hdds.utils.HddsServerUtil.toIOException(HddsServerUtil.java:628)
at
org.apache.hadoop.hdds.utils.db.RocksDatabase.toIOException(RocksDatabase.java:90)
at
org.apache.hadoop.hdds.utils.db.RocksDatabase.open(RocksDatabase.java:167) at
org.apache.hadoop.hdds.utils.db.RDBStore.<init>(RDBStore.java:110) ... 83
moreCaused by: org.rocksdb.RocksDBException: Corruption: IO error: No such file
or directory: While open a file for random read:
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/db.snapshots/checkpointState/om.db-aa3c01d4-a80a-47b1-8cde-979b62f0390d/000076.ldb:
No such file or directory in file
/Users/sbalachandran/Documents/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-2ff22622-5883-4e2b-8090-8276162d8e5a/omNode-3/db.snapshots/checkpointState/om.db-aa3c01d4-a80a-47b1-8cde-979b62f0390d/MANIFEST-000005
at org.rocksdb.RocksDB.open(Native Method) at
org.rocksdb.RocksDB.open(RocksDB.java:307) at
org.apache.hadoop.hdds.utils.db.managed.ManagedRocksDB.open(ManagedRocksDB.java:67)
at org.apache.hadoop.hdds.utils.db.RocksDatabase.open(RocksDatabase.java:153)
... 84 more
{code}
In order to avoid such race condition, sstfiltering service should be stopped
before copying the sst files.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
