Copilot commented on code in PR #10199:
URL: https://github.com/apache/ozone/pull/10199#discussion_r3216177800
##########
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestKeyValueContainer.java:
##########
@@ -492,6 +495,68 @@ public void
testContainerImportExport(ContainerTestVersionInfo versionInfo)
}
}
+ @ContainerTestVersionInfo.ContainerTest
+ public void testFailedImportCleanupMovesContainerBeforeDelete(
+ ContainerTestVersionInfo versionInfo) throws Exception {
+ init(versionInfo);
+
+ HddsVolume containerVolume = volumeChoosingPolicy.chooseVolume(
+ StorageVolumeUtil.getHddsVolumesList(volumeSet.getVolumesList()), 1);
+
+ KeyValueContainer container = new KeyValueContainer(
+ keyValueContainerData, CONF) {
+ @Override
+ void deleteDirectory(File directory) throws IOException {
+ File deletedContainerDir = KeyValueContainerUtil.getTmpDirectoryPath(
+ getContainerData(), getContainerData().getVolume()).toFile();
+ if (directory.equals(deletedContainerDir)) {
+ throw new IOException("Injected tmp cleanup failure");
+ }
+ super.deleteDirectory(directory);
+ }
+ };
+ container.populatePathFields(scmId, containerVolume);
+
+ ContainerPacker<KeyValueContainerData> failingPacker =
+ new ContainerPacker<KeyValueContainerData>() {
+ @Override
+ public byte[] unpackContainerData(
+ Container<KeyValueContainerData> containerToUnpack,
+ InputStream inputStream, Path tmpDir, Path destContainerDir)
+ throws IOException {
+ Files.createDirectories(new File(containerToUnpack
+ .getContainerData().getChunksPath()).toPath());
+ Files.createDirectories(new File(containerToUnpack
+ .getContainerData().getMetadataPath()).toPath());
+ throw new IOException("Injected import failure");
+ }
+
+ @Override
+ public void pack(Container<KeyValueContainerData> containerToPack,
+ OutputStream destination) {
+ }
+
+ @Override
+ public byte[] unpackContainerDescriptor(InputStream inputStream) {
+ return null;
+ }
+ };
+
+ assertThrows(IOException.class, () -> container.importContainerData(
+ new ByteArrayInputStream(new byte[0]), failingPacker));
+
+ assertFalse(new File(container.getContainerData().getContainerPath())
+ .exists());
+ File deletedContainerDir = KeyValueContainerUtil.getTmpDirectoryPath(
+ container.getContainerData(), container.getContainerData().getVolume())
+ .toFile();
+ assertTrue(deletedContainerDir.exists());
+ assertTrue(new File(deletedContainerDir, OzoneConsts.STORAGE_DIR_CHUNKS)
+ .exists());
+ assertTrue(new File(deletedContainerDir, OzoneConsts.CONTAINER_META_PATH)
+ .exists());
Review Comment:
This test currently injects a failure when deleting the deleted-container
tmp directory and then asserts that both chunks/ and metadata/ still exist
under that directory. That doesn’t validate the PR’s stated goal (clean up
chunks/ before metadata/ so that, on partial cleanup, metadata remains but
chunks is gone). To cover the intended behavior, inject the failure at metadata
deletion (or immediately after chunks deletion) and assert that chunks/ has
been removed while metadata/ is still present.
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueContainer.java:
##########
@@ -664,15 +664,24 @@ private void cleanupFailedImport() {
if (containerData.hasSchema(OzoneConsts.SCHEMA_V3)) {
BlockUtils.removeContainerFromDB(containerData, config);
}
- FileUtils.deleteDirectory(new File(containerData.getMetadataPath()));
- FileUtils.deleteDirectory(new File(containerData.getChunksPath()));
- FileUtils.deleteDirectory(new
File(getContainerData().getContainerPath()));
+ File containerDir = new File(getContainerData().getContainerPath());
+ if (containerDir.exists()) {
+ KeyValueContainerUtil.moveToDeletedContainerDir(containerData,
+ containerData.getVolume());
+
deleteDirectory(KeyValueContainerUtil.getTmpDirectoryPath(containerData,
+ containerData.getVolume()).toFile());
Review Comment:
cleanupFailedImport now moves the container dir to the deleted-container
area and then calls FileUtils.deleteDirectory on the whole directory. This does
not guarantee that chunks/ is deleted before metadata/ (Commons IO recursion
order depends on File.listFiles ordering), so an interrupted/partial delete can
still leave a chunks/-only residue—exactly the diagnostic issue this change is
meant to avoid. Consider explicitly deleting the chunks directory first and the
metadata directory second (then the container dir) in the deleted-container
location, so partial cleanup preferentially preserves metadata.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
