peterxcli commented on code in PR #7792:
URL: https://github.com/apache/ozone/pull/7792#discussion_r1949236481
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/fs/ozone/TestHSync.java:
##########
@@ -583,6 +583,49 @@ public void testHSyncDeletedKey() throws Exception {
}
}
+ static Stream<BucketLayout> bucketLayouts() {
+ return Stream.of(BucketLayout.FILE_SYSTEM_OPTIMIZED, BucketLayout.LEGACY);
+ }
+
+ @ParameterizedTest
+ @MethodSource("bucketLayouts")
+ public void testBatchKeyDeletionWithHSync(BucketLayout bucketLayout) throws
Exception {
+ // Create a bucket with the specified layout
+ OzoneBucket testBucket = TestDataUtil.createVolumeAndBucket(client,
bucketLayout);
+
+ final String rootPath = String.format("%s://%s/",
+ OZONE_OFS_URI_SCHEME, CONF.get(OZONE_OM_ADDRESS_KEY));
+ CONF.set(CommonConfigurationKeysPublic.FS_DEFAULT_NAME_KEY, rootPath);
+
+ final String dir = OZONE_ROOT + testBucket.getVolumeName()
+ + OZONE_URI_DELIMITER + testBucket.getName();
+
+ // Create multiple files, some with hsync
+ final int numFiles = 5;
+ List<String> openKeys = new ArrayList<>();
+
+ try (FileSystem fs = FileSystem.get(CONF)) {
+ for (int i = 0; i < numFiles; i++) {
+ String keyName = "batch-key-" + i;
+ Path file = new Path(dir, keyName);
+ try (FSDataOutputStream os = fs.create(file, true)) {
+ os.write(1);
+ os.hsync();
+ openKeys.add(keyName);
+ }
Review Comment:
I added some logs to the existing `testHsync#testHSyncDeletedKey` and found
the following:
```
// delete key
(OMKeyDeleteResponseWithFSO.java:addToDBBatch(126)) - Write open key:
/volume42541/bucket06654/key-hsync-del/113980087550083073 to DB with metadata:
{hsyncClientId=113980087550083073, deletedHsyncKey=true}
// second hsync
OMKeyCommitRequestWithFSO.java:validateAndUpdateCache(181)) - Read open key:
/-9223372036854775552/-9223372036854775040/-9223372036854775040/key-hsync-del/113980087550083073
from DB with metadata: {hsyncClientId=113980087550083073}
```
It appears that the key
`/-9223372036854775552/-9223372036854775040/-9223372036854775040/key-hsync-del/113980087550083073`
was written during the first hsync process:
https://github.com/apache/ozone/blob/0ba9147943cf2726f1a53ec64716ee8a4892ddc2/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/key/OMKeyCommitResponseWithFSO.java#L83-L84
Given this, I believe we should modify the approach to pass the complete key
name along with the desired open key/file info metadata.
I’d love to hear your thoughts on this. I really appreciate your input,
@jojochuang.
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
