iamcaoxudong commented on a change in pull request #830: HDDS-1530. Freon
support big files larger than 2GB and add --bufferSize and --validateWrites
options.
URL: https://github.com/apache/hadoop/pull/830#discussion_r285984744
##########
File path:
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/RandomKeyGenerator.java
##########
@@ -228,8 +243,20 @@ public Void call() throws Exception {
init(freon.createOzoneConfiguration());
}
- keyValue =
- DFSUtil.string2Bytes(RandomStringUtils.randomAscii(keySize - 36));
+ keyValueBuffer = DFSUtil.string2Bytes(
+ RandomStringUtils.randomAscii(bufferSize));
+
+ // Compute the common initial digest for all keys without their UUID
+ if (validateWrites) {
+ commonInitialMD = DigestUtils.getDigest(DIGEST_ALGORITHM);
+ int uuidLength = UUID.randomUUID().toString().length();
+ keySize = Math.max(uuidLength, keySize);
+ for (long nrRemaining = keySize - uuidLength; nrRemaining > 0;
+ nrRemaining -= bufferSize) {
+ int curSize = (int)Math.min(bufferSize, nrRemaining);
+ commonInitialMD.update(keyValueBuffer, 0, curSize);
Review comment:
Thank you for reviewing, but according to my test results, repeatedly
calculate the same content will change the digest, I test it by following
statement looply:
print ((MessageDigest)(commonInitialMD.clone())).digest()
The reason why we need to clone every time is that digest() is not an
idempotent operation, it will change the buffer of MessageDigest object (I
think it may be mainly due to padding)。
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
