iamcaoxudong commented on a change in pull request #830: HDDS-1530. Freon
support big files larger than 2GB and add --bufferSize and --validateWrites
options.
URL: https://github.com/apache/hadoop/pull/830#discussion_r285967860
##########
File path:
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/RandomKeyGenerator.java
##########
@@ -622,7 +642,11 @@ public void run() {
try (Scope writeScope = GlobalTracer.get()
.buildSpan("writeKeyData")
.startActive(true)) {
- os.write(keyValue);
+ for (long nrRemaining = keySize - randomValue.length;
+ nrRemaining > 0; nrRemaining -= bufferSize) {
+ int curSize = (int)Math.min(bufferSize, nrRemaining);
+ os.write(keyValueBuffer, 0, curSize);
Review comment:
Thank you for reviewing, but I have some different opinions:
1. On the client machine, the buffer here will be sent to the Ratis DataNode
via TCP later, so the file system buffer/cache is not involved.
2. On the DataNode, the file system cache is managed by the file's Inode
index. since each key is written on DataNode through different files, so I
think it will not affected by file system buffer/cache too significant.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
