[
https://issues.apache.org/jira/browse/HDDS-1530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16840300#comment-16840300
]
Hadoop QA commented on HDDS-1530:
---------------------------------
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
30s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} yetus {color} | {color:red} 0m 8s{color}
| {color:red} Unprocessed flag(s): --jenkins --skip-dir {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce base:
https://builds.apache.org/job/PreCommit-HDDS-Build/2691/artifact/out/Dockerfile
|
| JIRA Issue | HDDS-1530 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12968782/HDDS-1530.001.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HDDS-Build/2691/console |
| versions | git=2.7.4 |
| Powered by | Apache Yetus 0.11.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and
> "--validateWrites" options.
> ------------------------------------------------------------------------------------------------------
>
> Key: HDDS-1530
> URL: https://issues.apache.org/jira/browse/HDDS-1530
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: test
> Reporter: xudongcao
> Assignee: xudongcao
> Priority: Major
> Attachments: HDDS-1530.001.patch
>
>
> *Current problems:*
> 1. Freon does not support big files larger than 2GB because it use an int
> type "keySize" parameter and also "keyValue" buffer size.
> 2. Freon allocates a entire buffer for each key at once, so if the key size
> is large and the concurrency is high, freon will report OOM exception
> frequently.
> 3. Freon lacks option such as "--validateWrites", thus users cannot manually
> specify that verification is required after writing.
> *Some solutions:*
> 1. Use a long type "keySize" parameter, make sure freon can support big
> files larger than 2GB.
> 2. Use a small buffer repeatedly than allocating the entire key-size buffer
> at once, the default buffer size is 4K and can be configured by "–bufferSize"
> parameter.
> 3. Add a "--validateWrites" option to Freon command line, users can provide
> this option to indicate that a validation is required after write.
> 4. Remove the process of appending an uuid to each key, which is of little
> significance and complicates the code, especially when writting with a small
> buffer repeatedly.
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
