[
https://issues.apache.org/jira/browse/HDDS-1530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
xudongcao updated HDDS-1530:
----------------------------
Description:
*Current problems:*
1. Freon does not support big files larger than 2GB because it use an int type
"keySize" parameter and also "keyValue" buffer size.
2. Freon allocates a entire buffer for each key at once, so if the key size is
large and the concurrency is high, freon will report OOM exception frequently.
3. Freon lacks option such as "--validateWrites", thus users cannot manually
specify that verification is required after writing.
*Some solutions:*
1. Use a long type "keySize" parameter, make sure freon can support big files
larger than 2GB.
2. Use a small buffer repeatedly than allocating the entire key-size buffer at
once, the default buffer size is 4K and can be configured by "–bufferSize"
parameter.
3. Add a "--validateWrites" option to Freon command line, users can provide
this option to indicate that a validation is required after write.
was:
*Current problems:*
1. Freon does not support big files larger than 2GB because it use an int type
"keySize" parameter and also "keyValue" buffer size.
2. Freon allocates a entire buffer for each key at once, so if the key size is
large and the concurrency is high, freon will report OOM exception frequently.
3. Freon lacks option such as "--validateWrites", thus users cannot manually
specify that verification is required after writing.
*Some solutions:*
1. Use a long type "keySize" parameter, make sure freon can support big files
larger than 2GB.
2. Use a small buffer repeatedly than allocating the entire key-size buffer at
once, the default buffer size is 4K and can be configured by "–bufferSize"
parameter.
3. Add a "--validateWrites" option to Freon command line, users can provide
this option to indicate that a validation is required after write.
4. Remove the process of appending an uuid to each key, which is of little
significance and complicates the code, especially when writting with a small
buffer repeatedly.
> Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and
> "--validateWrites" options.
> ------------------------------------------------------------------------------------------------------
>
> Key: HDDS-1530
> URL: https://issues.apache.org/jira/browse/HDDS-1530
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: test
> Reporter: xudongcao
> Assignee: xudongcao
> Priority: Major
> Attachments: HDDS-1530.001.patch, HDDS-1530.002.patch
>
>
> *Current problems:*
> 1. Freon does not support big files larger than 2GB because it use an int
> type "keySize" parameter and also "keyValue" buffer size.
> 2. Freon allocates a entire buffer for each key at once, so if the key size
> is large and the concurrency is high, freon will report OOM exception
> frequently.
> 3. Freon lacks option such as "--validateWrites", thus users cannot manually
> specify that verification is required after writing.
> *Some solutions:*
> 1. Use a long type "keySize" parameter, make sure freon can support big
> files larger than 2GB.
> 2. Use a small buffer repeatedly than allocating the entire key-size buffer
> at once, the default buffer size is 4K and can be configured by "–bufferSize"
> parameter.
> 3. Add a "--validateWrites" option to Freon command line, users can provide
> this option to indicate that a validation is required after write.
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
