[
https://issues.apache.org/jira/browse/HDFS-14484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
xudongcao updated HDFS-14484:
-----------------------------
Summary: Ozone: Freon: Support big files larger than 2GB and improve
concurrent framework. (was: Ozone: Freon: Support big files larger than 2GB
and improve concurrent operations.)
> Ozone: Freon: Support big files larger than 2GB and improve concurrent
> framework.
> ---------------------------------------------------------------------------------
>
> Key: HDFS-14484
> URL: https://issues.apache.org/jira/browse/HDFS-14484
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: benchmarks
> Reporter: xudongcao
> Assignee: xudongcao
> Priority: Major
>
> *Current problems:*
> 1. Freon does not support big files larger than 2GB because it use an int
> type "keySize" parameter and also "keyValue" buffer size.
> 2. Freon allocates a entire buffer for each key at once, so if the key size
> is large and the concurrency is high, freon will report OOM exception
> frequently.
> 3. The existing concurrency framework only supports volume-level
> concurrency, but in actual testing, a user is likely to specify a small
> number of volumes, and a larger number of buckets and keys . At this point,
> it can't take full advantage of thread pool to speed up testing.
> *And we need:*
> 1. Use a long type "keySize" parameter, make sure freon can support big
> files larger than 2GB.
> 2. Use a small buffer repeatedly than allocating the entire key-size buffer
> at once, the default buffer size is 4K and can be configured by "–bufferSize"
> parameter.
> 3. Improve the concurrency framework. Now the volume creation task, bucket
> creation task, and key creation task all can be submitted to the thread pool
> concurrently as a general task.
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
