[
https://issues.apache.org/jira/browse/HDFS-11920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16092385#comment-16092385
]
Chen Liang commented on HDFS-11920:
-----------------------------------
Had an discussion with [~anu] today, mentioned that this key partition across
container change should enable further multi-threading optimization for
put/get. Although I think over-partitioned key is something to avoid, because
parallelism also comes with cost. For instance, to write a key that only has 10
bytes we probably should not try to divide the workload any further.
In the patches, there is a check that a scm block has a configurable max size,
say X. This means if writing is some size Y > X, then the key will be divided
into multiple blocks, each of size X except for last one. I think this just
happens to fit the demand for future parallelism: each thread picks up a size X
scm block at a time, write it until all the Y/X blocks are done. X is the size
that we don't bother to over-partition. An alternative can be having a
configurable partition factor, say Z, and given the size Y, we always divide
and make Y/Z partitions, and having Z threads each picking up one partition. I
don't see any significant difference between these two though.
Anything to add [~anu]?
(Whichever way we decide to go, the actual parallel read/write will need to be
added in a follow-up JIRA.)
> Ozone : add key partition
> -------------------------
>
> Key: HDFS-11920
> URL: https://issues.apache.org/jira/browse/HDFS-11920
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Chen Liang
> Assignee: Chen Liang
> Attachments: HDFS-11920-HDFS-7240.001.patch,
> HDFS-11920-HDFS-7240.002.patch, HDFS-11920-HDFS-7240.003.patch
>
>
> Currently, each key corresponds to one single SCM block, and putKey/getKey
> writes/reads to this single SCM block. This works fine for keys with
> reasonably small data size. However if the data is too huge, (e.g. not even
> fits into a single container), then we need to be able to partition the key
> data into multiple blocks, each in one container. This JIRA changes the
> key-related classes to support this.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
