[
https://issues.apache.org/jira/browse/HDFS-8998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745925#comment-14745925
]
Andrew Wang commented on HDFS-8998:
-----------------------------------
Hi Yong,
I brought up Ozone because of the idea of a block container format that allows
management of a number of blocks as a single container. If we have block
container management, then we could implement this idea as a LevelDB-backed
container that puts small files in LevelDB. Then compaction is all managed by
LevelDB (on the DN) rather than requiring NN coordination. The less work and
complexity we can add to the NN the better.
I'm also generally concerned about depending on append, since it's not used by
many applications and can lead to some very hard to debug situations.
We actually had a very related discussion about this topic as part of EC
development. FB's HDFS-RAID implementation combines many small files together
into larger EC files. This means we have an issue with compacting after a
delete (as we've discussed), but also for permissions. Security is done at the
level of blocks, so in this model a client could read all the other small files
in the block. How would this work here? Also setting different replication
factors.
> Small files storage supported inside HDFS
> -----------------------------------------
>
> Key: HDFS-8998
> URL: https://issues.apache.org/jira/browse/HDFS-8998
> Project: Hadoop HDFS
> Issue Type: New Feature
> Reporter: Yong Zhang
> Assignee: Yong Zhang
> Attachments: HDFS-8998.design.001.pdf
>
>
> HDFS has problems on store small files, just like this blog said
> (http://blog.cloudera.com/blog/2009/02/the-small-files-problem).
> This blog also tell us some way how to store small file in HDFS, but they are
> not good way, seems HAR files and Sequence Files are better for read-only
> files.
> Current each HDFS block is only for one HDFS file, if too many small file
> there, many small blocks will be in DataNode, which will make DataNode heavy
> loading.
> This jira will show how to online merge small blocks to big one, and how to
> delete small file, and so on.
> Cerrentlly we have many open jira for improving HDFS scalability on NameNode,
> such as HDFS-7836, HDFS-8286 and so on.
> So small file meta (INode and BlocksMap) will also be in NameNode.
> Design document will be uploaded soon.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
