[
https://issues.apache.org/jira/browse/HDFS-7244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14182313#comment-14182313
]
Colin Patrick McCabe commented on HDFS-7244:
--------------------------------------------
OK, I'll create a branch for this tomorrow and we can discuss whether to roll
HDFS-6658 in to it later. It would be nice to start moving on the off-heap
stuff-- there has been a lot of interest in this. The goal of a branch is to
move fast and not get bogged down in the process for getting things into trunk
or a release, which usually involves additional review.
We could certainly commit HDFS-6658 to the HDFS-7244 branch first, and then
later commit it to trunk (prior to anything else). It's pretty easy to
cherry-pick patches now that we're using git, and we've done things like this
in the past.
Thanks, guys.
> Reduce Namenode memory using Flyweight pattern
> ----------------------------------------------
>
> Key: HDFS-7244
> URL: https://issues.apache.org/jira/browse/HDFS-7244
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
> Reporter: Amir Langer
>
> Using the flyweight pattern can dramatically reduce memory usage in the
> Namenode. The pattern also abstracts the actual storage type and allows the
> decision of whether it is off-heap or not and what is the serialisation
> mechanism to be configured per deployment.
> The idea is to move all BlockInfo data (as a first step) to this storage
> using the Flyweight pattern. The cost to doing it will be in higher latency
> when accessing/modifying a block. The idea is that this will be offset with a
> reduction in memory and in the case of off-heap, a dramatic reduction in
> memory (effectively, memory used for BlockInfo would reduce to a very small
> constant value).
> This reduction will also have an huge impact on the latency as GC pauses will
> be reduced considerably and may even end up with better latency results than
> the original code.
> I wrote a stand-alone project as a proof of concept, to show the pattern, the
> data structure we can use and what will be the performance costs of this
> approach.
> see [Slab|https://github.com/langera/slab]
> and [Slab performance
> results|https://github.com/langera/slab/wiki/Performance-Results].
> Slab abstracts the storage, gives several storage implementations and
> implements the flyweight pattern for the application (Namenode in our case).
> The stages to incorporate Slab into the Namenode is outlined in the sub-tasks
> JIRAs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
