[
https://issues.apache.org/jira/browse/HDFS-12911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16295918#comment-16295918
]
Uma Maheswara Rao G commented on HDFS-12911:
--------------------------------------------
I don't like to repeat my answers again and again, as I tried to give my
explanations already in HDFS-10285.
However I would like to give a little clarifications on #1, To avoid confusions
for others.
{quote}
I consider this feature far from complete and I am worried that other feature
additions will hurt Namenode more and more.
{quote}
This could be your opinion. This feature does not intend to add any new
policies. Please look at JIRA description/design doc.
The goal of this work is to make the NN state and storage state to match.
Current policies are the basic core policies which we are persisting in NN. SPS
works to satisfy that core policy states. I am not sure whether you have more
policies which requires the persistence in NN at all. To the bottom line, SPS
works for the policies which you persisted in NN.
Fancier policies can be built anywhere outside as I think they don't need to
persist in NN, and I am not discussing that work today as that needs good time
to discuss that feature( IMO, its a non trivial feature) . Whatever fancy
policies you may define, they may be falling into the same existing fundamental
storage policies, they are HOT, WARM, COLD. SPS works only for the policies
which you persist in the NN. *Does not intend to add any new policies on its
own, This works under the NN exposed HSM options.*
{quote}
I completely disagree, this is a design choice that was made. Nothing prevents
this process from working outside the Namenode.
{quote}
Probably I have to say the same thing in other way. :-) Nothing much overhead
by running inside too.
{quote}
In my humble opinion, if you have not resolved the big picture, then proceeding
with things like optimizing lock is pointless
{quote}
Thanks for the respect on developers time. Devs will receive commonly agreed
feedback and work on, we don't see objection on lock improvement. And Currently
its already 47th task in HDFS-10285, titled *SPS in NN*. Right now, unable to
work on things which are not agreed by others yet.
Keeping arguments aside, from SPS dev POV, we would be happy to work on, if we
get commonly agreed feedbacks. As a community developer(with my experience) , I
would like to respect everyone's feedbacks.
Right now there is no common agreement on the approach how we start SPS.
Probably we should figure out a way how to satisfy all arguments. Probably
online meeting should help discuss on that steps. Thanks
> [SPS]: Fix review comments from discussions in HDFS-10285
> ---------------------------------------------------------
>
> Key: HDFS-12911
> URL: https://issues.apache.org/jira/browse/HDFS-12911
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
> Reporter: Uma Maheswara Rao G
> Assignee: Rakesh R
> Attachments: HDFS-12911.00.patch
>
>
> This is the JIRA for tracking the possible improvements or issues discussed
> in main JIRA
> So far comments to handle
> Daryn:
> # Lock should not kept while executing placement policy.
> # While starting up the NN, SPS Xattrs checks happen even if feature
> disabled. This could potentially impact the startup speed.
> UMA:
> # I am adding one more possible improvement to reduce Xattr objects
> significantly.
> SPS Xattr is constant object. So, we create one Xattr deduplication object
> once statically and use the same object reference when required to add SPS
> Xattr to Inode. So, here additional bytes required for storing SPS Xattr
> would turn to same as single object ref ( i.e 4 bytes in 32 bit). So Xattr
> overhead should come down significantly IMO. Lets explore the feasibility on
> this option.
> Xattr list Future will not be specially created for SPS, that list would have
> been created by SetStoragePolicy already on the same directory. So, no extra
> Feature creation because of SPS alone.
> # Currently SPS putting long id objects in Q for tracking SPS called Inodes.
> So, it is additional created and size of it would be (obj ref + value) = (8 +
> 8) bytes [ ignoring alignment for time being]
> So, the possible improvement here is, instead of creating new Long obj, we
> can keep existing inode object for tracking. Advantage is, Inode object
> already maintained in NN, so no new object creation is needed. So, we just
> need to maintain one obj ref. Above two points should significantly reduce
> the memory requirements of SPS. So, for SPS call: 8bytes for called inode
> tracking + 8 bytes for Xattr ref.
> # Use LightWeightLinkedSet instead of using LinkedList for from Q. This will
> reduce unnecessary Node creations inside LinkedList.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
