[
https://issues.apache.org/jira/browse/HDFS-12911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16291461#comment-16291461
]
Anu Engineer commented on HDFS-12911:
-------------------------------------
I feel like I am repeating myself, It must be pretty painful for the others :),
but I am going to answer each question since you asked.
bq. 1. With disable of SPS, absolutely no effect on NN, I have posted
benchmarks.
I consider this feature far from complete and I am worried that other feature
additions will hurt namenode more and more. Here is the bottom line, either you
are building an HSM -- when fully featured will have different policies, or
this is one off. Either one, I am against having this inside Namenode.
bq. 2. Front Q filing up is not real issue, as that is not going to be too much
memory consumption with realistic scenarios. To increase the speed of
processing, we have to tune DN bandwidths, but not NN Qs. So, with throttling,
processing memory will be constant.
*realistic scenarios* - You would be surprised by what you find in the field.
The notion of reasonable use goes out of the window as soon as you support
cases like, "I accidentally ran format on my production cluster, can you please
recover it".
bq. 3. It will be easy to respect replication tasks as high priority tasks if
we are inside Hadoop via keeping inline tasks controlling via xmits.
We have seen issues with IBR handling and hence things like lifeline protocol.
So I am not willing to buy that xmits or any such control will work effectively
if your Namenode is under severe load. Then you will be sitting around trying
to get data or maintaining large work queues which you cannot make any forward
progress.
bq. 4. Non of the tools like, Balancer/DiskBalancer, need to track or access
namespace, they just need utilization and DN block informations. So, this is
not exactly same as that.
I completely disagree, this is a design choice that was made. Nothing prevents
this process from working outside the Namenode.
bq. 5. We did not get consensus to add new process yet.
Agree, but we have no consensus to merge the code into Namenode either.
bq. 6. We got comments from other reviewer about lock and other stuff,
respecting their reviews, we are continuing to do this task. We can continue
discussions in HDFS-10285 if further issues.
In my humble opinion, if you have not resolved the big picture, then proceeding
with things like optimizing lock is pointless. Unless you are sure you *will be
able to merge* this into Namenode, optimizing a lock path that is inside
namenode is a waste of time.
And let me come back to the issue which does not seem to be discussed, we have
an existing pattern that works well. The only argument I have heard so far is
management tools have to do extra work if this feature is outside. I am sure
management tools have to do extra work even if this is inside. I have already
shared my point of view about that. I am still in the camp that this process
should be outside inside of Namenode.
> [SPS]: Fix review comments from discussions in HDFS-10285
> ---------------------------------------------------------
>
> Key: HDFS-12911
> URL: https://issues.apache.org/jira/browse/HDFS-12911
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
> Reporter: Uma Maheswara Rao G
> Assignee: Rakesh R
> Attachments: HDFS-12911.00.patch
>
>
> This is the JIRA for tracking the possible improvements or issues discussed
> in main JIRA
> So far comments to handle
> Daryn:
> # Lock should not kept while executing placement policy.
> # While starting up the NN, SPS Xattrs checks happen even if feature
> disabled. This could potentially impact the startup speed.
> UMA:
> # I am adding one more possible improvement to reduce Xattr objects
> significantly.
> SPS Xattr is constant object. So, we create one Xattr deduplication object
> once statically and use the same object reference when required to add SPS
> Xattr to Inode. So, here additional bytes required for storing SPS Xattr
> would turn to same as single object ref ( i.e 4 bytes in 32 bit). So Xattr
> overhead should come down significantly IMO. Lets explore the feasibility on
> this option.
> Xattr list Future will not be specially created for SPS, that list would have
> been created by SetStoragePolicy already on the same directory. So, no extra
> Feature creation because of SPS alone.
> # Currently SPS putting long id objects in Q for tracking SPS called Inodes.
> So, it is additional created and size of it would be (obj ref + value) = (8 +
> 8) bytes [ ignoring alignment for time being]
> So, the possible improvement here is, instead of creating new Long obj, we
> can keep existing inode object for tracking. Advantage is, Inode object
> already maintained in NN, so no new object creation is needed. So, we just
> need to maintain one obj ref. Above two points should significantly reduce
> the memory requirements of SPS. So, for SPS call: 8bytes for called inode
> tracking + 8 bytes for Xattr ref.
> # Use LightWeightLinkedSet instead of using LinkedList for from Q. This will
> reduce unnecessary Node creations inside LinkedList.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
