[
https://issues.apache.org/jira/browse/HDFS-12911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rakesh R updated HDFS-12911:
----------------------------
Attachment: HDFS-12911-HDFS-10285-01.patch
Thanks a lot [~chris.douglas] for the help and the useful patch. I have tried
an attempt incorporating the {{Context}} interface and separated out the
locking from the {{#analyseBlocksStorageMovementsAndAssignToDN}} function.
Following are the changes in the patch:
# Incorporated {{Context}} interface and {{IntraNNSPSContext}} impl.
# Separated out the querying and assigning block movements so that the lock can
be taken only for the querying logic.
# Moved the following sps classes to new package
{{'org.apache.hadoop.hdfs.server.namenode.sps'}}
- StoragePolicySatisfier
- BlockStorageMovementNeeded
- BlockStorageMovementAttemptedItems
Due to the package changes, the patch becomes quite large. Please apply the
changes to your env and I hope that will help you to understand the proposed
difference easily.
Appreciate your comments!
> [SPS]: Fix review comments from discussions in HDFS-10285
> ---------------------------------------------------------
>
> Key: HDFS-12911
> URL: https://issues.apache.org/jira/browse/HDFS-12911
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
> Reporter: Uma Maheswara Rao G
> Assignee: Rakesh R
> Attachments: HDFS-12911-HDFS-10285-01.patch, HDFS-12911.00.patch
>
>
> This is the JIRA for tracking the possible improvements or issues discussed
> in main JIRA
> So far comments to handle
> Daryn:
> # Lock should not kept while executing placement policy.
> # While starting up the NN, SPS Xattrs checks happen even if feature
> disabled. This could potentially impact the startup speed.
> UMA:
> # I am adding one more possible improvement to reduce Xattr objects
> significantly.
> SPS Xattr is constant object. So, we create one Xattr deduplication object
> once statically and use the same object reference when required to add SPS
> Xattr to Inode. So, here additional bytes required for storing SPS Xattr
> would turn to same as single object ref ( i.e 4 bytes in 32 bit). So Xattr
> overhead should come down significantly IMO. Lets explore the feasibility on
> this option.
> Xattr list Future will not be specially created for SPS, that list would have
> been created by SetStoragePolicy already on the same directory. So, no extra
> Feature creation because of SPS alone.
> # Currently SPS putting long id objects in Q for tracking SPS called Inodes.
> So, it is additional created and size of it would be (obj ref + value) = (8 +
> 8) bytes [ ignoring alignment for time being]
> So, the possible improvement here is, instead of creating new Long obj, we
> can keep existing inode object for tracking. Advantage is, Inode object
> already maintained in NN, so no new object creation is needed. So, we just
> need to maintain one obj ref. Above two points should significantly reduce
> the memory requirements of SPS. So, for SPS call: 8bytes for called inode
> tracking + 8 bytes for Xattr ref.
> # Use LightWeightLinkedSet instead of using LinkedList for from Q. This will
> reduce unnecessary Node creations inside LinkedList.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
