[
https://issues.apache.org/jira/browse/HDFS-12911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Douglas updated HDFS-12911:
---------------------------------
Attachment: HDFS-12911.00.patch
Tried a (very) naive refactoring to create an API between the SPS and the
namesystem/block manager, to get a sense of the coupling. Tried not to change
any of the logic or locking.
{code:java}
public interface Context {
Configuration getConf();
boolean isRunning();
boolean isInSafeMode();
boolean isMoverRunning();
INode getInode(long id);
INode getInode(String path) throws IOException;
void setSPSRunning(Supplier<Boolean> isSpsRunning);
void removeXattr(long id) throws IOException;
NetworkTopology getNetworkTopology();
DatanodeStorageInfo[] getStorages(BlockInfo blockInfo);
int getNumLiveDataNodes();
List<DatanodeDescriptor> getLiveDataNodes();
void addDropSPSWorkCommandsToAllDNs();
boolean hasLowRedundancyBlocks(BlockCollection blockCollection);
BlocksMovingAnalysis assignStorageMovement(long trackId);
}
{code}
{{FSTreeTraversal}} (and its subclasses) would not be simple to extract,
neither would it be simple to replace it with a generic traversal (so wrapping
{{FSDirectory}} is one example, not the only supported type).
The SPS does a lot of work holding the lock in
{{analyseBlocksStorageMovementsAndAssignToDN}}. It looks like it could be
broken up so the lock is only held for queries and not during processing.
> [SPS]: Fix review comments from discussions in HDFS-10285
> ---------------------------------------------------------
>
> Key: HDFS-12911
> URL: https://issues.apache.org/jira/browse/HDFS-12911
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
> Reporter: Uma Maheswara Rao G
> Assignee: Rakesh R
> Attachments: HDFS-12911.00.patch
>
>
> This is the JIRA for tracking the possible improvements or issues discussed
> in main JIRA
> So far comments to handle
> Daryn:
> # Lock should not kept while executing placement policy.
> # While starting up the NN, SPS Xattrs checks happen even if feature
> disabled. This could potentially impact the startup speed.
> UMA:
> # I am adding one more possible improvement to reduce Xattr objects
> significantly.
> SPS Xattr is constant object. So, we create one Xattr deduplication object
> once statically and use the same object reference when required to add SPS
> Xattr to Inode. So, here additional bytes required for storing SPS Xattr
> would turn to same as single object ref ( i.e 4 bytes in 32 bit). So Xattr
> overhead should come down significantly IMO. Lets explore the feasibility on
> this option.
> Xattr list Future will not be specially created for SPS, that list would have
> been created by SetStoragePolicy already on the same directory. So, no extra
> Feature creation because of SPS alone.
> # Currently SPS putting long id objects in Q for tracking SPS called Inodes.
> So, it is additional created and size of it would be (obj ref + value) = (8 +
> 8) bytes [ ignoring alignment for time being]
> So, the possible improvement here is, instead of creating new Long obj, we
> can keep existing inode object for tracking. Advantage is, Inode object
> already maintained in NN, so no new object creation is needed. So, we just
> need to maintain one obj ref. Above two points should significantly reduce
> the memory requirements of SPS. So, for SPS call: 8bytes for called inode
> tracking + 8 bytes for Xattr ref.
> # Use LightWeightLinkedSet instead of using LinkedList for from Q. This will
> reduce unnecessary Node creations inside LinkedList.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
