[ https://issues.apache.org/jira/browse/HDFS-12911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HDFS-12911: --------------------------------- Attachment: HDFS-12911.00.patch Tried a (very) naive refactoring to create an API between the SPS and the namesystem/block manager, to get a sense of the coupling. Tried not to change any of the logic or locking. {code:java} public interface Context { Configuration getConf(); boolean isRunning(); boolean isInSafeMode(); boolean isMoverRunning(); INode getInode(long id); INode getInode(String path) throws IOException; void setSPSRunning(Supplier<Boolean> isSpsRunning); void removeXattr(long id) throws IOException; NetworkTopology getNetworkTopology(); DatanodeStorageInfo[] getStorages(BlockInfo blockInfo); int getNumLiveDataNodes(); List<DatanodeDescriptor> getLiveDataNodes(); void addDropSPSWorkCommandsToAllDNs(); boolean hasLowRedundancyBlocks(BlockCollection blockCollection); BlocksMovingAnalysis assignStorageMovement(long trackId); } {code} {{FSTreeTraversal}} (and its subclasses) would not be simple to extract, neither would it be simple to replace it with a generic traversal (so wrapping {{FSDirectory}} is one example, not the only supported type). The SPS does a lot of work holding the lock in {{analyseBlocksStorageMovementsAndAssignToDN}}. It looks like it could be broken up so the lock is only held for queries and not during processing. > [SPS]: Fix review comments from discussions in HDFS-10285 > --------------------------------------------------------- > > Key: HDFS-12911 > URL: https://issues.apache.org/jira/browse/HDFS-12911 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode > Reporter: Uma Maheswara Rao G > Assignee: Rakesh R > Attachments: HDFS-12911.00.patch > > > This is the JIRA for tracking the possible improvements or issues discussed > in main JIRA > So far comments to handle > Daryn: > # Lock should not kept while executing placement policy. > # While starting up the NN, SPS Xattrs checks happen even if feature > disabled. This could potentially impact the startup speed. > UMA: > # I am adding one more possible improvement to reduce Xattr objects > significantly. > SPS Xattr is constant object. So, we create one Xattr deduplication object > once statically and use the same object reference when required to add SPS > Xattr to Inode. So, here additional bytes required for storing SPS Xattr > would turn to same as single object ref ( i.e 4 bytes in 32 bit). So Xattr > overhead should come down significantly IMO. Lets explore the feasibility on > this option. > Xattr list Future will not be specially created for SPS, that list would have > been created by SetStoragePolicy already on the same directory. So, no extra > Feature creation because of SPS alone. > # Currently SPS putting long id objects in Q for tracking SPS called Inodes. > So, it is additional created and size of it would be (obj ref + value) = (8 + > 8) bytes [ ignoring alignment for time being] > So, the possible improvement here is, instead of creating new Long obj, we > can keep existing inode object for tracking. Advantage is, Inode object > already maintained in NN, so no new object creation is needed. So, we just > need to maintain one obj ref. Above two points should significantly reduce > the memory requirements of SPS. So, for SPS call: 8bytes for called inode > tracking + 8 bytes for Xattr ref. > # Use LightWeightLinkedSet instead of using LinkedList for from Q. This will > reduce unnecessary Node creations inside LinkedList. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org