[jira] [Updated] (HDFS-12911) [SPS]: Fix review comments from discussions in HDFS-10285
[ https://issues.apache.org/jira/browse/HDFS-12911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-12911: --- Attachment: HDFS-12911-HDFS-10285-03.patch Attached another version of path. Introduced following Interfaces: SPSService:- Contains sps life cycle APIs and some of processing related APIs FileIdCollector:- An interface for plugging in the directory scanning functionality. BlockMoveTaskHandler:- An interface for implementing the core block movement logic. When users called SPS, BM adds the sps invoked pathids(a path id can be file/directoty) into SPSPathIds class. Now, SPSPathIdProcessor will pick the path ids and uses FileIdCollector to scan that paths. The scanned file ids will be added to SPSService for further processing. SPSService will take file ids and analyzes the block states and finds if any mismatches. If it finds mismatches, then it will use BlockMoveTaskHandler to move the blocks. > [SPS]: Fix review comments from discussions in HDFS-10285 > - > > Key: HDFS-12911 > URL: https://issues.apache.org/jira/browse/HDFS-12911 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Uma Maheswara Rao G >Assignee: Rakesh R > Attachments: HDFS-12911-HDFS-10285-01.patch, > HDFS-12911-HDFS-10285-02.patch, HDFS-12911-HDFS-10285-03.patch, > HDFS-12911.00.patch > > > This is the JIRA for tracking the possible improvements or issues discussed > in main JIRA > So far comments to handle > Daryn: > # Lock should not kept while executing placement policy. > # While starting up the NN, SPS Xattrs checks happen even if feature > disabled. This could potentially impact the startup speed. > UMA: > # I am adding one more possible improvement to reduce Xattr objects > significantly. > SPS Xattr is constant object. So, we create one Xattr deduplication object > once statically and use the same object reference when required to add SPS > Xattr to Inode. So, here additional bytes required for storing SPS Xattr > would turn to same as single object ref ( i.e 4 bytes in 32 bit). So Xattr > overhead should come down significantly IMO. Lets explore the feasibility on > this option. > Xattr list Future will not be specially created for SPS, that list would have > been created by SetStoragePolicy already on the same directory. So, no extra > Feature creation because of SPS alone. > # Currently SPS putting long id objects in Q for tracking SPS called Inodes. > So, it is additional created and size of it would be (obj ref + value) = (8 + > 8) bytes [ ignoring alignment for time being] > So, the possible improvement here is, instead of creating new Long obj, we > can keep existing inode object for tracking. Advantage is, Inode object > already maintained in NN, so no new object creation is needed. So, we just > need to maintain one obj ref. Above two points should significantly reduce > the memory requirements of SPS. So, for SPS call: 8bytes for called inode > tracking + 8 bytes for Xattr ref. > # Use LightWeightLinkedSet instead of using LinkedList for from Q. This will > reduce unnecessary Node creations inside LinkedList. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12911) [SPS]: Fix review comments from discussions in HDFS-10285
[ https://issues.apache.org/jira/browse/HDFS-12911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-12911: --- Status: Patch Available (was: Open) > [SPS]: Fix review comments from discussions in HDFS-10285 > - > > Key: HDFS-12911 > URL: https://issues.apache.org/jira/browse/HDFS-12911 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Uma Maheswara Rao G >Assignee: Rakesh R > Attachments: HDFS-12911-HDFS-10285-01.patch, > HDFS-12911-HDFS-10285-02.patch, HDFS-12911.00.patch > > > This is the JIRA for tracking the possible improvements or issues discussed > in main JIRA > So far comments to handle > Daryn: > # Lock should not kept while executing placement policy. > # While starting up the NN, SPS Xattrs checks happen even if feature > disabled. This could potentially impact the startup speed. > UMA: > # I am adding one more possible improvement to reduce Xattr objects > significantly. > SPS Xattr is constant object. So, we create one Xattr deduplication object > once statically and use the same object reference when required to add SPS > Xattr to Inode. So, here additional bytes required for storing SPS Xattr > would turn to same as single object ref ( i.e 4 bytes in 32 bit). So Xattr > overhead should come down significantly IMO. Lets explore the feasibility on > this option. > Xattr list Future will not be specially created for SPS, that list would have > been created by SetStoragePolicy already on the same directory. So, no extra > Feature creation because of SPS alone. > # Currently SPS putting long id objects in Q for tracking SPS called Inodes. > So, it is additional created and size of it would be (obj ref + value) = (8 + > 8) bytes [ ignoring alignment for time being] > So, the possible improvement here is, instead of creating new Long obj, we > can keep existing inode object for tracking. Advantage is, Inode object > already maintained in NN, so no new object creation is needed. So, we just > need to maintain one obj ref. Above two points should significantly reduce > the memory requirements of SPS. So, for SPS call: 8bytes for called inode > tracking + 8 bytes for Xattr ref. > # Use LightWeightLinkedSet instead of using LinkedList for from Q. This will > reduce unnecessary Node creations inside LinkedList. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12911) [SPS]: Fix review comments from discussions in HDFS-10285
[ https://issues.apache.org/jira/browse/HDFS-12911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-12911: --- Attachment: HDFS-12911-HDFS-10285-02.patch A patch attached with SPS modularization. # For Scanning: Added an interface, FileIDCollector. We can plugin specific scanning implementations. # For SPS-NN communication: Added interface Context, which is same as [~chris.douglas] proposed in first patch. All nn related interaction will happen via this interface. A specific implementations can be plugged in. One could directly access NN internals and other could use RPC to get info from NN. # For Block move tasks: Added an interface BlockMoveTaskHandler. We could plugin specific implementation for block movements. One can be heartbeat based assignment and other implementation can be sending block move ops directly to DNs. In this patch, I provided internal SPS implementation ( IntraSPSNameNodeContext, IntraSPSNameNodeFileIDCollector, IntraSPSNameNodeBlockMoveTaskHandler) and for external I just created dummy plugin classes (ExternalSPSContext, ExternalSPSFileIDCollector, ExternalSPSBlockMoveTaskHandler). Feedbacks are most welcomed. > [SPS]: Fix review comments from discussions in HDFS-10285 > - > > Key: HDFS-12911 > URL: https://issues.apache.org/jira/browse/HDFS-12911 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Uma Maheswara Rao G >Assignee: Rakesh R > Attachments: HDFS-12911-HDFS-10285-01.patch, > HDFS-12911-HDFS-10285-02.patch, HDFS-12911.00.patch > > > This is the JIRA for tracking the possible improvements or issues discussed > in main JIRA > So far comments to handle > Daryn: > # Lock should not kept while executing placement policy. > # While starting up the NN, SPS Xattrs checks happen even if feature > disabled. This could potentially impact the startup speed. > UMA: > # I am adding one more possible improvement to reduce Xattr objects > significantly. > SPS Xattr is constant object. So, we create one Xattr deduplication object > once statically and use the same object reference when required to add SPS > Xattr to Inode. So, here additional bytes required for storing SPS Xattr > would turn to same as single object ref ( i.e 4 bytes in 32 bit). So Xattr > overhead should come down significantly IMO. Lets explore the feasibility on > this option. > Xattr list Future will not be specially created for SPS, that list would have > been created by SetStoragePolicy already on the same directory. So, no extra > Feature creation because of SPS alone. > # Currently SPS putting long id objects in Q for tracking SPS called Inodes. > So, it is additional created and size of it would be (obj ref + value) = (8 + > 8) bytes [ ignoring alignment for time being] > So, the possible improvement here is, instead of creating new Long obj, we > can keep existing inode object for tracking. Advantage is, Inode object > already maintained in NN, so no new object creation is needed. So, we just > need to maintain one obj ref. Above two points should significantly reduce > the memory requirements of SPS. So, for SPS call: 8bytes for called inode > tracking + 8 bytes for Xattr ref. > # Use LightWeightLinkedSet instead of using LinkedList for from Q. This will > reduce unnecessary Node creations inside LinkedList. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12911) [SPS]: Fix review comments from discussions in HDFS-10285
[
https://issues.apache.org/jira/browse/HDFS-12911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rakesh R updated HDFS-12911:
Attachment: HDFS-12911-HDFS-10285-01.patch
Thanks a lot [~chris.douglas] for the help and the useful patch. I have tried
an attempt incorporating the {{Context}} interface and separated out the
locking from the {{#analyseBlocksStorageMovementsAndAssignToDN}} function.
Following are the changes in the patch:
# Incorporated {{Context}} interface and {{IntraNNSPSContext}} impl.
# Separated out the querying and assigning block movements so that the lock can
be taken only for the querying logic.
# Moved the following sps classes to new package
{{'org.apache.hadoop.hdfs.server.namenode.sps'}}
- StoragePolicySatisfier
- BlockStorageMovementNeeded
- BlockStorageMovementAttemptedItems
Due to the package changes, the patch becomes quite large. Please apply the
changes to your env and I hope that will help you to understand the proposed
difference easily.
Appreciate your comments!
> [SPS]: Fix review comments from discussions in HDFS-10285
> -
>
> Key: HDFS-12911
> URL: https://issues.apache.org/jira/browse/HDFS-12911
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Uma Maheswara Rao G
>Assignee: Rakesh R
> Attachments: HDFS-12911-HDFS-10285-01.patch, HDFS-12911.00.patch
>
>
> This is the JIRA for tracking the possible improvements or issues discussed
> in main JIRA
> So far comments to handle
> Daryn:
> # Lock should not kept while executing placement policy.
> # While starting up the NN, SPS Xattrs checks happen even if feature
> disabled. This could potentially impact the startup speed.
> UMA:
> # I am adding one more possible improvement to reduce Xattr objects
> significantly.
> SPS Xattr is constant object. So, we create one Xattr deduplication object
> once statically and use the same object reference when required to add SPS
> Xattr to Inode. So, here additional bytes required for storing SPS Xattr
> would turn to same as single object ref ( i.e 4 bytes in 32 bit). So Xattr
> overhead should come down significantly IMO. Lets explore the feasibility on
> this option.
> Xattr list Future will not be specially created for SPS, that list would have
> been created by SetStoragePolicy already on the same directory. So, no extra
> Feature creation because of SPS alone.
> # Currently SPS putting long id objects in Q for tracking SPS called Inodes.
> So, it is additional created and size of it would be (obj ref + value) = (8 +
> 8) bytes [ ignoring alignment for time being]
> So, the possible improvement here is, instead of creating new Long obj, we
> can keep existing inode object for tracking. Advantage is, Inode object
> already maintained in NN, so no new object creation is needed. So, we just
> need to maintain one obj ref. Above two points should significantly reduce
> the memory requirements of SPS. So, for SPS call: 8bytes for called inode
> tracking + 8 bytes for Xattr ref.
> # Use LightWeightLinkedSet instead of using LinkedList for from Q. This will
> reduce unnecessary Node creations inside LinkedList.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12911) [SPS]: Fix review comments from discussions in HDFS-10285
[
https://issues.apache.org/jira/browse/HDFS-12911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Douglas updated HDFS-12911:
-
Attachment: HDFS-12911.00.patch
Tried a (very) naive refactoring to create an API between the SPS and the
namesystem/block manager, to get a sense of the coupling. Tried not to change
any of the logic or locking.
{code:java}
public interface Context {
Configuration getConf();
boolean isRunning();
boolean isInSafeMode();
boolean isMoverRunning();
INode getInode(long id);
INode getInode(String path) throws IOException;
void setSPSRunning(Supplier isSpsRunning);
void removeXattr(long id) throws IOException;
NetworkTopology getNetworkTopology();
DatanodeStorageInfo[] getStorages(BlockInfo blockInfo);
int getNumLiveDataNodes();
List getLiveDataNodes();
void addDropSPSWorkCommandsToAllDNs();
boolean hasLowRedundancyBlocks(BlockCollection blockCollection);
BlocksMovingAnalysis assignStorageMovement(long trackId);
}
{code}
{{FSTreeTraversal}} (and its subclasses) would not be simple to extract,
neither would it be simple to replace it with a generic traversal (so wrapping
{{FSDirectory}} is one example, not the only supported type).
The SPS does a lot of work holding the lock in
{{analyseBlocksStorageMovementsAndAssignToDN}}. It looks like it could be
broken up so the lock is only held for queries and not during processing.
> [SPS]: Fix review comments from discussions in HDFS-10285
> -
>
> Key: HDFS-12911
> URL: https://issues.apache.org/jira/browse/HDFS-12911
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Uma Maheswara Rao G
>Assignee: Rakesh R
> Attachments: HDFS-12911.00.patch
>
>
> This is the JIRA for tracking the possible improvements or issues discussed
> in main JIRA
> So far comments to handle
> Daryn:
> # Lock should not kept while executing placement policy.
> # While starting up the NN, SPS Xattrs checks happen even if feature
> disabled. This could potentially impact the startup speed.
> UMA:
> # I am adding one more possible improvement to reduce Xattr objects
> significantly.
> SPS Xattr is constant object. So, we create one Xattr deduplication object
> once statically and use the same object reference when required to add SPS
> Xattr to Inode. So, here additional bytes required for storing SPS Xattr
> would turn to same as single object ref ( i.e 4 bytes in 32 bit). So Xattr
> overhead should come down significantly IMO. Lets explore the feasibility on
> this option.
> Xattr list Future will not be specially created for SPS, that list would have
> been created by SetStoragePolicy already on the same directory. So, no extra
> Feature creation because of SPS alone.
> # Currently SPS putting long id objects in Q for tracking SPS called Inodes.
> So, it is additional created and size of it would be (obj ref + value) = (8 +
> 8) bytes [ ignoring alignment for time being]
> So, the possible improvement here is, instead of creating new Long obj, we
> can keep existing inode object for tracking. Advantage is, Inode object
> already maintained in NN, so no new object creation is needed. So, we just
> need to maintain one obj ref. Above two points should significantly reduce
> the memory requirements of SPS. So, for SPS call: 8bytes for called inode
> tracking + 8 bytes for Xattr ref.
> # Use LightWeightLinkedSet instead of using LinkedList for from Q. This will
> reduce unnecessary Node creations inside LinkedList.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12911) [SPS]: Fix review comments from discussions in HDFS-10285
[ https://issues.apache.org/jira/browse/HDFS-12911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-12911: --- Description: This is the JIRA for tracking the possible improvements or issues discussed in main JIRA So far comments to handle Daryn: # Lock should not kept while executing placement policy. # While starting up the NN, SPS Xattrs checks happen even if feature disabled. This could potentially impact the startup speed. UMA: # I am adding one more possible improvement to reduce Xattr objects significantly. SPS Xattr is constant object. So, we create one Xattr deduplication object once statically and use the same object reference when required to add SPS Xattr to Inode. So, here additional bytes required for storing SPS Xattr would turn to same as single object ref ( i.e 4 bytes in 32 bit). So Xattr overhead should come down significantly IMO. Lets explore the feasibility on this option. Xattr list Future will not be specially created for SPS, that list would have been created by SetStoragePolicy already on the same directory. So, no extra Feature creation because of SPS alone. # Currently SPS putting long id objects in Q for tracking SPS called Inodes. So, it is additional created and size of it would be (obj ref + value) = (8 + 8) bytes [ ignoring alignment for time being] So, the possible improvement here is, instead of creating new Long obj, we can keep existing inode object for tracking. Advantage is, Inode object already maintained in NN, so no new object creation is needed. So, we just need to maintain one obj ref. Above two points should significantly reduce the memory requirements of SPS. So, for SPS call: 8bytes for called inode tracking + 8 bytes for Xattr ref. # Use LightWeightLinkedSet instead of using LinkedList for from Q. This will reduce unnecessary Node creations inside LinkedList. was: This is the JIRA for tracking the possible improvements or issues discussed in main JIRA So far comments to handle Daryn: # Lock should not kept while executing placement policy. # While starting up the NN, SPS Xattrs checks happen even if feature disabled. This could potentially impact the startup speed. UMA: # I am adding one more possible improvement to reduce Xattr objects significantly. SPS Xattr is constant object. So, we create one Xattr deduplication object once statically and use the same object reference when required to add SPS Xattr to Inode. So, here additional bytes required for storing SPS Xattr would turn to same as single object ref ( i.e 4 bytes in 32 bit). So Xattr overhead should come down significantly IMO. Lets explore the feasibility on this option. Xattr list Future will not be specially created for SPS, that list would have been created by SetStoragePolicy already on the same directory. So, no extra Feature creation because of SPS alone. # Currently SPS putting long id objects in Q for tracking SPS called Inodes. So, it is additional created and size of it would be (obj ref + value) = (8 + 8) bytes [ ignoring alignment for time being] So, the possible improvement here is, instead of creating new Long obj, we can keep existing inode object for tracking. Advantage is, Inode object already maintained in NN, so no new object creation is needed. So, we just need to maintain one obj ref. Above two points should significantly reduce the memory requirements of SPS. So, for SPS call: 8bytes for called inode tracking + 8 bytes for Xattr ref. > [SPS]: Fix review comments from discussions in HDFS-10285 > - > > Key: HDFS-12911 > URL: https://issues.apache.org/jira/browse/HDFS-12911 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Uma Maheswara Rao G >Assignee: Rakesh R > > This is the JIRA for tracking the possible improvements or issues discussed > in main JIRA > So far comments to handle > Daryn: > # Lock should not kept while executing placement policy. > # While starting up the NN, SPS Xattrs checks happen even if feature > disabled. This could potentially impact the startup speed. > UMA: > # I am adding one more possible improvement to reduce Xattr objects > significantly. > SPS Xattr is constant object. So, we create one Xattr deduplication object > once statically and use the same object reference when required to add SPS > Xattr to Inode. So, here additional bytes required for storing SPS Xattr > would turn to same as single object ref ( i.e 4 bytes in 32 bit). So Xattr > overhead should come down significantly IMO. Lets explore the feasibility on > this option. > Xattr list Future will not be specially created for SPS, that list would have > been created by SetStoragePolicy already on the same directory. So, no extra > Feature
[jira] [Updated] (HDFS-12911) [SPS]: Fix review comments from discussions in HDFS-10285
[ https://issues.apache.org/jira/browse/HDFS-12911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-12911: --- Description: This is the JIRA for tracking the possible improvements or issues discussed in main JIRA So far comments to handle Daryn: # Lock should not kept while executing placement policy. # While starting up the NN, SPS Xattrs checks happen even if feature disabled. This could potentially impact the startup speed. UMA: # I am adding one more possible improvement to reduce Xattr objects significantly. SPS Xattr is constant object. So, we create one Xattr deduplication object once statically and use the same object reference when required to add SPS Xattr to Inode. So, here additional bytes required for storing SPS Xattr would turn to same as single object ref ( i.e 4 bytes in 32 bit). So Xattr overhead should come down significantly IMO. Lets explore the feasibility on this option. Xattr list Future will not be specially created for SPS, that list would have been created by SetStoragePolicy already on the same directory. So, no extra Feature creation because of SPS alone. # Currently SPS putting long id objects in Q for tracking SPS called Inodes. So, it is additional created and size of it would be (obj ref + value) = (8 + 8) bytes [ ignoring alignment for time being] So, the possible improvement here is, instead of creating new Long obj, we can keep existing inode object for tracking. Advantage is, Inode object already maintained in NN, so no new object creation is needed. So, we just need to maintain one obj ref. Above two points should significantly reduce the memory requirements of SPS. So, for SPS call: 8bytes for called inode tracking + 8 bytes for Xattr ref. was: This is the JIRA for tracking the possible improvements or issues discussed in main JIRA So, far from Daryn: 1. Lock should not kept while executing placement policy. 2. While starting up the NN, SPS Xattrs checks happen even if feature disabled. This could potentially impact the startup speed. I am adding one more possible improvement to reduce Xattr objects significantly. SPS Xattr is constant object. So, we create one Xattr deduplication object once statically and use the same object reference when required to add SPS Xattr to Inode. So, here additional bytes required for storing SPS Xattr would turn to same as single object ref ( i.e 4 bytes in 32 bit). So Xattr overhead should come down significantly IMO. Lets explore the feasibility on this option. Xattr list Future will not be specially created for SPS, that list would have been created by SetStoragePolicy already on the same directory. So, no extra Future creation because of SPS alone. > [SPS]: Fix review comments from discussions in HDFS-10285 > - > > Key: HDFS-12911 > URL: https://issues.apache.org/jira/browse/HDFS-12911 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Uma Maheswara Rao G >Assignee: Rakesh R > > This is the JIRA for tracking the possible improvements or issues discussed > in main JIRA > So far comments to handle > Daryn: > # Lock should not kept while executing placement policy. > # While starting up the NN, SPS Xattrs checks happen even if feature > disabled. This could potentially impact the startup speed. > UMA: > # I am adding one more possible improvement to reduce Xattr objects > significantly. > SPS Xattr is constant object. So, we create one Xattr deduplication object > once statically and use the same object reference when required to add SPS > Xattr to Inode. So, here additional bytes required for storing SPS Xattr > would turn to same as single object ref ( i.e 4 bytes in 32 bit). So Xattr > overhead should come down significantly IMO. Lets explore the feasibility on > this option. > Xattr list Future will not be specially created for SPS, that list would have > been created by SetStoragePolicy already on the same directory. So, no extra > Feature creation because of SPS alone. > # Currently SPS putting long id objects in Q for tracking SPS called Inodes. > So, it is additional created and size of it would be (obj ref + value) = (8 + > 8) bytes [ ignoring alignment for time being] > So, the possible improvement here is, instead of creating new Long obj, we > can keep existing inode object for tracking. Advantage is, Inode object > already maintained in NN, so no new object creation is needed. So, we just > need to maintain one obj ref. Above two points should significantly reduce > the memory requirements of SPS. So, for SPS call: 8bytes for called inode > tracking + 8 bytes for Xattr ref. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
