[
https://issues.apache.org/jira/browse/HDFS-12911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uma Maheswara Rao G updated HDFS-12911:
---------------------------------------
Attachment: HDFS-12911-HDFS-10285-02.patch
A patch attached with SPS modularization.
# For Scanning: Added an interface, FileIDCollector. We can plugin specific
scanning implementations.
# For SPS-NN communication: Added interface Context, which is same as
[~chris.douglas] proposed in first patch. All nn related interaction will
happen via this interface. A specific implementations can be plugged in. One
could directly access NN internals and other could use RPC to get info from NN.
# For Block move tasks: Added an interface BlockMoveTaskHandler. We could
plugin specific implementation for block movements. One can be heartbeat based
assignment and other implementation can be sending block move ops directly to
DNs.
In this patch, I provided internal SPS implementation (
IntraSPSNameNodeContext, IntraSPSNameNodeFileIDCollector,
IntraSPSNameNodeBlockMoveTaskHandler) and for external I just created dummy
plugin classes (ExternalSPSContext, ExternalSPSFileIDCollector,
ExternalSPSBlockMoveTaskHandler).
Feedbacks are most welcomed.
> [SPS]: Fix review comments from discussions in HDFS-10285
> ---------------------------------------------------------
>
> Key: HDFS-12911
> URL: https://issues.apache.org/jira/browse/HDFS-12911
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
> Reporter: Uma Maheswara Rao G
> Assignee: Rakesh R
> Attachments: HDFS-12911-HDFS-10285-01.patch,
> HDFS-12911-HDFS-10285-02.patch, HDFS-12911.00.patch
>
>
> This is the JIRA for tracking the possible improvements or issues discussed
> in main JIRA
> So far comments to handle
> Daryn:
> # Lock should not kept while executing placement policy.
> # While starting up the NN, SPS Xattrs checks happen even if feature
> disabled. This could potentially impact the startup speed.
> UMA:
> # I am adding one more possible improvement to reduce Xattr objects
> significantly.
> SPS Xattr is constant object. So, we create one Xattr deduplication object
> once statically and use the same object reference when required to add SPS
> Xattr to Inode. So, here additional bytes required for storing SPS Xattr
> would turn to same as single object ref ( i.e 4 bytes in 32 bit). So Xattr
> overhead should come down significantly IMO. Lets explore the feasibility on
> this option.
> Xattr list Future will not be specially created for SPS, that list would have
> been created by SetStoragePolicy already on the same directory. So, no extra
> Feature creation because of SPS alone.
> # Currently SPS putting long id objects in Q for tracking SPS called Inodes.
> So, it is additional created and size of it would be (obj ref + value) = (8 +
> 8) bytes [ ignoring alignment for time being]
> So, the possible improvement here is, instead of creating new Long obj, we
> can keep existing inode object for tracking. Advantage is, Inode object
> already maintained in NN, so no new object creation is needed. So, we just
> need to maintain one obj ref. Above two points should significantly reduce
> the memory requirements of SPS. So, for SPS call: 8bytes for called inode
> tracking + 8 bytes for Xattr ref.
> # Use LightWeightLinkedSet instead of using LinkedList for from Q. This will
> reduce unnecessary Node creations inside LinkedList.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
