[
https://issues.apache.org/jira/browse/HDFS-12911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uma Maheswara Rao G updated HDFS-12911:
---------------------------------------
Description:
One of the key comment from the discussions was to modularize the SPS code, so
we can easily plug in the external/internal implementations.
So other comments to handle
Daryn:
# Lock should not kept while executing placement policy.
- already handled by
# While starting up the NN, SPS Xattrs checks happen even if feature disabled.
This could potentially impact the startup speed.
was:
This is the JIRA for tracking the possible improvements or issues discussed in
main JIRA
So far comments to handle
Daryn:
# Lock should not kept while executing placement policy.
# While starting up the NN, SPS Xattrs checks happen even if feature disabled.
This could potentially impact the startup speed.
UMA:
# I am adding one more possible improvement to reduce Xattr objects
significantly.
SPS Xattr is constant object. So, we create one Xattr deduplication object
once statically and use the same object reference when required to add SPS
Xattr to Inode. So, here additional bytes required for storing SPS Xattr would
turn to same as single object ref ( i.e 4 bytes in 32 bit). So Xattr overhead
should come down significantly IMO. Lets explore the feasibility on this option.
Xattr list Future will not be specially created for SPS, that list would have
been created by SetStoragePolicy already on the same directory. So, no extra
Feature creation because of SPS alone.
# Currently SPS putting long id objects in Q for tracking SPS called Inodes.
So, it is additional created and size of it would be (obj ref + value) = (8 +
8) bytes [ ignoring alignment for time being]
So, the possible improvement here is, instead of creating new Long obj, we can
keep existing inode object for tracking. Advantage is, Inode object already
maintained in NN, so no new object creation is needed. So, we just need to
maintain one obj ref. Above two points should significantly reduce the memory
requirements of SPS. So, for SPS call: 8bytes for called inode tracking + 8
bytes for Xattr ref.
# Use LightWeightLinkedSet instead of using LinkedList for from Q. This will
reduce unnecessary Node creations inside LinkedList.
> [SPS]: Modularize the SPS code and expose necessary interfaces for
> external/internal implementations.
> -----------------------------------------------------------------------------------------------------
>
> Key: HDFS-12911
> URL: https://issues.apache.org/jira/browse/HDFS-12911
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
> Reporter: Uma Maheswara Rao G
> Assignee: Rakesh R
> Attachments: HDFS-12911-HDFS-10285-01.patch,
> HDFS-12911-HDFS-10285-02.patch, HDFS-12911-HDFS-10285-03.patch,
> HDFS-12911.00.patch
>
>
> One of the key comment from the discussions was to modularize the SPS code,
> so we can easily plug in the external/internal implementations.
> So other comments to handle
> Daryn:
> # Lock should not kept while executing placement policy.
> - already handled by
> # While starting up the NN, SPS Xattrs checks happen even if feature
> disabled. This could potentially impact the startup speed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
