[ 
https://issues.apache.org/jira/browse/HDFS-10690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15536651#comment-15536651
 ] 

Xiaoyu Yao commented on HDFS-10690:
-----------------------------------

[~fenghua_hu], we will need to update {{TestEnhancedByteBufferAccess.java}} and 
{{TestShortCircuitCache.java}} now that the {{CacheVisitor}} interface has been 
changed. Otherwise, it will fail with compiler error like below. 

Not sure why Jenkins run did not catch this, but will file separate infra JIRA 
for that. 

{code}
[ERROR] 
/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestEnhancedByteBufferAccess.java:[291,18]
 org.apache.hadoop.fs.TestEnhancedByteBufferAccess.CountingVisitor is not 
abstract and does not override abstract method 
visit(int,java.util.Map<org.apache.hadoop.hdfs.ExtendedBlockId,org.apache.hadoop.hdfs.shortcircuit.ShortCircuitReplica>,java.util.Map<org.apache.hadoop.hdfs.ExtendedBlockId,org.apache.hadoop.security.token.SecretManager.InvalidToken>,org.apache.commons.collections.map.LinkedMap,org.apache.commons.collections.map.LinkedMap)
 in org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.CacheVisitor
[ERROR] 
/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestEnhancedByteBufferAccess.java:[306,5]
 method does not override or implement a method from a supertype
{code}

> Optimize insertion/removal of replica in ShortCircuitCache.java
> ---------------------------------------------------------------
>
>                 Key: HDFS-10690
>                 URL: https://issues.apache.org/jira/browse/HDFS-10690
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: hdfs-client
>    Affects Versions: 3.0.0-alpha2
>            Reporter: Fenghua Hu
>            Assignee: Fenghua Hu
>         Attachments: HDFS-10690.001.patch, HDFS-10690.002.patch, 
> HDFS-10690.003.patch, HDFS-10690.004.patch, HDFS-10690.005.patch, 
> HDFS-10690.006.patch, ShortCircuitCache_LinkedMap.patch
>
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> Currently in ShortCircuitCache, two TreeMap objects are used to track the 
> cached replicas.
> private final TreeMap<Long, ShortCircuitReplica> evictable = new TreeMap<>();
> private final TreeMap<Long, ShortCircuitReplica> evictableMmapped = new 
> TreeMap<>();
> TreeMap employs Red-Black tree for sorting. This isn't an issue when using 
> traditional HDD. But when using high-performance SSD/PCIe Flash, the cost 
> inserting/removing an entry  becomes considerable.
> To mitigate it, we designed a new list-based for replica tracking.
> The list is a double-linked FIFO. FIFO is time-based, thus insertion is a 
> very low cost operation. On the other hand, list is not lookup-friendly. To 
> address this issue, we introduce two references into ShortCircuitReplica 
> object.
> ShortCircuitReplica next = null;
> ShortCircuitReplica prev = null;
> In this way, lookup is not needed when removing a replica from the list. We 
> only need to modify its predecessor's and successor's references in the lists.
> Our tests showed up to 15-50% performance improvement when using PCIe flash 
> as storage media.
> The original patch is against 2.6.4, now I am porting to Hadoop trunk, and 
> patch will be posted soon.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

Reply via email to