[
https://issues.apache.org/jira/browse/HDFS-10690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15536651#comment-15536651
]
Xiaoyu Yao edited comment on HDFS-10690 at 9/30/16 6:34 PM:
------------------------------------------------------------
[~fenghua_hu], we will need to update the {{CacheVisistor}} instances in
{{TestEnhancedByteBufferAccess.java}} and {{TestShortCircuitCache.java}} now
that the {{CacheVisitor}} interface has been changed from using Map to
LinkedMap. Otherwise, it will fail with compiler error like below.
Not sure why Jenkins run did not catch this, but will file separate infra JIRA
for that.
{code}
[ERROR]
/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestEnhancedByteBufferAccess.java:[291,18]
org.apache.hadoop.fs.TestEnhancedByteBufferAccess.CountingVisitor is not
abstract and does not override abstract method
visit(int,java.util.Map<org.apache.hadoop.hdfs.ExtendedBlockId,org.apache.hadoop.hdfs.shortcircuit.ShortCircuitReplica>,java.util.Map<org.apache.hadoop.hdfs.ExtendedBlockId,org.apache.hadoop.security.token.SecretManager.InvalidToken>,org.apache.commons.collections.map.LinkedMap,org.apache.commons.collections.map.LinkedMap)
in org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.CacheVisitor
[ERROR]
/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestEnhancedByteBufferAccess.java:[306,5]
method does not override or implement a method from a supertype
{code}
was (Author: xyao):
[~fenghua_hu], we will need to update {{TestEnhancedByteBufferAccess.java}} and
{{TestShortCircuitCache.java}} now that the {{CacheVisitor}} interface has been
changed. Otherwise, it will fail with compiler error like below.
Not sure why Jenkins run did not catch this, but will file separate infra JIRA
for that.
{code}
[ERROR]
/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestEnhancedByteBufferAccess.java:[291,18]
org.apache.hadoop.fs.TestEnhancedByteBufferAccess.CountingVisitor is not
abstract and does not override abstract method
visit(int,java.util.Map<org.apache.hadoop.hdfs.ExtendedBlockId,org.apache.hadoop.hdfs.shortcircuit.ShortCircuitReplica>,java.util.Map<org.apache.hadoop.hdfs.ExtendedBlockId,org.apache.hadoop.security.token.SecretManager.InvalidToken>,org.apache.commons.collections.map.LinkedMap,org.apache.commons.collections.map.LinkedMap)
in org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.CacheVisitor
[ERROR]
/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestEnhancedByteBufferAccess.java:[306,5]
method does not override or implement a method from a supertype
{code}
> Optimize insertion/removal of replica in ShortCircuitCache.java
> ---------------------------------------------------------------
>
> Key: HDFS-10690
> URL: https://issues.apache.org/jira/browse/HDFS-10690
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
> Affects Versions: 3.0.0-alpha2
> Reporter: Fenghua Hu
> Assignee: Fenghua Hu
> Attachments: HDFS-10690.001.patch, HDFS-10690.002.patch,
> HDFS-10690.003.patch, HDFS-10690.004.patch, HDFS-10690.005.patch,
> HDFS-10690.006.patch, ShortCircuitCache_LinkedMap.patch
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> Currently in ShortCircuitCache, two TreeMap objects are used to track the
> cached replicas.
> private final TreeMap<Long, ShortCircuitReplica> evictable = new TreeMap<>();
> private final TreeMap<Long, ShortCircuitReplica> evictableMmapped = new
> TreeMap<>();
> TreeMap employs Red-Black tree for sorting. This isn't an issue when using
> traditional HDD. But when using high-performance SSD/PCIe Flash, the cost
> inserting/removing an entry becomes considerable.
> To mitigate it, we designed a new list-based for replica tracking.
> The list is a double-linked FIFO. FIFO is time-based, thus insertion is a
> very low cost operation. On the other hand, list is not lookup-friendly. To
> address this issue, we introduce two references into ShortCircuitReplica
> object.
> ShortCircuitReplica next = null;
> ShortCircuitReplica prev = null;
> In this way, lookup is not needed when removing a replica from the list. We
> only need to modify its predecessor's and successor's references in the lists.
> Our tests showed up to 15-50% performance improvement when using PCIe flash
> as storage media.
> The original patch is against 2.6.4, now I am porting to Hadoop trunk, and
> patch will be posted soon.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
