[
https://issues.apache.org/jira/browse/HDFS-7112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14161190#comment-14161190
]
Xiaoyu Yao commented on HDFS-7112:
----------------------------------
Thanks [~arpitagarwal] for reviewing the patch.
# FsDatasetImpl.java: We should avoid adding volumes to asyncDiskService unless
there is a RAM_DISK volume configured. This avoids creating unnecessary thread
pools on most deployments which will not have RAM_DISK.
Do you mean adding volumes to asyncLazyPersistService? For
asyncLazyPersistService, we will only call asyncLazyPersistService#addVolume
when the storage type of the volume is RAM_DISK.
# We don't need the counter and logging in {{addExecutorForVolume}}. We will
never add more than one thread per volume.
Fixed.
# {{onStartLazyPersist}} should be called by {{saveNextReplica}} before it
calls {{submitLazyPersistTask}}.
Fixed.
# {{onFailLazyPersist}} should never be called unless {{onStartLazyPersist}}
has been called. I think it can be called on some failure paths even if
submitLazyPersistTask was not called.
onFailLazyPersist should be called upon any failure after the block is dequeued
with the following call.
block = ramDiskReplicaTracker.dequeueNextReplicaToPersist();
The patch calls onFailLazyPersist in two error paths.
1) Failed to submit the request to thread pool when calling
submitLazyPersistTask
2) Failed during the thread pool thread execution.
# {{BlockPoolSlice#lazyPersistReplica}} is unused, can be removed.
Removed.
# {{RamDiskAsyncLazyPersistService#countPendingTasks}} is unused.
Removed.
# It would be good if RamDiskAsyncLazyPersistService did not have a dependency
on DataNode/FsDatasetImpl. It can accept the success and failure callbacks as
parameters. But it's okay to fix it later in a separate Jira.
That’s a good idea. I will file a separate JIRA for that.
> LazyWriter should use either async IO or one thread per physical disk
> ---------------------------------------------------------------------
>
> Key: HDFS-7112
> URL: https://issues.apache.org/jira/browse/HDFS-7112
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
> Affects Versions: HDFS-6581
> Reporter: Arpit Agarwal
> Assignee: Xiaoyu Yao
> Fix For: 2.6.0
>
> Attachments: HDFS-7112.0.patch
>
>
> The LazyWriter currently uses synchronous IO and a single thread. This limits
> the throughput to that of a single disk. Using either async overlapped IO or
> one thread per physical disk will improve the write throughput.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
