[
https://issues.apache.org/jira/browse/HDFS-5727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Richard Chen updated HDFS-5727:
-------------------------------
Description:
Currently the datanode read/write SLA is difficult to be guaranteed for HBase
online requirement. One of major reasons is we don't support io priority or io
request reorder inside datanode.
I propose introducing a self-maintain io queue mechanism to handle io request
priority. Imaging there're lots of concurrent read/write requests from HBase
side, and a background datanode block scanner is running(default is every 21
days, IIRC) just in time, then the HBase read/write 99% or 99.9% percentile
latency would be vulnerable despite we have a bg thread throttling...
the reorder stuff i have not thought clearly enough, but definitely the reorder
in the queue in the app side would beat the currently relying OS's io queue
merge.
was:
Currently the datanode read/write SLA is dfficult to be ganranteed for HBase
online requirement. One of major reasons is we don't support io priority or io
reqeust reorder inside datanode.
I proposal introducing a self-maintain io queue mechanism to handle io request
priority. Image there're lots of concurrent read/write reqeust from HBase side,
and a background datanode block scanner is running(default is every 21 days,
IIRC) just in time, then the HBase read/write 99% or 99.9% percentile latency
would be vulnerable despite we have a bg thread throttling...
the reorder stuf i have not thought clearly enough, but definitely the reorder
in the queue in the app side would beat the currently relying OS's io queue
merge.
> introduce a self-maintain io queue handling mechanism
> -----------------------------------------------------
>
> Key: HDFS-5727
> URL: https://issues.apache.org/jira/browse/HDFS-5727
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode
> Affects Versions: 3.0.0
> Reporter: Liang Xie
> Assignee: Liang Xie
>
> Currently the datanode read/write SLA is difficult to be guaranteed for HBase
> online requirement. One of major reasons is we don't support io priority or
> io request reorder inside datanode.
> I propose introducing a self-maintain io queue mechanism to handle io request
> priority. Imaging there're lots of concurrent read/write requests from HBase
> side, and a background datanode block scanner is running(default is every 21
> days, IIRC) just in time, then the HBase read/write 99% or 99.9% percentile
> latency would be vulnerable despite we have a bg thread throttling...
> the reorder stuff i have not thought clearly enough, but definitely the
> reorder in the queue in the app side would beat the currently relying OS's io
> queue merge.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
