[ https://issues.apache.org/jira/browse/HDFS-5727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Richard Chen updated HDFS-5727: ------------------------------- Description: Currently the datanode read/write SLA is difficult to be guaranteed for HBase online requirement. One of major reasons is we don't support io priority or io request reorder inside datanode. I propose introducing a self-maintain io queue mechanism to handle io request priority. Imaging there're lots of concurrent read/write requests from HBase side, and a background datanode block scanner is running(default is every 21 days, IIRC) just in time, then the HBase read/write 99% or 99.9% percentile latency would be vulnerable despite we have a bg thread throttling... the reorder stuff i have not thought clearly enough, but definitely the reorder in the queue in the app side would beat the currently relying OS's io queue merge. was: Currently the datanode read/write SLA is dfficult to be ganranteed for HBase online requirement. One of major reasons is we don't support io priority or io reqeust reorder inside datanode. I proposal introducing a self-maintain io queue mechanism to handle io request priority. Image there're lots of concurrent read/write reqeust from HBase side, and a background datanode block scanner is running(default is every 21 days, IIRC) just in time, then the HBase read/write 99% or 99.9% percentile latency would be vulnerable despite we have a bg thread throttling... the reorder stuf i have not thought clearly enough, but definitely the reorder in the queue in the app side would beat the currently relying OS's io queue merge. > introduce a self-maintain io queue handling mechanism > ----------------------------------------------------- > > Key: HDFS-5727 > URL: https://issues.apache.org/jira/browse/HDFS-5727 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode > Affects Versions: 3.0.0 > Reporter: Liang Xie > Assignee: Liang Xie > > Currently the datanode read/write SLA is difficult to be guaranteed for HBase > online requirement. One of major reasons is we don't support io priority or > io request reorder inside datanode. > I propose introducing a self-maintain io queue mechanism to handle io request > priority. Imaging there're lots of concurrent read/write requests from HBase > side, and a background datanode block scanner is running(default is every 21 > days, IIRC) just in time, then the HBase read/write 99% or 99.9% percentile > latency would be vulnerable despite we have a bg thread throttling... > the reorder stuff i have not thought clearly enough, but definitely the > reorder in the queue in the app side would beat the currently relying OS's io > queue merge. -- This message was sent by Atlassian JIRA (v6.1.5#6160)