[
https://issues.apache.org/jira/browse/HDFS-6109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liang Xie updated HDFS-6109:
----------------------------
Attachment: HDFS-6109.txt
Attached is a straight-forward trunk patch, please help to review, thanks.
I introduced a config per comment and the default behivor keeps the same as
before.
This change is only meaningful for latency sensitive application like HBase.
Every write stall from datanode layer will let HBase WAL sync operation hung
the similar time range at least.
I found this post from one fb mysql developer is really nice:
http://yoshinorimatsunobu.blogspot.hk/2014/03/how-syncfilerange-really-works.html
Especially the final "Solution for the stalls" section:
{code}
calling sync_file_range() from background threads (not from user-facing thread)
are important
{code}
> let sync_file_range() system call run in backgroud
> --------------------------------------------------
>
> Key: HDFS-6109
> URL: https://issues.apache.org/jira/browse/HDFS-6109
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
> Affects Versions: 3.0.0, 2.3.0
> Reporter: Liang Xie
> Assignee: Liang Xie
> Attachments: HDFS-6109.txt
>
>
> Through we passed SYNC_FILE_RANGE_WRITE to sync_file_range, to make it as
> asynchronous as possible, it still could be blocked, e.g. the os io request
> queue is full.
> Since we use sync_file_range just as a page cache advisor role:) it doesn't
> decide or guarantee the real durability, it would be nice if we could run it
> in backgroud. At least my test log showed, a few sync_file_range calls still
> cost tens of ms or more, due to the happened location is in the critical
> write path(BlockReceiver class), from a upper view, like HBase application,
> will "hung" tens of ms as well during Hlog syncing.
> Generally speaking, the patch could not improve too much, but, better than
> before, right ? :)
--
This message was sent by Atlassian JIRA
(v6.2#6252)
