[
https://issues.apache.org/jira/browse/HDFS-895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12800055#action_12800055
]
Jean-Daniel Cryans commented on HDFS-895:
-----------------------------------------
I did the experiment Joydeep described on 1 machine. I replaced the line where
we call hflush in the write-ahead-log with:
{code}
if(now % 3 == 0)
Thread.sleep(1);
{code}
Because first if I just slept for 1ms it was already 2-3 times slower then
normal sync time, I guess it's because it's very hard for the JVM to schedule
such a small sleep time. The "now" variable is a System.currentTimeInMillis
called just before and used for other metrics.
So with this modification a single client takes as much time inserting as with
normal sync and 4 clients take almost the same time on average to insert a
value. With sync and 4 clients, it takes twice the time to insert a single
value.
It would tend to confirm that the synchronization between append and sync costs
a lot for multi-threaded clients.
> Allow hflush/sync to occur in parallel with new writes to the file
> ------------------------------------------------------------------
>
> Key: HDFS-895
> URL: https://issues.apache.org/jira/browse/HDFS-895
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs client
> Reporter: dhruba borthakur
>
> In the current trunk, the HDFS client methods writeChunk() and hflush./sync
> are syncronized. This means that if a hflush/sync is in progress, an
> applicationn cannot write data to the HDFS client buffer. This reduces the
> write throughput of the transaction log in HBase.
> The hflush/sync should allow new writes to happen to the HDFS client even
> when a hflush/sync is in progress. It can record the seqno of the message for
> which it should receice the ack, indicate to the DataStream thread to star
> flushing those messages, exit the synchronized section and just wai for that
> ack to arrive.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
