[
https://issues.apache.org/jira/browse/HDDS-3155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17061019#comment-17061019
]
Shashikant Banerjee commented on HDDS-3155:
-------------------------------------------
Thanks [~micahzhao].
>> 1. If you do not actively call flush, after the client's buffer is full,
>>it will automatically brush to the DN's buffer through the DFSOutputStream's
>>hflush (no guarantee to write to disk). This is why HDFS behaves faster than
>>ozone.
In Ozone as well, once the client buffer gets full (64MB), it automatically
does flush and make sure the data is pushed to DN.
>> If I actively call flush, as I did in my test program above. The data in
>>the client buffer will be flushed directly to the DN buffer via hflush (no
>>writing to disk is guaranteed). This is why HDFS Mapreduce doesn't get stuck.
In this case as well, if my understanding is correct, it will flush the data
from the client buffer to DN in Ozone as well.
The difference between HDFS and Ozone is it has an additional RPC call called
"watchForCommit" which ensures whether data is replicated to all servers or
not.
Also, with less no of clients/mappers , HDFS seems to be better than Ozone but
as you increase the no of clients/mappers Ozone performs on par with HDFS as
observed in our tests. There is additional disk sync(raft log sync) involved
with Ratis for each transaction which does not seem to exist in HDFS .
> Improved ozone client flush implementation to make it faster.
> -------------------------------------------------------------
>
> Key: HDDS-3155
> URL: https://issues.apache.org/jira/browse/HDDS-3155
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Reporter: mingchao zhao
> Priority: Major
> Attachments: amlog, image-2020-03-12-16-48-08-391.png,
> image-2020-03-12-17-47-57-770.png, stdout
>
>
> Background:
> When we execute mapreduce in the ozone, we find that the task will be
> stuck for a long time after the completion of Map and Reduce. The log is as
> follows:
> {code:java}
> //Refer to the attachment: stdout
> 20/03/05 14:43:30 INFO mapreduce.Job: map 100% reduce 33%
> 20/03/05 14:43:33 INFO mapreduce.Job: map 100% reduce 100%
> 20/03/05 15:29:52 INFO mapreduce.Job: Job job_1583385253878_0002 completed
> successfully{code}
> By looking at AM's log(Refer to the amlog for details), we found that the
> time of over 40 minutes is AM writing a task log into ozone.
> At present, after MR execution, the Task information is recorded into the
> log on HDFS or ozone by AM. Moreover, the task information is flush to HDFS
> or ozone one by one
> ([details|https://github.com/apache/hadoop/blob/a55d6bba71c81c1c4e9d8cd11f55c78f10a548b0/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java#L1640]).
> The problem occurs when the number of task maps is large.
> Currently, each flush operation in ozone generates a new chunk file in
> real time on the disk. This approach is not very efficient at the moment. For
> this we can refer to the implementation of HDFS flush. Instead of writing to
> disk each time flush writes the contents of the buffer to the datanode's OS
> buffer. In the first place, we need to ensure that this content can be read
> by other datanodes.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
