[
https://issues.apache.org/jira/browse/HDDS-3155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17057814#comment-17057814
]
mingchao zhao edited comment on HDDS-3155 at 3/12/20, 11:06 AM:
----------------------------------------------------------------
Hi [~elek] , [~shashikant] and [~msingh] , very thanks for your discussing this
issue. Your Suggestions have helped me a lot.
Here is my previous test result, simulating the operation of MapReduce AM to
write Log. Contrast HDFS and ozone:
{code:java}
//
String dst = "o3fs://bucket.hadoop/test4.log";
FileSystem fs = FileSystem.get(URI.create(dst), conf,"root");
FSDataOutputStream fsOut = fs.create(new Path(dst), true);
String d1 = new Date().toString();
for(int i=0;i<8000;i++){
String str =
"org.apache.hadoop.mapreduce.jobhistory.TaskAttemporg.apache.hadoop.mapreduce.jobhistory.TaskAttempa";
fsOut.write(str.getBytes());
fsOut.flush();
}
fsOut.close();
{code}
When testing ozone, the previous phenomenon I saw was that a chunk file would
be generated each time we performed flush：
!image-2020-03-12-16-48-08-391.png|width=543,height=302!
And when testing hdfs, It generated only one data file.
!image-2020-03-12-17-47-57-770.png|width=552,height=94!
The actual execution time is also quite different. I found that HDFS is called
hflush([Refer to the description of this
method|https://github.com/apache/hadoop/blob/ac4b556e2d44d3cd10b81c190ecee23e2dd66c10/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java#L573])
when performing flush.
At present, I am not sure if ozone has a similar design. I will look at the
implementation of HDDS-2717 first to confirm whether this PR can solve the
current problem.
was (Author: micahzhao):
Hi [~elek] [~shashikant] [~msingh] very thanks for your discussing this issue.
Your Suggestions have helped me a lot.
Here is my previous test result, simulating the operation of MapReduce AM to
write Log. Contrast HDFS and ozone:
{code:java}
//
String dst = "o3fs://bucket.hadoop/test4.log";
FileSystem fs = FileSystem.get(URI.create(dst), conf,"root");
FSDataOutputStream fsOut = fs.create(new Path(dst), true);
String d1 = new Date().toString();
for(int i=0;i<8000;i++){
String str =
"org.apache.hadoop.mapreduce.jobhistory.TaskAttemporg.apache.hadoop.mapreduce.jobhistory.TaskAttempa";
fsOut.write(str.getBytes());
fsOut.flush();
}
fsOut.close();
{code}
When testing ozone, the previous phenomenon I saw was that a chunk file would
be generated each time we performed flush：
!image-2020-03-12-16-48-08-391.png|width=543,height=302!
And when testing hdfs, It generated only one data file.
!image-2020-03-12-17-47-57-770.png|width=552,height=94!
The actual execution time is also quite different. I found that HDFS is called
hflush([Refer to the description of this
method|https://github.com/apache/hadoop/blob/ac4b556e2d44d3cd10b81c190ecee23e2dd66c10/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java#L573])
when performing flush.
At present, I am not sure if ozone has a similar design. I will look at the
implementation of HDDS-2717 first to confirm whether this PR can solve the
current problem.
> Improved ozone client flush implementation to make it faster.
> -------------------------------------------------------------
>
> Key: HDDS-3155
> URL: https://issues.apache.org/jira/browse/HDDS-3155
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Reporter: mingchao zhao
> Priority: Major
> Attachments: amlog, image-2020-03-12-16-48-08-391.png,
> image-2020-03-12-17-47-57-770.png, stdout
>
>
> Background:
> When we execute mapreduce in the ozone, we find that the task will be
> stuck for a long time after the completion of Map and Reduce. The log is as
> follows:
> {code:java}
> //Refer to the attachment: stdout
> 20/03/05 14:43:30 INFO mapreduce.Job: map 100% reduce 33%
> 20/03/05 14:43:33 INFO mapreduce.Job: map 100% reduce 100%
> 20/03/05 15:29:52 INFO mapreduce.Job: Job job_1583385253878_0002 completed
> successfully{code}
> By looking at AM's log(Refer to the amlog for details), we found that the
> time of over 40 minutes is AM writing a task log into ozone.
> At present, after MR execution, the Task information is recorded into the
> log on HDFS or ozone by AM. Moreover, the task information is flush to HDFS
> or ozone one by one
> ([details|https://github.com/apache/hadoop/blob/a55d6bba71c81c1c4e9d8cd11f55c78f10a548b0/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java#L1640]).
> The problem occurs when the number of task maps is large.
> Currently, each flush operation in ozone generates a new chunk file in
> real time on the disk. This approach is not very efficient at the moment. For
> this we can refer to the implementation of HDFS flush. Instead of writing to
> disk each time flush writes the contents of the buffer to the datanode's OS
> buffer. In the first place, we need to ensure that this content can be read
> by other datanodes.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
