[jira] [Commented] (MAPREDUCE-7158) Inefficient Flush Logic in JobHistory EventWriter
[ https://issues.apache.org/jira/browse/MAPREDUCE-7158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685669#comment-16685669 ] Hudson commented on MAPREDUCE-7158: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15420 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/15420/]) MAPREDUCE-7158. Inefficient Flush Logic in JobHistory EventWriter. (wangda: rev 762a56cc64bc07d57f94e253920534b8e049f238) * (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/EventWriter.java > Inefficient Flush Logic in JobHistory EventWriter > - > > Key: MAPREDUCE-7158 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7158 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 3.2.0 >Reporter: Zichen Sun >Assignee: Zichen Sun >Priority: Major > Fix For: 3.1.2, 3.3.0, 3.2.1 > > Attachments: MAPREDUCE-7158-001.patch > > > In HDFS, if the flush is implemented to send server request to actually > commit the pending writes on the storage service side, we could observe in > the benchmark runs that the MR jobs are taking much longer. From > investigation we see the current implementation for writing events doesn't > look right: > EventWriter# write() > This flush is redundant and this statement should be removed. It defeats the > purpose of having a separate flush function itself. > Encoder.flush calls flush of the underlying output stream > After patching with the fix the MR jobs could complete normally, please > kindly find the patch in attached. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7158) Inefficient Flush Logic in JobHistory EventWriter
[ https://issues.apache.org/jira/browse/MAPREDUCE-7158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685550#comment-16685550 ] Wangda Tan commented on MAPREDUCE-7158: --- +1, patch LGTM, thanks [~zichensun]. Also added you to contributor list so you can assign tickets to yourself in the future. > Inefficient Flush Logic in JobHistory EventWriter > - > > Key: MAPREDUCE-7158 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7158 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 3.2.0 >Reporter: Zichen Sun >Assignee: Zichen Sun >Priority: Major > Attachments: MAPREDUCE-7158-001.patch > > > In HDFS, if the flush is implemented to send server request to actually > commit the pending writes on the storage service side, we could observe in > the benchmark runs that the MR jobs are taking much longer. From > investigation we see the current implementation for writing events doesn't > look right: > EventWriter# write() > This flush is redundant and this statement should be removed. It defeats the > purpose of having a separate flush function itself. > Encoder.flush calls flush of the underlying output stream > After patching with the fix the MR jobs could complete normally, please > kindly find the patch in attached. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7158) Inefficient Flush Logic in JobHistory EventWriter
[ https://issues.apache.org/jira/browse/MAPREDUCE-7158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16684546#comment-16684546 ] Zichen Sun commented on MAPREDUCE-7158: --- the patch has no functionality change, it's been verified in a test cluster using 10TB benchmark data and we the performance is much improved > Inefficient Flush Logic in JobHistory EventWriter > - > > Key: MAPREDUCE-7158 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7158 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 3.2.0 >Reporter: Zichen Sun >Priority: Major > Attachments: MAPREDUCE-7158-001.patch > > > In HDFS, if the flush is implemented to send server request to actually > commit the pending writes on the storage service side, we could observe in > the benchmark runs that the MR jobs are taking much longer. From > investigation we see the current implementation for writing events doesn't > look right: > EventWriter# write() > This flush is redundant and this statement should be removed. It defeats the > purpose of having a separate flush function itself. > Encoder.flush calls flush of the underlying output stream > After patching with the fix the MR jobs could complete normally, please > kindly find the patch in attached. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7158) Inefficient Flush Logic in JobHistory EventWriter
[
https://issues.apache.org/jira/browse/MAPREDUCE-7158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16684336#comment-16684336
]
Hadoop QA commented on MAPREDUCE-7158:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 34s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 7s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
19s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 54m 49s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | MAPREDUCE-7158 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12947880/MAPREDUCE-7158-001.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 6e1205c7a24c 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 1f9c4f3 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7526/testReport/ |
| Max. process+thread count | 1576 (vs. ulimit of 1) |
| modules | C:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
U:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7526/console |
| Powered by | Apache
