[ https://issues.apache.org/jira/browse/MAPREDUCE-7272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17080773#comment-17080773 ]
Hadoop QA commented on MAPREDUCE-7272: -------------------------------------- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 26s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 4 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 17s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m 1s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 43s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 51s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 7s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 15m 19s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 40s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 49s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 3s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 43s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 43s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 44s{color} | {color:green} hadoop-mapreduce-project/hadoop-mapreduce-client: The patch generated 0 new + 585 unchanged - 3 fixed = 585 total (was 588) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 59s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 14m 6s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 47s{color} | {color:red} hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 38s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 5m 20s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 20s{color} | {color:green} hadoop-mapreduce-client-app in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 30s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 75m 18s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app | | | org.apache.hadoop.mapreduce.v2.api.records.TaskAttemptId is incompatible with expected argument type TaskAttemptID in org.apache.hadoop.mapred.TaskAttemptListenerImpl.unregister(TaskAttemptId, WrappedJvmID) At TaskAttemptListenerImpl.java:argument type TaskAttemptID in org.apache.hadoop.mapred.TaskAttemptListenerImpl.unregister(TaskAttemptId, WrappedJvmID) At TaskAttemptListenerImpl.java:[line 583] | | | TaskAttemptListenerImpl$TaskProgressLogPair is incompatible with expected argument type TaskAttemptID in org.apache.hadoop.mapred.TaskAttemptListenerImpl$TaskProgressLogPair.update(float) At TaskAttemptListenerImpl.java:argument type TaskAttemptID in org.apache.hadoop.mapred.TaskAttemptListenerImpl$TaskProgressLogPair.update(float) At TaskAttemptListenerImpl.java:[line 712] | \\ \\ || Subsystem || Report/Notes || | Docker | Client=19.03.8 Server=19.03.8 Image:yetus/hadoop:e6455cc864d | | JIRA Issue | MAPREDUCE-7272 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12999561/MAPREDUCE-7272.002.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 36291c60b8eb 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 275c478 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_242 | | findbugs | v3.1.0-RC1 | | findbugs | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7759/artifact/out/new-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.html | | Test Results | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7759/testReport/ | | Max. process+thread count | 1294 (vs. ulimit of 5500) | | modules | C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client | | Console output | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7759/console | | Powered by | Apache Yetus 0.8.0 http://yetus.apache.org | This message was automatically generated. > TaskAttemptListenerImpl excessive log messages > ---------------------------------------------- > > Key: MAPREDUCE-7272 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7272 > Project: Hadoop Map/Reduce > Issue Type: Bug > Reporter: Ahmed Hussein > Assignee: Ahmed Hussein > Priority: Major > Attachments: MAPREDUCE-7272-branch-2.10.001.patch, > MAPREDUCE-7272-branch-2.10.002.patch, MAPREDUCE-7272-branch-2.10.003.patch, > MAPREDUCE-7272.001.patch, MAPREDUCE-7272.002.patch > > > {{TaskAttemptListenerImpl.statusUpdate()}} causes a bloating in log files. > One every call, the listener uses {{LOG.info()}} to printout the progress of > the {{TaskAttempt}}. > {code:java} > taskAttemptStatus.progress = taskStatus.getProgress(); > LOG.info("Progress of TaskAttempt " + taskAttemptID + " is : " > + taskStatus.getProgress()); > {code} > > {code:bash} > 2020-04-07 10:20:50,708 INFO [IPC Server handler 17 on 43926] > org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt > attempt_1586003420099_716645_m_007783_0 is : 0.40713295 > 2020-04-07 10:20:50,717 INFO [IPC Server handler 7 on 43926] > org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt > attempt_1586003420099_716645_m_020681_0 is : 0.55573714 > 2020-04-07 10:20:50,717 INFO [IPC Server handler 26 on 43926] > org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt > attempt_1586003420099_716645_m_024371_0 is : 0.54190344 > 2020-04-07 10:20:50,738 INFO [IPC Server handler 15 on 43926] > org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt > attempt_1586003420099_716645_m_033182_0 is : 0.50264555 > 2020-04-07 10:20:50,748 INFO [IPC Server handler 3 on 43926] > org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt > attempt_1586003420099_716645_m_022375_0 is : 0.5495565 > {code} > After discussing this issue with [~nroberts], [~ebadger], and [~epayne], we > thought that while it is helpful to have a log print of task progress, it is > still excessive to log the progress in every update. > This Jira is to suppress the excessive logging from TaskAttemptListener > without affecting the frequency of progress updates. > There are two flags: > * {{-Dmapreduce.task.log.progress.delta.threshold=0.10}}: means that the > task progress will be logged every 10% of delta progress. Default is 5%. > * {{-Dmapreduce.task.log.progress.wait.interval-seconds=120}}: means that if > the listener will log the progress every 2 minutes. This is helpful for long > running tasks that take long time to achieve the delta threshold. Default is > 1 minute. > The listener will long whichever of {{delta.threshold}} and > {{wait.interval-seconds}} is reached first. > Enabling {{LOG.DEBUG}} for {{TaskAttemptListenerImpl}} will override > those two flags and log the task progress on every update. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org