[jira] [Commented] (YARN-10975) EntityGroupFSTimelineStore#ActiveLogParser parses already processed files
[ https://issues.apache.org/jira/browse/YARN-10975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17746217#comment-17746217 ] Rakesh Radhakrishnan commented on YARN-10975: - [~prabhujoseph] I have updated the fix version to 3.4.0 as this is merged only to the trunk. > EntityGroupFSTimelineStore#ActiveLogParser parses already processed files > -- > > Key: YARN-10975 > URL: https://issues.apache.org/jira/browse/YARN-10975 > Project: Hadoop YARN > Issue Type: Bug > Components: timelineserver >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Ravuri Sushma sree >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Attachments: YARN-10975.001.patch, YARN-10975.002.patch, > YARN-10975.003.patch > > Time Spent: 0.5h > Remaining Estimate: 0h > > EntityGroupFSTimelineStore#ActiveLogParser parses already processed files > again and again even though there is no change in the file. This leads to > unnecessary load on DFS where summary files resides and Timeline Store where > timeline entities present. > {code} > 2021-10-10 19:20:43,940 INFO timeline.LogInfo - Parsed 6 entities from > hdfs:/prabhuJoseph/atshistory/active/application_1631559260564_0009/appattempt_1631559260564_0009_01/summarylog-appattempt_1631559260564_0009_01_2331123893 > in 275 msec > 2021-10-10 19:21:44,079 INFO timeline.LogInfo - Parsed 6 entities from > hdfs:/prabhuJoseph/atshistory/active/application_1631559260564_0009/appattempt_1631559260564_0009_01/summarylog-appattempt_1631559260564_0009_01_2331123893 > in 341 msec > 2021-10-10 19:22:44,065 INFO timeline.LogInfo - Parsed 6 entities from > hdfs:/prabhuJoseph/atshistory/active/application_1631559260564_0009/appattempt_1631559260564_0009_01/summarylog-appattempt_1631559260564_0009_01_2331123893 > in 335 msec > 2021-10-10 19:23:44,038 INFO timeline.LogInfo - Parsed 6 entities from > hdfs:/prabhuJoseph/atshistory/active/application_1631559260564_0009/appattempt_1631559260564_0009_01/summarylog-appattempt_1631559260564_0009_01_2331123893 > in 370 msec > 2021-10-10 19:24:44,087 INFO timeline.LogInfo - Parsed 6 entities from > hdfs:/prabhuJoseph/atshistory/active/application_1631559260564_0009/appattempt_1631559260564_0009_01/summarylog-appattempt_1631559260564_0009_01_2331123893 > in 317 msec > 2021-10-10 19:25:44,092 INFO timeline.LogInfo - Parsed 6 entities from > hdfs:/prabhuJoseph/atshistory/active/application_1631559260564_0009/appattempt_1631559260564_0009_01/summarylog-appattempt_1631559260564_0009_01_2331123893 > in 336 msec > {code} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10975) EntityGroupFSTimelineStore#ActiveLogParser parses already processed files
[ https://issues.apache.org/jira/browse/YARN-10975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17450654#comment-17450654 ] Prabhu Joseph commented on YARN-10975: -- Have committed the patch to trunk. Thanks [~Sushma_28]. > EntityGroupFSTimelineStore#ActiveLogParser parses already processed files > -- > > Key: YARN-10975 > URL: https://issues.apache.org/jira/browse/YARN-10975 > Project: Hadoop YARN > Issue Type: Bug > Components: timelineserver >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Ravuri Sushma sree >Priority: Major > Labels: pull-request-available > Attachments: YARN-10975.001.patch, YARN-10975.002.patch, > YARN-10975.003.patch > > Time Spent: 0.5h > Remaining Estimate: 0h > > EntityGroupFSTimelineStore#ActiveLogParser parses already processed files > again and again even though there is no change in the file. This leads to > unnecessary load on DFS where summary files resides and Timeline Store where > timeline entities present. > {code} > 2021-10-10 19:20:43,940 INFO timeline.LogInfo - Parsed 6 entities from > hdfs:/prabhuJoseph/atshistory/active/application_1631559260564_0009/appattempt_1631559260564_0009_01/summarylog-appattempt_1631559260564_0009_01_2331123893 > in 275 msec > 2021-10-10 19:21:44,079 INFO timeline.LogInfo - Parsed 6 entities from > hdfs:/prabhuJoseph/atshistory/active/application_1631559260564_0009/appattempt_1631559260564_0009_01/summarylog-appattempt_1631559260564_0009_01_2331123893 > in 341 msec > 2021-10-10 19:22:44,065 INFO timeline.LogInfo - Parsed 6 entities from > hdfs:/prabhuJoseph/atshistory/active/application_1631559260564_0009/appattempt_1631559260564_0009_01/summarylog-appattempt_1631559260564_0009_01_2331123893 > in 335 msec > 2021-10-10 19:23:44,038 INFO timeline.LogInfo - Parsed 6 entities from > hdfs:/prabhuJoseph/atshistory/active/application_1631559260564_0009/appattempt_1631559260564_0009_01/summarylog-appattempt_1631559260564_0009_01_2331123893 > in 370 msec > 2021-10-10 19:24:44,087 INFO timeline.LogInfo - Parsed 6 entities from > hdfs:/prabhuJoseph/atshistory/active/application_1631559260564_0009/appattempt_1631559260564_0009_01/summarylog-appattempt_1631559260564_0009_01_2331123893 > in 317 msec > 2021-10-10 19:25:44,092 INFO timeline.LogInfo - Parsed 6 entities from > hdfs:/prabhuJoseph/atshistory/active/application_1631559260564_0009/appattempt_1631559260564_0009_01/summarylog-appattempt_1631559260564_0009_01_2331123893 > in 336 msec > {code} -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10975) EntityGroupFSTimelineStore#ActiveLogParser parses already processed files
[ https://issues.apache.org/jira/browse/YARN-10975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17449430#comment-17449430 ] Prabhu Joseph commented on YARN-10975: -- Thanks [~Sushma_28] for the patch. The patch looks good to me. Will commit it shortly. > EntityGroupFSTimelineStore#ActiveLogParser parses already processed files > -- > > Key: YARN-10975 > URL: https://issues.apache.org/jira/browse/YARN-10975 > Project: Hadoop YARN > Issue Type: Bug > Components: timelineserver >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Ravuri Sushma sree >Priority: Major > Attachments: YARN-10975.001.patch, YARN-10975.002.patch, > YARN-10975.003.patch > > > EntityGroupFSTimelineStore#ActiveLogParser parses already processed files > again and again even though there is no change in the file. This leads to > unnecessary load on DFS where summary files resides and Timeline Store where > timeline entities present. > {code} > 2021-10-10 19:20:43,940 INFO timeline.LogInfo - Parsed 6 entities from > hdfs:/prabhuJoseph/atshistory/active/application_1631559260564_0009/appattempt_1631559260564_0009_01/summarylog-appattempt_1631559260564_0009_01_2331123893 > in 275 msec > 2021-10-10 19:21:44,079 INFO timeline.LogInfo - Parsed 6 entities from > hdfs:/prabhuJoseph/atshistory/active/application_1631559260564_0009/appattempt_1631559260564_0009_01/summarylog-appattempt_1631559260564_0009_01_2331123893 > in 341 msec > 2021-10-10 19:22:44,065 INFO timeline.LogInfo - Parsed 6 entities from > hdfs:/prabhuJoseph/atshistory/active/application_1631559260564_0009/appattempt_1631559260564_0009_01/summarylog-appattempt_1631559260564_0009_01_2331123893 > in 335 msec > 2021-10-10 19:23:44,038 INFO timeline.LogInfo - Parsed 6 entities from > hdfs:/prabhuJoseph/atshistory/active/application_1631559260564_0009/appattempt_1631559260564_0009_01/summarylog-appattempt_1631559260564_0009_01_2331123893 > in 370 msec > 2021-10-10 19:24:44,087 INFO timeline.LogInfo - Parsed 6 entities from > hdfs:/prabhuJoseph/atshistory/active/application_1631559260564_0009/appattempt_1631559260564_0009_01/summarylog-appattempt_1631559260564_0009_01_2331123893 > in 317 msec > 2021-10-10 19:25:44,092 INFO timeline.LogInfo - Parsed 6 entities from > hdfs:/prabhuJoseph/atshistory/active/application_1631559260564_0009/appattempt_1631559260564_0009_01/summarylog-appattempt_1631559260564_0009_01_2331123893 > in 336 msec > {code} -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10975) EntityGroupFSTimelineStore#ActiveLogParser parses already processed files
[ https://issues.apache.org/jira/browse/YARN-10975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17449349#comment-17449349 ] Hadoop QA commented on YARN-10975: -- | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Logfile || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 18m 39s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || || | {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} No case conflicting files found. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color} | {color:green}test4tests{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 12m 44s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m 23s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m 11s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 41s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 43s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 48s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 19m 0s{color} | {color:green}{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 42s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 31s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} | | {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 25m 21s{color} | {color:blue}{color} | {color:blue} Both FindBugs and SpotBugs are enabled, using SpotBugs. {color} | | {color:green}+1{color} | {color:green} spotbugs {color} | {color:green} 3m 8s{color} | {color:green}{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 28s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 3s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m 36s{color} | {color:green}{color} | {color:green} the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m 36s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 47s{color} | {color:green}{color} | {color:green} the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m 47s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 1m 48s{color} | {color:orange}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/1243/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch generated 1 new + 177 unchanged - 1 fixed = 178 total (was 178) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 37s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch has no whitespace issues. {color} | |
[jira] [Commented] (YARN-10975) EntityGroupFSTimelineStore#ActiveLogParser parses already processed files
[ https://issues.apache.org/jira/browse/YARN-10975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17444513#comment-17444513 ] Hadoop QA commented on YARN-10975: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Logfile || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 18m 25s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || || | {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} No case conflicting files found. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color} | {color:green}test4tests{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 50s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m 42s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m 35s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m 3s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 35s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 33s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 18m 41s{color} | {color:green}{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 23s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 18s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} | | {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 24m 12s{color} | {color:blue}{color} | {color:blue} Both FindBugs and SpotBugs are enabled, using SpotBugs. {color} | | {color:green}+1{color} | {color:green} spotbugs {color} | {color:green} 2m 50s{color} | {color:green}{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 28s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 3s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m 46s{color} | {color:green}{color} | {color:green} the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 9m 46s{color} | {color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/1242/artifact/out/diff-compile-javac-hadoop-yarn-project_hadoop-yarn-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt{color} | {color:red} hadoop-yarn-project_hadoop-yarn-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 generated 1 new + 399 unchanged - 0 fixed = 400 total (was 399) {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 54s{color} | {color:green}{color} | {color:green} the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 8m 54s{color} | {color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/1242/artifact/out/diff-compile-javac-hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt{color} | {color:red} hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 generated 1 new + 346 unchanged - 0 fixed = 347 total (was 346) {color} | |
[jira] [Commented] (YARN-10975) EntityGroupFSTimelineStore#ActiveLogParser parses already processed files
[ https://issues.apache.org/jira/browse/YARN-10975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17442785#comment-17442785 ] Hadoop QA commented on YARN-10975: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Logfile || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 19m 49s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || || | {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} No case conflicting files found. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color} | {color:green}test4tests{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 12m 42s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m 29s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m 15s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 39s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 42s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 33s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 20m 42s{color} | {color:green}{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 23s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 10s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} | | {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 26m 6s{color} | {color:blue}{color} | {color:blue} Both FindBugs and SpotBugs are enabled, using SpotBugs. {color} | | {color:green}+1{color} | {color:green} spotbugs {color} | {color:green} 2m 53s{color} | {color:green}{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 24s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 12s{color} | {color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/1241/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timeline-pluginstorage.txt{color} | {color:red} hadoop-yarn-server-timeline-pluginstorage in the patch failed. {color} | | {color:red}-1{color} | {color:red} compile {color} | {color:red} 4m 19s{color} | {color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/1241/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt{color} | {color:red} hadoop-yarn in the patch failed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 4m 19s{color} | {color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/1241/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt{color} | {color:red} hadoop-yarn in the patch failed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. {color} | | {color:red}-1{color} | {color:red} compile {color} | {color:red} 3m 36s{color} | {color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/1241/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt{color} | {color:red} hadoop-yarn in the patch failed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 3m
[jira] [Commented] (YARN-10975) EntityGroupFSTimelineStore#ActiveLogParser parses already processed files
[ https://issues.apache.org/jira/browse/YARN-10975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17442711#comment-17442711 ] Ravuri Sushma sree commented on YARN-10975: --- Thanks for filing this Jira [~prabhujoseph] . I have attached a patch which ensures * The last processed offset of the entity file is tracked * Skips the entity file if it is already processed and there is no new modification in it by checking the last modification time Please review, thanks! > EntityGroupFSTimelineStore#ActiveLogParser parses already processed files > -- > > Key: YARN-10975 > URL: https://issues.apache.org/jira/browse/YARN-10975 > Project: Hadoop YARN > Issue Type: Bug > Components: timelineserver >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Ravuri Sushma sree >Priority: Major > Attachments: YARN-10975.001.patch > > > EntityGroupFSTimelineStore#ActiveLogParser parses already processed files > again and again even though there is no change in the file. This leads to > unnecessary load on DFS where summary files resides and Timeline Store where > timeline entities present. > {code} > 2021-10-10 19:20:43,940 INFO timeline.LogInfo - Parsed 6 entities from > hdfs:/prabhuJoseph/atshistory/active/application_1631559260564_0009/appattempt_1631559260564_0009_01/summarylog-appattempt_1631559260564_0009_01_2331123893 > in 275 msec > 2021-10-10 19:21:44,079 INFO timeline.LogInfo - Parsed 6 entities from > hdfs:/prabhuJoseph/atshistory/active/application_1631559260564_0009/appattempt_1631559260564_0009_01/summarylog-appattempt_1631559260564_0009_01_2331123893 > in 341 msec > 2021-10-10 19:22:44,065 INFO timeline.LogInfo - Parsed 6 entities from > hdfs:/prabhuJoseph/atshistory/active/application_1631559260564_0009/appattempt_1631559260564_0009_01/summarylog-appattempt_1631559260564_0009_01_2331123893 > in 335 msec > 2021-10-10 19:23:44,038 INFO timeline.LogInfo - Parsed 6 entities from > hdfs:/prabhuJoseph/atshistory/active/application_1631559260564_0009/appattempt_1631559260564_0009_01/summarylog-appattempt_1631559260564_0009_01_2331123893 > in 370 msec > 2021-10-10 19:24:44,087 INFO timeline.LogInfo - Parsed 6 entities from > hdfs:/prabhuJoseph/atshistory/active/application_1631559260564_0009/appattempt_1631559260564_0009_01/summarylog-appattempt_1631559260564_0009_01_2331123893 > in 317 msec > 2021-10-10 19:25:44,092 INFO timeline.LogInfo - Parsed 6 entities from > hdfs:/prabhuJoseph/atshistory/active/application_1631559260564_0009/appattempt_1631559260564_0009_01/summarylog-appattempt_1631559260564_0009_01_2331123893 > in 336 msec > {code} -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10975) EntityGroupFSTimelineStore#ActiveLogParser parses already processed files
[ https://issues.apache.org/jira/browse/YARN-10975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17441137#comment-17441137 ] Prabhu Joseph commented on YARN-10975: -- The main issue is in the below code which returns 0 always. Every time the file is processed, the offset is set to 0 and next time it starts processing from 0. {code} bytesParsed = parser.getCurrentLocation().getCharOffset() + 1; LOG.trace("Parser now at offset {}", bytesParsed); {code} > EntityGroupFSTimelineStore#ActiveLogParser parses already processed files > -- > > Key: YARN-10975 > URL: https://issues.apache.org/jira/browse/YARN-10975 > Project: Hadoop YARN > Issue Type: Bug > Components: timelineserver >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Ravuri Sushma sree >Priority: Major > > EntityGroupFSTimelineStore#ActiveLogParser parses already processed files > again and again even though there is no change in the file. This leads to > unnecessary load on DFS where summary files resides and Timeline Store where > timeline entities present. > {code} > 2021-10-10 19:20:43,940 INFO timeline.LogInfo - Parsed 6 entities from > hdfs:/prabhuJoseph/atshistory/active/application_1631559260564_0009/appattempt_1631559260564_0009_01/summarylog-appattempt_1631559260564_0009_01_2331123893 > in 275 msec > 2021-10-10 19:21:44,079 INFO timeline.LogInfo - Parsed 6 entities from > hdfs:/prabhuJoseph/atshistory/active/application_1631559260564_0009/appattempt_1631559260564_0009_01/summarylog-appattempt_1631559260564_0009_01_2331123893 > in 341 msec > 2021-10-10 19:22:44,065 INFO timeline.LogInfo - Parsed 6 entities from > hdfs:/prabhuJoseph/atshistory/active/application_1631559260564_0009/appattempt_1631559260564_0009_01/summarylog-appattempt_1631559260564_0009_01_2331123893 > in 335 msec > 2021-10-10 19:23:44,038 INFO timeline.LogInfo - Parsed 6 entities from > hdfs:/prabhuJoseph/atshistory/active/application_1631559260564_0009/appattempt_1631559260564_0009_01/summarylog-appattempt_1631559260564_0009_01_2331123893 > in 370 msec > 2021-10-10 19:24:44,087 INFO timeline.LogInfo - Parsed 6 entities from > hdfs:/prabhuJoseph/atshistory/active/application_1631559260564_0009/appattempt_1631559260564_0009_01/summarylog-appattempt_1631559260564_0009_01_2331123893 > in 317 msec > 2021-10-10 19:25:44,092 INFO timeline.LogInfo - Parsed 6 entities from > hdfs:/prabhuJoseph/atshistory/active/application_1631559260564_0009/appattempt_1631559260564_0009_01/summarylog-appattempt_1631559260564_0009_01_2331123893 > in 336 msec > {code} -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org