[
https://issues.apache.org/jira/browse/YARN-10975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17442711#comment-17442711
]
Ravuri Sushma sree commented on YARN-10975:
-------------------------------------------
Thanks for filing this Jira [~prabhujoseph] . I have attached a patch which
ensures
* The last processed offset of the entity file is tracked
* Skips the entity file if it is already processed and there is no new
modification in it by checking the last modification time
Please review, thanks!
> EntityGroupFSTimelineStore#ActiveLogParser parses already processed files
> --------------------------------------------------------------------------
>
> Key: YARN-10975
> URL: https://issues.apache.org/jira/browse/YARN-10975
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineserver
> Affects Versions: 3.3.0
> Reporter: Prabhu Joseph
> Assignee: Ravuri Sushma sree
> Priority: Major
> Attachments: YARN-10975.001.patch
>
>
> EntityGroupFSTimelineStore#ActiveLogParser parses already processed files
> again and again even though there is no change in the file. This leads to
> unnecessary load on DFS where summary files resides and Timeline Store where
> timeline entities present.
> {code}
> 2021-10-10 19:20:43,940 INFO timeline.LogInfo - Parsed 6 entities from
> hdfs:/prabhuJoseph/atshistory/active/application_1631559260564_0009/appattempt_1631559260564_0009_000001/summarylog-appattempt_1631559260564_0009_000001_2331123893
> in 275 msec
> 2021-10-10 19:21:44,079 INFO timeline.LogInfo - Parsed 6 entities from
> hdfs:/prabhuJoseph/atshistory/active/application_1631559260564_0009/appattempt_1631559260564_0009_000001/summarylog-appattempt_1631559260564_0009_000001_2331123893
> in 341 msec
> 2021-10-10 19:22:44,065 INFO timeline.LogInfo - Parsed 6 entities from
> hdfs:/prabhuJoseph/atshistory/active/application_1631559260564_0009/appattempt_1631559260564_0009_000001/summarylog-appattempt_1631559260564_0009_000001_2331123893
> in 335 msec
> 2021-10-10 19:23:44,038 INFO timeline.LogInfo - Parsed 6 entities from
> hdfs:/prabhuJoseph/atshistory/active/application_1631559260564_0009/appattempt_1631559260564_0009_000001/summarylog-appattempt_1631559260564_0009_000001_2331123893
> in 370 msec
> 2021-10-10 19:24:44,087 INFO timeline.LogInfo - Parsed 6 entities from
> hdfs:/prabhuJoseph/atshistory/active/application_1631559260564_0009/appattempt_1631559260564_0009_000001/summarylog-appattempt_1631559260564_0009_000001_2331123893
> in 317 msec
> 2021-10-10 19:25:44,092 INFO timeline.LogInfo - Parsed 6 entities from
> hdfs:/prabhuJoseph/atshistory/active/application_1631559260564_0009/appattempt_1631559260564_0009_000001/summarylog-appattempt_1631559260564_0009_000001_2331123893
> in 336 msec
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
