[
https://issues.apache.org/jira/browse/YARN-11093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17800274#comment-17800274
]
ASF GitHub Bot commented on YARN-11093:
---------------------------------------
guptashailesh92 closed pull request #4093: YARN-11093: fix for reading files in
timeline server in chronological order in case of fs-support-append to false
URL: https://github.com/apache/hadoop/pull/4093
> In fs-support-append false case, timelineserver reads event files in
> arbitrary order
> ------------------------------------------------------------------------------------
>
> Key: YARN-11093
> URL: https://issues.apache.org/jira/browse/YARN-11093
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineserver
> Affects Versions: 3.3.2
> Reporter: shailesh gupta
> Priority: Critical
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> In our setup, we are using Hive and Tez and using offline tez-ui mode, where
> we copy the ATS events files to other place and start tez-ui. To keep the
> event files small, we are using
> fs-support-append = false, and creating a new file for writing the events. We
> can see that, in this mode, file name contains
> [suffix|https://github.com/apache/hadoop/blob/1d5650c4d0acf33f141d593762682b3603523104/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/impl/FileSystemTimelineWriter.java#L382]
> as timestamp.
> But at the time of
> [read|https://github.com/apache/hadoop/blob/1d5650c4d0acf33f141d593762682b3603523104/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/src/main/java/org/apache/hadoop/yarn/server/timeline/EntityGroupFSTimelineStore.java#L844],
> we are relying on the FS to provide correct file order. If that fails, we
> are adding event out of order, leading to event being discarded, or incorrect
> information.
> Fix could be sorting of the file names, based on suffix if append mode is not
> used.
> sample file names:
> * summarylog-appattempt_1647348120288_0001_000001_460237
> *
> entitylog-timelineEntityGroupId_1647348120288_1_dag_1647348120288_0001_1_673147
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
