[
https://issues.apache.org/jira/browse/YARN-4265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15092255#comment-15092255
]
Junping Du commented on YARN-4265:
----------------------------------
Thanks [~gtCarrera9] for reply my comments above.
bq. This is not possible because in the first part of cleanLogs(), we're only
doing a DFS to decide if we need to remove this dir. If any file in the
directory is new, we will not remove it. The detailed remove logic happens
after the DFS process.
If our intention is to a DFS (deep first search), then I would say the
implementation of cleanLogs() failed to achieve this because the flag of
{{shouldClean}} in recursively calling cleanLogs() doesn't passed up to upper
caller to reflect {{shouldClean}} flag there. We should fix it with a unit test.
bq. I believe we cannot. We're trying to return the latest time any file within
a directory has been changed to decide if the app is in UNKNOWN state for long
enough in parseSummaryLogs.
For HDFS, any file update should reflect the modification time to parent
directory. Any exception case here?
> Provide new timeline plugin storage to support fine-grained entity caching
> --------------------------------------------------------------------------
>
> Key: YARN-4265
> URL: https://issues.apache.org/jira/browse/YARN-4265
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
> Reporter: Li Lu
> Assignee: Li Lu
> Attachments: YARN-4265-trunk.001.patch, YARN-4265-trunk.002.patch,
> YARN-4265-trunk.003.patch, YARN-4265-trunk.004.patch,
> YARN-4265.YARN-4234.001.patch, YARN-4265.YARN-4234.002.patch
>
>
> To support the newly proposed APIs in YARN-4234, we need to create a new
> plugin timeline store. The store may have similar behavior as the
> EntityFileTimelineStore proposed in YARN-3942, but cache date in cache id
> granularity, instead of application id granularity. Let's have this storage
> as a standalone one, instead of updating EntityFileTimelineStore, to keep the
> existing store (EntityFileTimelineStore) stable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
