[
https://issues.apache.org/jira/browse/YARN-6396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15969197#comment-15969197
]
zhihai xu edited comment on YARN-6396 at 4/14/17 4:02 PM:
----------------------------------------------------------
Thanks for the review [~jianhe] and [~rkanter]! if some one deletes the remote
log dir, all the old log will disappear. That will be a more serious issue,
recreating the remote log dir won't save the old log data. This looks like a
monitor problem, I think it will be better to do it in some tool outside the
NM. It will be more efficient to do it at one place instead of on each NM,
which could be many thousands in a large cluster. Yes, it's a trade off between
validation and efficiency. Also restarting the NM will help recreate the remote
log dir.
was (Author: zxu):
Thanks for the review [~jianhe] and [~rkanter]! if some one deletes the remote
log dir, all the old log will disappear. That will be a more serious issue,
recreating the remote log dir won't save the old log data. This looks like a
monitor problem, I think it will be better to do it in some tool outside the
NM. It will be more efficient to do it at one place instead of on each NM,
which could be many thousands in a large cluster. Yes, it's a trade off between
validation and efficiency.
> Call verifyAndCreateRemoteLogDir at service initialization instead of
> application initialization to decrease load for name node
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: YARN-6396
> URL: https://issues.apache.org/jira/browse/YARN-6396
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: log-aggregation
> Affects Versions: 3.0.0-alpha2
> Reporter: zhihai xu
> Assignee: zhihai xu
> Priority: Minor
> Attachments: YARN-6396.000.patch
>
>
> Call verifyAndCreateRemoteLogDir at service initialization instead of
> application initialization to decrease load for name node.
> Currently for every application at each Node, verifyAndCreateRemoteLogDir
> will be called before doing log aggregation, This will be a non trivial
> overhead for name node in a large cluster since verifyAndCreateRemoteLogDir
> calls getFileStatus. Once the remote log directory is created successfully,
> it is not necessary to call it again. It will be better to call
> verifyAndCreateRemoteLogDir at LogAggregationService service initialization.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
