[
https://issues.apache.org/jira/browse/TEZ-1106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13999178#comment-13999178
]
Hitesh Shah commented on TEZ-1106:
----------------------------------
[~kamrul] There was a comment on getTezSystemStagingPath() being called in each
and every function in TezCommonUtils. This function is heavy and goes to the NN
each time to resolve the path, create dirs, etc.This needs to be addressed.
Other comments:
- typo at "Path tezSysStagingPat,"
{code}
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("AppMaster binaryConfPath = " + binaryConfPath);
+ }
{code}
- there are a lot of debug statements such as this. 2 options: Option is only
log the staging dir once and remove others. Option 2 is add more useful info
and combine all into a single log message. Also, you should log appId and
staging dir with this log message as the same log file could have multiple
applications ( example hive-server.log which uses tez client )
> Tez framework should use a unique subdir when creating new files in staging
> -----------------------------------------------------------------------------
>
> Key: TEZ-1106
> URL: https://issues.apache.org/jira/browse/TEZ-1106
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Mohammad Kamrul Islam
> Assignee: Mohammad Kamrul Islam
> Attachments: TEZ-1106.1.patch, TEZ-1106.2.patch, TEZ-1106.3.patch
>
>
> Currently the files are created in different sub-directories. It is hard to
> manage and cleanup at the end.
> The proposal is to create a new subdir : $STAGE_DIR/<APP_ID>/
> All recovery files will go under : $STAGE_DIR/<APP_ID>/recovery/<attemp_num>/
> All confs will go under: $STAGE_DIR/<APP_ID>/conf/
> All dagplans will go: $STAGE_DIR/<APP_ID>/dag_id/plan/
--
This message was sent by Atlassian JIRA
(v6.2#6252)
