[
https://issues.apache.org/jira/browse/TEZ-1106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14011532#comment-14011532
]
Hitesh Shah commented on TEZ-1106:
----------------------------------
bq. I thought we should not modify the conf. If you think, we can do it
otherwise.
In this case, I think it might be useful as we should be trying to resolve the
path only once.
bq. They are two different items. First one is for original staging directory
provided by user. The second one is for Tez system internal dir which append
_tez and application id at the end of the path.
Understood. However, why does the user or any of the framework code need to
access basePath()? Shouldn't all public access be to the system path dir i.e
the one with .tez/appId/ ?
bq. It creates if it doesn't exist. Yes, the Tez system directory needs to be
created at least in the first call. User can't or shouldn't create this.
My assumption here is that that once the AM is running, there should never be
any situation where the staging dir does not exist. If it has disappeared, it
means something went wrong or someone else manually deleted it. Would I be
correct in the above assumption? If yes, this implies that the code in the AM
should never be silently creating the dir. Likewise, are there any points in
the client where the expectation is that the dir should already exist?
> Tez framework should use a unique subdir when creating new files in staging
> -----------------------------------------------------------------------------
>
> Key: TEZ-1106
> URL: https://issues.apache.org/jira/browse/TEZ-1106
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Mohammad Kamrul Islam
> Assignee: Mohammad Kamrul Islam
> Attachments: TEZ-1106.1.patch, TEZ-1106.2.patch, TEZ-1106.3.patch,
> TEZ-1106.4.patch
>
>
> Currently the files are created in different sub-directories. It is hard to
> manage and cleanup at the end.
> The proposal is to create a new subdir : $STAGE_DIR/<APP_ID>/
> All recovery files will go under : $STAGE_DIR/<APP_ID>/recovery/<attemp_num>/
> All confs will go under: $STAGE_DIR/<APP_ID>/conf/
> All dagplans will go: $STAGE_DIR/<APP_ID>/dag_id/plan/
--
This message was sent by Atlassian JIRA
(v6.2#6252)
