[
https://issues.apache.org/jira/browse/TEZ-1106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14011890#comment-14011890
]
Hitesh Shah commented on TEZ-1106:
----------------------------------
bq. I think we can ensure it either by adding a new property into Conf or by
updating the existing Conf property for staging dir that you originally hinted.
This can probably be skipped for now. If you believe there is a case to do it,
I would prefer that we update the existing value in the conf to the fully
resolved path. One thing to note though is that the value in the config is the
base path and code should ensure that it sets only the base path back into the
config.
bq. Yes. we should create it only once and first time. How can we make sure if
the creation is the first time?
I believe a slight modification of Option 2 should work. Support both
create_dir function and a get_dir functions. createDir() should be called once
and only once in the client. It will create the dir if it doesn't exist and do
the necessary permission checks ( logging messages when either the dir is not
present or perms need to be reset). createDir() could also return the fully
resolved path back to the user. getDir() should be called after this point in
the AM and the path obtained re-used elsewhere as needed.
> Tez framework should use a unique subdir when creating new files in staging
> -----------------------------------------------------------------------------
>
> Key: TEZ-1106
> URL: https://issues.apache.org/jira/browse/TEZ-1106
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Mohammad Kamrul Islam
> Assignee: Mohammad Kamrul Islam
> Attachments: TEZ-1106.1.patch, TEZ-1106.2.patch, TEZ-1106.3.patch,
> TEZ-1106.4.patch
>
>
> Currently the files are created in different sub-directories. It is hard to
> manage and cleanup at the end.
> The proposal is to create a new subdir : $STAGE_DIR/<APP_ID>/
> All recovery files will go under : $STAGE_DIR/<APP_ID>/recovery/<attemp_num>/
> All confs will go under: $STAGE_DIR/<APP_ID>/conf/
> All dagplans will go: $STAGE_DIR/<APP_ID>/dag_id/plan/
--
This message was sent by Atlassian JIRA
(v6.2#6252)
