[
https://issues.apache.org/jira/browse/TEZ-1106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14000705#comment-14000705
]
Mohammad Kamrul Islam commented on TEZ-1106:
--------------------------------------------
Very good comments.
Few clarifications would be helpful:
> set permissions on staging dir related code in TezClientUtils.java could be
> moved into the same class as other helper functions
They are mainly used in ensureStagingDirExists() code. Do you propose to move
it into TezCommonUtils? Btw this method is called by almost all examples (12
references).
>can any of these values ever be null?
I don't see that possibility.
>TezCommonUtils:: getTezBaseStagingPath() vs getTezSystemStagingPath()
The first one returns the stage dir defined by user. The second one is for TEZ
internals. If you prefer, we can rename the second one something else.
>What is the expectation for a user when configuring the staging dir?
Currently looks like there is no expectation. User can (not must) define it in
the conf or TEZ default is applied. Last time, i gave you wrong related JIRA,
it should TEZ-792 where the default or not default is discussed.
>When the staging dir is deleted by the AM, what is deleted? the .tez subdir or
>the base dir?
I'm not sure which one will be better. Do you have any preference? I believe in
Hadoop 1.x, the framework deletes the job specific stuffs during cleanup not
the root stage dir. (Related JIRA: TEZ-693).
> Tez framework should use a unique subdir when creating new files in staging
> -----------------------------------------------------------------------------
>
> Key: TEZ-1106
> URL: https://issues.apache.org/jira/browse/TEZ-1106
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Mohammad Kamrul Islam
> Assignee: Mohammad Kamrul Islam
> Attachments: TEZ-1106.1.patch, TEZ-1106.2.patch, TEZ-1106.3.patch
>
>
> Currently the files are created in different sub-directories. It is hard to
> manage and cleanup at the end.
> The proposal is to create a new subdir : $STAGE_DIR/<APP_ID>/
> All recovery files will go under : $STAGE_DIR/<APP_ID>/recovery/<attemp_num>/
> All confs will go under: $STAGE_DIR/<APP_ID>/conf/
> All dagplans will go: $STAGE_DIR/<APP_ID>/dag_id/plan/
--
This message was sent by Atlassian JIRA
(v6.2#6252)
