[
https://issues.apache.org/jira/browse/TEZ-1106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14010788#comment-14010788
]
Mohammad Kamrul Islam commented on TEZ-1106:
--------------------------------------------
Thanks [~hitesh] for the comments.
>In getTezBaseStagingPath(), is there a reason that the resolved path is not
>being updated back into the config?
I thought we should not modify the conf. If you think, we can do it otherwise.
>In getTezSystemStagingPath(), why is getTezBaseStagingPath() being called? Why
>are 2 public functions needed instead of just exposing a single one?
They are two different items. First one is for original staging directory
provided by user. The second one is for Tez system internal dir which append
_tez and application id at the end of the path.
>Also, for getTezSystemStagingPath(), is there a reason why it always has to do
>a create? To clarify, are there any cases that you believe the directory
>should always exist and not be silently created?
It creates if it doesn't exist. Yes, the Tez system directory needs to be
created at least in the first call. User can't or shouldn't create this.
>Thanks for patiently addressing the review comments
I also appreciate your, Bikas and Sid's effort to make the system standard
higher.
After getting your reply, i will upload a new patch.
> Tez framework should use a unique subdir when creating new files in staging
> -----------------------------------------------------------------------------
>
> Key: TEZ-1106
> URL: https://issues.apache.org/jira/browse/TEZ-1106
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Mohammad Kamrul Islam
> Assignee: Mohammad Kamrul Islam
> Attachments: TEZ-1106.1.patch, TEZ-1106.2.patch, TEZ-1106.3.patch,
> TEZ-1106.4.patch
>
>
> Currently the files are created in different sub-directories. It is hard to
> manage and cleanup at the end.
> The proposal is to create a new subdir : $STAGE_DIR/<APP_ID>/
> All recovery files will go under : $STAGE_DIR/<APP_ID>/recovery/<attemp_num>/
> All confs will go under: $STAGE_DIR/<APP_ID>/conf/
> All dagplans will go: $STAGE_DIR/<APP_ID>/dag_id/plan/
--
This message was sent by Atlassian JIRA
(v6.2#6252)
