[
https://issues.apache.org/jira/browse/YARN-6811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16110452#comment-16110452
]
Rohith Sharma K S commented on YARN-6811:
-----------------------------------------
Thanks [~djp] for the reviews. Couple of items to clear.
# *TimelineClient*: The configuration exposed is only used at client side
configuration. TimelineClient can decide whether to write inside user-dir or
not. It does not impact running timeline clients during rolling upgrade.
Running timelineclient still write into old path after upgrade.
# *TimelineServer* : After this patch, TimelineServer supports reading data
from active-dir and user-active-dir which is subfolder of active-dir. So, newly
added configuration is not needed at server side.
## Scanning active directory and move completed application into done
directory. It happens for every regular interval. Scanning happens recursively
under active path which takes care of moving data from user directory as well.
## If read request came while application is still running which need to be
cached
##* The existing behavior i.e reading active application data from active-path
still continue to support. No change for this code path. _It do not impact
rolling upgrade_
##* If server do not find application data under active-path directory then try
to find under user directory which are sub folder of active path. This would
affect little bit performance since active application path need to searched
under active path sub folders.
Overall
# Rolling upgrade do not affect with this patch. Downgrading the TimelineServer
would be concern that server will not serve from user-dir. IIUC, downgrading
would always result in loosing some of the existing support. I think we need
not much worry about it.
# Performance: Given timeline clients are written inside user-dir then impact
is there on performance if read request come for active application. Once
application is finished, then there is NO impact on performance.
bq. I think one improve could we don't search user directory when
"keep-under-user-dir" set to false.
It is client side configuration as I explained above
bq. The name of new added configuration is too long, can it simply be
"with-user-dir"?
agree
bq. We should document the new configuration in yarn-default.xml
agree
bq. Like my comments offline, createUserDir(String user) should have a better
name given it doesn't already create user dir
I missed to update it. I will do in next path.
bq. I think we can add a unit test here as we can write app log
Yep, to read from active-path and sub folder of it as well.
> [ATS1.5] All history logs should be kept under its own User Directory.
> -----------------------------------------------------------------------
>
> Key: YARN-6811
> URL: https://issues.apache.org/jira/browse/YARN-6811
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: timelineclient, timelineserver
> Reporter: Rohith Sharma K S
> Assignee: Rohith Sharma K S
> Attachments: YARN-6811.01.patch
>
>
> ATS1.5 allows to store history data in underlying FileSystem folder path i.e
> */acitve-dir* and */done-dir*. These base directories are protected for
> unauthorized user access for other users data by setting sticky bit for
> /active-dir.
> But object store filesystems such as WASB does not have user access control
> on folders and files. When WASB are used as underlying file system for
> ATS1.5, the history data which are stored in FS are accessible to all users.
> *This would be a security risk*
> I would propose to keep history data under its own user directory i.e
> */active-dir/$USER*. Even this do not solve basic user access from FS, but it
> provides capability to plugin Apache Ranger policies for each user folders.
> One thing to note that setting policies to each user folder is admin
> responsibility. But grouping all history data of one user folder allows to
> set policies so that user access control is achieved.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
