[jira] [Commented] (YARN-6811) [ATS1.5] All history logs should be kept under its own User Directory.
[ https://issues.apache.org/jira/browse/YARN-6811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16174101#comment-16174101 ] Rohith Sharma K S commented on YARN-6811: - bq. Can we simply not change the active-dir configuration to have a $user.name path? This configuration is referred by TimelineSever to retrieve unfinished entities via REST service. So this can't be changed directly in timelineclient. This active base path should be same in timeline server and timeline client as well. bq. why doesn't the same logic apply for the done-directory? DONE directory is accessed only by TimelineServer. So, it is not required to create user directory under done directory path! > [ATS1.5] All history logs should be kept under its own User Directory. > --- > > Key: YARN-6811 > URL: https://issues.apache.org/jira/browse/YARN-6811 > Project: Hadoop YARN > Issue Type: Improvement > Components: timelineclient, timelineserver >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Fix For: 2.9.0, 3.0.0-beta1 > > Attachments: YARN-6811.01.patch, YARN-6811.02.patch, > YARN-6811-branch-2.01.patch > > > ATS1.5 allows to store history data in underlying FileSystem folder path i.e > */acitve-dir* and */done-dir*. These base directories are protected for > unauthorized user access for other users data by setting sticky bit for > /active-dir. > But object store filesystems such as WASB does not have user access control > on folders and files. When WASB are used as underlying file system for > ATS1.5, the history data which are stored in FS are accessible to all users. > *This would be a security risk* > I would propose to keep history data under its own user directory i.e > */active-dir/$USER*. Even this do not solve basic user access from FS, but it > provides capability to plugin Apache Ranger policies for each user folders. > One thing to note that setting policies to each user folder is admin > responsibility. But grouping all history data of one user folder allows to > set policies so that user access control is achieved. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6811) [ATS1.5] All history logs should be kept under its own User Directory.
[ https://issues.apache.org/jira/browse/YARN-6811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16173995#comment-16173995 ] Vinod Kumar Vavilapalli commented on YARN-6811: --- I don't understand why this patch is needed. Can we simply not change the active-dir configuration to have a $user.name path? Also, why doesn't the same logic apply for the done-directory? I don't see corresponding changes. > [ATS1.5] All history logs should be kept under its own User Directory. > --- > > Key: YARN-6811 > URL: https://issues.apache.org/jira/browse/YARN-6811 > Project: Hadoop YARN > Issue Type: Improvement > Components: timelineclient, timelineserver >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Fix For: 2.9.0, 3.0.0-beta1 > > Attachments: YARN-6811.01.patch, YARN-6811.02.patch, > YARN-6811-branch-2.01.patch > > > ATS1.5 allows to store history data in underlying FileSystem folder path i.e > */acitve-dir* and */done-dir*. These base directories are protected for > unauthorized user access for other users data by setting sticky bit for > /active-dir. > But object store filesystems such as WASB does not have user access control > on folders and files. When WASB are used as underlying file system for > ATS1.5, the history data which are stored in FS are accessible to all users. > *This would be a security risk* > I would propose to keep history data under its own user directory i.e > */active-dir/$USER*. Even this do not solve basic user access from FS, but it > provides capability to plugin Apache Ranger policies for each user folders. > One thing to note that setting policies to each user folder is admin > responsibility. But grouping all history data of one user folder allows to > set policies so that user access control is achieved. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6811) [ATS1.5] All history logs should be kept under its own User Directory.
[ https://issues.apache.org/jira/browse/YARN-6811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118930#comment-16118930 ] Junping Du commented on YARN-6811: -- Thanks [~rohithsharma]. I have commit the patch to trunk and branch-2. > [ATS1.5] All history logs should be kept under its own User Directory. > --- > > Key: YARN-6811 > URL: https://issues.apache.org/jira/browse/YARN-6811 > Project: Hadoop YARN > Issue Type: Improvement > Components: timelineclient, timelineserver >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: YARN-6811.01.patch, YARN-6811.02.patch, > YARN-6811-branch-2.01.patch > > > ATS1.5 allows to store history data in underlying FileSystem folder path i.e > */acitve-dir* and */done-dir*. These base directories are protected for > unauthorized user access for other users data by setting sticky bit for > /active-dir. > But object store filesystems such as WASB does not have user access control > on folders and files. When WASB are used as underlying file system for > ATS1.5, the history data which are stored in FS are accessible to all users. > *This would be a security risk* > I would propose to keep history data under its own user directory i.e > */active-dir/$USER*. Even this do not solve basic user access from FS, but it > provides capability to plugin Apache Ranger policies for each user folders. > One thing to note that setting policies to each user folder is admin > responsibility. But grouping all history data of one user folder allows to > set policies so that user access control is achieved. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6811) [ATS1.5] All history logs should be kept under its own User Directory.
[ https://issues.apache.org/jira/browse/YARN-6811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117790#comment-16117790 ] Rohith Sharma K S commented on YARN-6811: - cc :/ [~djp] Updated the patch for branch-2 > [ATS1.5] All history logs should be kept under its own User Directory. > --- > > Key: YARN-6811 > URL: https://issues.apache.org/jira/browse/YARN-6811 > Project: Hadoop YARN > Issue Type: Improvement > Components: timelineclient, timelineserver >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: YARN-6811.01.patch, YARN-6811.02.patch, > YARN-6811-branch-2.01.patch > > > ATS1.5 allows to store history data in underlying FileSystem folder path i.e > */acitve-dir* and */done-dir*. These base directories are protected for > unauthorized user access for other users data by setting sticky bit for > /active-dir. > But object store filesystems such as WASB does not have user access control > on folders and files. When WASB are used as underlying file system for > ATS1.5, the history data which are stored in FS are accessible to all users. > *This would be a security risk* > I would propose to keep history data under its own user directory i.e > */active-dir/$USER*. Even this do not solve basic user access from FS, but it > provides capability to plugin Apache Ranger policies for each user folders. > One thing to note that setting policies to each user folder is admin > responsibility. But grouping all history data of one user folder allows to > set policies so that user access control is achieved. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6811) [ATS1.5] All history logs should be kept under its own User Directory.
[
https://issues.apache.org/jira/browse/YARN-6811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16116392#comment-16116392
]
Hadoop QA commented on YARN-6811:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 12m
27s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
47s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
50s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
52s{color} | {color:green} branch-2 passed with JDK v1.8.0_144 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
14s{color} | {color:green} branch-2 passed with JDK v1.7.0_131 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
21s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
53s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} branch-2 passed with JDK v1.8.0_144 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
7s{color} | {color:green} branch-2 passed with JDK v1.7.0_131 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
49s{color} | {color:green} the patch passed with JDK v1.8.0_144 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
15s{color} | {color:green} the patch passed with JDK v1.7.0_131 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
15s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 41s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 5 new + 231 unchanged - 10 fixed = 236 total (was 241) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} the patch passed with JDK v1.8.0_144 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} the patch passed with JDK v1.7.0_131 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
26s{color} | {color:green} hadoop-yarn-api in the patch passed with JDK
v1.7.0_131. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
23s{color} | {color:green} hadoop-yarn-common in the patch passed with JDK
v1.7.0_131. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
51s{color} | {color:green} hadoop-yarn-server-timeline-pluginstorage in the
patch passed with JDK v1.7.0_131. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License
[jira] [Commented] (YARN-6811) [ATS1.5] All history logs should be kept under its own User Directory.
[ https://issues.apache.org/jira/browse/YARN-6811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16115124#comment-16115124 ] Junping Du commented on YARN-6811: -- I have commit the patch to trunk. For branch-2, my cherry-pick has several conflicts and the build still get failed even after I fix these conflicts. [~rohithsharma], can you upload a patch for branch-2? > [ATS1.5] All history logs should be kept under its own User Directory. > --- > > Key: YARN-6811 > URL: https://issues.apache.org/jira/browse/YARN-6811 > Project: Hadoop YARN > Issue Type: Improvement > Components: timelineclient, timelineserver >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: YARN-6811.01.patch, YARN-6811.02.patch > > > ATS1.5 allows to store history data in underlying FileSystem folder path i.e > */acitve-dir* and */done-dir*. These base directories are protected for > unauthorized user access for other users data by setting sticky bit for > /active-dir. > But object store filesystems such as WASB does not have user access control > on folders and files. When WASB are used as underlying file system for > ATS1.5, the history data which are stored in FS are accessible to all users. > *This would be a security risk* > I would propose to keep history data under its own user directory i.e > */active-dir/$USER*. Even this do not solve basic user access from FS, but it > provides capability to plugin Apache Ranger policies for each user folders. > One thing to note that setting policies to each user folder is admin > responsibility. But grouping all history data of one user folder allows to > set policies so that user access control is achieved. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6811) [ATS1.5] All history logs should be kept under its own User Directory.
[ https://issues.apache.org/jira/browse/YARN-6811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16115099#comment-16115099 ] Hudson commented on YARN-6811: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12122 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/12122/]) YARN-6811. [ATS1.5] All history logs should be kept under its own User (junping_du: rev f44b349b813508f0f6d99ca10bddba683dedf6c4) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/src/main/java/org/apache/hadoop/yarn/server/timeline/EntityGroupFSTimelineStore.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/impl/FileSystemTimelineWriter.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestTimelineClientForATS1_5.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/src/test/java/org/apache/hadoop/yarn/server/timeline/TestEntityGroupFSTimelineStore.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml > [ATS1.5] All history logs should be kept under its own User Directory. > --- > > Key: YARN-6811 > URL: https://issues.apache.org/jira/browse/YARN-6811 > Project: Hadoop YARN > Issue Type: Improvement > Components: timelineclient, timelineserver >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: YARN-6811.01.patch, YARN-6811.02.patch > > > ATS1.5 allows to store history data in underlying FileSystem folder path i.e > */acitve-dir* and */done-dir*. These base directories are protected for > unauthorized user access for other users data by setting sticky bit for > /active-dir. > But object store filesystems such as WASB does not have user access control > on folders and files. When WASB are used as underlying file system for > ATS1.5, the history data which are stored in FS are accessible to all users. > *This would be a security risk* > I would propose to keep history data under its own user directory i.e > */active-dir/$USER*. Even this do not solve basic user access from FS, but it > provides capability to plugin Apache Ranger policies for each user folders. > One thing to note that setting policies to each user folder is admin > responsibility. But grouping all history data of one user folder allows to > set policies so that user access control is achieved. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6811) [ATS1.5] All history logs should be kept under its own User Directory.
[ https://issues.apache.org/jira/browse/YARN-6811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113524#comment-16113524 ] Junping Du commented on YARN-6811: -- Thanks for updating the patch and addressing my comments, [~rohithsharma]. Latest patch LGTM. +1. Will commit it tomorrow if no other comments. > [ATS1.5] All history logs should be kept under its own User Directory. > --- > > Key: YARN-6811 > URL: https://issues.apache.org/jira/browse/YARN-6811 > Project: Hadoop YARN > Issue Type: Improvement > Components: timelineclient, timelineserver >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: YARN-6811.01.patch, YARN-6811.02.patch > > > ATS1.5 allows to store history data in underlying FileSystem folder path i.e > */acitve-dir* and */done-dir*. These base directories are protected for > unauthorized user access for other users data by setting sticky bit for > /active-dir. > But object store filesystems such as WASB does not have user access control > on folders and files. When WASB are used as underlying file system for > ATS1.5, the history data which are stored in FS are accessible to all users. > *This would be a security risk* > I would propose to keep history data under its own user directory i.e > */active-dir/$USER*. Even this do not solve basic user access from FS, but it > provides capability to plugin Apache Ranger policies for each user folders. > One thing to note that setting policies to each user folder is admin > responsibility. But grouping all history data of one user folder allows to > set policies so that user access control is achieved. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6811) [ATS1.5] All history logs should be kept under its own User Directory.
[
https://issues.apache.org/jira/browse/YARN-6811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113181#comment-16113181
]
Hadoop QA commented on YARN-6811:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
32s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m
38s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 0s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 6 new + 231 unchanged - 11 fixed = 237 total (was 242) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
33s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
30s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
17s{color} | {color:green} hadoop-yarn-server-timeline-pluginstorage in the
patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 60m 22s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-6811 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12880260/YARN-6811.02.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux 8e14ba642a8e 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / c5d256c |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
[jira] [Commented] (YARN-6811) [ATS1.5] All history logs should be kept under its own User Directory.
[ https://issues.apache.org/jira/browse/YARN-6811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16111732#comment-16111732 ] Junping Du commented on YARN-6811: -- Thanks [~rohithsharma] for more clarification. That sounds like a reasonable design to me and address all my concern for rolling upgrade. Look forward to new patch to address my other comments. > [ATS1.5] All history logs should be kept under its own User Directory. > --- > > Key: YARN-6811 > URL: https://issues.apache.org/jira/browse/YARN-6811 > Project: Hadoop YARN > Issue Type: Improvement > Components: timelineclient, timelineserver >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: YARN-6811.01.patch > > > ATS1.5 allows to store history data in underlying FileSystem folder path i.e > */acitve-dir* and */done-dir*. These base directories are protected for > unauthorized user access for other users data by setting sticky bit for > /active-dir. > But object store filesystems such as WASB does not have user access control > on folders and files. When WASB are used as underlying file system for > ATS1.5, the history data which are stored in FS are accessible to all users. > *This would be a security risk* > I would propose to keep history data under its own user directory i.e > */active-dir/$USER*. Even this do not solve basic user access from FS, but it > provides capability to plugin Apache Ranger policies for each user folders. > One thing to note that setting policies to each user folder is admin > responsibility. But grouping all history data of one user folder allows to > set policies so that user access control is achieved. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6811) [ATS1.5] All history logs should be kept under its own User Directory.
[ https://issues.apache.org/jira/browse/YARN-6811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16110452#comment-16110452 ] Rohith Sharma K S commented on YARN-6811: - Thanks [~djp] for the reviews. Couple of items to clear. # *TimelineClient*: The configuration exposed is only used at client side configuration. TimelineClient can decide whether to write inside user-dir or not. It does not impact running timeline clients during rolling upgrade. Running timelineclient still write into old path after upgrade. # *TimelineServer* : After this patch, TimelineServer supports reading data from active-dir and user-active-dir which is subfolder of active-dir. So, newly added configuration is not needed at server side. ## Scanning active directory and move completed application into done directory. It happens for every regular interval. Scanning happens recursively under active path which takes care of moving data from user directory as well. ## If read request came while application is still running which need to be cached ##* The existing behavior i.e reading active application data from active-path still continue to support. No change for this code path. _It do not impact rolling upgrade_ ##* If server do not find application data under active-path directory then try to find under user directory which are sub folder of active path. This would affect little bit performance since active application path need to searched under active path sub folders. Overall # Rolling upgrade do not affect with this patch. Downgrading the TimelineServer would be concern that server will not serve from user-dir. IIUC, downgrading would always result in loosing some of the existing support. I think we need not much worry about it. # Performance: Given timeline clients are written inside user-dir then impact is there on performance if read request come for active application. Once application is finished, then there is NO impact on performance. bq. I think one improve could we don't search user directory when "keep-under-user-dir" set to false. It is client side configuration as I explained above bq. The name of new added configuration is too long, can it simply be "with-user-dir"? agree bq. We should document the new configuration in yarn-default.xml agree bq. Like my comments offline, createUserDir(String user) should have a better name given it doesn't already create user dir I missed to update it. I will do in next path. bq. I think we can add a unit test here as we can write app log Yep, to read from active-path and sub folder of it as well. > [ATS1.5] All history logs should be kept under its own User Directory. > --- > > Key: YARN-6811 > URL: https://issues.apache.org/jira/browse/YARN-6811 > Project: Hadoop YARN > Issue Type: Improvement > Components: timelineclient, timelineserver >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: YARN-6811.01.patch > > > ATS1.5 allows to store history data in underlying FileSystem folder path i.e > */acitve-dir* and */done-dir*. These base directories are protected for > unauthorized user access for other users data by setting sticky bit for > /active-dir. > But object store filesystems such as WASB does not have user access control > on folders and files. When WASB are used as underlying file system for > ATS1.5, the history data which are stored in FS are accessible to all users. > *This would be a security risk* > I would propose to keep history data under its own user directory i.e > */active-dir/$USER*. Even this do not solve basic user access from FS, but it > provides capability to plugin Apache Ranger policies for each user folders. > One thing to note that setting policies to each user folder is admin > responsibility. But grouping all history data of one user folder allows to > set policies so that user access control is achieved. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6811) [ATS1.5] All history logs should be kept under its own User Directory.
[
https://issues.apache.org/jira/browse/YARN-6811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16109903#comment-16109903
]
Junping Du commented on YARN-6811:
--
Thanks [~rohithsharma] for contributing the patch! The approach here looks
general good to me. The only concern here is it could be a bit performance
impact as it will search two directories (with User and without user). I think
one improve could we don't search user directory when "keep-under-user-dir" set
to false. The verse is not true because we need to handle rolling upgrade case.
Some detail comments:
{noformat}
public static final String
+ TIMELINE_SERVICE_ENTITYGROUP_FS_STORE_KEEP_UNDER_USER_DIR =
+ TIMELINE_SERVICE_ENTITYGROUP_FS_STORE_PREFIX + "keep-under-user-dir"
{noformat}
The name of new added configuration is too long, can it simply be
"with-user-dir"?
We should document the new configuration in yarn-default.xml with proper
explanation of how this configuration is used for.
Like my comments offline, {{createUserDir(String user)}} should have a better
name given it doesn't already create user dir (depends on configuration). May
be better to call it {{getAppRootDir()}}?
We need to handle rolling upgrade case. I think we can add a unit test here as
we can write app log with "keep-under-user-dir" = false for writing to old
location, and try to read it out when set "keep-under-user-dir" = true.
> [ATS1.5] All history logs should be kept under its own User Directory.
> ---
>
> Key: YARN-6811
> URL: https://issues.apache.org/jira/browse/YARN-6811
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: timelineclient, timelineserver
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
> Attachments: YARN-6811.01.patch
>
>
> ATS1.5 allows to store history data in underlying FileSystem folder path i.e
> */acitve-dir* and */done-dir*. These base directories are protected for
> unauthorized user access for other users data by setting sticky bit for
> /active-dir.
> But object store filesystems such as WASB does not have user access control
> on folders and files. When WASB are used as underlying file system for
> ATS1.5, the history data which are stored in FS are accessible to all users.
> *This would be a security risk*
> I would propose to keep history data under its own user directory i.e
> */active-dir/$USER*. Even this do not solve basic user access from FS, but it
> provides capability to plugin Apache Ranger policies for each user folders.
> One thing to note that setting policies to each user folder is admin
> responsibility. But grouping all history data of one user folder allows to
> set policies so that user access control is achieved.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6811) [ATS1.5] All history logs should be kept under its own User Directory.
[
https://issues.apache.org/jira/browse/YARN-6811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16107277#comment-16107277
]
Hadoop QA commented on YARN-6811:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
25s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m
14s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 55s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 7 new + 229 unchanged - 11 fixed = 236 total (was 240) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
31s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
27s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
6s{color} | {color:green} hadoop-yarn-server-timeline-pluginstorage in the
patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 55m 7s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-6811 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12879617/YARN-6811.01.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 970ca0826447 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 0fd6d0f |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/16619/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/16619/testReport/ |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
