subject:"\[jira\] \[Commented\] \(YARN\-2424\) LCE should support non\-cgroups, non\-secure mode"


[ 
https://issues.apache.org/jira/browse/YARN-2424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1410#comment-1410
 ] 

Hudson commented on YARN-2424:
--

FAILURE: Integrated in Hadoop-Yarn-trunk #653 (See 
[https://builds.apache.org/job/Hadoop-Yarn-trunk/653/])
YARN-2424. LCE should support non-cgroups, non-secure mode (Chris Douglas via 
aw) (aw: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1619421)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutor.java


 LCE should support non-cgroups, non-secure mode
 ---

 Key: YARN-2424
 URL: https://issues.apache.org/jira/browse/YARN-2424
 Project: Hadoop YARN
  Issue Type: Bug
  Components: nodemanager
Affects Versions: 2.3.0, 2.4.0, 2.5.0, 2.4.1
Reporter: Allen Wittenauer
Assignee: Chris Douglas
Priority: Blocker
 Attachments: Y2424-1.patch, YARN-2424.patch


 After YARN-1253, LCE no longer works for non-secure, non-cgroup scenarios.  
 This is a fairly serious regression, as turning on LCE prior to turning on 
 full-blown security is a fairly standard procedure.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (YARN-2424) LCE should support non-cgroups, non-secure mode


[ 
https://issues.apache.org/jira/browse/YARN-2424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105777#comment-14105777
 ] 

Hudson commented on YARN-2424:
--

FAILURE: Integrated in Hadoop-Hdfs-trunk #1844 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1844/])
YARN-2424. LCE should support non-cgroups, non-secure mode (Chris Douglas via 
aw) (aw: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1619421)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutor.java


 LCE should support non-cgroups, non-secure mode
 ---

 Key: YARN-2424
 URL: https://issues.apache.org/jira/browse/YARN-2424
 Project: Hadoop YARN
  Issue Type: Bug
  Components: nodemanager
Affects Versions: 2.3.0, 2.4.0, 2.5.0, 2.4.1
Reporter: Allen Wittenauer
Assignee: Allen Wittenauer
Priority: Blocker
 Attachments: Y2424-1.patch, YARN-2424.patch


 After YARN-1253, LCE no longer works for non-secure, non-cgroup scenarios.  
 This is a fairly serious regression, as turning on LCE prior to turning on 
 full-blown security is a fairly standard procedure.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (YARN-2424) LCE should support non-cgroups, non-secure mode


[ 
https://issues.apache.org/jira/browse/YARN-2424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105774#comment-14105774
 ] 

Hudson commented on YARN-2424:
--

FAILURE: Integrated in Hadoop-trunk-Commit #6092 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/6092/])
YARN-2424. LCE should support non-cgroups, non-secure mode (Chris Douglas via 
aw) (aw: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1619421)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutor.java


 LCE should support non-cgroups, non-secure mode
 ---

 Key: YARN-2424
 URL: https://issues.apache.org/jira/browse/YARN-2424
 Project: Hadoop YARN
  Issue Type: Bug
  Components: nodemanager
Affects Versions: 2.3.0, 2.4.0, 2.5.0, 2.4.1
Reporter: Allen Wittenauer
Assignee: Allen Wittenauer
Priority: Blocker
 Attachments: Y2424-1.patch, YARN-2424.patch


 After YARN-1253, LCE no longer works for non-secure, non-cgroup scenarios.  
 This is a fairly serious regression, as turning on LCE prior to turning on 
 full-blown security is a fairly standard procedure.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (YARN-2424) LCE should support non-cgroups, non-secure mode


[ 
https://issues.apache.org/jira/browse/YARN-2424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105845#comment-14105845
 ] 

Hudson commented on YARN-2424:
--

SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1870 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1870/])
YARN-2424. LCE should support non-cgroups, non-secure mode (Chris Douglas via 
aw) (aw: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1619421)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutor.java


 LCE should support non-cgroups, non-secure mode
 ---

 Key: YARN-2424
 URL: https://issues.apache.org/jira/browse/YARN-2424
 Project: Hadoop YARN
  Issue Type: Bug
  Components: nodemanager
Affects Versions: 2.3.0, 2.4.0, 2.5.0, 2.4.1
Reporter: Allen Wittenauer
Assignee: Allen Wittenauer
Priority: Blocker
 Attachments: Y2424-1.patch, YARN-2424.patch


 After YARN-1253, LCE no longer works for non-secure, non-cgroup scenarios.  
 This is a fairly serious regression, as turning on LCE prior to turning on 
 full-blown security is a fairly standard procedure.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (YARN-2424) LCE should support non-cgroups, non-secure mode

2014-08-20 Thread Owen O'Malley (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-2424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14104155#comment-14104155
 ] 

Owen O'Malley commented on YARN-2424:
-

This is a pretty clear case of trying to fix the breakage from YARN-1253. Yahoo 
ran clusters for a year with LCE before security was turned on and got 
significant value from that. The largest being that it prevents killall -9 
java type mistakes on the part of users. (Yes that did actually happen.)

 LCE should support non-cgroups, non-secure mode
 ---

 Key: YARN-2424
 URL: https://issues.apache.org/jira/browse/YARN-2424
 Project: Hadoop YARN
  Issue Type: Bug
  Components: nodemanager
Affects Versions: 2.3.0, 2.4.0, 2.5.0, 2.4.1
Reporter: Allen Wittenauer
Priority: Blocker
 Attachments: YARN-2424.patch


 After YARN-1253, LCE no longer works for non-secure, non-cgroup scenarios.  
 This is a fairly serious regression, as turning on LCE prior to turning on 
 full-blown security is a fairly standard procedure.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (YARN-2424) LCE should support non-cgroups, non-secure mode


[ 
https://issues.apache.org/jira/browse/YARN-2424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14104179#comment-14104179
 ] 

Alejandro Abdelnur commented on YARN-2424:
--

I disagree on YARN-1253 being a breakage. 

Personally, I would never recommend using this in production. Given that, I'm 
OK with the patch if:

* the NM logs print a WARN at startup stating the setting.
* the container stdout/stderr also prints a WARN to alert the user of the 
setting.



 LCE should support non-cgroups, non-secure mode
 ---

 Key: YARN-2424
 URL: https://issues.apache.org/jira/browse/YARN-2424
 Project: Hadoop YARN
  Issue Type: Bug
  Components: nodemanager
Affects Versions: 2.3.0, 2.4.0, 2.5.0, 2.4.1
Reporter: Allen Wittenauer
Priority: Blocker
 Attachments: YARN-2424.patch


 After YARN-1253, LCE no longer works for non-secure, non-cgroup scenarios.  
 This is a fairly serious regression, as turning on LCE prior to turning on 
 full-blown security is a fairly standard procedure.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (YARN-2424) LCE should support non-cgroups, non-secure mode

2014-08-20 Thread Owen O'Malley (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-2424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14104196#comment-14104196
 ] 

Owen O'Malley commented on YARN-2424:
-

Alejandro, after I told you that users have run in production with that 
setting, it is very rude to say that removing the feature is not breakage. It 
is *obviously* breakage.

A warning makes sense, but it should only be once when the ResourceManager 
boots. It is a system level configuration and warning more than once is wrong.

 LCE should support non-cgroups, non-secure mode
 ---

 Key: YARN-2424
 URL: https://issues.apache.org/jira/browse/YARN-2424
 Project: Hadoop YARN
  Issue Type: Bug
  Components: nodemanager
Affects Versions: 2.3.0, 2.4.0, 2.5.0, 2.4.1
Reporter: Allen Wittenauer
Priority: Blocker
 Attachments: YARN-2424.patch


 After YARN-1253, LCE no longer works for non-secure, non-cgroup scenarios.  
 This is a fairly serious regression, as turning on LCE prior to turning on 
 full-blown security is a fairly standard procedure.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (YARN-2424) LCE should support non-cgroups, non-secure mode


[ 
https://issues.apache.org/jira/browse/YARN-2424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14104257#comment-14104257
 ] 

Alejandro Abdelnur commented on YARN-2424:
--

I disagree in me being rude (or very rude) just for disagreeing with something. 
IMO security fixes trump backwards compatibility.

Anyway, I'm -0 with the patch if the WARNs are printed in in the RM at startup 
as Owen suggests. I insists that the WARN should be in the stderr/stdout of 
every container. Otherwise this will go completely unnoticed to users running 
apps. It should be obvious to them that they are exposed.


 LCE should support non-cgroups, non-secure mode
 ---

 Key: YARN-2424
 URL: https://issues.apache.org/jira/browse/YARN-2424
 Project: Hadoop YARN
  Issue Type: Bug
  Components: nodemanager
Affects Versions: 2.3.0, 2.4.0, 2.5.0, 2.4.1
Reporter: Allen Wittenauer
Priority: Blocker
 Attachments: YARN-2424.patch


 After YARN-1253, LCE no longer works for non-secure, non-cgroup scenarios.  
 This is a fairly serious regression, as turning on LCE prior to turning on 
 full-blown security is a fairly standard procedure.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (YARN-2424) LCE should support non-cgroups, non-secure mode

2014-08-20 Thread Allen Wittenauer (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-2424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14104303#comment-14104303
 ] 

Allen Wittenauer commented on YARN-2424:


bq. It should be obvious to them that they are exposed.

Then we should return a WARN whenever isSecurityEnabled returns false since 
that's the only way they are secure.

 LCE should support non-cgroups, non-secure mode
 ---

 Key: YARN-2424
 URL: https://issues.apache.org/jira/browse/YARN-2424
 Project: Hadoop YARN
  Issue Type: Bug
  Components: nodemanager
Affects Versions: 2.3.0, 2.4.0, 2.5.0, 2.4.1
Reporter: Allen Wittenauer
Priority: Blocker
 Attachments: YARN-2424.patch


 After YARN-1253, LCE no longer works for non-secure, non-cgroup scenarios.  
 This is a fairly serious regression, as turning on LCE prior to turning on 
 full-blown security is a fairly standard procedure.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (YARN-2424) LCE should support non-cgroups, non-secure mode


[ 
https://issues.apache.org/jira/browse/YARN-2424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14104314#comment-14104314
 ] 

Alejandro Abdelnur commented on YARN-2424:
--

if you don't have to kinit it is obvious security is OFF, no?

 LCE should support non-cgroups, non-secure mode
 ---

 Key: YARN-2424
 URL: https://issues.apache.org/jira/browse/YARN-2424
 Project: Hadoop YARN
  Issue Type: Bug
  Components: nodemanager
Affects Versions: 2.3.0, 2.4.0, 2.5.0, 2.4.1
Reporter: Allen Wittenauer
Priority: Blocker
 Attachments: YARN-2424.patch


 After YARN-1253, LCE no longer works for non-secure, non-cgroup scenarios.  
 This is a fairly serious regression, as turning on LCE prior to turning on 
 full-blown security is a fairly standard procedure.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (YARN-2424) LCE should support non-cgroups, non-secure mode

2014-08-20 Thread Allen Wittenauer (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-2424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14104322#comment-14104322
 ] 

Allen Wittenauer commented on YARN-2424:


Apparently not, given:

bq. Otherwise this will go completely unnoticed to users running apps.



 LCE should support non-cgroups, non-secure mode
 ---

 Key: YARN-2424
 URL: https://issues.apache.org/jira/browse/YARN-2424
 Project: Hadoop YARN
  Issue Type: Bug
  Components: nodemanager
Affects Versions: 2.3.0, 2.4.0, 2.5.0, 2.4.1
Reporter: Allen Wittenauer
Priority: Blocker
 Attachments: YARN-2424.patch


 After YARN-1253, LCE no longer works for non-secure, non-cgroup scenarios.  
 This is a fairly serious regression, as turning on LCE prior to turning on 
 full-blown security is a fairly standard procedure.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (YARN-2424) LCE should support non-cgroups, non-secure mode

2014-08-20 Thread Ravi Prakash (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-2424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14104893#comment-14104893
 ] 

Ravi Prakash commented on YARN-2424:


I reviewed the code and the changes make sense to me. I'm a +1 on the patch as 
is.

 LCE should support non-cgroups, non-secure mode
 ---

 Key: YARN-2424
 URL: https://issues.apache.org/jira/browse/YARN-2424
 Project: Hadoop YARN
  Issue Type: Bug
  Components: nodemanager
Affects Versions: 2.3.0, 2.4.0, 2.5.0, 2.4.1
Reporter: Allen Wittenauer
Priority: Blocker
 Attachments: YARN-2424.patch


 After YARN-1253, LCE no longer works for non-secure, non-cgroup scenarios.  
 This is a fairly serious regression, as turning on LCE prior to turning on 
 full-blown security is a fairly standard procedure.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (YARN-2424) LCE should support non-cgroups, non-secure mode


[ 
https://issues.apache.org/jira/browse/YARN-2424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14104993#comment-14104993
 ] 

Alejandro Abdelnur commented on YARN-2424:
--

sure, fine, enough cycles spent on this, thx.

 LCE should support non-cgroups, non-secure mode
 ---

 Key: YARN-2424
 URL: https://issues.apache.org/jira/browse/YARN-2424
 Project: Hadoop YARN
  Issue Type: Bug
  Components: nodemanager
Affects Versions: 2.3.0, 2.4.0, 2.5.0, 2.4.1
Reporter: Allen Wittenauer
Priority: Blocker
 Attachments: Y2424-1.patch, YARN-2424.patch


 After YARN-1253, LCE no longer works for non-secure, non-cgroup scenarios.  
 This is a fairly serious regression, as turning on LCE prior to turning on 
 full-blown security is a fairly standard procedure.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (YARN-2424) LCE should support non-cgroups, non-secure mode


[ 
https://issues.apache.org/jira/browse/YARN-2424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14102357#comment-14102357
 ] 

Alejandro Abdelnur commented on YARN-2424:
--

[~raviprak], allow to sudo to more than one user in unsecure mode, it doesn't 
give you any extra security. Actually, it may give you a sense of false 
security.

On using groups in the LCE blacklist/whitelist, i'll comment in YARN-2429.

 LCE should support non-cgroups, non-secure mode
 ---

 Key: YARN-2424
 URL: https://issues.apache.org/jira/browse/YARN-2424
 Project: Hadoop YARN
  Issue Type: Bug
  Components: nodemanager
Affects Versions: 2.3.0, 2.4.0, 2.5.0, 2.4.1
Reporter: Allen Wittenauer
Priority: Blocker
 Attachments: YARN-2424.patch


 After YARN-1253, LCE no longer works for non-secure, non-cgroup scenarios.  
 This is a fairly serious regression, as turning on LCE prior to turning on 
 full-blown security is a fairly standard procedure.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (YARN-2424) LCE should support non-cgroups, non-secure mode

2014-08-19 Thread Allen Wittenauer (JIRA)

[
https://issues.apache.org/jira/browse/YARN-2424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14102416#comment-14102416
]

Allen Wittenauer commented on YARN-2424:

BTW, it should be pointed out that the current code doesn't actually protect
non-RPCSEC NFSv3/v2 directories. It only prevents them from being mounted
using system facilities. (I'll leave it up to the reader to see how to
implement an exploit not that it's particularly hard.)

The only security thing the current code does is limit containers to run as
one uid which in turn means preventing access to any elevated privs that any
other user might have. That's it. So if you have too many users with, say,
passwordless sudo or if you don't want to publish user names to your compute
nodes, the current code helps. Otherwise, you're getting zero benefits. For
example, YARN scheduling and HDFS writes are still being done by the originally
requested user.

The security aspects, as pointed out in the original JIRA, are a red herring.

LCE should support non-cgroups, non-secure mode
---

Key: YARN-2424
URL: https://issues.apache.org/jira/browse/YARN-2424
Project: Hadoop YARN
Issue Type: Bug
Components: nodemanager
Affects Versions: 2.3.0, 2.4.0, 2.5.0, 2.4.1
Reporter: Allen Wittenauer
Priority: Blocker
Attachments: YARN-2424.patch

After YARN-1253, LCE no longer works for non-secure, non-cgroup scenarios.
This is a fairly serious regression, as turning on LCE prior to turning on
full-blown security is a fairly standard procedure.

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (YARN-2424) LCE should support non-cgroups, non-secure mode


[ 
https://issues.apache.org/jira/browse/YARN-2424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14102450#comment-14102450
 ] 

Alejandro Abdelnur commented on YARN-2424:
--

If security is OFF, I can simply submit a job as ANY user by simply doing 
-Duser.name=ANY. User ANY will be the one used by YARN and HDFS (I'll leave it 
up to the reader to see how to do this).

I really don't like what this JIRA is proposing, and I've indicated what it 
would have to be done for me not to -1.



 LCE should support non-cgroups, non-secure mode
 ---

 Key: YARN-2424
 URL: https://issues.apache.org/jira/browse/YARN-2424
 Project: Hadoop YARN
  Issue Type: Bug
  Components: nodemanager
Affects Versions: 2.3.0, 2.4.0, 2.5.0, 2.4.1
Reporter: Allen Wittenauer
Priority: Blocker
 Attachments: YARN-2424.patch


 After YARN-1253, LCE no longer works for non-secure, non-cgroup scenarios.  
 This is a fairly serious regression, as turning on LCE prior to turning on 
 full-blown security is a fairly standard procedure.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (YARN-2424) LCE should support non-cgroups, non-secure mode


[ 
https://issues.apache.org/jira/browse/YARN-2424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14102538#comment-14102538
 ] 

Alejandro Abdelnur commented on YARN-2424:
--

[~aw], I think you are missing the point.

I know that in an un-secure cluster you can fake the user name to interact with 
HDFS or YARN from anywhere at anytime. 

YARN-1253 is not about protecting HDFS or YARN, it is about protecting the node 
at OS level by enforcing the use of a least privileged user.

 LCE should support non-cgroups, non-secure mode
 ---

 Key: YARN-2424
 URL: https://issues.apache.org/jira/browse/YARN-2424
 Project: Hadoop YARN
  Issue Type: Bug
  Components: nodemanager
Affects Versions: 2.3.0, 2.4.0, 2.5.0, 2.4.1
Reporter: Allen Wittenauer
Priority: Blocker
 Attachments: YARN-2424.patch


 After YARN-1253, LCE no longer works for non-secure, non-cgroup scenarios.  
 This is a fairly serious regression, as turning on LCE prior to turning on 
 full-blown security is a fairly standard procedure.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (YARN-2424) LCE should support non-cgroups, non-secure mode

[
https://issues.apache.org/jira/browse/YARN-2424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14102554#comment-14102554
]

Ravi Prakash commented on YARN-2424:

Hi Tucu!
bq. If security is OFF, I can simply submit a job as ANY user by simply doing
-Duser.name=ANY. User ANY will be the one used by YARN and HDFS
Is this true? Are you suggesting that the blacklist (banner.users) in
container-executor.cfg does not work? Could you not blacklist root, hdfs,
mapred and yarn?

We are not doing this for security. We understand that +*with the right
configuration*+, the level of security you provide is exactly the same as you
would have in an unsecure cluster. If only the users of the cluster are
whitelisted and all other users like root / mapred / yarn / hdfs are
blacklisted, and the users which are whitelisted don't enjoy any elevated
privilidges on the slave nodes. This is a perfectly valid configuration with
the same level of security as would be provided if all yarn tasks ran one user.

Could you please point out a technical concern with the security in this
configuration? This would not be a configuration for troubleshooting only.
This would be a perfectly valid configuration.

LCE should support non-cgroups, non-secure mode
---

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (YARN-2424) LCE should support non-cgroups, non-secure mode


[ 
https://issues.apache.org/jira/browse/YARN-2424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14102629#comment-14102629
 ] 

Alejandro Abdelnur commented on YARN-2424:
--

Ravi, all the config in the container-executor.cfg is EXCLUSIVELY for enforcing 
constraints on the process to be launched, it does not restrict a launched JVM 
process from doing a {{System.setProperty(user.name, ANY)}} to gain access 
to HDFS as user ANY (if Kerberos is ON, setting 'user.name' property has no 
effect).

BTW, I'm not OK with making this a valid configuration, it is not.

 LCE should support non-cgroups, non-secure mode
 ---

 Key: YARN-2424
 URL: https://issues.apache.org/jira/browse/YARN-2424
 Project: Hadoop YARN
  Issue Type: Bug
  Components: nodemanager
Affects Versions: 2.3.0, 2.4.0, 2.5.0, 2.4.1
Reporter: Allen Wittenauer
Priority: Blocker
 Attachments: YARN-2424.patch


 After YARN-1253, LCE no longer works for non-secure, non-cgroup scenarios.  
 This is a fairly serious regression, as turning on LCE prior to turning on 
 full-blown security is a fairly standard procedure.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (YARN-2424) LCE should support non-cgroups, non-secure mode


[ 
https://issues.apache.org/jira/browse/YARN-2424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14102684#comment-14102684
 ] 

Ravi Prakash commented on YARN-2424:


Thanks Tucu! In YARN-1253, Vinod wrote:
bq. Even if the jobs run as a single 'yarnuser', security isn't still there - 
like Arun said, any body can bomb HDFS directories of other users, any user can 
kill any other user's tasks/containers, any one can delete any one else's local 
dirs, log-dir and so on
Is that not true?


 LCE should support non-cgroups, non-secure mode
 ---

 Key: YARN-2424
 URL: https://issues.apache.org/jira/browse/YARN-2424
 Project: Hadoop YARN
  Issue Type: Bug
  Components: nodemanager
Affects Versions: 2.3.0, 2.4.0, 2.5.0, 2.4.1
Reporter: Allen Wittenauer
Priority: Blocker
 Attachments: YARN-2424.patch


 After YARN-1253, LCE no longer works for non-secure, non-cgroup scenarios.  
 This is a fairly serious regression, as turning on LCE prior to turning on 
 full-blown security is a fairly standard procedure.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (YARN-2424) LCE should support non-cgroups, non-secure mode


[ 
https://issues.apache.org/jira/browse/YARN-2424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14102794#comment-14102794
 ] 

Alejandro Abdelnur commented on YARN-2424:
--

Repeating myself from a previous comment: YARN-1253 is not about protecting 
HDFS or YARN, it is about protecting the node at OS level by enforcing the use 
of a least privileged user.


 LCE should support non-cgroups, non-secure mode
 ---

 Key: YARN-2424
 URL: https://issues.apache.org/jira/browse/YARN-2424
 Project: Hadoop YARN
  Issue Type: Bug
  Components: nodemanager
Affects Versions: 2.3.0, 2.4.0, 2.5.0, 2.4.1
Reporter: Allen Wittenauer
Priority: Blocker
 Attachments: YARN-2424.patch


 After YARN-1253, LCE no longer works for non-secure, non-cgroup scenarios.  
 This is a fairly serious regression, as turning on LCE prior to turning on 
 full-blown security is a fairly standard procedure.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (YARN-2424) LCE should support non-cgroups, non-secure mode

[
https://issues.apache.org/jira/browse/YARN-2424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14102873#comment-14102873
]

Ravi Prakash commented on YARN-2424:

Hi Tucu! I'd brought it up only because in the earlier comment you'd said
bq. Ravi, all the config in the container-executor.cfg is EXCLUSIVELY for
enforcing constraints on the process to be launched, it does not restrict a
launched JVM process from doing a System.setProperty(user.name, ANY) to
gain access to +*HDFS*+ as user ANY (if Kerberos is ON, setting 'user.name'
property has no effect).
I'm glad we agree that YARN-1253 wasn't about protecting HDFS or YARN.

bq. it is about protecting the node at OS level by enforcing the use of a least
privileged user.
So if we enforced the use of several least privileged users (instead of only
1), is that not just as secure? Would you argue that with the proper use of
blacklists and whitelists this cannot be achieved?

LCE should support non-cgroups, non-secure mode
---

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (YARN-2424) LCE should support non-cgroups, non-secure mode


[ 
https://issues.apache.org/jira/browse/YARN-2424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14102941#comment-14102941
 ] 

Alejandro Abdelnur commented on YARN-2424:
--

Having more than one 'least privileged' user does not bring you any benefit as 
they can always step on each other by faking the username at job submission.


 LCE should support non-cgroups, non-secure mode
 ---

 Key: YARN-2424
 URL: https://issues.apache.org/jira/browse/YARN-2424
 Project: Hadoop YARN
  Issue Type: Bug
  Components: nodemanager
Affects Versions: 2.3.0, 2.4.0, 2.5.0, 2.4.1
Reporter: Allen Wittenauer
Priority: Blocker
 Attachments: YARN-2424.patch


 After YARN-1253, LCE no longer works for non-secure, non-cgroup scenarios.  
 This is a fairly serious regression, as turning on LCE prior to turning on 
 full-blown security is a fairly standard procedure.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (YARN-2424) LCE should support non-cgroups, non-secure mode


[ 
https://issues.apache.org/jira/browse/YARN-2424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14103046#comment-14103046
 ] 

Ravi Prakash commented on YARN-2424:


Hi Tucu! The intention was for this to be a useful thing to do as an 
intermediate step when migrating from an insecure cluster to a Kerberized 
cluster. This would let people test their provisioning of unix users without 
having to deal with Kerberos issues.
Could you please answer my question?
bq. So if we enforced the use of several least privileged users (instead of 
only 1), is that not just as secure?

 LCE should support non-cgroups, non-secure mode
 ---

 Key: YARN-2424
 URL: https://issues.apache.org/jira/browse/YARN-2424
 Project: Hadoop YARN
  Issue Type: Bug
  Components: nodemanager
Affects Versions: 2.3.0, 2.4.0, 2.5.0, 2.4.1
Reporter: Allen Wittenauer
Priority: Blocker
 Attachments: YARN-2424.patch


 After YARN-1253, LCE no longer works for non-secure, non-cgroup scenarios.  
 This is a fairly serious regression, as turning on LCE prior to turning on 
 full-blown security is a fairly standard procedure.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (YARN-2424) LCE should support non-cgroups, non-secure mode


[ 
https://issues.apache.org/jira/browse/YARN-2424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14103063#comment-14103063
 ] 

Alejandro Abdelnur commented on YARN-2424:
--

You are saying this is proactive troubleshooting then, not meant for 
production? If so, then, as I said before:

* the property has 'use-only-for-troubleshooting' in its name.
* the NM logs print a WARN at startup and on every started container stating 
the flag and its un-secure nature
* the container stdout/stderr also print a WARN to alert the user of the 
cluster setup.

 LCE should support non-cgroups, non-secure mode
 ---

 Key: YARN-2424
 URL: https://issues.apache.org/jira/browse/YARN-2424
 Project: Hadoop YARN
  Issue Type: Bug
  Components: nodemanager
Affects Versions: 2.3.0, 2.4.0, 2.5.0, 2.4.1
Reporter: Allen Wittenauer
Priority: Blocker
 Attachments: YARN-2424.patch


 After YARN-1253, LCE no longer works for non-secure, non-cgroup scenarios.  
 This is a fairly serious regression, as turning on LCE prior to turning on 
 full-blown security is a fairly standard procedure.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (YARN-2424) LCE should support non-cgroups, non-secure mode