[jira] [Commented] (YARN-1063) Winutils needs ability to create task as domain user
[ https://issues.apache.org/jira/browse/YARN-1063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14180735#comment-14180735 ] Jian He commented on YARN-1063: --- I merged this to 2.6, as this was marked for 2.6 Winutils needs ability to create task as domain user Key: YARN-1063 URL: https://issues.apache.org/jira/browse/YARN-1063 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Environment: Windows Reporter: Kyle Leckie Assignee: Remus Rusanu Labels: security, windows Fix For: 2.6.0 Attachments: YARN-1063.2.patch, YARN-1063.3.patch, YARN-1063.4.patch, YARN-1063.5.patch, YARN-1063.6.patch, YARN-1063.patch h1. Summary: Securing a Hadoop cluster requires constructing some form of security boundary around the processes executed in YARN containers. Isolation based on Windows user isolation seems most feasible. This approach is similar to the approach taken by the existing LinuxContainerExecutor. The current patch to winutils.exe adds the ability to create a process as a domain user. h1. Alternative Methods considered: h2. Process rights limited by security token restriction: On Windows access decisions are made by examining the security token of a process. It is possible to spawn a process with a restricted security token. Any of the rights granted by SIDs of the default token may be restricted. It is possible to see this in action by examining the security tone of a sandboxed process launch be a web browser. Typically the launched process will have a fully restricted token and need to access machine resources through a dedicated broker process that enforces a custom security policy. This broker process mechanism would break compatibility with the typical Hadoop container process. The Container process must be able to utilize standard function calls for disk and network IO. I performed some work looking at ways to ACL the local files to the specific launched without granting rights to other processes launched on the same machine but found this to be an overly complex solution. h2. Relying on APP containers: Recent versions of windows have the ability to launch processes within an isolated container. Application containers are supported for execution of WinRT based executables. This method was ruled out due to the lack of official support for standard windows APIs. At some point in the future windows may support functionality similar to BSD jails or Linux containers, at that point support for containers should be added. h1. Create As User Feature Description: h2. Usage: A new sub command was added to the set of task commands. Here is the syntax: winutils task createAsUser [TASKNAME] [USERNAME] [COMMAND_LINE] Some notes: * The username specified is in the format of user@domain * The machine executing this command must be joined to the domain of the user specified * The domain controller must allow the account executing the command access to the user information. For this join the account to the predefined group labeled Pre-Windows 2000 Compatible Access * The account running the command must have several rights on the local machine. These can be managed manually using secpol.msc: ** Act as part of the operating system - SE_TCB_NAME ** Replace a process-level token - SE_ASSIGNPRIMARYTOKEN_NAME ** Adjust memory quotas for a process - SE_INCREASE_QUOTA_NAME * The launched process will not have rights to the desktop so will not be able to display any information or create UI. * The launched process will have no network credentials. Any access of network resources that requires domain authentication will fail. h2. Implementation: Winutils performs the following steps: # Enable the required privileges for the current process. # Register as a trusted process with the Local Security Authority (LSA). # Create a new logon for the user passed on the command line. # Load/Create a profile on the local machine for the new logon. # Create a new environment for the new logon. # Launch the new process in a job with the task name specified and using the created logon. # Wait for the JOB to exit. h2. Future work: The following work was scoped out of this check in: * Support for non-domain users or machine that are not domain joined. * Support for privilege isolation by running the task launcher in a high privilege service with access over an ACLed named pipe. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1063) Winutils needs ability to create task as domain user
[ https://issues.apache.org/jira/browse/YARN-1063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14156357#comment-14156357 ] Hudson commented on YARN-1063: -- FAILURE: Integrated in Hadoop-Yarn-trunk #698 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/698/]) YARN-1063. Augmented Hadoop common winutils to have the ability to create containers as domain users. Contributed by Remus Rusanu. (vinodkv: rev 5ca97f1e60b8a7848f6eadd15f6c08ed390a8cda) * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestWinUtils.java * hadoop-common-project/hadoop-common/src/main/winutils/symlink.c * hadoop-common-project/hadoop-common/src/main/winutils/chown.c * hadoop-common-project/hadoop-common/src/main/winutils/task.c * hadoop-yarn-project/CHANGES.txt * hadoop-common-project/hadoop-common/src/main/winutils/include/winutils.h * hadoop-common-project/hadoop-common/src/main/winutils/libwinutils.c Winutils needs ability to create task as domain user Key: YARN-1063 URL: https://issues.apache.org/jira/browse/YARN-1063 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Environment: Windows Reporter: Kyle Leckie Assignee: Remus Rusanu Labels: security, windows Fix For: 2.6.0 Attachments: YARN-1063.2.patch, YARN-1063.3.patch, YARN-1063.4.patch, YARN-1063.5.patch, YARN-1063.6.patch, YARN-1063.patch h1. Summary: Securing a Hadoop cluster requires constructing some form of security boundary around the processes executed in YARN containers. Isolation based on Windows user isolation seems most feasible. This approach is similar to the approach taken by the existing LinuxContainerExecutor. The current patch to winutils.exe adds the ability to create a process as a domain user. h1. Alternative Methods considered: h2. Process rights limited by security token restriction: On Windows access decisions are made by examining the security token of a process. It is possible to spawn a process with a restricted security token. Any of the rights granted by SIDs of the default token may be restricted. It is possible to see this in action by examining the security tone of a sandboxed process launch be a web browser. Typically the launched process will have a fully restricted token and need to access machine resources through a dedicated broker process that enforces a custom security policy. This broker process mechanism would break compatibility with the typical Hadoop container process. The Container process must be able to utilize standard function calls for disk and network IO. I performed some work looking at ways to ACL the local files to the specific launched without granting rights to other processes launched on the same machine but found this to be an overly complex solution. h2. Relying on APP containers: Recent versions of windows have the ability to launch processes within an isolated container. Application containers are supported for execution of WinRT based executables. This method was ruled out due to the lack of official support for standard windows APIs. At some point in the future windows may support functionality similar to BSD jails or Linux containers, at that point support for containers should be added. h1. Create As User Feature Description: h2. Usage: A new sub command was added to the set of task commands. Here is the syntax: winutils task createAsUser [TASKNAME] [USERNAME] [COMMAND_LINE] Some notes: * The username specified is in the format of user@domain * The machine executing this command must be joined to the domain of the user specified * The domain controller must allow the account executing the command access to the user information. For this join the account to the predefined group labeled Pre-Windows 2000 Compatible Access * The account running the command must have several rights on the local machine. These can be managed manually using secpol.msc: ** Act as part of the operating system - SE_TCB_NAME ** Replace a process-level token - SE_ASSIGNPRIMARYTOKEN_NAME ** Adjust memory quotas for a process - SE_INCREASE_QUOTA_NAME * The launched process will not have rights to the desktop so will not be able to display any information or create UI. * The launched process will have no network credentials. Any access of network resources that requires domain authentication will fail. h2. Implementation: Winutils performs the following steps: # Enable the required privileges for the current process. # Register as a trusted process with the Local Security Authority (LSA). # Create a new logon for the user passed on the command line. # Load/Create a profile on the local machine for the new logon. # Create a new environment
[jira] [Commented] (YARN-1063) Winutils needs ability to create task as domain user
[ https://issues.apache.org/jira/browse/YARN-1063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14156435#comment-14156435 ] Hudson commented on YARN-1063: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #1889 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1889/]) YARN-1063. Augmented Hadoop common winutils to have the ability to create containers as domain users. Contributed by Remus Rusanu. (vinodkv: rev 5ca97f1e60b8a7848f6eadd15f6c08ed390a8cda) * hadoop-common-project/hadoop-common/src/main/winutils/symlink.c * hadoop-common-project/hadoop-common/src/main/winutils/libwinutils.c * hadoop-common-project/hadoop-common/src/main/winutils/chown.c * hadoop-common-project/hadoop-common/src/main/winutils/include/winutils.h * hadoop-common-project/hadoop-common/src/main/winutils/task.c * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestWinUtils.java * hadoop-yarn-project/CHANGES.txt Winutils needs ability to create task as domain user Key: YARN-1063 URL: https://issues.apache.org/jira/browse/YARN-1063 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Environment: Windows Reporter: Kyle Leckie Assignee: Remus Rusanu Labels: security, windows Fix For: 2.6.0 Attachments: YARN-1063.2.patch, YARN-1063.3.patch, YARN-1063.4.patch, YARN-1063.5.patch, YARN-1063.6.patch, YARN-1063.patch h1. Summary: Securing a Hadoop cluster requires constructing some form of security boundary around the processes executed in YARN containers. Isolation based on Windows user isolation seems most feasible. This approach is similar to the approach taken by the existing LinuxContainerExecutor. The current patch to winutils.exe adds the ability to create a process as a domain user. h1. Alternative Methods considered: h2. Process rights limited by security token restriction: On Windows access decisions are made by examining the security token of a process. It is possible to spawn a process with a restricted security token. Any of the rights granted by SIDs of the default token may be restricted. It is possible to see this in action by examining the security tone of a sandboxed process launch be a web browser. Typically the launched process will have a fully restricted token and need to access machine resources through a dedicated broker process that enforces a custom security policy. This broker process mechanism would break compatibility with the typical Hadoop container process. The Container process must be able to utilize standard function calls for disk and network IO. I performed some work looking at ways to ACL the local files to the specific launched without granting rights to other processes launched on the same machine but found this to be an overly complex solution. h2. Relying on APP containers: Recent versions of windows have the ability to launch processes within an isolated container. Application containers are supported for execution of WinRT based executables. This method was ruled out due to the lack of official support for standard windows APIs. At some point in the future windows may support functionality similar to BSD jails or Linux containers, at that point support for containers should be added. h1. Create As User Feature Description: h2. Usage: A new sub command was added to the set of task commands. Here is the syntax: winutils task createAsUser [TASKNAME] [USERNAME] [COMMAND_LINE] Some notes: * The username specified is in the format of user@domain * The machine executing this command must be joined to the domain of the user specified * The domain controller must allow the account executing the command access to the user information. For this join the account to the predefined group labeled Pre-Windows 2000 Compatible Access * The account running the command must have several rights on the local machine. These can be managed manually using secpol.msc: ** Act as part of the operating system - SE_TCB_NAME ** Replace a process-level token - SE_ASSIGNPRIMARYTOKEN_NAME ** Adjust memory quotas for a process - SE_INCREASE_QUOTA_NAME * The launched process will not have rights to the desktop so will not be able to display any information or create UI. * The launched process will have no network credentials. Any access of network resources that requires domain authentication will fail. h2. Implementation: Winutils performs the following steps: # Enable the required privileges for the current process. # Register as a trusted process with the Local Security Authority (LSA). # Create a new logon for the user passed on the command line. # Load/Create a profile on the local machine for the new logon. # Create a new environment
[jira] [Commented] (YARN-1063) Winutils needs ability to create task as domain user
[ https://issues.apache.org/jira/browse/YARN-1063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14156552#comment-14156552 ] Hudson commented on YARN-1063: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1914 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1914/]) YARN-1063. Augmented Hadoop common winutils to have the ability to create containers as domain users. Contributed by Remus Rusanu. (vinodkv: rev 5ca97f1e60b8a7848f6eadd15f6c08ed390a8cda) * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestWinUtils.java * hadoop-common-project/hadoop-common/src/main/winutils/chown.c * hadoop-common-project/hadoop-common/src/main/winutils/symlink.c * hadoop-common-project/hadoop-common/src/main/winutils/libwinutils.c * hadoop-common-project/hadoop-common/src/main/winutils/include/winutils.h * hadoop-common-project/hadoop-common/src/main/winutils/task.c * hadoop-yarn-project/CHANGES.txt Winutils needs ability to create task as domain user Key: YARN-1063 URL: https://issues.apache.org/jira/browse/YARN-1063 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Environment: Windows Reporter: Kyle Leckie Assignee: Remus Rusanu Labels: security, windows Fix For: 2.6.0 Attachments: YARN-1063.2.patch, YARN-1063.3.patch, YARN-1063.4.patch, YARN-1063.5.patch, YARN-1063.6.patch, YARN-1063.patch h1. Summary: Securing a Hadoop cluster requires constructing some form of security boundary around the processes executed in YARN containers. Isolation based on Windows user isolation seems most feasible. This approach is similar to the approach taken by the existing LinuxContainerExecutor. The current patch to winutils.exe adds the ability to create a process as a domain user. h1. Alternative Methods considered: h2. Process rights limited by security token restriction: On Windows access decisions are made by examining the security token of a process. It is possible to spawn a process with a restricted security token. Any of the rights granted by SIDs of the default token may be restricted. It is possible to see this in action by examining the security tone of a sandboxed process launch be a web browser. Typically the launched process will have a fully restricted token and need to access machine resources through a dedicated broker process that enforces a custom security policy. This broker process mechanism would break compatibility with the typical Hadoop container process. The Container process must be able to utilize standard function calls for disk and network IO. I performed some work looking at ways to ACL the local files to the specific launched without granting rights to other processes launched on the same machine but found this to be an overly complex solution. h2. Relying on APP containers: Recent versions of windows have the ability to launch processes within an isolated container. Application containers are supported for execution of WinRT based executables. This method was ruled out due to the lack of official support for standard windows APIs. At some point in the future windows may support functionality similar to BSD jails or Linux containers, at that point support for containers should be added. h1. Create As User Feature Description: h2. Usage: A new sub command was added to the set of task commands. Here is the syntax: winutils task createAsUser [TASKNAME] [USERNAME] [COMMAND_LINE] Some notes: * The username specified is in the format of user@domain * The machine executing this command must be joined to the domain of the user specified * The domain controller must allow the account executing the command access to the user information. For this join the account to the predefined group labeled Pre-Windows 2000 Compatible Access * The account running the command must have several rights on the local machine. These can be managed manually using secpol.msc: ** Act as part of the operating system - SE_TCB_NAME ** Replace a process-level token - SE_ASSIGNPRIMARYTOKEN_NAME ** Adjust memory quotas for a process - SE_INCREASE_QUOTA_NAME * The launched process will not have rights to the desktop so will not be able to display any information or create UI. * The launched process will have no network credentials. Any access of network resources that requires domain authentication will fail. h2. Implementation: Winutils performs the following steps: # Enable the required privileges for the current process. # Register as a trusted process with the Local Security Authority (LSA). # Create a new logon for the user passed on the command line. # Load/Create a profile on the local machine for the new logon. # Create a new
[jira] [Commented] (YARN-1063) Winutils needs ability to create task as domain user
[ https://issues.apache.org/jira/browse/YARN-1063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14155093#comment-14155093 ] Vinod Kumar Vavilapalli commented on YARN-1063: --- Tx for the updates [~rusanu]! I am committing this now to unblock the follow up patches, trusting [~ivanmi]'s reviews on the Windows side of things. Winutils needs ability to create task as domain user Key: YARN-1063 URL: https://issues.apache.org/jira/browse/YARN-1063 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Environment: Windows Reporter: Kyle Leckie Assignee: Remus Rusanu Labels: security, windows Attachments: YARN-1063.2.patch, YARN-1063.3.patch, YARN-1063.4.patch, YARN-1063.5.patch, YARN-1063.6.patch, YARN-1063.patch h1. Summary: Securing a Hadoop cluster requires constructing some form of security boundary around the processes executed in YARN containers. Isolation based on Windows user isolation seems most feasible. This approach is similar to the approach taken by the existing LinuxContainerExecutor. The current patch to winutils.exe adds the ability to create a process as a domain user. h1. Alternative Methods considered: h2. Process rights limited by security token restriction: On Windows access decisions are made by examining the security token of a process. It is possible to spawn a process with a restricted security token. Any of the rights granted by SIDs of the default token may be restricted. It is possible to see this in action by examining the security tone of a sandboxed process launch be a web browser. Typically the launched process will have a fully restricted token and need to access machine resources through a dedicated broker process that enforces a custom security policy. This broker process mechanism would break compatibility with the typical Hadoop container process. The Container process must be able to utilize standard function calls for disk and network IO. I performed some work looking at ways to ACL the local files to the specific launched without granting rights to other processes launched on the same machine but found this to be an overly complex solution. h2. Relying on APP containers: Recent versions of windows have the ability to launch processes within an isolated container. Application containers are supported for execution of WinRT based executables. This method was ruled out due to the lack of official support for standard windows APIs. At some point in the future windows may support functionality similar to BSD jails or Linux containers, at that point support for containers should be added. h1. Create As User Feature Description: h2. Usage: A new sub command was added to the set of task commands. Here is the syntax: winutils task createAsUser [TASKNAME] [USERNAME] [COMMAND_LINE] Some notes: * The username specified is in the format of user@domain * The machine executing this command must be joined to the domain of the user specified * The domain controller must allow the account executing the command access to the user information. For this join the account to the predefined group labeled Pre-Windows 2000 Compatible Access * The account running the command must have several rights on the local machine. These can be managed manually using secpol.msc: ** Act as part of the operating system - SE_TCB_NAME ** Replace a process-level token - SE_ASSIGNPRIMARYTOKEN_NAME ** Adjust memory quotas for a process - SE_INCREASE_QUOTA_NAME * The launched process will not have rights to the desktop so will not be able to display any information or create UI. * The launched process will have no network credentials. Any access of network resources that requires domain authentication will fail. h2. Implementation: Winutils performs the following steps: # Enable the required privileges for the current process. # Register as a trusted process with the Local Security Authority (LSA). # Create a new logon for the user passed on the command line. # Load/Create a profile on the local machine for the new logon. # Create a new environment for the new logon. # Launch the new process in a job with the task name specified and using the created logon. # Wait for the JOB to exit. h2. Future work: The following work was scoped out of this check in: * Support for non-domain users or machine that are not domain joined. * Support for privilege isolation by running the task launcher in a high privilege service with access over an ACLed named pipe. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1063) Winutils needs ability to create task as domain user
[ https://issues.apache.org/jira/browse/YARN-1063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14155116#comment-14155116 ] Hudson commented on YARN-1063: -- FAILURE: Integrated in Hadoop-trunk-Commit #6164 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/6164/]) YARN-1063. Augmented Hadoop common winutils to have the ability to create containers as domain users. Contributed by Remus Rusanu. (vinodkv: rev 5ca97f1e60b8a7848f6eadd15f6c08ed390a8cda) * hadoop-yarn-project/CHANGES.txt * hadoop-common-project/hadoop-common/src/main/winutils/libwinutils.c * hadoop-common-project/hadoop-common/src/main/winutils/symlink.c * hadoop-common-project/hadoop-common/src/main/winutils/include/winutils.h * hadoop-common-project/hadoop-common/src/main/winutils/task.c * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestWinUtils.java * hadoop-common-project/hadoop-common/src/main/winutils/chown.c Winutils needs ability to create task as domain user Key: YARN-1063 URL: https://issues.apache.org/jira/browse/YARN-1063 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Environment: Windows Reporter: Kyle Leckie Assignee: Remus Rusanu Labels: security, windows Fix For: 2.6.0 Attachments: YARN-1063.2.patch, YARN-1063.3.patch, YARN-1063.4.patch, YARN-1063.5.patch, YARN-1063.6.patch, YARN-1063.patch h1. Summary: Securing a Hadoop cluster requires constructing some form of security boundary around the processes executed in YARN containers. Isolation based on Windows user isolation seems most feasible. This approach is similar to the approach taken by the existing LinuxContainerExecutor. The current patch to winutils.exe adds the ability to create a process as a domain user. h1. Alternative Methods considered: h2. Process rights limited by security token restriction: On Windows access decisions are made by examining the security token of a process. It is possible to spawn a process with a restricted security token. Any of the rights granted by SIDs of the default token may be restricted. It is possible to see this in action by examining the security tone of a sandboxed process launch be a web browser. Typically the launched process will have a fully restricted token and need to access machine resources through a dedicated broker process that enforces a custom security policy. This broker process mechanism would break compatibility with the typical Hadoop container process. The Container process must be able to utilize standard function calls for disk and network IO. I performed some work looking at ways to ACL the local files to the specific launched without granting rights to other processes launched on the same machine but found this to be an overly complex solution. h2. Relying on APP containers: Recent versions of windows have the ability to launch processes within an isolated container. Application containers are supported for execution of WinRT based executables. This method was ruled out due to the lack of official support for standard windows APIs. At some point in the future windows may support functionality similar to BSD jails or Linux containers, at that point support for containers should be added. h1. Create As User Feature Description: h2. Usage: A new sub command was added to the set of task commands. Here is the syntax: winutils task createAsUser [TASKNAME] [USERNAME] [COMMAND_LINE] Some notes: * The username specified is in the format of user@domain * The machine executing this command must be joined to the domain of the user specified * The domain controller must allow the account executing the command access to the user information. For this join the account to the predefined group labeled Pre-Windows 2000 Compatible Access * The account running the command must have several rights on the local machine. These can be managed manually using secpol.msc: ** Act as part of the operating system - SE_TCB_NAME ** Replace a process-level token - SE_ASSIGNPRIMARYTOKEN_NAME ** Adjust memory quotas for a process - SE_INCREASE_QUOTA_NAME * The launched process will not have rights to the desktop so will not be able to display any information or create UI. * The launched process will have no network credentials. Any access of network resources that requires domain authentication will fail. h2. Implementation: Winutils performs the following steps: # Enable the required privileges for the current process. # Register as a trusted process with the Local Security Authority (LSA). # Create a new logon for the user passed on the command line. # Load/Create a profile on the local machine for the new logon. # Create a new
[jira] [Commented] (YARN-1063) Winutils needs ability to create task as domain user
[
https://issues.apache.org/jira/browse/YARN-1063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14155169#comment-14155169
]
Hadoop QA commented on YARN-1063:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12657587/YARN-1063.6.patch
against trunk revision 04b0843.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/5197//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5197//console
This message is automatically generated.
Winutils needs ability to create task as domain user
Key: YARN-1063
URL: https://issues.apache.org/jira/browse/YARN-1063
Project: Hadoop YARN
Issue Type: Sub-task
Components: nodemanager
Environment: Windows
Reporter: Kyle Leckie
Assignee: Remus Rusanu
Labels: security, windows
Fix For: 2.6.0
Attachments: YARN-1063.2.patch, YARN-1063.3.patch, YARN-1063.4.patch,
YARN-1063.5.patch, YARN-1063.6.patch, YARN-1063.patch
h1. Summary:
Securing a Hadoop cluster requires constructing some form of security
boundary around the processes executed in YARN containers. Isolation based on
Windows user isolation seems most feasible. This approach is similar to the
approach taken by the existing LinuxContainerExecutor. The current patch to
winutils.exe adds the ability to create a process as a domain user.
h1. Alternative Methods considered:
h2. Process rights limited by security token restriction:
On Windows access decisions are made by examining the security token of a
process. It is possible to spawn a process with a restricted security token.
Any of the rights granted by SIDs of the default token may be restricted. It
is possible to see this in action by examining the security tone of a
sandboxed process launch be a web browser. Typically the launched process
will have a fully restricted token and need to access machine resources
through a dedicated broker process that enforces a custom security policy.
This broker process mechanism would break compatibility with the typical
Hadoop container process. The Container process must be able to utilize
standard function calls for disk and network IO. I performed some work
looking at ways to ACL the local files to the specific launched without
granting rights to other processes launched on the same machine but found
this to be an overly complex solution.
h2. Relying on APP containers:
Recent versions of windows have the ability to launch processes within an
isolated container. Application containers are supported for execution of
WinRT based executables. This method was ruled out due to the lack of
official support for standard windows APIs. At some point in the future
windows may support functionality similar to BSD jails or Linux containers,
at that point support for containers should be added.
h1. Create As User Feature Description:
h2. Usage:
A new sub command was added to the set of task commands. Here is the syntax:
winutils task createAsUser [TASKNAME] [USERNAME] [COMMAND_LINE]
Some notes:
* The username specified is in the format of user@domain
* The machine executing this command must be joined to the domain of the user
specified
* The domain controller must allow the account executing the command access
to the user information. For this join the account to the predefined group
labeled Pre-Windows 2000 Compatible Access
* The account running the command must have several rights on the local
machine. These can be managed manually using secpol.msc:
** Act as part of the operating system - SE_TCB_NAME
** Replace a process-level token - SE_ASSIGNPRIMARYTOKEN_NAME
** Adjust memory quotas for a process - SE_INCREASE_QUOTA_NAME
* The launched process will not have rights to the desktop so
[jira] [Commented] (YARN-1063) Winutils needs ability to create task as domain user
[ https://issues.apache.org/jira/browse/YARN-1063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14153490#comment-14153490 ] Craig Welch commented on YARN-1063: --- [~rusanu] [~vinodkv], we can go ahead and address these comments as part of the [YARN-2198] effort, it's not necessary to resolve these before these patches are committed. Winutils needs ability to create task as domain user Key: YARN-1063 URL: https://issues.apache.org/jira/browse/YARN-1063 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Environment: Windows Reporter: Kyle Leckie Assignee: Remus Rusanu Labels: security, windows Attachments: YARN-1063.2.patch, YARN-1063.3.patch, YARN-1063.4.patch, YARN-1063.5.patch, YARN-1063.6.patch, YARN-1063.patch h1. Summary: Securing a Hadoop cluster requires constructing some form of security boundary around the processes executed in YARN containers. Isolation based on Windows user isolation seems most feasible. This approach is similar to the approach taken by the existing LinuxContainerExecutor. The current patch to winutils.exe adds the ability to create a process as a domain user. h1. Alternative Methods considered: h2. Process rights limited by security token restriction: On Windows access decisions are made by examining the security token of a process. It is possible to spawn a process with a restricted security token. Any of the rights granted by SIDs of the default token may be restricted. It is possible to see this in action by examining the security tone of a sandboxed process launch be a web browser. Typically the launched process will have a fully restricted token and need to access machine resources through a dedicated broker process that enforces a custom security policy. This broker process mechanism would break compatibility with the typical Hadoop container process. The Container process must be able to utilize standard function calls for disk and network IO. I performed some work looking at ways to ACL the local files to the specific launched without granting rights to other processes launched on the same machine but found this to be an overly complex solution. h2. Relying on APP containers: Recent versions of windows have the ability to launch processes within an isolated container. Application containers are supported for execution of WinRT based executables. This method was ruled out due to the lack of official support for standard windows APIs. At some point in the future windows may support functionality similar to BSD jails or Linux containers, at that point support for containers should be added. h1. Create As User Feature Description: h2. Usage: A new sub command was added to the set of task commands. Here is the syntax: winutils task createAsUser [TASKNAME] [USERNAME] [COMMAND_LINE] Some notes: * The username specified is in the format of user@domain * The machine executing this command must be joined to the domain of the user specified * The domain controller must allow the account executing the command access to the user information. For this join the account to the predefined group labeled Pre-Windows 2000 Compatible Access * The account running the command must have several rights on the local machine. These can be managed manually using secpol.msc: ** Act as part of the operating system - SE_TCB_NAME ** Replace a process-level token - SE_ASSIGNPRIMARYTOKEN_NAME ** Adjust memory quotas for a process - SE_INCREASE_QUOTA_NAME * The launched process will not have rights to the desktop so will not be able to display any information or create UI. * The launched process will have no network credentials. Any access of network resources that requires domain authentication will fail. h2. Implementation: Winutils performs the following steps: # Enable the required privileges for the current process. # Register as a trusted process with the Local Security Authority (LSA). # Create a new logon for the user passed on the command line. # Load/Create a profile on the local machine for the new logon. # Create a new environment for the new logon. # Launch the new process in a job with the task name specified and using the created logon. # Wait for the JOB to exit. h2. Future work: The following work was scoped out of this check in: * Support for non-domain users or machine that are not domain joined. * Support for privilege isolation by running the task launcher in a high privilege service with access over an ACLed named pipe. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1063) Winutils needs ability to create task as domain user
[ https://issues.apache.org/jira/browse/YARN-1063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14152504#comment-14152504 ] Craig Welch commented on YARN-1063: --- When looking this over to pickup context for 2198, I noticed a couple things: libwinutils.c CreateLogonForUser - confusing name, makes me think a new account is being created - CreateLogonTokenForUser? LogonUser? TestWinUtils - can we add testing specific to security? Winutils needs ability to create task as domain user Key: YARN-1063 URL: https://issues.apache.org/jira/browse/YARN-1063 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Environment: Windows Reporter: Kyle Leckie Assignee: Remus Rusanu Labels: security, windows Attachments: YARN-1063.2.patch, YARN-1063.3.patch, YARN-1063.4.patch, YARN-1063.5.patch, YARN-1063.6.patch, YARN-1063.patch h1. Summary: Securing a Hadoop cluster requires constructing some form of security boundary around the processes executed in YARN containers. Isolation based on Windows user isolation seems most feasible. This approach is similar to the approach taken by the existing LinuxContainerExecutor. The current patch to winutils.exe adds the ability to create a process as a domain user. h1. Alternative Methods considered: h2. Process rights limited by security token restriction: On Windows access decisions are made by examining the security token of a process. It is possible to spawn a process with a restricted security token. Any of the rights granted by SIDs of the default token may be restricted. It is possible to see this in action by examining the security tone of a sandboxed process launch be a web browser. Typically the launched process will have a fully restricted token and need to access machine resources through a dedicated broker process that enforces a custom security policy. This broker process mechanism would break compatibility with the typical Hadoop container process. The Container process must be able to utilize standard function calls for disk and network IO. I performed some work looking at ways to ACL the local files to the specific launched without granting rights to other processes launched on the same machine but found this to be an overly complex solution. h2. Relying on APP containers: Recent versions of windows have the ability to launch processes within an isolated container. Application containers are supported for execution of WinRT based executables. This method was ruled out due to the lack of official support for standard windows APIs. At some point in the future windows may support functionality similar to BSD jails or Linux containers, at that point support for containers should be added. h1. Create As User Feature Description: h2. Usage: A new sub command was added to the set of task commands. Here is the syntax: winutils task createAsUser [TASKNAME] [USERNAME] [COMMAND_LINE] Some notes: * The username specified is in the format of user@domain * The machine executing this command must be joined to the domain of the user specified * The domain controller must allow the account executing the command access to the user information. For this join the account to the predefined group labeled Pre-Windows 2000 Compatible Access * The account running the command must have several rights on the local machine. These can be managed manually using secpol.msc: ** Act as part of the operating system - SE_TCB_NAME ** Replace a process-level token - SE_ASSIGNPRIMARYTOKEN_NAME ** Adjust memory quotas for a process - SE_INCREASE_QUOTA_NAME * The launched process will not have rights to the desktop so will not be able to display any information or create UI. * The launched process will have no network credentials. Any access of network resources that requires domain authentication will fail. h2. Implementation: Winutils performs the following steps: # Enable the required privileges for the current process. # Register as a trusted process with the Local Security Authority (LSA). # Create a new logon for the user passed on the command line. # Load/Create a profile on the local machine for the new logon. # Create a new environment for the new logon. # Launch the new process in a job with the task name specified and using the created logon. # Wait for the JOB to exit. h2. Future work: The following work was scoped out of this check in: * Support for non-domain users or machine that are not domain joined. * Support for privilege isolation by running the task launcher in a high privilege service with access over an ACLed named pipe. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1063) Winutils needs ability to create task as domain user
[ https://issues.apache.org/jira/browse/YARN-1063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14129755#comment-14129755 ] Remus Rusanu commented on YARN-1063: I have tested against today trunk and the patch is identical, no need for any rebase on this. Winutils needs ability to create task as domain user Key: YARN-1063 URL: https://issues.apache.org/jira/browse/YARN-1063 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Environment: Windows Reporter: Kyle Leckie Assignee: Remus Rusanu Labels: security, windows Attachments: YARN-1063.2.patch, YARN-1063.3.patch, YARN-1063.4.patch, YARN-1063.5.patch, YARN-1063.6.patch, YARN-1063.patch h1. Summary: Securing a Hadoop cluster requires constructing some form of security boundary around the processes executed in YARN containers. Isolation based on Windows user isolation seems most feasible. This approach is similar to the approach taken by the existing LinuxContainerExecutor. The current patch to winutils.exe adds the ability to create a process as a domain user. h1. Alternative Methods considered: h2. Process rights limited by security token restriction: On Windows access decisions are made by examining the security token of a process. It is possible to spawn a process with a restricted security token. Any of the rights granted by SIDs of the default token may be restricted. It is possible to see this in action by examining the security tone of a sandboxed process launch be a web browser. Typically the launched process will have a fully restricted token and need to access machine resources through a dedicated broker process that enforces a custom security policy. This broker process mechanism would break compatibility with the typical Hadoop container process. The Container process must be able to utilize standard function calls for disk and network IO. I performed some work looking at ways to ACL the local files to the specific launched without granting rights to other processes launched on the same machine but found this to be an overly complex solution. h2. Relying on APP containers: Recent versions of windows have the ability to launch processes within an isolated container. Application containers are supported for execution of WinRT based executables. This method was ruled out due to the lack of official support for standard windows APIs. At some point in the future windows may support functionality similar to BSD jails or Linux containers, at that point support for containers should be added. h1. Create As User Feature Description: h2. Usage: A new sub command was added to the set of task commands. Here is the syntax: winutils task createAsUser [TASKNAME] [USERNAME] [COMMAND_LINE] Some notes: * The username specified is in the format of user@domain * The machine executing this command must be joined to the domain of the user specified * The domain controller must allow the account executing the command access to the user information. For this join the account to the predefined group labeled Pre-Windows 2000 Compatible Access * The account running the command must have several rights on the local machine. These can be managed manually using secpol.msc: ** Act as part of the operating system - SE_TCB_NAME ** Replace a process-level token - SE_ASSIGNPRIMARYTOKEN_NAME ** Adjust memory quotas for a process - SE_INCREASE_QUOTA_NAME * The launched process will not have rights to the desktop so will not be able to display any information or create UI. * The launched process will have no network credentials. Any access of network resources that requires domain authentication will fail. h2. Implementation: Winutils performs the following steps: # Enable the required privileges for the current process. # Register as a trusted process with the Local Security Authority (LSA). # Create a new logon for the user passed on the command line. # Load/Create a profile on the local machine for the new logon. # Create a new environment for the new logon. # Launch the new process in a job with the task name specified and using the created logon. # Wait for the JOB to exit. h2. Future work: The following work was scoped out of this check in: * Support for non-domain users or machine that are not domain joined. * Support for privilege isolation by running the task launcher in a high privilege service with access over an ACLed named pipe. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1063) Winutils needs ability to create task as domain user
[
https://issues.apache.org/jira/browse/YARN-1063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14073008#comment-14073008
]
Hadoop QA commented on YARN-1063:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12657572/YARN-1063.5.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 1259 javac
compiler warnings (more than the trunk's current 1258 warnings).
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common:
org.apache.hadoop.ipc.TestIPC
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4412//testReport/
Javac warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/4412//artifact/trunk/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4412//console
This message is automatically generated.
Winutils needs ability to create task as domain user
Key: YARN-1063
URL: https://issues.apache.org/jira/browse/YARN-1063
Project: Hadoop YARN
Issue Type: Sub-task
Components: nodemanager
Environment: Windows
Reporter: Kyle Leckie
Assignee: Remus Rusanu
Labels: security, windows
Attachments: YARN-1063.2.patch, YARN-1063.3.patch, YARN-1063.4.patch,
YARN-1063.5.patch, YARN-1063.patch
h1. Summary:
Securing a Hadoop cluster requires constructing some form of security
boundary around the processes executed in YARN containers. Isolation based on
Windows user isolation seems most feasible. This approach is similar to the
approach taken by the existing LinuxContainerExecutor. The current patch to
winutils.exe adds the ability to create a process as a domain user.
h1. Alternative Methods considered:
h2. Process rights limited by security token restriction:
On Windows access decisions are made by examining the security token of a
process. It is possible to spawn a process with a restricted security token.
Any of the rights granted by SIDs of the default token may be restricted. It
is possible to see this in action by examining the security tone of a
sandboxed process launch be a web browser. Typically the launched process
will have a fully restricted token and need to access machine resources
through a dedicated broker process that enforces a custom security policy.
This broker process mechanism would break compatibility with the typical
Hadoop container process. The Container process must be able to utilize
standard function calls for disk and network IO. I performed some work
looking at ways to ACL the local files to the specific launched without
granting rights to other processes launched on the same machine but found
this to be an overly complex solution.
h2. Relying on APP containers:
Recent versions of windows have the ability to launch processes within an
isolated container. Application containers are supported for execution of
WinRT based executables. This method was ruled out due to the lack of
official support for standard windows APIs. At some point in the future
windows may support functionality similar to BSD jails or Linux containers,
at that point support for containers should be added.
h1. Create As User Feature Description:
h2. Usage:
A new sub command was added to the set of task commands. Here is the syntax:
winutils task createAsUser [TASKNAME] [USERNAME] [COMMAND_LINE]
Some notes:
* The username specified is in the format of user@domain
* The machine executing this command must be joined to the domain of the user
specified
* The domain controller must allow the account executing the command access
to the user information. For this join the account to the predefined group
labeled Pre-Windows 2000 Compatible Access
* The account running the command must have several rights on the local
machine. These can be managed manually using secpol.msc:
** Act as part of the operating system - SE_TCB_NAME
** Replace a process-level token -
[jira] [Commented] (YARN-1063) Winutils needs ability to create task as domain user
[
https://issues.apache.org/jira/browse/YARN-1063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14073123#comment-14073123
]
Hadoop QA commented on YARN-1063:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12657587/YARN-1063.6.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common:
org.apache.hadoop.ipc.TestIPC
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4413//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4413//console
This message is automatically generated.
Winutils needs ability to create task as domain user
Key: YARN-1063
URL: https://issues.apache.org/jira/browse/YARN-1063
Project: Hadoop YARN
Issue Type: Sub-task
Components: nodemanager
Environment: Windows
Reporter: Kyle Leckie
Assignee: Remus Rusanu
Labels: security, windows
Attachments: YARN-1063.2.patch, YARN-1063.3.patch, YARN-1063.4.patch,
YARN-1063.5.patch, YARN-1063.6.patch, YARN-1063.patch
h1. Summary:
Securing a Hadoop cluster requires constructing some form of security
boundary around the processes executed in YARN containers. Isolation based on
Windows user isolation seems most feasible. This approach is similar to the
approach taken by the existing LinuxContainerExecutor. The current patch to
winutils.exe adds the ability to create a process as a domain user.
h1. Alternative Methods considered:
h2. Process rights limited by security token restriction:
On Windows access decisions are made by examining the security token of a
process. It is possible to spawn a process with a restricted security token.
Any of the rights granted by SIDs of the default token may be restricted. It
is possible to see this in action by examining the security tone of a
sandboxed process launch be a web browser. Typically the launched process
will have a fully restricted token and need to access machine resources
through a dedicated broker process that enforces a custom security policy.
This broker process mechanism would break compatibility with the typical
Hadoop container process. The Container process must be able to utilize
standard function calls for disk and network IO. I performed some work
looking at ways to ACL the local files to the specific launched without
granting rights to other processes launched on the same machine but found
this to be an overly complex solution.
h2. Relying on APP containers:
Recent versions of windows have the ability to launch processes within an
isolated container. Application containers are supported for execution of
WinRT based executables. This method was ruled out due to the lack of
official support for standard windows APIs. At some point in the future
windows may support functionality similar to BSD jails or Linux containers,
at that point support for containers should be added.
h1. Create As User Feature Description:
h2. Usage:
A new sub command was added to the set of task commands. Here is the syntax:
winutils task createAsUser [TASKNAME] [USERNAME] [COMMAND_LINE]
Some notes:
* The username specified is in the format of user@domain
* The machine executing this command must be joined to the domain of the user
specified
* The domain controller must allow the account executing the command access
to the user information. For this join the account to the predefined group
labeled Pre-Windows 2000 Compatible Access
* The account running the command must have several rights on the local
machine. These can be managed manually using secpol.msc:
** Act as part of the operating system - SE_TCB_NAME
** Replace a process-level token - SE_ASSIGNPRIMARYTOKEN_NAME
** Adjust memory quotas for a process - SE_INCREASE_QUOTA_NAME
* The launched process will not have rights to
[jira] [Commented] (YARN-1063) Winutils needs ability to create task as domain user
[ https://issues.apache.org/jira/browse/YARN-1063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14073143#comment-14073143 ] Remus Rusanu commented on YARN-1063: TestIPC.testRetryProxy passes for me locally with the patch applied. The test does not exercise in any way the winutils. Winutils needs ability to create task as domain user Key: YARN-1063 URL: https://issues.apache.org/jira/browse/YARN-1063 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Environment: Windows Reporter: Kyle Leckie Assignee: Remus Rusanu Labels: security, windows Attachments: YARN-1063.2.patch, YARN-1063.3.patch, YARN-1063.4.patch, YARN-1063.5.patch, YARN-1063.6.patch, YARN-1063.patch h1. Summary: Securing a Hadoop cluster requires constructing some form of security boundary around the processes executed in YARN containers. Isolation based on Windows user isolation seems most feasible. This approach is similar to the approach taken by the existing LinuxContainerExecutor. The current patch to winutils.exe adds the ability to create a process as a domain user. h1. Alternative Methods considered: h2. Process rights limited by security token restriction: On Windows access decisions are made by examining the security token of a process. It is possible to spawn a process with a restricted security token. Any of the rights granted by SIDs of the default token may be restricted. It is possible to see this in action by examining the security tone of a sandboxed process launch be a web browser. Typically the launched process will have a fully restricted token and need to access machine resources through a dedicated broker process that enforces a custom security policy. This broker process mechanism would break compatibility with the typical Hadoop container process. The Container process must be able to utilize standard function calls for disk and network IO. I performed some work looking at ways to ACL the local files to the specific launched without granting rights to other processes launched on the same machine but found this to be an overly complex solution. h2. Relying on APP containers: Recent versions of windows have the ability to launch processes within an isolated container. Application containers are supported for execution of WinRT based executables. This method was ruled out due to the lack of official support for standard windows APIs. At some point in the future windows may support functionality similar to BSD jails or Linux containers, at that point support for containers should be added. h1. Create As User Feature Description: h2. Usage: A new sub command was added to the set of task commands. Here is the syntax: winutils task createAsUser [TASKNAME] [USERNAME] [COMMAND_LINE] Some notes: * The username specified is in the format of user@domain * The machine executing this command must be joined to the domain of the user specified * The domain controller must allow the account executing the command access to the user information. For this join the account to the predefined group labeled Pre-Windows 2000 Compatible Access * The account running the command must have several rights on the local machine. These can be managed manually using secpol.msc: ** Act as part of the operating system - SE_TCB_NAME ** Replace a process-level token - SE_ASSIGNPRIMARYTOKEN_NAME ** Adjust memory quotas for a process - SE_INCREASE_QUOTA_NAME * The launched process will not have rights to the desktop so will not be able to display any information or create UI. * The launched process will have no network credentials. Any access of network resources that requires domain authentication will fail. h2. Implementation: Winutils performs the following steps: # Enable the required privileges for the current process. # Register as a trusted process with the Local Security Authority (LSA). # Create a new logon for the user passed on the command line. # Load/Create a profile on the local machine for the new logon. # Create a new environment for the new logon. # Launch the new process in a job with the task name specified and using the created logon. # Wait for the JOB to exit. h2. Future work: The following work was scoped out of this check in: * Support for non-domain users or machine that are not domain joined. * Support for privilege isolation by running the task launcher in a high privilege service with access over an ACLed named pipe. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1063) Winutils needs ability to create task as domain user
[
https://issues.apache.org/jira/browse/YARN-1063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14071781#comment-14071781
]
Hadoop QA commented on YARN-1063:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12657356/YARN-1063.5.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4404//console
This message is automatically generated.
Winutils needs ability to create task as domain user
Key: YARN-1063
URL: https://issues.apache.org/jira/browse/YARN-1063
Project: Hadoop YARN
Issue Type: Sub-task
Components: nodemanager
Environment: Windows
Reporter: Kyle Leckie
Assignee: Remus Rusanu
Labels: security, windows
Attachments: YARN-1063.2.patch, YARN-1063.3.patch, YARN-1063.4.patch,
YARN-1063.5.patch, YARN-1063.patch
h1. Summary:
Securing a Hadoop cluster requires constructing some form of security
boundary around the processes executed in YARN containers. Isolation based on
Windows user isolation seems most feasible. This approach is similar to the
approach taken by the existing LinuxContainerExecutor. The current patch to
winutils.exe adds the ability to create a process as a domain user.
h1. Alternative Methods considered:
h2. Process rights limited by security token restriction:
On Windows access decisions are made by examining the security token of a
process. It is possible to spawn a process with a restricted security token.
Any of the rights granted by SIDs of the default token may be restricted. It
is possible to see this in action by examining the security tone of a
sandboxed process launch be a web browser. Typically the launched process
will have a fully restricted token and need to access machine resources
through a dedicated broker process that enforces a custom security policy.
This broker process mechanism would break compatibility with the typical
Hadoop container process. The Container process must be able to utilize
standard function calls for disk and network IO. I performed some work
looking at ways to ACL the local files to the specific launched without
granting rights to other processes launched on the same machine but found
this to be an overly complex solution.
h2. Relying on APP containers:
Recent versions of windows have the ability to launch processes within an
isolated container. Application containers are supported for execution of
WinRT based executables. This method was ruled out due to the lack of
official support for standard windows APIs. At some point in the future
windows may support functionality similar to BSD jails or Linux containers,
at that point support for containers should be added.
h1. Create As User Feature Description:
h2. Usage:
A new sub command was added to the set of task commands. Here is the syntax:
winutils task createAsUser [TASKNAME] [USERNAME] [COMMAND_LINE]
Some notes:
* The username specified is in the format of user@domain
* The machine executing this command must be joined to the domain of the user
specified
* The domain controller must allow the account executing the command access
to the user information. For this join the account to the predefined group
labeled Pre-Windows 2000 Compatible Access
* The account running the command must have several rights on the local
machine. These can be managed manually using secpol.msc:
** Act as part of the operating system - SE_TCB_NAME
** Replace a process-level token - SE_ASSIGNPRIMARYTOKEN_NAME
** Adjust memory quotas for a process - SE_INCREASE_QUOTA_NAME
* The launched process will not have rights to the desktop so will not be
able to display any information or create UI.
* The launched process will have no network credentials. Any access of
network resources that requires domain authentication will fail.
h2. Implementation:
Winutils performs the following steps:
# Enable the required privileges for the current process.
# Register as a trusted process with the Local Security Authority (LSA).
# Create a new logon for the user passed on the command line.
# Load/Create a profile on the local machine for the new logon.
# Create a new environment for the new logon.
# Launch the new process in a job with the task name specified and using the
created logon.
# Wait for the JOB to exit.
h2. Future work:
The following work was scoped out of this check in:
* Support for non-domain users or machine that are not domain joined.
* Support for privilege isolation by running the task launcher in a high
privilege service with access over an ACLed named
[jira] [Commented] (YARN-1063) Winutils needs ability to create task as domain user
[ https://issues.apache.org/jira/browse/YARN-1063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14011524#comment-14011524 ] Vinod Kumar Vavilapalli commented on YARN-1063: --- Scanned through the patch. It's dense and full of windows related stuff which I am not entirely familiar with. Looked at the code from YARN container localization and launch POV. I have posted some comments on YARN-1972 which may cause some changes here too. Winutils needs ability to create task as domain user Key: YARN-1063 URL: https://issues.apache.org/jira/browse/YARN-1063 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Environment: Windows Reporter: Kyle Leckie Assignee: Remus Rusanu Labels: security, windows Attachments: YARN-1063.2.patch, YARN-1063.3.patch, YARN-1063.4.patch, YARN-1063.patch h1. Summary: Securing a Hadoop cluster requires constructing some form of security boundary around the processes executed in YARN containers. Isolation based on Windows user isolation seems most feasible. This approach is similar to the approach taken by the existing LinuxContainerExecutor. The current patch to winutils.exe adds the ability to create a process as a domain user. h1. Alternative Methods considered: h2. Process rights limited by security token restriction: On Windows access decisions are made by examining the security token of a process. It is possible to spawn a process with a restricted security token. Any of the rights granted by SIDs of the default token may be restricted. It is possible to see this in action by examining the security tone of a sandboxed process launch be a web browser. Typically the launched process will have a fully restricted token and need to access machine resources through a dedicated broker process that enforces a custom security policy. This broker process mechanism would break compatibility with the typical Hadoop container process. The Container process must be able to utilize standard function calls for disk and network IO. I performed some work looking at ways to ACL the local files to the specific launched without granting rights to other processes launched on the same machine but found this to be an overly complex solution. h2. Relying on APP containers: Recent versions of windows have the ability to launch processes within an isolated container. Application containers are supported for execution of WinRT based executables. This method was ruled out due to the lack of official support for standard windows APIs. At some point in the future windows may support functionality similar to BSD jails or Linux containers, at that point support for containers should be added. h1. Create As User Feature Description: h2. Usage: A new sub command was added to the set of task commands. Here is the syntax: winutils task createAsUser [TASKNAME] [USERNAME] [COMMAND_LINE] Some notes: * The username specified is in the format of user@domain * The machine executing this command must be joined to the domain of the user specified * The domain controller must allow the account executing the command access to the user information. For this join the account to the predefined group labeled Pre-Windows 2000 Compatible Access * The account running the command must have several rights on the local machine. These can be managed manually using secpol.msc: ** Act as part of the operating system - SE_TCB_NAME ** Replace a process-level token - SE_ASSIGNPRIMARYTOKEN_NAME ** Adjust memory quotas for a process - SE_INCREASE_QUOTA_NAME * The launched process will not have rights to the desktop so will not be able to display any information or create UI. * The launched process will have no network credentials. Any access of network resources that requires domain authentication will fail. h2. Implementation: Winutils performs the following steps: # Enable the required privileges for the current process. # Register as a trusted process with the Local Security Authority (LSA). # Create a new logon for the user passed on the command line. # Load/Create a profile on the local machine for the new logon. # Create a new environment for the new logon. # Launch the new process in a job with the task name specified and using the created logon. # Wait for the JOB to exit. h2. Future work: The following work was scoped out of this check in: * Support for non-domain users or machine that are not domain joined. * Support for privilege isolation by running the task launcher in a high privilege service with access over an ACLed named pipe. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1063) Winutils needs ability to create task as domain user
[
https://issues.apache.org/jira/browse/YARN-1063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14007485#comment-14007485
]
Ivan Mitic commented on YARN-1063:
--
I have already reviewed a version of the initial patch (YARN-1063.patch). Copy
pasting the full list of comments for documentation purposes on the Jira.
First round:
1. winutils.h: You have a duplicate EnablePrivilege declaration. Please remove
the one with BOOL.
2. winutils.h: Convention in the file is to use CamelCased function names.
Please name your functions appropriately. Also, a nit, no space between '(' and
function args. Same comment across the board.
3. libwinutils.c#EnablePrivilege:
{code}
{
ReportErrorCode(LLookupPrivilegeValue, GetLastError());
CloseHandle(hToken);
return GetLastError();
}
{code}
You shouldn't be calling GetLastError() twice above. CloseHandle() might reset
it to 0 or some other value. Can you change all error code paths in this
function to first assign GetLastError() to a local variable, and then log it
and do other things. E.g.
{code}
{
dwErrorCode = GetLastError();
ReportErrorCode(LLookupPrivilegeValue, dwErrorCode);
CloseHandle(hToken);
return dwErrorCode;
}
{code}
3. Something is wrong with this comment
{code}
// Function: assignLsaString
//
// Description:
// fills in values of LSA_STRING struct to point to a string buffer
{code}
4. void assignLsaString( __in LSA_STRING * target, __in const char *strBuf )
Is target an __inout parameter?
5. libwinutils.c: Mixed tabs and spaces. Please use 2 space ident across the
board.
6. libwinutils.c: authentication pacakage - typo
7. libwinutils.c: Should the constant be named MICROSOFT_KERBEROS_NAME or there
is something else more appropriate?
8. libwinutils.c: GetNameFromLogonToken: You can assert that first
GetTokenInformation returns false. Don't do assert(GetTokenInformation() ==
FALSE) :)
9. I don't believe that calloc() sets the last error. If the return value is
NULL, you should assume/error-with ERROR_NOT_ENOUGH_MEMORY. Applies to all
places.
10. libwinutils.c: LookupAccountSid: Do you need to allocate userNameSize+1 and
domainNameSize+1 buffer sizes, or is this already accounted for?
11. task.c: You are not checking result of GetCurrentDirectory(). If you expect
the method to always succeed, you can assert that the result 0.
12. task.c: Please keep CreateProcessAsUser and old CreateProcess codepaths
separate. I don't think it is trivial to prove that the new code with
CreateProcessAsUser has exactly the same sematic as the old code.
13. task.c: I don't understand the need to TerminateJobObject when
CreateProcessAsUser failed? Why wouldn't the regular return code path exit the
process with non zero code?
14. Consider using dwErrorCode in your functions to track status of the win
error codes.
15. task.c: createTaskAsUser: Do you need to check if lsaHandle is valid to be
sent to unregisterWithLsa. You can initialize to INVALID_HANDLE_VALUE
16. task.c: Task: Can you please rename size to cmdLineSize. argLen to
crtArgLen. Btw, is size needed?
17. task.c: Task: Can you please rename ARGC_GROUPID to ARGC_JOBOBJECTNAME or
something alike.
18. task.c: TaskUsage: Your command line format does not seem to be valid. You
are missing jobobject and pidfile.
19. task.c:
{code}
cmdLine = argv[ARGC_COMMAND];
if (argc ARGC_COMMAND_ARGS) {
crtArg = ARGC_COMMAND;
insertHere = buffer;
while (crtArg argc) {
argLen = wcslen(argv[crtArg]);
wcscat(insertHere, argv[crtArg]);
insertHere += argLen;
size -= argLen;
insertHere[0] = L' ';
insertHere += 1;
size -= 1;
insertHere[0] = 0;
++crtArg;
}
cmdLine = buffer;
}
{code}
Do you mind adding a short comment on what you're doing.
20.
bq. add PIDFILE argument to task createAsUser. Pidfile must be created by the
task controller just before launching the process.
Can you comment on the rationale?
21.
bq. accept arbitrary arguments to pass to the process launched. We use the
launcher for both container and localizer and that requires variable arguments.
How this works today? Is the localizer using something else?
22. We haven't written a whole lot of unittests for winutils so far (Check
testwinutils.java). I'll let you make the call on whether this could/should be
unittested and if yes, what is appropriate. I will sign off anyhow.
Second round:
1. I still see GetLastError() being called twice in EnablePrivilege. Can you
please use dwErrorCode instead of the second call.
{code}
dwErrCode = GetLastError();
ReportErrorCode(LOpenProcessToken, GetLastError());
{code}
2. You missed changing the SAL annoation in the header file
{code}
void AssignLsaString(__inout LSA_STRING * target, __in const char *strBuf)
{code}
3.
[jira] [Commented] (YARN-1063) Winutils needs ability to create task as domain user
[ https://issues.apache.org/jira/browse/YARN-1063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14007490#comment-14007490 ] Ivan Mitic commented on YARN-1063: -- Thanks Remus for addressing all comments in the latest patch (YARN-1063.4.patch). Looks good to me, +1 I will give others a couple of days to provide additional feedback and then commit the patch. Winutils needs ability to create task as domain user Key: YARN-1063 URL: https://issues.apache.org/jira/browse/YARN-1063 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Environment: Windows Reporter: Kyle Leckie Assignee: Remus Rusanu Labels: security, windows Attachments: YARN-1063.2.patch, YARN-1063.3.patch, YARN-1063.4.patch, YARN-1063.patch h1. Summary: Securing a Hadoop cluster requires constructing some form of security boundary around the processes executed in YARN containers. Isolation based on Windows user isolation seems most feasible. This approach is similar to the approach taken by the existing LinuxContainerExecutor. The current patch to winutils.exe adds the ability to create a process as a domain user. h1. Alternative Methods considered: h2. Process rights limited by security token restriction: On Windows access decisions are made by examining the security token of a process. It is possible to spawn a process with a restricted security token. Any of the rights granted by SIDs of the default token may be restricted. It is possible to see this in action by examining the security tone of a sandboxed process launch be a web browser. Typically the launched process will have a fully restricted token and need to access machine resources through a dedicated broker process that enforces a custom security policy. This broker process mechanism would break compatibility with the typical Hadoop container process. The Container process must be able to utilize standard function calls for disk and network IO. I performed some work looking at ways to ACL the local files to the specific launched without granting rights to other processes launched on the same machine but found this to be an overly complex solution. h2. Relying on APP containers: Recent versions of windows have the ability to launch processes within an isolated container. Application containers are supported for execution of WinRT based executables. This method was ruled out due to the lack of official support for standard windows APIs. At some point in the future windows may support functionality similar to BSD jails or Linux containers, at that point support for containers should be added. h1. Create As User Feature Description: h2. Usage: A new sub command was added to the set of task commands. Here is the syntax: winutils task createAsUser [TASKNAME] [USERNAME] [COMMAND_LINE] Some notes: * The username specified is in the format of user@domain * The machine executing this command must be joined to the domain of the user specified * The domain controller must allow the account executing the command access to the user information. For this join the account to the predefined group labeled Pre-Windows 2000 Compatible Access * The account running the command must have several rights on the local machine. These can be managed manually using secpol.msc: ** Act as part of the operating system - SE_TCB_NAME ** Replace a process-level token - SE_ASSIGNPRIMARYTOKEN_NAME ** Adjust memory quotas for a process - SE_INCREASE_QUOTA_NAME * The launched process will not have rights to the desktop so will not be able to display any information or create UI. * The launched process will have no network credentials. Any access of network resources that requires domain authentication will fail. h2. Implementation: Winutils performs the following steps: # Enable the required privileges for the current process. # Register as a trusted process with the Local Security Authority (LSA). # Create a new logon for the user passed on the command line. # Load/Create a profile on the local machine for the new logon. # Create a new environment for the new logon. # Launch the new process in a job with the task name specified and using the created logon. # Wait for the JOB to exit. h2. Future work: The following work was scoped out of this check in: * Support for non-domain users or machine that are not domain joined. * Support for privilege isolation by running the task launcher in a high privilege service with access over an ACLed named pipe. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1063) Winutils needs ability to create task as domain user
[
https://issues.apache.org/jira/browse/YARN-1063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13983136#comment-13983136
]
Hadoop QA commented on YARN-1063:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12642248/YARN-1063.4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3640//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3640//console
This message is automatically generated.
Winutils needs ability to create task as domain user
Key: YARN-1063
URL: https://issues.apache.org/jira/browse/YARN-1063
Project: Hadoop YARN
Issue Type: Sub-task
Components: nodemanager
Environment: Windows
Reporter: Kyle Leckie
Assignee: Remus Rusanu
Labels: security, windows
Attachments: YARN-1063.2.patch, YARN-1063.3.patch, YARN-1063.4.patch,
YARN-1063.patch
h1. Summary:
Securing a Hadoop cluster requires constructing some form of security
boundary around the processes executed in YARN containers. Isolation based on
Windows user isolation seems most feasible. This approach is similar to the
approach taken by the existing LinuxContainerExecutor. The current patch to
winutils.exe adds the ability to create a process as a domain user.
h1. Alternative Methods considered:
h2. Process rights limited by security token restriction:
On Windows access decisions are made by examining the security token of a
process. It is possible to spawn a process with a restricted security token.
Any of the rights granted by SIDs of the default token may be restricted. It
is possible to see this in action by examining the security tone of a
sandboxed process launch be a web browser. Typically the launched process
will have a fully restricted token and need to access machine resources
through a dedicated broker process that enforces a custom security policy.
This broker process mechanism would break compatibility with the typical
Hadoop container process. The Container process must be able to utilize
standard function calls for disk and network IO. I performed some work
looking at ways to ACL the local files to the specific launched without
granting rights to other processes launched on the same machine but found
this to be an overly complex solution.
h2. Relying on APP containers:
Recent versions of windows have the ability to launch processes within an
isolated container. Application containers are supported for execution of
WinRT based executables. This method was ruled out due to the lack of
official support for standard windows APIs. At some point in the future
windows may support functionality similar to BSD jails or Linux containers,
at that point support for containers should be added.
h1. Create As User Feature Description:
h2. Usage:
A new sub command was added to the set of task commands. Here is the syntax:
winutils task createAsUser [TASKNAME] [USERNAME] [COMMAND_LINE]
Some notes:
* The username specified is in the format of user@domain
* The machine executing this command must be joined to the domain of the user
specified
* The domain controller must allow the account executing the command access
to the user information. For this join the account to the predefined group
labeled Pre-Windows 2000 Compatible Access
* The account running the command must have several rights on the local
machine. These can be managed manually using secpol.msc:
** Act as part of the operating system - SE_TCB_NAME
** Replace a process-level token - SE_ASSIGNPRIMARYTOKEN_NAME
** Adjust memory quotas for a process - SE_INCREASE_QUOTA_NAME
* The launched process will not have rights to the
[jira] [Commented] (YARN-1063) Winutils needs ability to create task as domain user
[ https://issues.apache.org/jira/browse/YARN-1063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13980754#comment-13980754 ] Remus Rusanu commented on YARN-1063: patch applies fine on trunk for me. Not sure why it failed for Mr. Jenkins. I removed the trunk-win tags. Winutils needs ability to create task as domain user Key: YARN-1063 URL: https://issues.apache.org/jira/browse/YARN-1063 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Environment: Windows Reporter: Kyle Leckie Assignee: Remus Rusanu Labels: security, windows Attachments: YARN-1063.2.patch, YARN-1063.patch h1. Summary: Securing a Hadoop cluster requires constructing some form of security boundary around the processes executed in YARN containers. Isolation based on Windows user isolation seems most feasible. This approach is similar to the approach taken by the existing LinuxContainerExecutor. The current patch to winutils.exe adds the ability to create a process as a domain user. h1. Alternative Methods considered: h2. Process rights limited by security token restriction: On Windows access decisions are made by examining the security token of a process. It is possible to spawn a process with a restricted security token. Any of the rights granted by SIDs of the default token may be restricted. It is possible to see this in action by examining the security tone of a sandboxed process launch be a web browser. Typically the launched process will have a fully restricted token and need to access machine resources through a dedicated broker process that enforces a custom security policy. This broker process mechanism would break compatibility with the typical Hadoop container process. The Container process must be able to utilize standard function calls for disk and network IO. I performed some work looking at ways to ACL the local files to the specific launched without granting rights to other processes launched on the same machine but found this to be an overly complex solution. h2. Relying on APP containers: Recent versions of windows have the ability to launch processes within an isolated container. Application containers are supported for execution of WinRT based executables. This method was ruled out due to the lack of official support for standard windows APIs. At some point in the future windows may support functionality similar to BSD jails or Linux containers, at that point support for containers should be added. h1. Create As User Feature Description: h2. Usage: A new sub command was added to the set of task commands. Here is the syntax: winutils task createAsUser [TASKNAME] [USERNAME] [COMMAND_LINE] Some notes: * The username specified is in the format of user@domain * The machine executing this command must be joined to the domain of the user specified * The domain controller must allow the account executing the command access to the user information. For this join the account to the predefined group labeled Pre-Windows 2000 Compatible Access * The account running the command must have several rights on the local machine. These can be managed manually using secpol.msc: ** Act as part of the operating system - SE_TCB_NAME ** Replace a process-level token - SE_ASSIGNPRIMARYTOKEN_NAME ** Adjust memory quotas for a process - SE_INCREASE_QUOTA_NAME * The launched process will not have rights to the desktop so will not be able to display any information or create UI. * The launched process will have no network credentials. Any access of network resources that requires domain authentication will fail. h2. Implementation: Winutils performs the following steps: # Enable the required privileges for the current process. # Register as a trusted process with the Local Security Authority (LSA). # Create a new logon for the user passed on the command line. # Load/Create a profile on the local machine for the new logon. # Create a new environment for the new logon. # Launch the new process in a job with the task name specified and using the created logon. # Wait for the JOB to exit. h2. Future work: The following work was scoped out of this check in: * Support for non-domain users or machine that are not domain joined. * Support for privilege isolation by running the task launcher in a high privilege service with access over an ACLed named pipe. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1063) Winutils needs ability to create task as domain user
[
https://issues.apache.org/jira/browse/YARN-1063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13981445#comment-13981445
]
Hadoop QA commented on YARN-1063:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12641970/YARN-1063.3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 1279 javac
compiler warnings (more than the trunk's current 1278 warnings).
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3630//testReport/
Javac warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/3630//artifact/trunk/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3630//console
This message is automatically generated.
Winutils needs ability to create task as domain user
Key: YARN-1063
URL: https://issues.apache.org/jira/browse/YARN-1063
Project: Hadoop YARN
Issue Type: Sub-task
Components: nodemanager
Environment: Windows
Reporter: Kyle Leckie
Assignee: Remus Rusanu
Labels: security, windows
Attachments: YARN-1063.2.patch, YARN-1063.3.patch, YARN-1063.patch
h1. Summary:
Securing a Hadoop cluster requires constructing some form of security
boundary around the processes executed in YARN containers. Isolation based on
Windows user isolation seems most feasible. This approach is similar to the
approach taken by the existing LinuxContainerExecutor. The current patch to
winutils.exe adds the ability to create a process as a domain user.
h1. Alternative Methods considered:
h2. Process rights limited by security token restriction:
On Windows access decisions are made by examining the security token of a
process. It is possible to spawn a process with a restricted security token.
Any of the rights granted by SIDs of the default token may be restricted. It
is possible to see this in action by examining the security tone of a
sandboxed process launch be a web browser. Typically the launched process
will have a fully restricted token and need to access machine resources
through a dedicated broker process that enforces a custom security policy.
This broker process mechanism would break compatibility with the typical
Hadoop container process. The Container process must be able to utilize
standard function calls for disk and network IO. I performed some work
looking at ways to ACL the local files to the specific launched without
granting rights to other processes launched on the same machine but found
this to be an overly complex solution.
h2. Relying on APP containers:
Recent versions of windows have the ability to launch processes within an
isolated container. Application containers are supported for execution of
WinRT based executables. This method was ruled out due to the lack of
official support for standard windows APIs. At some point in the future
windows may support functionality similar to BSD jails or Linux containers,
at that point support for containers should be added.
h1. Create As User Feature Description:
h2. Usage:
A new sub command was added to the set of task commands. Here is the syntax:
winutils task createAsUser [TASKNAME] [USERNAME] [COMMAND_LINE]
Some notes:
* The username specified is in the format of user@domain
* The machine executing this command must be joined to the domain of the user
specified
* The domain controller must allow the account executing the command access
to the user information. For this join the account to the predefined group
labeled Pre-Windows 2000 Compatible Access
* The account running the command must have several rights on the local
machine. These can be managed manually using secpol.msc:
** Act as part of the operating system - SE_TCB_NAME
** Replace a process-level token - SE_ASSIGNPRIMARYTOKEN_NAME
** Adjust memory quotas for a process - SE_INCREASE_QUOTA_NAME
* The
[jira] [Commented] (YARN-1063) Winutils needs ability to create task as domain user
[ https://issues.apache.org/jira/browse/YARN-1063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13979795#comment-13979795 ] Remus Rusanu commented on YARN-1063: I've uploaded a refreshed patch. Together with the patch that I'll (soon) upload on YARN-1972 it provides the necessary core changes to support secure clusters on Windows. Winutils needs ability to create task as domain user Key: YARN-1063 URL: https://issues.apache.org/jira/browse/YARN-1063 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Affects Versions: trunk-win Environment: Windows Reporter: Kyle Leckie Labels: security Fix For: trunk-win Attachments: YARN-1063.2.patch, YARN-1063.patch h1. Summary: Securing a Hadoop cluster requires constructing some form of security boundary around the processes executed in YARN containers. Isolation based on Windows user isolation seems most feasible. This approach is similar to the approach taken by the existing LinuxContainerExecutor. The current patch to winutils.exe adds the ability to create a process as a domain user. h1. Alternative Methods considered: h2. Process rights limited by security token restriction: On Windows access decisions are made by examining the security token of a process. It is possible to spawn a process with a restricted security token. Any of the rights granted by SIDs of the default token may be restricted. It is possible to see this in action by examining the security tone of a sandboxed process launch be a web browser. Typically the launched process will have a fully restricted token and need to access machine resources through a dedicated broker process that enforces a custom security policy. This broker process mechanism would break compatibility with the typical Hadoop container process. The Container process must be able to utilize standard function calls for disk and network IO. I performed some work looking at ways to ACL the local files to the specific launched without granting rights to other processes launched on the same machine but found this to be an overly complex solution. h2. Relying on APP containers: Recent versions of windows have the ability to launch processes within an isolated container. Application containers are supported for execution of WinRT based executables. This method was ruled out due to the lack of official support for standard windows APIs. At some point in the future windows may support functionality similar to BSD jails or Linux containers, at that point support for containers should be added. h1. Create As User Feature Description: h2. Usage: A new sub command was added to the set of task commands. Here is the syntax: winutils task createAsUser [TASKNAME] [USERNAME] [COMMAND_LINE] Some notes: * The username specified is in the format of user@domain * The machine executing this command must be joined to the domain of the user specified * The domain controller must allow the account executing the command access to the user information. For this join the account to the predefined group labeled Pre-Windows 2000 Compatible Access * The account running the command must have several rights on the local machine. These can be managed manually using secpol.msc: ** Act as part of the operating system - SE_TCB_NAME ** Replace a process-level token - SE_ASSIGNPRIMARYTOKEN_NAME ** Adjust memory quotas for a process - SE_INCREASE_QUOTA_NAME * The launched process will not have rights to the desktop so will not be able to display any information or create UI. * The launched process will have no network credentials. Any access of network resources that requires domain authentication will fail. h2. Implementation: Winutils performs the following steps: # Enable the required privileges for the current process. # Register as a trusted process with the Local Security Authority (LSA). # Create a new logon for the user passed on the command line. # Load/Create a profile on the local machine for the new logon. # Create a new environment for the new logon. # Launch the new process in a job with the task name specified and using the created logon. # Wait for the JOB to exit. h2. Future work: The following work was scoped out of this check in: * Support for non-domain users or machine that are not domain joined. * Support for privilege isolation by running the task launcher in a high privilege service with access over an ACLed named pipe. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1063) Winutils needs ability to create task as domain user
[
https://issues.apache.org/jira/browse/YARN-1063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13979800#comment-13979800
]
Hadoop QA commented on YARN-1063:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12641722/YARN-1063.2.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3622//console
This message is automatically generated.
Winutils needs ability to create task as domain user
Key: YARN-1063
URL: https://issues.apache.org/jira/browse/YARN-1063
Project: Hadoop YARN
Issue Type: Sub-task
Components: nodemanager
Affects Versions: trunk-win
Environment: Windows
Reporter: Kyle Leckie
Labels: security
Fix For: trunk-win
Attachments: YARN-1063.2.patch, YARN-1063.patch
h1. Summary:
Securing a Hadoop cluster requires constructing some form of security
boundary around the processes executed in YARN containers. Isolation based on
Windows user isolation seems most feasible. This approach is similar to the
approach taken by the existing LinuxContainerExecutor. The current patch to
winutils.exe adds the ability to create a process as a domain user.
h1. Alternative Methods considered:
h2. Process rights limited by security token restriction:
On Windows access decisions are made by examining the security token of a
process. It is possible to spawn a process with a restricted security token.
Any of the rights granted by SIDs of the default token may be restricted. It
is possible to see this in action by examining the security tone of a
sandboxed process launch be a web browser. Typically the launched process
will have a fully restricted token and need to access machine resources
through a dedicated broker process that enforces a custom security policy.
This broker process mechanism would break compatibility with the typical
Hadoop container process. The Container process must be able to utilize
standard function calls for disk and network IO. I performed some work
looking at ways to ACL the local files to the specific launched without
granting rights to other processes launched on the same machine but found
this to be an overly complex solution.
h2. Relying on APP containers:
Recent versions of windows have the ability to launch processes within an
isolated container. Application containers are supported for execution of
WinRT based executables. This method was ruled out due to the lack of
official support for standard windows APIs. At some point in the future
windows may support functionality similar to BSD jails or Linux containers,
at that point support for containers should be added.
h1. Create As User Feature Description:
h2. Usage:
A new sub command was added to the set of task commands. Here is the syntax:
winutils task createAsUser [TASKNAME] [USERNAME] [COMMAND_LINE]
Some notes:
* The username specified is in the format of user@domain
* The machine executing this command must be joined to the domain of the user
specified
* The domain controller must allow the account executing the command access
to the user information. For this join the account to the predefined group
labeled Pre-Windows 2000 Compatible Access
* The account running the command must have several rights on the local
machine. These can be managed manually using secpol.msc:
** Act as part of the operating system - SE_TCB_NAME
** Replace a process-level token - SE_ASSIGNPRIMARYTOKEN_NAME
** Adjust memory quotas for a process - SE_INCREASE_QUOTA_NAME
* The launched process will not have rights to the desktop so will not be
able to display any information or create UI.
* The launched process will have no network credentials. Any access of
network resources that requires domain authentication will fail.
h2. Implementation:
Winutils performs the following steps:
# Enable the required privileges for the current process.
# Register as a trusted process with the Local Security Authority (LSA).
# Create a new logon for the user passed on the command line.
# Load/Create a profile on the local machine for the new logon.
# Create a new environment for the new logon.
# Launch the new process in a job with the task name specified and using the
created logon.
# Wait for the JOB to exit.
h2. Future work:
The following work was scoped out of this check in:
* Support for non-domain users or machine that are not domain joined.
* Support for privilege isolation by running the task launcher in a high
privilege service with access over an ACLed named pipe.
--
This message was sent by
[jira] [Commented] (YARN-1063) Winutils needs ability to create task as domain user
[
https://issues.apache.org/jira/browse/YARN-1063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13976628#comment-13976628
]
Hadoop QA commented on YARN-1063:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12600298/YARN-1063.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3606//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3606//console
This message is automatically generated.
Winutils needs ability to create task as domain user
Key: YARN-1063
URL: https://issues.apache.org/jira/browse/YARN-1063
Project: Hadoop YARN
Issue Type: Sub-task
Components: nodemanager
Affects Versions: trunk-win
Environment: Windows
Reporter: Kyle Leckie
Labels: security
Fix For: trunk-win
Attachments: YARN-1063.patch
h1. Summary:
Securing a Hadoop cluster requires constructing some form of security
boundary around the processes executed in YARN containers. Isolation based on
Windows user isolation seems most feasible. This approach is similar to the
approach taken by the existing LinuxContainerExecutor. The current patch to
winutils.exe adds the ability to create a process as a domain user.
h1. Alternative Methods considered:
h2. Process rights limited by security token restriction:
On Windows access decisions are made by examining the security token of a
process. It is possible to spawn a process with a restricted security token.
Any of the rights granted by SIDs of the default token may be restricted. It
is possible to see this in action by examining the security tone of a
sandboxed process launch be a web browser. Typically the launched process
will have a fully restricted token and need to access machine resources
through a dedicated broker process that enforces a custom security policy.
This broker process mechanism would break compatibility with the typical
Hadoop container process. The Container process must be able to utilize
standard function calls for disk and network IO. I performed some work
looking at ways to ACL the local files to the specific launched without
granting rights to other processes launched on the same machine but found
this to be an overly complex solution.
h2. Relying on APP containers:
Recent versions of windows have the ability to launch processes within an
isolated container. Application containers are supported for execution of
WinRT based executables. This method was ruled out due to the lack of
official support for standard windows APIs. At some point in the future
windows may support functionality similar to BSD jails or Linux containers,
at that point support for containers should be added.
h1. Create As User Feature Description:
h2. Usage:
A new sub command was added to the set of task commands. Here is the syntax:
winutils task createAsUser [TASKNAME] [USERNAME] [COMMAND_LINE]
Some notes:
* The username specified is in the format of user@domain
* The machine executing this command must be joined to the domain of the user
specified
* The domain controller must allow the account executing the command access
to the user information. For this join the account to the predefined group
labeled Pre-Windows 2000 Compatible Access
* The account running the command must have several rights on the local
machine. These can be managed manually using secpol.msc:
** Act as part of the operating system - SE_TCB_NAME
** Replace a process-level token - SE_ASSIGNPRIMARYTOKEN_NAME
** Adjust memory quotas for a process -
[jira] [Commented] (YARN-1063) Winutils needs ability to create task as domain user
[ https://issues.apache.org/jira/browse/YARN-1063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13827776#comment-13827776 ] Larry McCay commented on YARN-1063: --- Hi Kyle, I am curious about the status of this work. Winutils needs ability to create task as domain user Key: YARN-1063 URL: https://issues.apache.org/jira/browse/YARN-1063 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Affects Versions: trunk-win Environment: Windows Reporter: Kyle Leckie Labels: security Fix For: trunk-win Attachments: YARN-1063.patch h1. Summary: Securing a Hadoop cluster requires constructing some form of security boundary around the processes executed in YARN containers. Isolation based on Windows user isolation seems most feasible. This approach is similar to the approach taken by the existing LinuxContainerExecutor. The current patch to winutils.exe adds the ability to create a process as a domain user. h1. Alternative Methods considered: h2. Process rights limited by security token restriction: On Windows access decisions are made by examining the security token of a process. It is possible to spawn a process with a restricted security token. Any of the rights granted by SIDs of the default token may be restricted. It is possible to see this in action by examining the security tone of a sandboxed process launch be a web browser. Typically the launched process will have a fully restricted token and need to access machine resources through a dedicated broker process that enforces a custom security policy. This broker process mechanism would break compatibility with the typical Hadoop container process. The Container process must be able to utilize standard function calls for disk and network IO. I performed some work looking at ways to ACL the local files to the specific launched without granting rights to other processes launched on the same machine but found this to be an overly complex solution. h2. Relying on APP containers: Recent versions of windows have the ability to launch processes within an isolated container. Application containers are supported for execution of WinRT based executables. This method was ruled out due to the lack of official support for standard windows APIs. At some point in the future windows may support functionality similar to BSD jails or Linux containers, at that point support for containers should be added. h1. Create As User Feature Description: h2. Usage: A new sub command was added to the set of task commands. Here is the syntax: winutils task createAsUser [TASKNAME] [USERNAME] [COMMAND_LINE] Some notes: * The username specified is in the format of user@domain * The machine executing this command must be joined to the domain of the user specified * The domain controller must allow the account executing the command access to the user information. For this join the account to the predefined group labeled Pre-Windows 2000 Compatible Access * The account running the command must have several rights on the local machine. These can be managed manually using secpol.msc: ** Act as part of the operating system - SE_TCB_NAME ** Replace a process-level token - SE_ASSIGNPRIMARYTOKEN_NAME ** Adjust memory quotas for a process - SE_INCREASE_QUOTA_NAME * The launched process will not have rights to the desktop so will not be able to display any information or create UI. * The launched process will have no network credentials. Any access of network resources that requires domain authentication will fail. h2. Implementation: Winutils performs the following steps: # Enable the required privileges for the current process. # Register as a trusted process with the Local Security Authority (LSA). # Create a new logon for the user passed on the command line. # Load/Create a profile on the local machine for the new logon. # Create a new environment for the new logon. # Launch the new process in a job with the task name specified and using the created logon. # Wait for the JOB to exit. h2. Future work: The following work was scoped out of this check in: * Support for non-domain users or machine that are not domain joined. * Support for privilege isolation by running the task launcher in a high privilege service with access over an ACLed named pipe. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1063) Winutils needs ability to create task as domain user
[ https://issues.apache.org/jira/browse/YARN-1063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13770984#comment-13770984 ] Kyle Leckie commented on YARN-1063: --- Hi Bikas, An isolated container is one of the requirements for a secure cluster. I have not evaluated utilizing this in a non-secure cluster. The issue you would run into is that in a non-secure cluster a user may run jobs in a user context that does not match a user on the domain controller. The Node Manager runs the winutils exe in order to perform the task launch. This mirrors the behaviour in a secure Linux cluster with the container-executor binary that has setuid enabled. Winutils needs ability to create task as domain user Key: YARN-1063 URL: https://issues.apache.org/jira/browse/YARN-1063 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Affects Versions: trunk-win Environment: Windows Reporter: Kyle Leckie Labels: security Fix For: trunk-win Attachments: YARN-1063.patch h1. Summary: Securing a Hadoop cluster requires constructing some form of security boundary around the processes executed in YARN containers. Isolation based on Windows user isolation seems most feasible. This approach is similar to the approach taken by the existing LinuxContainerExecutor. The current patch to winutils.exe adds the ability to create a process as a domain user. h1. Alternative Methods considered: h2. Process rights limited by security token restriction: On Windows access decisions are made by examining the security token of a process. It is possible to spawn a process with a restricted security token. Any of the rights granted by SIDs of the default token may be restricted. It is possible to see this in action by examining the security tone of a sandboxed process launch be a web browser. Typically the launched process will have a fully restricted token and need to access machine resources through a dedicated broker process that enforces a custom security policy. This broker process mechanism would break compatibility with the typical Hadoop container process. The Container process must be able to utilize standard function calls for disk and network IO. I performed some work looking at ways to ACL the local files to the specific launched without granting rights to other processes launched on the same machine but found this to be an overly complex solution. h2. Relying on APP containers: Recent versions of windows have the ability to launch processes within an isolated container. Application containers are supported for execution of WinRT based executables. This method was ruled out due to the lack of official support for standard windows APIs. At some point in the future windows may support functionality similar to BSD jails or Linux containers, at that point support for containers should be added. h1. Create As User Feature Description: h2. Usage: A new sub command was added to the set of task commands. Here is the syntax: winutils task createAsUser [TASKNAME] [USERNAME] [COMMAND_LINE] Some notes: * The username specified is in the format of user@domain * The machine executing this command must be joined to the domain of the user specified * The domain controller must allow the account executing the command access to the user information. For this join the account to the predefined group labeled Pre-Windows 2000 Compatible Access * The account running the command must have several rights on the local machine. These can be managed manually using secpol.msc: ** Act as part of the operating system - SE_TCB_NAME ** Replace a process-level token - SE_ASSIGNPRIMARYTOKEN_NAME ** Adjust memory quotas for a process - SE_INCREASE_QUOTA_NAME * The launched process will not have rights to the desktop so will not be able to display any information or create UI. * The launched process will have no network credentials. Any access of network resources that requires domain authentication will fail. h2. Implementation: Winutils performs the following steps: # Enable the required privileges for the current process. # Register as a trusted process with the Local Security Authority (LSA). # Create a new logon for the user passed on the command line. # Load/Create a profile on the local machine for the new logon. # Create a new environment for the new logon. # Launch the new process in a job with the task name specified and using the created logon. # Wait for the JOB to exit. h2. Future work: The following work was scoped out of this check in: * Support for non-domain users or machine that are not domain joined. * Support for privilege isolation by running the task launcher in a
[jira] [Commented] (YARN-1063) Winutils needs ability to create task as domain user
[ https://issues.apache.org/jira/browse/YARN-1063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13757994#comment-13757994 ] Bikas Saha commented on YARN-1063: -- Will this be used in secure and non-secure clusters? I dont think I fully understood the privileges of the launcher. By that do we mean the TaskTracker/NodeManager or the winutils process that is launched by the TT/NM. If its the TT/NM then do we end up having a long-running Hadoop service with elevated privileges? Winutils needs ability to create task as domain user Key: YARN-1063 URL: https://issues.apache.org/jira/browse/YARN-1063 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Affects Versions: trunk-win Environment: Windows Reporter: Kyle Leckie Labels: security Fix For: trunk-win Attachments: YARN-1063.patch h1. Summary: Securing a Hadoop cluster requires constructing some form of security boundary around the processes executed in YARN containers. Isolation based on Windows user isolation seems most feasible. This approach is similar to the approach taken by the existing LinuxContainerExecutor. The current patch to winutils.exe adds the ability to create a process as a domain user. h1. Alternative Methods considered: h2. Process rights limited by security token restriction: On Windows access decisions are made by examining the security token of a process. It is possible to spawn a process with a restricted security token. Any of the rights granted by SIDs of the default token may be restricted. It is possible to see this in action by examining the security tone of a sandboxed process launch be a web browser. Typically the launched process will have a fully restricted token and need to access machine resources through a dedicated broker process that enforces a custom security policy. This broker process mechanism would break compatibility with the typical Hadoop container process. The Container process must be able to utilize standard function calls for disk and network IO. I performed some work looking at ways to ACL the local files to the specific launched without granting rights to other processes launched on the same machine but found this to be an overly complex solution. h2. Relying on APP containers: Recent versions of windows have the ability to launch processes within an isolated container. Application containers are supported for execution of WinRT based executables. This method was ruled out due to the lack of official support for standard windows APIs. At some point in the future windows may support functionality similar to BSD jails or Linux containers, at that point support for containers should be added. h1. Create As User Feature Description: h2. Usage: A new sub command was added to the set of task commands. Here is the syntax: winutils task createAsUser [TASKNAME] [USERNAME] [COMMAND_LINE] Some notes: * The username specified is in the format of user@domain * The machine executing this command must be joined to the domain of the user specified * The domain controller must allow the account executing the command access to the user information. For this join the account to the predefined group labeled Pre-Windows 2000 Compatible Access * The account running the command must have several rights on the local machine. These can be managed manually using secpol.msc: ** Act as part of the operating system - SE_TCB_NAME ** Replace a process-level token - SE_ASSIGNPRIMARYTOKEN_NAME ** Adjust memory quotas for a process - SE_INCREASE_QUOTA_NAME * The launched process will not have rights to the desktop so will not be able to display any information or create UI. * The launched process will have no network credentials. Any access of network resources that requires domain authentication will fail. h2. Implementation: Winutils performs the following steps: # Enable the required privileges for the current process. # Register as a trusted process with the Local Security Authority (LSA). # Create a new logon for the user passed on the command line. # Load/Create a profile on the local machine for the new logon. # Create a new environment for the new logon. # Launch the new process in a job with the task name specified and using the created logon. # Wait for the JOB to exit. h2. Future work: The following work was scoped out of this check in: * Support for non-domain users or machine that are not domain joined. * Support for privilege isolation by running the task launcher in a high privilege service with access over an ACLed named pipe. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact
[jira] [Commented] (YARN-1063) Winutils needs ability to create task as domain user
[ https://issues.apache.org/jira/browse/YARN-1063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13752656#comment-13752656 ] Kyle Leckie commented on YARN-1063: --- Thanks Chuan, A rebase into my branch fixed the patch issue. Winutils needs ability to create task as domain user Key: YARN-1063 URL: https://issues.apache.org/jira/browse/YARN-1063 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Affects Versions: trunk-win Environment: Windows Reporter: Kyle Leckie Labels: security Fix For: trunk-win Attachments: YARN-1063.patch h1. Summary: Securing a Hadoop cluster requires constructing some form of security boundary around the processes executed in YARN containers. Isolation based on Windows user isolation seems most feasible. This approach is similar to the approach taken by the existing LinuxContainerExecutor. The current patch to winutils.exe adds the ability to create a process as a domain user. h1. Alternative Methods considered: h2. Process rights limited by security token restriction: On Windows access decisions are made by examining the security token of a process. It is possible to spawn a process with a restricted security token. Any of the rights granted by SIDs of the default token may be restricted. It is possible to see this in action by examining the security tone of a sandboxed process launch be a web browser. Typically the launched process will have a fully restricted token and need to access machine resources through a dedicated broker process that enforces a custom security policy. This broker process mechanism would break compatibility with the typical Hadoop container process. The Container process must be able to utilize standard function calls for disk and network IO. I performed some work looking at ways to ACL the local files to the specific launched without granting rights to other processes launched on the same machine but found this to be an overly complex solution. h2. Relying on APP containers: Recent versions of windows have the ability to launch processes within an isolated container. Application containers are supported for execution of WinRT based executables. This method was ruled out due to the lack of official support for standard windows APIs. At some point in the future windows may support functionality similar to BSD jails or Linux containers, at that point support for containers should be added. h1. Create As User Feature Description: h2. Usage: A new sub command was added to the set of task commands. Here is the syntax: winutils task createAsUser [TASKNAME] [USERNAME] [COMMAND_LINE] Some notes: * The username specified is in the format of user@domain * The machine executing this command must be joined to the domain of the user specified * The domain controller must allow the account executing the command access to the user information. For this join the account to the predefined group labeled Pre-Windows 2000 Compatible Access * The account running the command must have several rights on the local machine. These can be managed manually using secpol.msc: ** Act as part of the operating system - SE_TCB_NAME ** Replace a process-level token - SE_ASSIGNPRIMARYTOKEN_NAME ** Adjust memory quotas for a process - SE_INCREASE_QUOTA_NAME * The launched process will not have rights to the desktop so will not be able to display any information or create UI. * The launched process will have no network credentials. Any access of network resources that requires domain authentication will fail. h2. Implementation: Winutils performs the following steps: # Enable the required privileges for the current process. # Register as a trusted process with the Local Security Authority (LSA). # Create a new logon for the user passed on the command line. # Load/Create a profile on the local machine for the new logon. # Create a new environment for the new logon. # Launch the new process in a job with the task name specified and using the created logon. # Wait for the JOB to exit. h2. Future work: The following work was scoped out of this check in: * Support for non-domain users or machine that are not domain joined. * Support for privilege isolation by running the task launcher in a high privilege service with access over an ACLed named pipe. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1063) Winutils needs ability to create task as domain user
[
https://issues.apache.org/jira/browse/YARN-1063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13751990#comment-13751990
]
Hadoop QA commented on YARN-1063:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12600298/YARN-1063.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1778//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1778//console
This message is automatically generated.
Winutils needs ability to create task as domain user
Key: YARN-1063
URL: https://issues.apache.org/jira/browse/YARN-1063
Project: Hadoop YARN
Issue Type: Sub-task
Components: nodemanager
Affects Versions: trunk-win
Environment: Windows
Reporter: Kyle Leckie
Labels: security
Fix For: trunk-win
Attachments: YARN-1063.patch
h1. Summary:
Securing a Hadoop cluster requires constructing some form of security
boundary around the processes executed in YARN containers. Isolation based on
Windows user isolation seems most feasible. This approach is similar to the
approach taken by the existing LinuxContainerExecutor. The current patch to
winutils.exe adds the ability to create a process as a domain user.
h1. Alternative Methods considered:
h2. Process rights limited by security token restriction:
On Windows access decisions are made by examining the security token of a
process. It is possible to spawn a process with a restricted security token.
Any of the rights granted by SIDs of the default token may be restricted. It
is possible to see this in action by examining the security tone of a
sandboxed process launch be a web browser. Typically the launched process
will have a fully restricted token and need to access machine resources
through a dedicated broker process that enforces a custom security policy.
This broker process mechanism would break compatibility with the typical
Hadoop container process. The Container process must be able to utilize
standard function calls for disk and network IO. I performed some work
looking at ways to ACL the local files to the specific launched without
granting rights to other processes launched on the same machine but found
this to be an overly complex solution.
h2. Relying on APP containers:
Recent versions of windows have the ability to launch processes within an
isolated container. Application containers are supported for execution of
WinRT based executables. This method was ruled out due to the lack of
official support for standard windows APIs. At some point in the future
windows may support functionality similar to BSD jails or Linux containers,
at that point support for containers should be added.
h1. Create As User Feature Description:
h2. Usage:
A new sub command was added to the set of task commands. Here is the syntax:
winutils task createAsUser [TASKNAME] [USERNAME] [COMMAND_LINE]
Some notes:
* The username specified is in the format of user@domain
* The machine executing this command must be joined to the domain of the user
specified
* The domain controller must allow the account executing the command access
to the user information. For this join the account to the predefined group
labeled Pre-Windows 2000 Compatible Access
* The account running the command must have several rights on the local
machine. These can be managed manually using secpol.msc:
** Act as part of the operating system - SE_TCB_NAME
** Replace a process-level token - SE_ASSIGNPRIMARYTOKEN_NAME
** Adjust memory quotas for a
[jira] [Commented] (YARN-1063) Winutils needs ability to create task as domain user
[
https://issues.apache.org/jira/browse/YARN-1063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13750814#comment-13750814
]
Hadoop QA commented on YARN-1063:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12600064/YARN-1063.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1772//console
This message is automatically generated.
Winutils needs ability to create task as domain user
Key: YARN-1063
URL: https://issues.apache.org/jira/browse/YARN-1063
Project: Hadoop YARN
Issue Type: Sub-task
Components: nodemanager
Affects Versions: trunk-win
Environment: Windows
Reporter: Kyle Leckie
Labels: security
Fix For: trunk-win
Attachments: YARN-1063.patch
h1. Summary:
Securing a Hadoop cluster requires constructing some form of security
boundary around the processes executed in YARN containers. Isolation based on
Windows user isolation seems most feasible. This approach is similar to the
approach taken by the existing LinuxContainerExecutor. The current patch to
winutils.exe adds the ability to create a process as a domain user.
h1. Alternative Methods considered:
h2. Process rights limited by security token restriction:
On Windows access decisions are made by examining the security token of a
process. It is possible to spawn a process with a restricted security token.
Any of the rights granted by SIDs of the default token may be restricted. It
is possible to see this in action by examining the security tone of a
sandboxed process launch be a web browser. Typically the launched process
will have a fully restricted token and need to access machine resources
through a dedicated broker process that enforces a custom security policy.
This broker process mechanism would break compatibility with the typical
Hadoop container process. The Container process must be able to utilize
standard function calls for disk and network IO. I performed some work
looking at ways to ACL the local files to the specific launched without
granting rights to other processes launched on the same machine but found
this to be an overly complex solution.
h2. Relying on APP containers:
Recent versions of windows have the ability to launch processes within an
isolated container. Application containers are supported for execution of
WinRT based executables. This method was ruled out due to the lack of
official support for standard windows APIs. At some point in the future
windows may support functionality similar to BSD jails or Linux containers,
at that point support for containers should be added.
h1. Create As User Feature Description:
h2. Usage:
A new sub command was added to the set of task commands. Here is the syntax:
winutils task createAsUser [TASKNAME] [USERNAME] [COMMAND_LINE]
Some notes:
* The username specified is in the format of user@domain
* The machine executing this command must be joined to the domain of the user
specified
* The domain controller must allow the account executing the command access
to the user information. For this join the account to the predefined group
labeled Pre-Windows 2000 Compatible Access
* The account running the command must have several rights on the local
machine. These can be managed manually using secpol.msc:
** Act as part of the operating system - SE_TCB_NAME
** Replace a process-level token - SE_ASSIGNPRIMARYTOKEN_NAME
** Adjust memory quotas for a process - SE_INCREASE_QUOTA_NAME
* The launched process will not have rights to the desktop so will not be
able to display any information or create UI.
* The launched process will have no network credentials. Any access of
network resources that requires domain authentication will fail.
h2. Implementation:
Winutils performs the following steps:
# Enable the required privileges for the current process.
# Register as a trusted process with the Local Security Authority (LSA).
# Create a new logon for the user passed on the command line.
# Load/Create a profile on the local machine for the new logon.
# Create a new environment for the new logon.
# Launch the new process in a job with the task name specified and using the
created logon.
# Wait for the JOB to exit.
h2. Future work:
The following work was scoped out of this check in:
* Support for non-domain users or machine that are not domain joined.
* Support for privilege isolation by running the task launcher in a high
privilege service with access over an ACLed named pipe.
--
This message is automatically
[jira] [Commented] (YARN-1063) Winutils needs ability to create task as domain user
[
https://issues.apache.org/jira/browse/YARN-1063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13750823#comment-13750823
]
Chuan Liu commented on YARN-1063:
-
bq. -1 patch. The patch command could not apply the patch.
[~kyleckie], I think you need to sync to the latest code and create a new patch.
In your patch, I saw the following code.
{noformat}
-BOOL EnablePrivilege(__in LPCWSTR privilegeName);
\ No newline at end of file
{noformat}
There is a recently added function {{GetLibraryName}} after
{{EnablePrivilege}}. The code should not affect you patch.
Winutils needs ability to create task as domain user
Key: YARN-1063
URL: https://issues.apache.org/jira/browse/YARN-1063
Project: Hadoop YARN
Issue Type: Sub-task
Components: nodemanager
Affects Versions: trunk-win
Environment: Windows
Reporter: Kyle Leckie
Labels: security
Fix For: trunk-win
Attachments: YARN-1063.patch
h1. Summary:
Securing a Hadoop cluster requires constructing some form of security
boundary around the processes executed in YARN containers. Isolation based on
Windows user isolation seems most feasible. This approach is similar to the
approach taken by the existing LinuxContainerExecutor. The current patch to
winutils.exe adds the ability to create a process as a domain user.
h1. Alternative Methods considered:
h2. Process rights limited by security token restriction:
On Windows access decisions are made by examining the security token of a
process. It is possible to spawn a process with a restricted security token.
Any of the rights granted by SIDs of the default token may be restricted. It
is possible to see this in action by examining the security tone of a
sandboxed process launch be a web browser. Typically the launched process
will have a fully restricted token and need to access machine resources
through a dedicated broker process that enforces a custom security policy.
This broker process mechanism would break compatibility with the typical
Hadoop container process. The Container process must be able to utilize
standard function calls for disk and network IO. I performed some work
looking at ways to ACL the local files to the specific launched without
granting rights to other processes launched on the same machine but found
this to be an overly complex solution.
h2. Relying on APP containers:
Recent versions of windows have the ability to launch processes within an
isolated container. Application containers are supported for execution of
WinRT based executables. This method was ruled out due to the lack of
official support for standard windows APIs. At some point in the future
windows may support functionality similar to BSD jails or Linux containers,
at that point support for containers should be added.
h1. Create As User Feature Description:
h2. Usage:
A new sub command was added to the set of task commands. Here is the syntax:
winutils task createAsUser [TASKNAME] [USERNAME] [COMMAND_LINE]
Some notes:
* The username specified is in the format of user@domain
* The machine executing this command must be joined to the domain of the user
specified
* The domain controller must allow the account executing the command access
to the user information. For this join the account to the predefined group
labeled Pre-Windows 2000 Compatible Access
* The account running the command must have several rights on the local
machine. These can be managed manually using secpol.msc:
** Act as part of the operating system - SE_TCB_NAME
** Replace a process-level token - SE_ASSIGNPRIMARYTOKEN_NAME
** Adjust memory quotas for a process - SE_INCREASE_QUOTA_NAME
* The launched process will not have rights to the desktop so will not be
able to display any information or create UI.
* The launched process will have no network credentials. Any access of
network resources that requires domain authentication will fail.
h2. Implementation:
Winutils performs the following steps:
# Enable the required privileges for the current process.
# Register as a trusted process with the Local Security Authority (LSA).
# Create a new logon for the user passed on the command line.
# Load/Create a profile on the local machine for the new logon.
# Create a new environment for the new logon.
# Launch the new process in a job with the task name specified and using the
created logon.
# Wait for the JOB to exit.
h2. Future work:
The following work was scoped out of this check in:
* Support for non-domain users or machine that are not domain joined.
* Support for privilege isolation by running the task launcher in a high
privilege service with access over an ACLed named pipe.
--
This message is
[jira] [Commented] (YARN-1063) Winutils needs ability to create task as domain user
[ https://issues.apache.org/jira/browse/YARN-1063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13741717#comment-13741717 ] Kyle Leckie commented on YARN-1063: --- Hi Bikas, I have updated the Description of this JIRA to include more information. Please let me know if you have any questions/comments. Winutils needs ability to create task as domain user Key: YARN-1063 URL: https://issues.apache.org/jira/browse/YARN-1063 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Affects Versions: trunk-win Environment: Windows Reporter: Kyle Leckie Labels: security Fix For: trunk-win Attachments: YARN-732.patch h1. Summary: Securing a Hadoop cluster requires constructing some form of security boundary around the processes executed in YARN containers. Isolation based on Windows user isolation seems most feasible. This approach is similar to the approach taken by the existing LinuxContainerExecutor. The current patch to winutils.exe adds the ability to create a process as a domain user. h1. Alternative Methods considered: h2. Process rights limited by security token restriction: On Windows access decisions are made by examining the security token of a process. It is possible to spawn a process with a restricted security token. Any of the rights granted by SIDs of the default token may be restricted. It is possible to see this in action by examining the security tone of a sandboxed process launch be a web browser. Typically the launched process will have a fully restricted token and need to access machine resources through a dedicated broker process that enforces a custom security policy. This broker process mechanism would break compatibility with the typical Hadoop container process. The Container process must be able to utilize standard function calls for disk and network IO. I performed some work looking at ways to ACL the local files to the specific launched without granting rights to other processes launched on the same machine but found this to be an overly complex solution. h2. Relying on APP containers: Recent versions of windows have the ability to launch processes within an isolated container. Application containers are supported for execution of WinRT based executables. This method was ruled out due to the lack of official support for standard windows APIs. At some point in the future windows may support functionality similar to BSD jails or Linux containers, at that point support for containers should be added. h1. Create As User Feature Description: h2. Usage: A new sub command was added to the set of task commands. Here is the syntax: winutils task createAsUser [TASKNAME] [USERNAME] [COMMAND_LINE] Some notes: * The username specified is in the format of user@domain * The machine executing this command must be joined to the domain of the user specified * The domain controller must allow the account executing the command access to the user information. For this join the account to the predefined group labeled Pre-Windows 2000 Compatible Access * The account running the command must have several rights on the local machine. These can be managed manually using secpol.msc: ** Act as part of the operating system - SE_TCB_NAME ** Replace a process-level token - SE_ASSIGNPRIMARYTOKEN_NAME ** Adjust memory quotas for a process - SE_INCREASE_QUOTA_NAME * The launched process will not have rights to the desktop so will not be able to display any information or create UI. * The launched process will have no network credentials. Any access of network resources that requires domain authentication will fail. h2. Implementation: Winutils performs the following steps: # Enable the required privileges for the current process. # Register as a trusted process with the Local Security Authority (LSA). # Create a new logon for the user passed on the command line. # Load/Create a profile on the local machine for the new logon. # Create a new environment for the new logon. # Launch the new process in a job with the task name specified and using the created logon. # Wait for the JOB to exit. h2. Future work: The following work was scoped out of this check in: * Support for non-domain users or machine that are not domain joined. * Support for privilege isolation by running the task launcher in a high privilege service with access over an ACLed named pipe. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1063) Winutils needs ability to create task as domain user
[ https://issues.apache.org/jira/browse/YARN-1063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13740167#comment-13740167 ] Bikas Saha commented on YARN-1063: -- Can you please provide some overall design approach. Pros cons etc. Winutils needs ability to create task as domain user Key: YARN-1063 URL: https://issues.apache.org/jira/browse/YARN-1063 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Affects Versions: trunk-win Environment: Windows Reporter: Kyle Leckie Labels: security Fix For: trunk-win Attachments: YARN-732.patch Task isolation requires the ability to launch tasks in the context of a particular domain user. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
