[jira] [Commented] (YARN-1971) WindowsLocalWrapperScriptBuilder does not check for errors in generated script
[
https://issues.apache.org/jira/browse/YARN-1971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14526344#comment-14526344
]
Remus Rusanu commented on YARN-1971:
The problem is that there is no error check in the generated script. For
comparison the ContainerLaunch.WindowsShellScriptBuilder will check each line
in the generated script by adding this line automatically in the script, after
each command:
{code}
@if %errorlevel% neq 0 exit /b %errorlevel%
{code}
I'm not advocating checking for various error conditions before launching the
script, I'm saying the generated script itself should have error checking and
handling.
> WindowsLocalWrapperScriptBuilder does not check for errors in generated script
> --
>
> Key: YARN-1971
> URL: https://issues.apache.org/jira/browse/YARN-1971
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Reporter: Remus Rusanu
>Assignee: Remus Rusanu
>Priority: Minor
>
> Similar to YARN-1865. The
> DefaultContainerExecutor.WindowsLocalWrapperScriptBuilder builds a shell
> script that contains commands that potentially may fail:
> {code}
> pout.println("@echo " + containerIdStr + " > " + normalizedPidFile +".tmp");
> pout.println("@move /Y " + normalizedPidFile + ".tmp " + normalizedPidFile);
> {code}
> These can fail due to access permissions, disc out of space, bad hardware,
> cosmic rays etc etc. There should be proper error checking to ease
> troubleshooting.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2190) Provide a Windows container executor that can limit memory and CPU
[ https://issues.apache.org/jira/browse/YARN-2190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14347431#comment-14347431 ] Remus Rusanu commented on YARN-2190: >From my experience patches containing .sln or .vcxproj changes need to have >Windows style CRLF terminators *for the lines in the .sln/.vcxproj thunks*. >The rest of the patch should be normal Unix style terminators. If this is not >true then the patch will apply fine on Windows, but fail on Linux/Mac. > Provide a Windows container executor that can limit memory and CPU > -- > > Key: YARN-2190 > URL: https://issues.apache.org/jira/browse/YARN-2190 > Project: Hadoop YARN > Issue Type: New Feature > Components: nodemanager >Reporter: Chuan Liu >Assignee: Chuan Liu > Attachments: YARN-2190-prototype.patch, YARN-2190.1.patch, > YARN-2190.10.patch, YARN-2190.2.patch, YARN-2190.3.patch, YARN-2190.4.patch, > YARN-2190.5.patch, YARN-2190.6.patch, YARN-2190.7.patch, YARN-2190.8.patch, > YARN-2190.9.patch > > > Yarn default container executor on Windows does not set the resource limit on > the containers currently. The memory limit is enforced by a separate > monitoring thread. The container implementation on Windows uses Job Object > right now. The latest Windows (8 or later) API allows CPU and memory limits > on the job objects. We want to create a Windows container executor that sets > the limits on job objects thus provides resource enforcement at OS level. > http://msdn.microsoft.com/en-us/library/windows/desktop/ms686216(v=vs.85).aspx -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-2956) Some yarn-site index linked pages are difficult to discover because are not in the side bar
Remus Rusanu created YARN-2956: -- Summary: Some yarn-site index linked pages are difficult to discover because are not in the side bar Key: YARN-2956 URL: https://issues.apache.org/jira/browse/YARN-2956 Project: Hadoop YARN Issue Type: Bug Components: documentation Affects Versions: 2.6.0 Reporter: Remus Rusanu Priority: Minor The yarn-site index.apt.vm page is difficult to 'stumble upon' because the hadoop.apache.org/ sidebar navigation does not link to it. One needs to know the URL http://hadoop.apache.org/docs/r2.6.0/hadoop-yarn/hadoop-yarn-site/ to land on it. The links from the index page do not match the links from the side bar, so some pages are quickly accessible (from sidebar). I propose the links from the index.apt.vm to match the links from the YARN side bar subsection (Ideally through one single definition file, but I don't understand the APT generation process well enough to call out how this can be achieved. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2190) Provide a Windows container executor that can limit memory and CPU
[
https://issues.apache.org/jira/browse/YARN-2190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14231073#comment-14231073
]
Remus Rusanu commented on YARN-2190:
{
+#ifdef NTDDI_WIN8
}
I think you need a runtime version check too. Is possible to compile against
Win8 SDK but run on lower version.
> Provide a Windows container executor that can limit memory and CPU
> --
>
> Key: YARN-2190
> URL: https://issues.apache.org/jira/browse/YARN-2190
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: nodemanager
>Reporter: Chuan Liu
>Assignee: Chuan Liu
> Attachments: YARN-2190-prototype.patch, YARN-2190.1.patch,
> YARN-2190.2.patch, YARN-2190.3.patch, YARN-2190.4.patch, YARN-2190.5.patch,
> YARN-2190.6.patch, YARN-2190.7.patch
>
>
> Yarn default container executor on Windows does not set the resource limit on
> the containers currently. The memory limit is enforced by a separate
> monitoring thread. The container implementation on Windows uses Job Object
> right now. The latest Windows (8 or later) API allows CPU and memory limits
> on the job objects. We want to create a Windows container executor that sets
> the limits on job objects thus provides resource enforcement at OS level.
> http://msdn.microsoft.com/en-us/library/windows/desktop/ms686216(v=vs.85).aspx
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2739) winutils task: unsecure path should not call AddNodeManagerAndUserACEsToObject
[ https://issues.apache.org/jira/browse/YARN-2739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183160#comment-14183160 ] Remus Rusanu commented on YARN-2739: [~cwelch] just tested on secure cluster, passed fine > winutils task: unsecure path should not call AddNodeManagerAndUserACEsToObject > -- > > Key: YARN-2739 > URL: https://issues.apache.org/jira/browse/YARN-2739 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.6.0 >Reporter: Remus Rusanu >Assignee: Remus Rusanu > Labels: security, windows > Attachments: YARN-2739.000.patch > > > winutils task create path is broken after YARN-2198 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2739) winutils task: unsecure path should not call AddNodeManagerAndUserACEsToObject
[ https://issues.apache.org/jira/browse/YARN-2739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183040#comment-14183040 ] Remus Rusanu commented on YARN-2739: Yes, non-secure Windows clusters are broken as the code will unnecessarily check for the wsce-site.xml file and throw an error. > winutils task: unsecure path should not call AddNodeManagerAndUserACEsToObject > -- > > Key: YARN-2739 > URL: https://issues.apache.org/jira/browse/YARN-2739 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.6.0 >Reporter: Remus Rusanu >Assignee: Remus Rusanu > Labels: security, windows > Attachments: YARN-2739.000.patch > > > winutils task create path is broken after YARN-2198 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2739) winutils task: unsecure path should not call AddNodeManagerAndUserACEsToObject
[ https://issues.apache.org/jira/browse/YARN-2739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remus Rusanu updated YARN-2739: --- Target Version/s: 2.6.0 Affects Version/s: 2.6.0 > winutils task: unsecure path should not call AddNodeManagerAndUserACEsToObject > -- > > Key: YARN-2739 > URL: https://issues.apache.org/jira/browse/YARN-2739 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.6.0 >Reporter: Remus Rusanu >Assignee: Remus Rusanu > Labels: security, windows > Attachments: YARN-2739.000.patch > > > winutils task create path is broken after YARN-2198 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2739) winutils task: unsecure path should not call AddNodeManagerAndUserACEsToObject
[ https://issues.apache.org/jira/browse/YARN-2739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remus Rusanu updated YARN-2739: --- Attachment: YARN-2739.000.patch > winutils task: unsecure path should not call AddNodeManagerAndUserACEsToObject > -- > > Key: YARN-2739 > URL: https://issues.apache.org/jira/browse/YARN-2739 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Reporter: Remus Rusanu >Assignee: Remus Rusanu > Labels: security, windows > Attachments: YARN-2739.000.patch > > > winutils task create path is broken after YARN-2198 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-2739) winutils task: unsecure path should not call AddNodeManagerAndUserACEsToObject
Remus Rusanu created YARN-2739: -- Summary: winutils task: unsecure path should not call AddNodeManagerAndUserACEsToObject Key: YARN-2739 URL: https://issues.apache.org/jira/browse/YARN-2739 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Reporter: Remus Rusanu Assignee: Remus Rusanu winutils task create path is broken after YARN-2198 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2198) Remove the need to run NodeManager as privileged account for Windows Secure Container Executor
[ https://issues.apache.org/jira/browse/YARN-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remus Rusanu updated YARN-2198: --- Attachment: YARN-2198.16.patch .16.patch rebased to current trunk and resolves the conflict from YARN-2720 > Remove the need to run NodeManager as privileged account for Windows Secure > Container Executor > -- > > Key: YARN-2198 > URL: https://issues.apache.org/jira/browse/YARN-2198 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Remus Rusanu >Assignee: Remus Rusanu > Labels: security, windows > Attachments: .YARN-2198.delta.10.patch, YARN-2198.1.patch, > YARN-2198.11.patch, YARN-2198.12.patch, YARN-2198.13.patch, > YARN-2198.14.patch, YARN-2198.15.patch, YARN-2198.16.patch, > YARN-2198.2.patch, YARN-2198.3.patch, YARN-2198.delta.4.patch, > YARN-2198.delta.5.patch, YARN-2198.delta.6.patch, YARN-2198.delta.7.patch, > YARN-2198.separation.patch, YARN-2198.trunk.10.patch, > YARN-2198.trunk.4.patch, YARN-2198.trunk.5.patch, YARN-2198.trunk.6.patch, > YARN-2198.trunk.8.patch, YARN-2198.trunk.9.patch > > > YARN-1972 introduces a Secure Windows Container Executor. However this > executor requires the process launching the container to be LocalSystem or a > member of the a local Administrators group. Since the process in question is > the NodeManager, the requirement translates to the entire NM to run as a > privileged account, a very large surface area to review and protect. > This proposal is to move the privileged operations into a dedicated NT > service. The NM can run as a low privilege account and communicate with the > privileged NT service when it needs to launch a container. This would reduce > the surface exposed to the high privileges. > There has to exist a secure, authenticated and authorized channel of > communication between the NM and the privileged NT service. Possible > alternatives are a new TCP endpoint, Java RPC etc. My proposal though would > be to use Windows LPC (Local Procedure Calls), which is a Windows platform > specific inter-process communication channel that satisfies all requirements > and is easy to deploy. The privileged NT service would register and listen on > an LPC port (NtCreatePort, NtListenPort). The NM would use JNI to interop > with libwinutils which would host the LPC client code. The client would > connect to the LPC port (NtConnectPort) and send a message requesting a > container launch (NtRequestWaitReplyPort). LPC provides authentication and > the privileged NT service can use authorization API (AuthZ) to validate the > caller. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2198) Remove the need to run NodeManager as privileged account for Windows Secure Container Executor
[ https://issues.apache.org/jira/browse/YARN-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14178218#comment-14178218 ] Remus Rusanu commented on YARN-2198: The 2 new hadoop-common Findbugs are unrelated to the patch: - Inconsistent synchronization of org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.delegationTokenSequenceNumber; locked 71% of time - Dereference of the result of readLine() without nullcheck in org.apache.hadoop.tracing.SpanReceiverHost.getUniqueLocalTraceFileName() > Remove the need to run NodeManager as privileged account for Windows Secure > Container Executor > -- > > Key: YARN-2198 > URL: https://issues.apache.org/jira/browse/YARN-2198 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Remus Rusanu >Assignee: Remus Rusanu > Labels: security, windows > Attachments: .YARN-2198.delta.10.patch, YARN-2198.1.patch, > YARN-2198.11.patch, YARN-2198.12.patch, YARN-2198.13.patch, > YARN-2198.14.patch, YARN-2198.15.patch, YARN-2198.2.patch, YARN-2198.3.patch, > YARN-2198.delta.4.patch, YARN-2198.delta.5.patch, YARN-2198.delta.6.patch, > YARN-2198.delta.7.patch, YARN-2198.separation.patch, > YARN-2198.trunk.10.patch, YARN-2198.trunk.4.patch, YARN-2198.trunk.5.patch, > YARN-2198.trunk.6.patch, YARN-2198.trunk.8.patch, YARN-2198.trunk.9.patch > > > YARN-1972 introduces a Secure Windows Container Executor. However this > executor requires the process launching the container to be LocalSystem or a > member of the a local Administrators group. Since the process in question is > the NodeManager, the requirement translates to the entire NM to run as a > privileged account, a very large surface area to review and protect. > This proposal is to move the privileged operations into a dedicated NT > service. The NM can run as a low privilege account and communicate with the > privileged NT service when it needs to launch a container. This would reduce > the surface exposed to the high privileges. > There has to exist a secure, authenticated and authorized channel of > communication between the NM and the privileged NT service. Possible > alternatives are a new TCP endpoint, Java RPC etc. My proposal though would > be to use Windows LPC (Local Procedure Calls), which is a Windows platform > specific inter-process communication channel that satisfies all requirements > and is easy to deploy. The privileged NT service would register and listen on > an LPC port (NtCreatePort, NtListenPort). The NM would use JNI to interop > with libwinutils which would host the LPC client code. The client would > connect to the LPC port (NtConnectPort) and send a message requesting a > container launch (NtRequestWaitReplyPort). LPC provides authentication and > the privileged NT service can use authorization API (AuthZ) to validate the > caller. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2198) Remove the need to run NodeManager as privileged account for Windows Secure Container Executor
[ https://issues.apache.org/jira/browse/YARN-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remus Rusanu updated YARN-2198: --- Attachment: YARN-2198.15.patch Reload .15.patch with TestLCE fix > Remove the need to run NodeManager as privileged account for Windows Secure > Container Executor > -- > > Key: YARN-2198 > URL: https://issues.apache.org/jira/browse/YARN-2198 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Remus Rusanu >Assignee: Remus Rusanu > Labels: security, windows > Attachments: .YARN-2198.delta.10.patch, YARN-2198.1.patch, > YARN-2198.11.patch, YARN-2198.12.patch, YARN-2198.13.patch, > YARN-2198.14.patch, YARN-2198.15.patch, YARN-2198.2.patch, YARN-2198.3.patch, > YARN-2198.delta.4.patch, YARN-2198.delta.5.patch, YARN-2198.delta.6.patch, > YARN-2198.delta.7.patch, YARN-2198.separation.patch, > YARN-2198.trunk.10.patch, YARN-2198.trunk.4.patch, YARN-2198.trunk.5.patch, > YARN-2198.trunk.6.patch, YARN-2198.trunk.8.patch, YARN-2198.trunk.9.patch > > > YARN-1972 introduces a Secure Windows Container Executor. However this > executor requires the process launching the container to be LocalSystem or a > member of the a local Administrators group. Since the process in question is > the NodeManager, the requirement translates to the entire NM to run as a > privileged account, a very large surface area to review and protect. > This proposal is to move the privileged operations into a dedicated NT > service. The NM can run as a low privilege account and communicate with the > privileged NT service when it needs to launch a container. This would reduce > the surface exposed to the high privileges. > There has to exist a secure, authenticated and authorized channel of > communication between the NM and the privileged NT service. Possible > alternatives are a new TCP endpoint, Java RPC etc. My proposal though would > be to use Windows LPC (Local Procedure Calls), which is a Windows platform > specific inter-process communication channel that satisfies all requirements > and is easy to deploy. The privileged NT service would register and listen on > an LPC port (NtCreatePort, NtListenPort). The NM would use JNI to interop > with libwinutils which would host the LPC client code. The client would > connect to the LPC port (NtConnectPort) and send a message requesting a > container launch (NtRequestWaitReplyPort). LPC provides authentication and > the privileged NT service can use authorization API (AuthZ) to validate the > caller. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2198) Remove the need to run NodeManager as privileged account for Windows Secure Container Executor
[ https://issues.apache.org/jira/browse/YARN-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remus Rusanu updated YARN-2198: --- Attachment: (was: YARN-2198.15.patch) > Remove the need to run NodeManager as privileged account for Windows Secure > Container Executor > -- > > Key: YARN-2198 > URL: https://issues.apache.org/jira/browse/YARN-2198 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Remus Rusanu >Assignee: Remus Rusanu > Labels: security, windows > Attachments: .YARN-2198.delta.10.patch, YARN-2198.1.patch, > YARN-2198.11.patch, YARN-2198.12.patch, YARN-2198.13.patch, > YARN-2198.14.patch, YARN-2198.2.patch, YARN-2198.3.patch, > YARN-2198.delta.4.patch, YARN-2198.delta.5.patch, YARN-2198.delta.6.patch, > YARN-2198.delta.7.patch, YARN-2198.separation.patch, > YARN-2198.trunk.10.patch, YARN-2198.trunk.4.patch, YARN-2198.trunk.5.patch, > YARN-2198.trunk.6.patch, YARN-2198.trunk.8.patch, YARN-2198.trunk.9.patch > > > YARN-1972 introduces a Secure Windows Container Executor. However this > executor requires the process launching the container to be LocalSystem or a > member of the a local Administrators group. Since the process in question is > the NodeManager, the requirement translates to the entire NM to run as a > privileged account, a very large surface area to review and protect. > This proposal is to move the privileged operations into a dedicated NT > service. The NM can run as a low privilege account and communicate with the > privileged NT service when it needs to launch a container. This would reduce > the surface exposed to the high privileges. > There has to exist a secure, authenticated and authorized channel of > communication between the NM and the privileged NT service. Possible > alternatives are a new TCP endpoint, Java RPC etc. My proposal though would > be to use Windows LPC (Local Procedure Calls), which is a Windows platform > specific inter-process communication channel that satisfies all requirements > and is easy to deploy. The privileged NT service would register and listen on > an LPC port (NtCreatePort, NtListenPort). The NM would use JNI to interop > with libwinutils which would host the LPC client code. The client would > connect to the LPC port (NtConnectPort) and send a message requesting a > container launch (NtRequestWaitReplyPort). LPC provides authentication and > the privileged NT service can use authorization API (AuthZ) to validate the > caller. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2198) Remove the need to run NodeManager as privileged account for Windows Secure Container Executor
[ https://issues.apache.org/jira/browse/YARN-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remus Rusanu updated YARN-2198: --- Attachment: YARN-2198.15.patch .15.patch rebased to current trunk, .gitignore conflict resolved > Remove the need to run NodeManager as privileged account for Windows Secure > Container Executor > -- > > Key: YARN-2198 > URL: https://issues.apache.org/jira/browse/YARN-2198 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Remus Rusanu >Assignee: Remus Rusanu > Labels: security, windows > Attachments: .YARN-2198.delta.10.patch, YARN-2198.1.patch, > YARN-2198.11.patch, YARN-2198.12.patch, YARN-2198.13.patch, > YARN-2198.14.patch, YARN-2198.15.patch, YARN-2198.2.patch, YARN-2198.3.patch, > YARN-2198.delta.4.patch, YARN-2198.delta.5.patch, YARN-2198.delta.6.patch, > YARN-2198.delta.7.patch, YARN-2198.separation.patch, > YARN-2198.trunk.10.patch, YARN-2198.trunk.4.patch, YARN-2198.trunk.5.patch, > YARN-2198.trunk.6.patch, YARN-2198.trunk.8.patch, YARN-2198.trunk.9.patch > > > YARN-1972 introduces a Secure Windows Container Executor. However this > executor requires the process launching the container to be LocalSystem or a > member of the a local Administrators group. Since the process in question is > the NodeManager, the requirement translates to the entire NM to run as a > privileged account, a very large surface area to review and protect. > This proposal is to move the privileged operations into a dedicated NT > service. The NM can run as a low privilege account and communicate with the > privileged NT service when it needs to launch a container. This would reduce > the surface exposed to the high privileges. > There has to exist a secure, authenticated and authorized channel of > communication between the NM and the privileged NT service. Possible > alternatives are a new TCP endpoint, Java RPC etc. My proposal though would > be to use Windows LPC (Local Procedure Calls), which is a Windows platform > specific inter-process communication channel that satisfies all requirements > and is easy to deploy. The privileged NT service would register and listen on > an LPC port (NtCreatePort, NtListenPort). The NM would use JNI to interop > with libwinutils which would host the LPC client code. The client would > connect to the LPC port (NtConnectPort) and send a message requesting a > container launch (NtRequestWaitReplyPort). LPC provides authentication and > the privileged NT service can use authorization API (AuthZ) to validate the > caller. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2198) Remove the need to run NodeManager as privileged account for Windows Secure Container Executor
[ https://issues.apache.org/jira/browse/YARN-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14174992#comment-14174992 ] Remus Rusanu commented on YARN-2198: Neither the findbug nor the core test failures are related to the patch. > Remove the need to run NodeManager as privileged account for Windows Secure > Container Executor > -- > > Key: YARN-2198 > URL: https://issues.apache.org/jira/browse/YARN-2198 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Remus Rusanu >Assignee: Remus Rusanu > Labels: security, windows > Attachments: .YARN-2198.delta.10.patch, YARN-2198.1.patch, > YARN-2198.11.patch, YARN-2198.12.patch, YARN-2198.13.patch, > YARN-2198.14.patch, YARN-2198.2.patch, YARN-2198.3.patch, > YARN-2198.delta.4.patch, YARN-2198.delta.5.patch, YARN-2198.delta.6.patch, > YARN-2198.delta.7.patch, YARN-2198.separation.patch, > YARN-2198.trunk.10.patch, YARN-2198.trunk.4.patch, YARN-2198.trunk.5.patch, > YARN-2198.trunk.6.patch, YARN-2198.trunk.8.patch, YARN-2198.trunk.9.patch > > > YARN-1972 introduces a Secure Windows Container Executor. However this > executor requires the process launching the container to be LocalSystem or a > member of the a local Administrators group. Since the process in question is > the NodeManager, the requirement translates to the entire NM to run as a > privileged account, a very large surface area to review and protect. > This proposal is to move the privileged operations into a dedicated NT > service. The NM can run as a low privilege account and communicate with the > privileged NT service when it needs to launch a container. This would reduce > the surface exposed to the high privileges. > There has to exist a secure, authenticated and authorized channel of > communication between the NM and the privileged NT service. Possible > alternatives are a new TCP endpoint, Java RPC etc. My proposal though would > be to use Windows LPC (Local Procedure Calls), which is a Windows platform > specific inter-process communication channel that satisfies all requirements > and is easy to deploy. The privileged NT service would register and listen on > an LPC port (NtCreatePort, NtListenPort). The NM would use JNI to interop > with libwinutils which would host the LPC client code. The client would > connect to the LPC port (NtConnectPort) and send a message requesting a > container launch (NtRequestWaitReplyPort). LPC provides authentication and > the privileged NT service can use authorization API (AuthZ) to validate the > caller. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2198) Remove the need to run NodeManager as privileged account for Windows Secure Container Executor
[ https://issues.apache.org/jira/browse/YARN-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remus Rusanu updated YARN-2198: --- Attachment: YARN-2198.14.patch .14.patch is rebased to current trunk and fixes the conflict with YARN-2682 > Remove the need to run NodeManager as privileged account for Windows Secure > Container Executor > -- > > Key: YARN-2198 > URL: https://issues.apache.org/jira/browse/YARN-2198 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Remus Rusanu >Assignee: Remus Rusanu > Labels: security, windows > Attachments: .YARN-2198.delta.10.patch, YARN-2198.1.patch, > YARN-2198.11.patch, YARN-2198.12.patch, YARN-2198.13.patch, > YARN-2198.14.patch, YARN-2198.2.patch, YARN-2198.3.patch, > YARN-2198.delta.4.patch, YARN-2198.delta.5.patch, YARN-2198.delta.6.patch, > YARN-2198.delta.7.patch, YARN-2198.separation.patch, > YARN-2198.trunk.10.patch, YARN-2198.trunk.4.patch, YARN-2198.trunk.5.patch, > YARN-2198.trunk.6.patch, YARN-2198.trunk.8.patch, YARN-2198.trunk.9.patch > > > YARN-1972 introduces a Secure Windows Container Executor. However this > executor requires the process launching the container to be LocalSystem or a > member of the a local Administrators group. Since the process in question is > the NodeManager, the requirement translates to the entire NM to run as a > privileged account, a very large surface area to review and protect. > This proposal is to move the privileged operations into a dedicated NT > service. The NM can run as a low privilege account and communicate with the > privileged NT service when it needs to launch a container. This would reduce > the surface exposed to the high privileges. > There has to exist a secure, authenticated and authorized channel of > communication between the NM and the privileged NT service. Possible > alternatives are a new TCP endpoint, Java RPC etc. My proposal though would > be to use Windows LPC (Local Procedure Calls), which is a Windows platform > specific inter-process communication channel that satisfies all requirements > and is easy to deploy. The privileged NT service would register and listen on > an LPC port (NtCreatePort, NtListenPort). The NM would use JNI to interop > with libwinutils which would host the LPC client code. The client would > connect to the LPC port (NtConnectPort) and send a message requesting a > container launch (NtRequestWaitReplyPort). LPC provides authentication and > the privileged NT service can use authorization API (AuthZ) to validate the > caller. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2198) Remove the need to run NodeManager as privileged account for Windows Secure Container Executor
[ https://issues.apache.org/jira/browse/YARN-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14172223#comment-14172223 ] Remus Rusanu commented on YARN-2198: The findbugs issues are unrelated to the patch: - Inconsistent synchronization of org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.delegationTokenSequenceNumber; locked 71% of time - Dereference of the result of readLine() without nullcheck in org.apache.hadoop.tracing.SpanReceiverHost.getUniqueLocalTraceFileName() > Remove the need to run NodeManager as privileged account for Windows Secure > Container Executor > -- > > Key: YARN-2198 > URL: https://issues.apache.org/jira/browse/YARN-2198 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Remus Rusanu >Assignee: Remus Rusanu > Labels: security, windows > Attachments: .YARN-2198.delta.10.patch, YARN-2198.1.patch, > YARN-2198.11.patch, YARN-2198.12.patch, YARN-2198.13.patch, > YARN-2198.2.patch, YARN-2198.3.patch, YARN-2198.delta.4.patch, > YARN-2198.delta.5.patch, YARN-2198.delta.6.patch, YARN-2198.delta.7.patch, > YARN-2198.separation.patch, YARN-2198.trunk.10.patch, > YARN-2198.trunk.4.patch, YARN-2198.trunk.5.patch, YARN-2198.trunk.6.patch, > YARN-2198.trunk.8.patch, YARN-2198.trunk.9.patch > > > YARN-1972 introduces a Secure Windows Container Executor. However this > executor requires the process launching the container to be LocalSystem or a > member of the a local Administrators group. Since the process in question is > the NodeManager, the requirement translates to the entire NM to run as a > privileged account, a very large surface area to review and protect. > This proposal is to move the privileged operations into a dedicated NT > service. The NM can run as a low privilege account and communicate with the > privileged NT service when it needs to launch a container. This would reduce > the surface exposed to the high privileges. > There has to exist a secure, authenticated and authorized channel of > communication between the NM and the privileged NT service. Possible > alternatives are a new TCP endpoint, Java RPC etc. My proposal though would > be to use Windows LPC (Local Procedure Calls), which is a Windows platform > specific inter-process communication channel that satisfies all requirements > and is easy to deploy. The privileged NT service would register and listen on > an LPC port (NtCreatePort, NtListenPort). The NM would use JNI to interop > with libwinutils which would host the LPC client code. The client would > connect to the LPC port (NtConnectPort) and send a message requesting a > container launch (NtRequestWaitReplyPort). LPC provides authentication and > the privileged NT service can use authorization API (AuthZ) to validate the > caller. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2682) WindowsSecureContainerExecutor should not depend on DefaultContainerExecutor#getFirstApplicationDir.
[ https://issues.apache.org/jira/browse/YARN-2682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14172125#comment-14172125 ] Remus Rusanu commented on YARN-2682: Thanks [~zxu]! YARN-2198 will probably conflict with this but is only cosmetic. as soon as you commit, I'll resolve the conflicts. > WindowsSecureContainerExecutor should not depend on > DefaultContainerExecutor#getFirstApplicationDir. > - > > Key: YARN-2682 > URL: https://issues.apache.org/jira/browse/YARN-2682 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.5.0 >Reporter: zhihai xu >Assignee: zhihai xu >Priority: Minor > Attachments: YARN-2682.000.patch, YARN-2682.001.patch > > > DefaultContainerExecutor won't use getFirstApplicationDir any more. But we > can't delete getFirstApplicationDir in DefaultContainerExecutor because > WindowsSecureContainerExecutor uses it. > We should move getFirstApplicationDir function from DefaultContainerExecutor > to WindowsSecureContainerExecutor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2198) Remove the need to run NodeManager as privileged account for Windows Secure Container Executor
[ https://issues.apache.org/jira/browse/YARN-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remus Rusanu updated YARN-2198: --- Attachment: YARN-2198.13.patch .13.patch rebased to current trunk fixes. > Remove the need to run NodeManager as privileged account for Windows Secure > Container Executor > -- > > Key: YARN-2198 > URL: https://issues.apache.org/jira/browse/YARN-2198 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Remus Rusanu >Assignee: Remus Rusanu > Labels: security, windows > Attachments: .YARN-2198.delta.10.patch, YARN-2198.1.patch, > YARN-2198.11.patch, YARN-2198.12.patch, YARN-2198.13.patch, > YARN-2198.2.patch, YARN-2198.3.patch, YARN-2198.delta.4.patch, > YARN-2198.delta.5.patch, YARN-2198.delta.6.patch, YARN-2198.delta.7.patch, > YARN-2198.separation.patch, YARN-2198.trunk.10.patch, > YARN-2198.trunk.4.patch, YARN-2198.trunk.5.patch, YARN-2198.trunk.6.patch, > YARN-2198.trunk.8.patch, YARN-2198.trunk.9.patch > > > YARN-1972 introduces a Secure Windows Container Executor. However this > executor requires the process launching the container to be LocalSystem or a > member of the a local Administrators group. Since the process in question is > the NodeManager, the requirement translates to the entire NM to run as a > privileged account, a very large surface area to review and protect. > This proposal is to move the privileged operations into a dedicated NT > service. The NM can run as a low privilege account and communicate with the > privileged NT service when it needs to launch a container. This would reduce > the surface exposed to the high privileges. > There has to exist a secure, authenticated and authorized channel of > communication between the NM and the privileged NT service. Possible > alternatives are a new TCP endpoint, Java RPC etc. My proposal though would > be to use Windows LPC (Local Procedure Calls), which is a Windows platform > specific inter-process communication channel that satisfies all requirements > and is easy to deploy. The privileged NT service would register and listen on > an LPC port (NtCreatePort, NtListenPort). The NM would use JNI to interop > with libwinutils which would host the LPC client code. The client would > connect to the LPC port (NtConnectPort) and send a message requesting a > container launch (NtRequestWaitReplyPort). LPC provides authentication and > the privileged NT service can use authorization API (AuthZ) to validate the > caller. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2198) Remove the need to run NodeManager as privileged account for Windows Secure Container Executor
[ https://issues.apache.org/jira/browse/YARN-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remus Rusanu updated YARN-2198: --- Attachment: YARN-2198.12.patch > Remove the need to run NodeManager as privileged account for Windows Secure > Container Executor > -- > > Key: YARN-2198 > URL: https://issues.apache.org/jira/browse/YARN-2198 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Remus Rusanu >Assignee: Remus Rusanu > Labels: security, windows > Attachments: .YARN-2198.delta.10.patch, YARN-2198.1.patch, > YARN-2198.11.patch, YARN-2198.12.patch, YARN-2198.2.patch, YARN-2198.3.patch, > YARN-2198.delta.4.patch, YARN-2198.delta.5.patch, YARN-2198.delta.6.patch, > YARN-2198.delta.7.patch, YARN-2198.separation.patch, > YARN-2198.trunk.10.patch, YARN-2198.trunk.4.patch, YARN-2198.trunk.5.patch, > YARN-2198.trunk.6.patch, YARN-2198.trunk.8.patch, YARN-2198.trunk.9.patch > > > YARN-1972 introduces a Secure Windows Container Executor. However this > executor requires the process launching the container to be LocalSystem or a > member of the a local Administrators group. Since the process in question is > the NodeManager, the requirement translates to the entire NM to run as a > privileged account, a very large surface area to review and protect. > This proposal is to move the privileged operations into a dedicated NT > service. The NM can run as a low privilege account and communicate with the > privileged NT service when it needs to launch a container. This would reduce > the surface exposed to the high privileges. > There has to exist a secure, authenticated and authorized channel of > communication between the NM and the privileged NT service. Possible > alternatives are a new TCP endpoint, Java RPC etc. My proposal though would > be to use Windows LPC (Local Procedure Calls), which is a Windows platform > specific inter-process communication channel that satisfies all requirements > and is easy to deploy. The privileged NT service would register and listen on > an LPC port (NtCreatePort, NtListenPort). The NM would use JNI to interop > with libwinutils which would host the LPC client code. The client would > connect to the LPC port (NtConnectPort) and send a message requesting a > container launch (NtRequestWaitReplyPort). LPC provides authentication and > the privileged NT service can use authorization API (AuthZ) to validate the > caller. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2198) Remove the need to run NodeManager as privileged account for Windows Secure Container Executor
[ https://issues.apache.org/jira/browse/YARN-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remus Rusanu updated YARN-2198: --- Attachment: (was: YARN-2198.12.patch) > Remove the need to run NodeManager as privileged account for Windows Secure > Container Executor > -- > > Key: YARN-2198 > URL: https://issues.apache.org/jira/browse/YARN-2198 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Remus Rusanu >Assignee: Remus Rusanu > Labels: security, windows > Attachments: .YARN-2198.delta.10.patch, YARN-2198.1.patch, > YARN-2198.11.patch, YARN-2198.2.patch, YARN-2198.3.patch, > YARN-2198.delta.4.patch, YARN-2198.delta.5.patch, YARN-2198.delta.6.patch, > YARN-2198.delta.7.patch, YARN-2198.separation.patch, > YARN-2198.trunk.10.patch, YARN-2198.trunk.4.patch, YARN-2198.trunk.5.patch, > YARN-2198.trunk.6.patch, YARN-2198.trunk.8.patch, YARN-2198.trunk.9.patch > > > YARN-1972 introduces a Secure Windows Container Executor. However this > executor requires the process launching the container to be LocalSystem or a > member of the a local Administrators group. Since the process in question is > the NodeManager, the requirement translates to the entire NM to run as a > privileged account, a very large surface area to review and protect. > This proposal is to move the privileged operations into a dedicated NT > service. The NM can run as a low privilege account and communicate with the > privileged NT service when it needs to launch a container. This would reduce > the surface exposed to the high privileges. > There has to exist a secure, authenticated and authorized channel of > communication between the NM and the privileged NT service. Possible > alternatives are a new TCP endpoint, Java RPC etc. My proposal though would > be to use Windows LPC (Local Procedure Calls), which is a Windows platform > specific inter-process communication channel that satisfies all requirements > and is easy to deploy. The privileged NT service would register and listen on > an LPC port (NtCreatePort, NtListenPort). The NM would use JNI to interop > with libwinutils which would host the LPC client code. The client would > connect to the LPC port (NtConnectPort) and send a message requesting a > container launch (NtRequestWaitReplyPort). LPC provides authentication and > the privileged NT service can use authorization API (AuthZ) to validate the > caller. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-2687) WindowsSecureContainerExecutor hadoopwinutilsvc is difficult to troubleshoot
Remus Rusanu created YARN-2687: -- Summary: WindowsSecureContainerExecutor hadoopwinutilsvc is difficult to troubleshoot Key: YARN-2687 URL: https://issues.apache.org/jira/browse/YARN-2687 Project: Hadoop YARN Issue Type: Improvement Components: nodemanager Reporter: Remus Rusanu Assignee: Remus Rusanu The hadoopwinutilsvc logs using the NT service logging infrastructure (ie. Event Viewer). Ideally it should log within the Hadoop logging expected location/format, and be configured via the same parameters. As native C++ code it cannot leverage directly the log4j (and log4c++ is rather different config etc). I'm thinking that the hadoopwinutilsvc could establish a communication channel with NM itself and log via the NM. We already have the infrastructure in place (RPC, IDL etc). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2552) Windows Secure Container Executor: the privileged file operations of hadoopwinutilsvc should be constrained to localdirs only
[
https://issues.apache.org/jira/browse/YARN-2552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14170907#comment-14170907
]
Remus Rusanu commented on YARN-2552:
Copying here the patch' apt.vm update:
> 'yarn.nodemanager.windows-secure-container-executor.local-dirs' should
> contain the nodemanager local dirs. hadoopwinutilsvc will allow only file
> operations under these directories. This should contain the same values as
> '${yarn.nodemanager.local-dirs}, ${yarn.nodemanager.log-dirs}' but note that
> hadoopwinutilsvc XML configuration processing does not do substitutions so
> the value must be the final value. All paths must be absolute and no
> environment variable substitution will be performed. The paths are compared
> LOCAL_INVARIANT case insensitive string comparison, the file path validated
> must start with one of the paths listed in local-dirs configuration. Use
> comma as path separator.
> Windows Secure Container Executor: the privileged file operations of
> hadoopwinutilsvc should be constrained to localdirs only
> -
>
> Key: YARN-2552
> URL: https://issues.apache.org/jira/browse/YARN-2552
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Remus Rusanu
>Assignee: Remus Rusanu
> Labels: security, windows, wsce
> Attachments: YARN-2552.1.patch
>
>
> YARN-2458 added file manipulation operations executed in an elevated context
> by hadoopwinutilsvc. W/o any constraint, the NM (or a hijacker that takes
> over the NM) can manipulate arbitrary OS files under highest possible
> privileges, an easy elevation attack vector. The service should only allow
> operations on files/directories that are under the configured NM localdirs.
> It should read this value from wsce-site.xml, as the yarn-site.xml cannot be
> trusted, being writable by Hadoop admins (YARN-2551 ensures wsce-site.xml is
> only writable by system Administrators, not Hadoop admins).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2682) WindowsSecureContainerExecutor should not depend on DefaultContainerExecutor#getFirstApplicationDir.
[ https://issues.apache.org/jira/browse/YARN-2682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14170716#comment-14170716 ] Remus Rusanu commented on YARN-2682: WSCE should behave the same as DCE. If getFirstApplicationDir() was removed from DCE and getApplicationDir() is used instead, then WSCE should also use getApplicationDir(), w/o a need to define getFirstApplicationDir() in WSCE. > WindowsSecureContainerExecutor should not depend on > DefaultContainerExecutor#getFirstApplicationDir. > - > > Key: YARN-2682 > URL: https://issues.apache.org/jira/browse/YARN-2682 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.5.0 >Reporter: zhihai xu >Assignee: zhihai xu >Priority: Minor > Attachments: YARN-2682.000.patch > > > DefaultContainerExecutor won't use getFirstApplicationDir any more. But we > can't delete getFirstApplicationDir in DefaultContainerExecutor because > WindowsSecureContainerExecutor uses it. > We should move getFirstApplicationDir function from DefaultContainerExecutor > to WindowsSecureContainerExecutor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2198) Remove the need to run NodeManager as privileged account for Windows Secure Container Executor
[ https://issues.apache.org/jira/browse/YARN-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14169132#comment-14169132 ] Remus Rusanu commented on YARN-2198: [~cwelch] I have added the suggested WSCE unit test in YARN-2636 > Remove the need to run NodeManager as privileged account for Windows Secure > Container Executor > -- > > Key: YARN-2198 > URL: https://issues.apache.org/jira/browse/YARN-2198 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Remus Rusanu >Assignee: Remus Rusanu > Labels: security, windows > Attachments: .YARN-2198.delta.10.patch, YARN-2198.1.patch, > YARN-2198.11.patch, YARN-2198.12.patch, YARN-2198.2.patch, YARN-2198.3.patch, > YARN-2198.delta.4.patch, YARN-2198.delta.5.patch, YARN-2198.delta.6.patch, > YARN-2198.delta.7.patch, YARN-2198.separation.patch, > YARN-2198.trunk.10.patch, YARN-2198.trunk.4.patch, YARN-2198.trunk.5.patch, > YARN-2198.trunk.6.patch, YARN-2198.trunk.8.patch, YARN-2198.trunk.9.patch > > > YARN-1972 introduces a Secure Windows Container Executor. However this > executor requires the process launching the container to be LocalSystem or a > member of the a local Administrators group. Since the process in question is > the NodeManager, the requirement translates to the entire NM to run as a > privileged account, a very large surface area to review and protect. > This proposal is to move the privileged operations into a dedicated NT > service. The NM can run as a low privilege account and communicate with the > privileged NT service when it needs to launch a container. This would reduce > the surface exposed to the high privileges. > There has to exist a secure, authenticated and authorized channel of > communication between the NM and the privileged NT service. Possible > alternatives are a new TCP endpoint, Java RPC etc. My proposal though would > be to use Windows LPC (Local Procedure Calls), which is a Windows platform > specific inter-process communication channel that satisfies all requirements > and is easy to deploy. The privileged NT service would register and listen on > an LPC port (NtCreatePort, NtListenPort). The NM would use JNI to interop > with libwinutils which would host the LPC client code. The client would > connect to the LPC port (NtConnectPort) and send a message requesting a > container launch (NtRequestWaitReplyPort). LPC provides authentication and > the privileged NT service can use authorization API (AuthZ) to validate the > caller. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2636) Windows Secure Container Executor: add unit tests for WSCE
[ https://issues.apache.org/jira/browse/YARN-2636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remus Rusanu updated YARN-2636: --- Attachment: YARN-2636.delta.1.patch delta.1.path requires YARN-2198. Adds new unit test for WindowsSecureContainerExecutor. > Windows Secure Container Executor: add unit tests for WSCE > -- > > Key: YARN-2636 > URL: https://issues.apache.org/jira/browse/YARN-2636 > Project: Hadoop YARN > Issue Type: Test >Reporter: Remus Rusanu >Assignee: Remus Rusanu >Priority: Critical > Attachments: YARN-2636.delta.1.patch > > > As title says. > The WSCE has no check-in unit tests. Much of the functionality depends on > elevated hadoopwinutilsvc service and cannot be tested, but lets test what is > possible to be mocked in Java. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2636) Windows Secure Container Executor: add unit tests for WSCE
[ https://issues.apache.org/jira/browse/YARN-2636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remus Rusanu updated YARN-2636: --- Attachment: (was: YARN-2636.doesnotwork.0.patch) > Windows Secure Container Executor: add unit tests for WSCE > -- > > Key: YARN-2636 > URL: https://issues.apache.org/jira/browse/YARN-2636 > Project: Hadoop YARN > Issue Type: Test >Reporter: Remus Rusanu >Assignee: Remus Rusanu >Priority: Critical > > As title says. > The WSCE has no check-in unit tests. Much of the functionality depends on > elevated hadoopwinutilsvc service and cannot be tested, but lets test what is > possible to be mocked in Java. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2636) Windows Secure Container Executor: add unit tests for WSCE
[ https://issues.apache.org/jira/browse/YARN-2636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remus Rusanu updated YARN-2636: --- Attachment: YARN-2636.doesnotwork.0.patch > Windows Secure Container Executor: add unit tests for WSCE > -- > > Key: YARN-2636 > URL: https://issues.apache.org/jira/browse/YARN-2636 > Project: Hadoop YARN > Issue Type: Test >Reporter: Remus Rusanu >Assignee: Remus Rusanu >Priority: Critical > Attachments: YARN-2636.doesnotwork.0.patch > > > As title says. > The WSCE has no check-in unit tests. Much of the functionality depends on > elevated hadoopwinutilsvc service and cannot be tested, but lets test what is > possible to be mocked in Java. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2636) Windows Secure Container Executor: add unit tests for WSCE
[ https://issues.apache.org/jira/browse/YARN-2636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remus Rusanu updated YARN-2636: --- Issue Type: Test (was: Sub-task) Parent: (was: YARN-2198) > Windows Secure Container Executor: add unit tests for WSCE > -- > > Key: YARN-2636 > URL: https://issues.apache.org/jira/browse/YARN-2636 > Project: Hadoop YARN > Issue Type: Test >Reporter: Remus Rusanu >Assignee: Remus Rusanu >Priority: Critical > > As title says. > The WSCE has no check-in unit tests. Much of the functionality depends on > elevated hadoopwinutilsvc service and cannot be tested, but lets test what is > possible to be mocked in Java. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2198) Remove the need to run NodeManager as privileged account for Windows Secure Container Executor
[
https://issues.apache.org/jira/browse/YARN-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14166751#comment-14166751
]
Remus Rusanu commented on YARN-2198:
.12.patch:
> streamReaderThread.run - using the readLine() instead of following the simple
> buffer copy idiom in ShellCommandExecutor has some efficiency issues, granted
> it looks to be reading "memory sized" data so it may be no big deal, but it
> would be nice to follow the buffer-copy pattern instead
RR: Fixed
> ContainerLocalizer LOG.info(String.format("nRet: %d", nRet)); - not sure this
> should be "info" level
RR: Fixed
> Remove the need to run NodeManager as privileged account for Windows Secure
> Container Executor
> --
>
> Key: YARN-2198
> URL: https://issues.apache.org/jira/browse/YARN-2198
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Remus Rusanu
>Assignee: Remus Rusanu
> Labels: security, windows
> Attachments: .YARN-2198.delta.10.patch, YARN-2198.1.patch,
> YARN-2198.11.patch, YARN-2198.12.patch, YARN-2198.2.patch, YARN-2198.3.patch,
> YARN-2198.delta.4.patch, YARN-2198.delta.5.patch, YARN-2198.delta.6.patch,
> YARN-2198.delta.7.patch, YARN-2198.separation.patch,
> YARN-2198.trunk.10.patch, YARN-2198.trunk.4.patch, YARN-2198.trunk.5.patch,
> YARN-2198.trunk.6.patch, YARN-2198.trunk.8.patch, YARN-2198.trunk.9.patch
>
>
> YARN-1972 introduces a Secure Windows Container Executor. However this
> executor requires the process launching the container to be LocalSystem or a
> member of the a local Administrators group. Since the process in question is
> the NodeManager, the requirement translates to the entire NM to run as a
> privileged account, a very large surface area to review and protect.
> This proposal is to move the privileged operations into a dedicated NT
> service. The NM can run as a low privilege account and communicate with the
> privileged NT service when it needs to launch a container. This would reduce
> the surface exposed to the high privileges.
> There has to exist a secure, authenticated and authorized channel of
> communication between the NM and the privileged NT service. Possible
> alternatives are a new TCP endpoint, Java RPC etc. My proposal though would
> be to use Windows LPC (Local Procedure Calls), which is a Windows platform
> specific inter-process communication channel that satisfies all requirements
> and is easy to deploy. The privileged NT service would register and listen on
> an LPC port (NtCreatePort, NtListenPort). The NM would use JNI to interop
> with libwinutils which would host the LPC client code. The client would
> connect to the LPC port (NtConnectPort) and send a message requesting a
> container launch (NtRequestWaitReplyPort). LPC provides authentication and
> the privileged NT service can use authorization API (AuthZ) to validate the
> caller.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-2198) Remove the need to run NodeManager as privileged account for Windows Secure Container Executor
[ https://issues.apache.org/jira/browse/YARN-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remus Rusanu updated YARN-2198: --- Attachment: YARN-2198.12.patch .12.patch is from trunk and addresses the two TODO review items > Remove the need to run NodeManager as privileged account for Windows Secure > Container Executor > -- > > Key: YARN-2198 > URL: https://issues.apache.org/jira/browse/YARN-2198 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Remus Rusanu >Assignee: Remus Rusanu > Labels: security, windows > Attachments: .YARN-2198.delta.10.patch, YARN-2198.1.patch, > YARN-2198.11.patch, YARN-2198.12.patch, YARN-2198.2.patch, YARN-2198.3.patch, > YARN-2198.delta.4.patch, YARN-2198.delta.5.patch, YARN-2198.delta.6.patch, > YARN-2198.delta.7.patch, YARN-2198.separation.patch, > YARN-2198.trunk.10.patch, YARN-2198.trunk.4.patch, YARN-2198.trunk.5.patch, > YARN-2198.trunk.6.patch, YARN-2198.trunk.8.patch, YARN-2198.trunk.9.patch > > > YARN-1972 introduces a Secure Windows Container Executor. However this > executor requires the process launching the container to be LocalSystem or a > member of the a local Administrators group. Since the process in question is > the NodeManager, the requirement translates to the entire NM to run as a > privileged account, a very large surface area to review and protect. > This proposal is to move the privileged operations into a dedicated NT > service. The NM can run as a low privilege account and communicate with the > privileged NT service when it needs to launch a container. This would reduce > the surface exposed to the high privileges. > There has to exist a secure, authenticated and authorized channel of > communication between the NM and the privileged NT service. Possible > alternatives are a new TCP endpoint, Java RPC etc. My proposal though would > be to use Windows LPC (Local Procedure Calls), which is a Windows platform > specific inter-process communication channel that satisfies all requirements > and is easy to deploy. The privileged NT service would register and listen on > an LPC port (NtCreatePort, NtListenPort). The NM would use JNI to interop > with libwinutils which would host the LPC client code. The client would > connect to the LPC port (NtConnectPort) and send a message requesting a > container launch (NtRequestWaitReplyPort). LPC provides authentication and > the privileged NT service can use authorization API (AuthZ) to validate the > caller. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2198) Remove the need to run NodeManager as privileged account for Windows Secure Container Executor
[
https://issues.apache.org/jira/browse/YARN-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14166718#comment-14166718
]
Remus Rusanu commented on YARN-2198:
> libwinutils.c CreateLogonForUser - confusing name, makes me think a new- >
> CreateLogonTokenForUser?
RR: Fixed
> TestWinUtils - can we add testing specific to security?
RR: Tracked by YARN-2636
> ContainerLaunch launchContainer - nit, why "userName" here, it's user
> everywhere else
RR fixed
> getLocalWrapperScriptBuilder - why not an override instead of conditional
> (see below wrt WindowsContainerExecutor)
>WindowsSecureContainerExecutor - I really think there should be a
>"WindowsContainerExecutor"
RR: I left both as is, predates the WSCE
> it looks like this is only a 64 bit build now, where it used to be 64 and 32.
> I assume this is intentional and ok?
RR: correct. x86 was not possible to build from mvn, and was not required.
> It would be really nice if we could start to separate out some of this new
> functionality from winutils, e.g., make the elevated service functionality
> independent. I see that there is a jira for doing so down the road, which is
> good... it looks like the elevated privilages are just around creating local
> directories and (obviously) spawning the process. If a stand-alone service
> just created and set permissions on those directories, and the java code
> simply checked for their existance and then moved on if they were present, I
> think that a lot of the back-and-forth of the elevation could be dropped to
> just one call to create the base directory and a second to spawn/hand back
> the output handles. Is that correct?
RR: I actually intentionally avoided that. The LCE does it, and the result is a
lot of duplication between Java code in Default Container Executor and C code
in the native container-executor. With the WSCE I preferred to keep the logic
in Java and use native methods just for primitive operations.
>service.c
> // We're now transfering ownership of the duplicated handles to the caller
> + // If the RPC call fails after this point the handles are leaked inside the
> NM process
> this is a little alarming. Doesn't the close() call clean this up, regardless
> of success/ fail?
RR: I added some more comments to clarify that only a process kill or hardware
error can fail after this point. An atomic transfer is not possible.
> why is this conditional check different from all the others?
RR: fixed
> nit anonimous sp anonymous
RR: fixed
> just a line added, pls revert
RR: fixed
> ElevatedFileSystem:delete()
> it appears that the tests for existance, etc, are run in a non-elevated way,
> while the actions are elevated. Is it possible for permissions to be such
> that the non-elevated tests do not see files/directories which are present
> for permission reasons? should those not be elevated also?
RR: It is not possible under a correct configured deployment. Explicit
overwriting permissions can deny this, but that will always be possible (eg.
deny permission explicitly to LocalSystem).
> streamReaderThread.run - using the readLine() instead of following the simple
> buffer copy idiom in ShellCommandExecutor has some efficiency issues, granted
> it looks to be reading "memory sized" data so it may be no big deal, but it
> would be nice to follow the buffer-copy pattern instead
RR: I forgot to address this. Todo.
> ContainerExecutor comment on comment:
RR: Fixed
> ContainerLaunch public void sanitizeEnv(...)
RR: This predates WSCE, I left it as is
> ContainerLocalizer LOG.info(String.format("nRet: %d", nRet)); - not sure this
> should be "info" level
RR: todo, forgot to address it
> getContainerClasspathJarPrivateDir not used in ContainerExecutor.java, we can
> remove that.
RR: fixed
> Unnecessary format change only in YarnConfiguration.we can revert
RR: fixed
> Multiple places exceed 80 column limit code convention.
RR: I think I fixed all new Java code
> DefaultContainerExecutor#buildCommandExecutor
RR: Fixed
> Remove the need to run NodeManager as privileged account for Windows Secure
> Container Executor
> --
>
> Key: YARN-2198
> URL: https://issues.apache.org/jira/browse/YARN-2198
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Remus Rusanu
>Assignee: Remus Rusanu
> Labels: security, windows
> Attachments: .YARN-2198.delta.10.patch, YARN-2198.1.patch,
> YARN-2198.11.patch, YARN-2198.2.patch, YARN-2198.3.patch,
> YARN-2198.delta.4.patch, YARN-2198.delta.5.patch, YARN-2198.delta.6.patch,
> YARN-2198.delta.7.patch, YARN-2198.separation.patch,
> YARN-2198.trunk.10.patch, YARN-2198.trunk.4.patch, YARN-2198.trunk.5.patch,
> YARN-2198.trunk.6.patch, YARN-2198.trun
[jira] [Updated] (YARN-2198) Remove the need to run NodeManager as privileged account for Windows Secure Container Executor
[ https://issues.apache.org/jira/browse/YARN-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remus Rusanu updated YARN-2198: --- Attachment: YARN-2198.11.patch 11.patch is diff from trunk and implements the code review feedback. > Remove the need to run NodeManager as privileged account for Windows Secure > Container Executor > -- > > Key: YARN-2198 > URL: https://issues.apache.org/jira/browse/YARN-2198 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Remus Rusanu >Assignee: Remus Rusanu > Labels: security, windows > Attachments: .YARN-2198.delta.10.patch, YARN-2198.1.patch, > YARN-2198.11.patch, YARN-2198.2.patch, YARN-2198.3.patch, > YARN-2198.delta.4.patch, YARN-2198.delta.5.patch, YARN-2198.delta.6.patch, > YARN-2198.delta.7.patch, YARN-2198.separation.patch, > YARN-2198.trunk.10.patch, YARN-2198.trunk.4.patch, YARN-2198.trunk.5.patch, > YARN-2198.trunk.6.patch, YARN-2198.trunk.8.patch, YARN-2198.trunk.9.patch > > > YARN-1972 introduces a Secure Windows Container Executor. However this > executor requires the process launching the container to be LocalSystem or a > member of the a local Administrators group. Since the process in question is > the NodeManager, the requirement translates to the entire NM to run as a > privileged account, a very large surface area to review and protect. > This proposal is to move the privileged operations into a dedicated NT > service. The NM can run as a low privilege account and communicate with the > privileged NT service when it needs to launch a container. This would reduce > the surface exposed to the high privileges. > There has to exist a secure, authenticated and authorized channel of > communication between the NM and the privileged NT service. Possible > alternatives are a new TCP endpoint, Java RPC etc. My proposal though would > be to use Windows LPC (Local Procedure Calls), which is a Windows platform > specific inter-process communication channel that satisfies all requirements > and is easy to deploy. The privileged NT service would register and listen on > an LPC port (NtCreatePort, NtListenPort). The NM would use JNI to interop > with libwinutils which would host the LPC client code. The client would > connect to the LPC port (NtConnectPort) and send a message requesting a > container launch (NtRequestWaitReplyPort). LPC provides authentication and > the privileged NT service can use authorization API (AuthZ) to validate the > caller. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2198) Remove the need to run NodeManager as privileged account for Windows Secure Container Executor
[ https://issues.apache.org/jira/browse/YARN-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14158337#comment-14158337 ] Remus Rusanu commented on YARN-2198: [~jianhe] I considered investigating that, but the classpath-jar creation is a common feature and may be used by 3rd parties unrelated to Windows security. YARN-358 already talks about using the classpath jar on all platforms, and HBase is looking at taking a dependency on it. So I preferred to stay with the copy approach. > Remove the need to run NodeManager as privileged account for Windows Secure > Container Executor > -- > > Key: YARN-2198 > URL: https://issues.apache.org/jira/browse/YARN-2198 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Remus Rusanu >Assignee: Remus Rusanu > Labels: security, windows > Attachments: .YARN-2198.delta.10.patch, YARN-2198.1.patch, > YARN-2198.2.patch, YARN-2198.3.patch, YARN-2198.delta.4.patch, > YARN-2198.delta.5.patch, YARN-2198.delta.6.patch, YARN-2198.delta.7.patch, > YARN-2198.separation.patch, YARN-2198.trunk.10.patch, > YARN-2198.trunk.4.patch, YARN-2198.trunk.5.patch, YARN-2198.trunk.6.patch, > YARN-2198.trunk.8.patch, YARN-2198.trunk.9.patch > > > YARN-1972 introduces a Secure Windows Container Executor. However this > executor requires the process launching the container to be LocalSystem or a > member of the a local Administrators group. Since the process in question is > the NodeManager, the requirement translates to the entire NM to run as a > privileged account, a very large surface area to review and protect. > This proposal is to move the privileged operations into a dedicated NT > service. The NM can run as a low privilege account and communicate with the > privileged NT service when it needs to launch a container. This would reduce > the surface exposed to the high privileges. > There has to exist a secure, authenticated and authorized channel of > communication between the NM and the privileged NT service. Possible > alternatives are a new TCP endpoint, Java RPC etc. My proposal though would > be to use Windows LPC (Local Procedure Calls), which is a Windows platform > specific inter-process communication channel that satisfies all requirements > and is easy to deploy. The privileged NT service would register and listen on > an LPC port (NtCreatePort, NtListenPort). The NM would use JNI to interop > with libwinutils which would host the LPC client code. The client would > connect to the LPC port (NtConnectPort) and send a message requesting a > container launch (NtRequestWaitReplyPort). LPC provides authentication and > the privileged NT service can use authorization API (AuthZ) to validate the > caller. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2198) Remove the need to run NodeManager as privileged account for Windows Secure Container Executor
[ https://issues.apache.org/jira/browse/YARN-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14158260#comment-14158260 ] Remus Rusanu commented on YARN-2198: thanks for the review [~jianhe]. I will address the issues, but here is the answer to the nmPrivateClasspathJarDir : The NM process does not have the privilege to write into the pwd folder. And the container process does not have the privilege of reading from the NM private dir. Hence the ping-pong of NM writing it in it's private dir, then using the elevated copy operation to move it into the pwd. > Remove the need to run NodeManager as privileged account for Windows Secure > Container Executor > -- > > Key: YARN-2198 > URL: https://issues.apache.org/jira/browse/YARN-2198 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Remus Rusanu >Assignee: Remus Rusanu > Labels: security, windows > Attachments: .YARN-2198.delta.10.patch, YARN-2198.1.patch, > YARN-2198.2.patch, YARN-2198.3.patch, YARN-2198.delta.4.patch, > YARN-2198.delta.5.patch, YARN-2198.delta.6.patch, YARN-2198.delta.7.patch, > YARN-2198.separation.patch, YARN-2198.trunk.10.patch, > YARN-2198.trunk.4.patch, YARN-2198.trunk.5.patch, YARN-2198.trunk.6.patch, > YARN-2198.trunk.8.patch, YARN-2198.trunk.9.patch > > > YARN-1972 introduces a Secure Windows Container Executor. However this > executor requires the process launching the container to be LocalSystem or a > member of the a local Administrators group. Since the process in question is > the NodeManager, the requirement translates to the entire NM to run as a > privileged account, a very large surface area to review and protect. > This proposal is to move the privileged operations into a dedicated NT > service. The NM can run as a low privilege account and communicate with the > privileged NT service when it needs to launch a container. This would reduce > the surface exposed to the high privileges. > There has to exist a secure, authenticated and authorized channel of > communication between the NM and the privileged NT service. Possible > alternatives are a new TCP endpoint, Java RPC etc. My proposal though would > be to use Windows LPC (Local Procedure Calls), which is a Windows platform > specific inter-process communication channel that satisfies all requirements > and is easy to deploy. The privileged NT service would register and listen on > an LPC port (NtCreatePort, NtListenPort). The NM would use JNI to interop > with libwinutils which would host the LPC client code. The client would > connect to the LPC port (NtConnectPort) and send a message requesting a > container launch (NtRequestWaitReplyPort). LPC provides authentication and > the privileged NT service can use authorization API (AuthZ) to validate the > caller. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1063) Winutils needs ability to create task as domain user
[ https://issues.apache.org/jira/browse/YARN-1063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14155373#comment-14155373 ] Remus Rusanu commented on YARN-1063: Contributor credit should also got to Kyle Leckie > Winutils needs ability to create task as domain user > > > Key: YARN-1063 > URL: https://issues.apache.org/jira/browse/YARN-1063 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager > Environment: Windows >Reporter: Kyle Leckie >Assignee: Remus Rusanu > Labels: security, windows > Fix For: 2.6.0 > > Attachments: YARN-1063.2.patch, YARN-1063.3.patch, YARN-1063.4.patch, > YARN-1063.5.patch, YARN-1063.6.patch, YARN-1063.patch > > > h1. Summary: > Securing a Hadoop cluster requires constructing some form of security > boundary around the processes executed in YARN containers. Isolation based on > Windows user isolation seems most feasible. This approach is similar to the > approach taken by the existing LinuxContainerExecutor. The current patch to > winutils.exe adds the ability to create a process as a domain user. > h1. Alternative Methods considered: > h2. Process rights limited by security token restriction: > On Windows access decisions are made by examining the security token of a > process. It is possible to spawn a process with a restricted security token. > Any of the rights granted by SIDs of the default token may be restricted. It > is possible to see this in action by examining the security tone of a > sandboxed process launch be a web browser. Typically the launched process > will have a fully restricted token and need to access machine resources > through a dedicated broker process that enforces a custom security policy. > This broker process mechanism would break compatibility with the typical > Hadoop container process. The Container process must be able to utilize > standard function calls for disk and network IO. I performed some work > looking at ways to ACL the local files to the specific launched without > granting rights to other processes launched on the same machine but found > this to be an overly complex solution. > h2. Relying on APP containers: > Recent versions of windows have the ability to launch processes within an > isolated container. Application containers are supported for execution of > WinRT based executables. This method was ruled out due to the lack of > official support for standard windows APIs. At some point in the future > windows may support functionality similar to BSD jails or Linux containers, > at that point support for containers should be added. > h1. Create As User Feature Description: > h2. Usage: > A new sub command was added to the set of task commands. Here is the syntax: > winutils task createAsUser [TASKNAME] [USERNAME] [COMMAND_LINE] > Some notes: > * The username specified is in the format of "user@domain" > * The machine executing this command must be joined to the domain of the user > specified > * The domain controller must allow the account executing the command access > to the user information. For this join the account to the predefined group > labeled "Pre-Windows 2000 Compatible Access" > * The account running the command must have several rights on the local > machine. These can be managed manually using secpol.msc: > ** "Act as part of the operating system" - SE_TCB_NAME > ** "Replace a process-level token" - SE_ASSIGNPRIMARYTOKEN_NAME > ** "Adjust memory quotas for a process" - SE_INCREASE_QUOTA_NAME > * The launched process will not have rights to the desktop so will not be > able to display any information or create UI. > * The launched process will have no network credentials. Any access of > network resources that requires domain authentication will fail. > h2. Implementation: > Winutils performs the following steps: > # Enable the required privileges for the current process. > # Register as a trusted process with the Local Security Authority (LSA). > # Create a new logon for the user passed on the command line. > # Load/Create a profile on the local machine for the new logon. > # Create a new environment for the new logon. > # Launch the new process in a job with the task name specified and using the > created logon. > # Wait for the JOB to exit. > h2. Future work: > The following work was scoped out of this check in: > * Support for non-domain users or machine that are not domain joined. > * Support for privilege isolation by running the task launcher in a high > privilege service with access over an ACLed named pipe. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-2636) Windows Secure Container Executor: add unit tests for WSCE
Remus Rusanu created YARN-2636: -- Summary: Windows Secure Container Executor: add unit tests for WSCE Key: YARN-2636 URL: https://issues.apache.org/jira/browse/YARN-2636 Project: Hadoop YARN Issue Type: Sub-task Reporter: Remus Rusanu Assignee: Remus Rusanu Priority: Critical As title says. The WSCE has no check-in unit tests. Much of the functionality depends on elevated hadoopwinutilsvc service and cannot be tested, but lets test what is possible to be mocked in Java. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2198) Remove the need to run NodeManager as privileged account for Windows Secure Container Executor
[ https://issues.apache.org/jira/browse/YARN-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14153611#comment-14153611 ] Remus Rusanu commented on YARN-2198: [~cwelch]: thanks for the review! I will address many of the comments with new patch, meantime some reply on issues I won't address: > pom.xml - don’t see a /etc/hadoop or a wsce-site.xml, missed? RR: Not sure what you mean. Do you expect a default wcse-site.xml in hadoop-common/src/conf ? > return (parent == null || parent2f.exists() || mkdirs(parent)) && > (mkOneDir(p2f) || p2f.isDirectory()); > so, I don't get this logic, & believe it will fail if the path exists and is > not a directory. Why not just do if p2f doesn't exist mkdirs(p2f)? seems much > simpler, and drops the need for mkOneDir RR: This is actually the result of a problem Kevin hit during test deployments when NM has access to child dirs but is access denied to parent dirs. Old NM code would attempt to mkdir ever dir in the parent path, all the way to /. With existing dirs with access denied, this would fail, hence the need for my change. There is already a check in the unmodified code for the parent existing and not being a dir, couple of lines above my change. > TestWinUtils: can we add testing specific to security? RR: I would like to add some, but is not at all easy. The core tenet of the WSCE is the elevated privilege required for S4U impersonation and having tests depend on that would pose many problems (false failures). Basically, starting the hadoopwinutilsvc service on the test box is unfeasable. > WindowsSecureContainerExecutor - I really think there should be a > "WindowsContainerExecutor" RR: While I agree that the class architecture separation of secure vs. non-secure and Windows vs. Linux leaves room for improvement, it is not my goal with these JIRAs to address that problem. In fact I do have an explicit opposite mandate, to disturb all the non-secure code paths as little as possible, to minimize regression risks. > Remove the need to run NodeManager as privileged account for Windows Secure > Container Executor > -- > > Key: YARN-2198 > URL: https://issues.apache.org/jira/browse/YARN-2198 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Remus Rusanu >Assignee: Remus Rusanu > Labels: security, windows > Attachments: .YARN-2198.delta.10.patch, YARN-2198.1.patch, > YARN-2198.2.patch, YARN-2198.3.patch, YARN-2198.delta.4.patch, > YARN-2198.delta.5.patch, YARN-2198.delta.6.patch, YARN-2198.delta.7.patch, > YARN-2198.separation.patch, YARN-2198.trunk.10.patch, > YARN-2198.trunk.4.patch, YARN-2198.trunk.5.patch, YARN-2198.trunk.6.patch, > YARN-2198.trunk.8.patch, YARN-2198.trunk.9.patch > > > YARN-1972 introduces a Secure Windows Container Executor. However this > executor requires the process launching the container to be LocalSystem or a > member of the a local Administrators group. Since the process in question is > the NodeManager, the requirement translates to the entire NM to run as a > privileged account, a very large surface area to review and protect. > This proposal is to move the privileged operations into a dedicated NT > service. The NM can run as a low privilege account and communicate with the > privileged NT service when it needs to launch a container. This would reduce > the surface exposed to the high privileges. > There has to exist a secure, authenticated and authorized channel of > communication between the NM and the privileged NT service. Possible > alternatives are a new TCP endpoint, Java RPC etc. My proposal though would > be to use Windows LPC (Local Procedure Calls), which is a Windows platform > specific inter-process communication channel that satisfies all requirements > and is easy to deploy. The privileged NT service would register and listen on > an LPC port (NtCreatePort, NtListenPort). The NM would use JNI to interop > with libwinutils which would host the LPC client code. The client would > connect to the LPC port (NtConnectPort) and send a message requesting a > container launch (NtRequestWaitReplyPort). LPC provides authentication and > the privileged NT service can use authorization API (AuthZ) to validate the > caller. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2623) Linux container executor only use the first local directory to copy token file in container-executor.c.
[
https://issues.apache.org/jira/browse/YARN-2623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14153023#comment-14153023
]
Remus Rusanu commented on YARN-2623:
Note that DCE also picks first local dir, DefaultContainerExecutor.java@99:
{code}
// TODO: Why pick first app dir. The same in LCE why not random?
Path appStorageDir = getFirstApplicationDir(localDirs, user, appId);
{code}
> Linux container executor only use the first local directory to copy token
> file in container-executor.c.
> ---
>
> Key: YARN-2623
> URL: https://issues.apache.org/jira/browse/YARN-2623
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 2.5.0
> Environment: Linux container executor only use the first local
> directory to copy token file in container-executor.c.
>Reporter: zhihai xu
>Assignee: zhihai xu
>
> Linux container executor only use the first local directory to copy token
> file in container-executor.c. if It failed to copy token file to the first
> local directory, the localization failure event will happen. Even though it
> can copy token file to the other local directory successfully. The correct
> way should be to copy token file to the next local directory if the first
> one failed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2198) Remove the need to run NodeManager as privileged account for Windows Secure Container Executor
[ https://issues.apache.org/jira/browse/YARN-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14152042#comment-14152042 ] Remus Rusanu commented on YARN-2198: the last QA -1 is for delta.10.patch, which is not trunk diff. > Remove the need to run NodeManager as privileged account for Windows Secure > Container Executor > -- > > Key: YARN-2198 > URL: https://issues.apache.org/jira/browse/YARN-2198 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Remus Rusanu >Assignee: Remus Rusanu > Labels: security, windows > Attachments: .YARN-2198.delta.10.patch, YARN-2198.1.patch, > YARN-2198.2.patch, YARN-2198.3.patch, YARN-2198.delta.4.patch, > YARN-2198.delta.5.patch, YARN-2198.delta.6.patch, YARN-2198.delta.7.patch, > YARN-2198.separation.patch, YARN-2198.trunk.10.patch, > YARN-2198.trunk.4.patch, YARN-2198.trunk.5.patch, YARN-2198.trunk.6.patch, > YARN-2198.trunk.8.patch, YARN-2198.trunk.9.patch > > > YARN-1972 introduces a Secure Windows Container Executor. However this > executor requires the process launching the container to be LocalSystem or a > member of the a local Administrators group. Since the process in question is > the NodeManager, the requirement translates to the entire NM to run as a > privileged account, a very large surface area to review and protect. > This proposal is to move the privileged operations into a dedicated NT > service. The NM can run as a low privilege account and communicate with the > privileged NT service when it needs to launch a container. This would reduce > the surface exposed to the high privileges. > There has to exist a secure, authenticated and authorized channel of > communication between the NM and the privileged NT service. Possible > alternatives are a new TCP endpoint, Java RPC etc. My proposal though would > be to use Windows LPC (Local Procedure Calls), which is a Windows platform > specific inter-process communication channel that satisfies all requirements > and is easy to deploy. The privileged NT service would register and listen on > an LPC port (NtCreatePort, NtListenPort). The NM would use JNI to interop > with libwinutils which would host the LPC client code. The client would > connect to the LPC port (NtConnectPort) and send a message requesting a > container launch (NtRequestWaitReplyPort). LPC provides authentication and > the privileged NT service can use authorization API (AuthZ) to validate the > caller. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2198) Remove the need to run NodeManager as privileged account for Windows Secure Container Executor
[ https://issues.apache.org/jira/browse/YARN-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remus Rusanu updated YARN-2198: --- Description: YARN-1972 introduces a Secure Windows Container Executor. However this executor requires the process launching the container to be LocalSystem or a member of the a local Administrators group. Since the process in question is the NodeManager, the requirement translates to the entire NM to run as a privileged account, a very large surface area to review and protect. This proposal is to move the privileged operations into a dedicated NT service. The NM can run as a low privilege account and communicate with the privileged NT service when it needs to launch a container. This would reduce the surface exposed to the high privileges. There has to exist a secure, authenticated and authorized channel of communication between the NM and the privileged NT service. Possible alternatives are a new TCP endpoint, Java RPC etc. My proposal though would be to use Windows LPC (Local Procedure Calls), which is a Windows platform specific inter-process communication channel that satisfies all requirements and is easy to deploy. The privileged NT service would register and listen on an LPC port (NtCreatePort, NtListenPort). The NM would use JNI to interop with libwinutils which would host the LPC client code. The client would connect to the LPC port (NtConnectPort) and send a message requesting a container launch (NtRequestWaitReplyPort). LPC provides authentication and the privileged NT service can use authorization API (AuthZ) to validate the caller. was: YARN-1972 introduces a Secure Windows Container Executor. However this executor requires a process launching the container to be LocalSystem or a member of the a local Administrators group. Since the process in question is the NodeManager, the requirement translates to the entire NM to run as a privileged account, a very large surface area to review and protect. This proposal is to move the privileged operations into a dedicated NT service. The NM can run as a low privilege account and communicate with the privileged NT service when it needs to launch a container. This would reduce the surface exposed to the high privileges. There has to exist a secure, authenticated and authorized channel of communication between the NM and the privileged NT service. Possible alternatives are a new TCP endpoint, Java RPC etc. My proposal though would be to use Windows LPC (Local Procedure Calls), which is a Windows platform specific inter-process communication channel that satisfies all requirements and is easy to deploy. The privileged NT service would register and listen on an LPC port (NtCreatePort, NtListenPort). The NM would use JNI to interop with libwinutils which would host the LPC client code. The client would connect to the LPC port (NtConnectPort) and send a message requesting a container launch (NtRequestWaitReplyPort). LPC provides authentication and the privileged NT service can use authorization API (AuthZ) to validate the caller. > Remove the need to run NodeManager as privileged account for Windows Secure > Container Executor > -- > > Key: YARN-2198 > URL: https://issues.apache.org/jira/browse/YARN-2198 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Remus Rusanu >Assignee: Remus Rusanu > Labels: security, windows > Attachments: .YARN-2198.delta.10.patch, YARN-2198.1.patch, > YARN-2198.2.patch, YARN-2198.3.patch, YARN-2198.delta.4.patch, > YARN-2198.delta.5.patch, YARN-2198.delta.6.patch, YARN-2198.delta.7.patch, > YARN-2198.separation.patch, YARN-2198.trunk.10.patch, > YARN-2198.trunk.4.patch, YARN-2198.trunk.5.patch, YARN-2198.trunk.6.patch, > YARN-2198.trunk.8.patch, YARN-2198.trunk.9.patch > > > YARN-1972 introduces a Secure Windows Container Executor. However this > executor requires the process launching the container to be LocalSystem or a > member of the a local Administrators group. Since the process in question is > the NodeManager, the requirement translates to the entire NM to run as a > privileged account, a very large surface area to review and protect. > This proposal is to move the privileged operations into a dedicated NT > service. The NM can run as a low privilege account and communicate with the > privileged NT service when it needs to launch a container. This would reduce > the surface exposed to the high privileges. > There has to exist a secure, authenticated and authorized channel of > communication between the NM and the privileged NT service. Possible > alternatives are a new TCP endpoint, Java RPC etc. My proposal though would > be to use Windows LPC (Local Procedure Calls), which i
[jira] [Updated] (YARN-2198) Remove the need to run NodeManager as privileged account for Windows Secure Container Executor
[ https://issues.apache.org/jira/browse/YARN-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remus Rusanu updated YARN-2198: --- Attachment: .YARN-2198.delta.10.patch delta.10 is the delta from YANR-1972 corresponding to .trunk.10 > Remove the need to run NodeManager as privileged account for Windows Secure > Container Executor > -- > > Key: YARN-2198 > URL: https://issues.apache.org/jira/browse/YARN-2198 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Remus Rusanu >Assignee: Remus Rusanu > Labels: security, windows > Attachments: .YARN-2198.delta.10.patch, YARN-2198.1.patch, > YARN-2198.2.patch, YARN-2198.3.patch, YARN-2198.delta.4.patch, > YARN-2198.delta.5.patch, YARN-2198.delta.6.patch, YARN-2198.delta.7.patch, > YARN-2198.separation.patch, YARN-2198.trunk.10.patch, > YARN-2198.trunk.4.patch, YARN-2198.trunk.5.patch, YARN-2198.trunk.6.patch, > YARN-2198.trunk.8.patch, YARN-2198.trunk.9.patch > > > YARN-1972 introduces a Secure Windows Container Executor. However this > executor requires a the process launching the container to be LocalSystem or > a member of the a local Administrators group. Since the process in question > is the NodeManager, the requirement translates to the entire NM to run as a > privileged account, a very large surface area to review and protect. > This proposal is to move the privileged operations into a dedicated NT > service. The NM can run as a low privilege account and communicate with the > privileged NT service when it needs to launch a container. This would reduce > the surface exposed to the high privileges. > There has to exist a secure, authenticated and authorized channel of > communication between the NM and the privileged NT service. Possible > alternatives are a new TCP endpoint, Java RPC etc. My proposal though would > be to use Windows LPC (Local Procedure Calls), which is a Windows platform > specific inter-process communication channel that satisfies all requirements > and is easy to deploy. The privileged NT service would register and listen on > an LPC port (NtCreatePort, NtListenPort). The NM would use JNI to interop > with libwinutils which would host the LPC client code. The client would > connect to the LPC port (NtConnectPort) and send a message requesting a > container launch (NtRequestWaitReplyPort). LPC provides authentication and > the privileged NT service can use authorization API (AuthZ) to validate the > caller. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2357) Port Windows Secure Container Executor YARN-1063, YARN-1972, YARN-2198 changes to branch-2
[ https://issues.apache.org/jira/browse/YARN-2357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remus Rusanu updated YARN-2357: --- Attachment: YARN-2357.3.patch > Port Windows Secure Container Executor YARN-1063, YARN-1972, YARN-2198 > changes to branch-2 > -- > > Key: YARN-2357 > URL: https://issues.apache.org/jira/browse/YARN-2357 > Project: Hadoop YARN > Issue Type: Task > Components: nodemanager >Affects Versions: 2.4.0 >Reporter: Remus Rusanu >Assignee: Remus Rusanu >Priority: Critical > Labels: security, windows > Attachments: YARN-2357.1.patch, YARN-2357.2.patch, YARN-2357.3.patch > > > As title says. Once YARN-1063, YARN-1972 and YARN-2198 are committed to > trunk, they need to be backported to branch-2 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2357) Port Windows Secure Container Executor YARN-1063, YARN-1972, YARN-2198 changes to branch-2
[ https://issues.apache.org/jira/browse/YARN-2357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remus Rusanu updated YARN-2357: --- Attachment: (was: YARN-2357.3.patch) > Port Windows Secure Container Executor YARN-1063, YARN-1972, YARN-2198 > changes to branch-2 > -- > > Key: YARN-2357 > URL: https://issues.apache.org/jira/browse/YARN-2357 > Project: Hadoop YARN > Issue Type: Task > Components: nodemanager >Affects Versions: 2.4.0 >Reporter: Remus Rusanu >Assignee: Remus Rusanu >Priority: Critical > Labels: security, windows > Attachments: YARN-2357.1.patch, YARN-2357.2.patch, YARN-2357.3.patch > > > As title says. Once YARN-1063, YARN-1972 and YARN-2198 are committed to > trunk, they need to be backported to branch-2 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2357) Port Windows Secure Container Executor YARN-1063, YARN-1972, YARN-2198 changes to branch-2
[ https://issues.apache.org/jira/browse/YARN-2357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remus Rusanu updated YARN-2357: --- Attachment: YARN-2357.3.patch .3.patch is the port of YARN-2198.trunk.10.patch > Port Windows Secure Container Executor YARN-1063, YARN-1972, YARN-2198 > changes to branch-2 > -- > > Key: YARN-2357 > URL: https://issues.apache.org/jira/browse/YARN-2357 > Project: Hadoop YARN > Issue Type: Task > Components: nodemanager >Affects Versions: 2.4.0 >Reporter: Remus Rusanu >Assignee: Remus Rusanu >Priority: Critical > Labels: security, windows > Attachments: YARN-2357.1.patch, YARN-2357.2.patch, YARN-2357.3.patch > > > As title says. Once YARN-1063, YARN-1972 and YARN-2198 are committed to > trunk, they need to be backported to branch-2 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2198) Remove the need to run NodeManager as privileged account for Windows Secure Container Executor
[ https://issues.apache.org/jira/browse/YARN-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14149002#comment-14149002 ] Remus Rusanu commented on YARN-2198: The findbugs issue is HADOOP-11122 > Remove the need to run NodeManager as privileged account for Windows Secure > Container Executor > -- > > Key: YARN-2198 > URL: https://issues.apache.org/jira/browse/YARN-2198 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Remus Rusanu >Assignee: Remus Rusanu > Labels: security, windows > Attachments: YARN-2198.1.patch, YARN-2198.2.patch, YARN-2198.3.patch, > YARN-2198.delta.4.patch, YARN-2198.delta.5.patch, YARN-2198.delta.6.patch, > YARN-2198.delta.7.patch, YARN-2198.separation.patch, > YARN-2198.trunk.10.patch, YARN-2198.trunk.4.patch, YARN-2198.trunk.5.patch, > YARN-2198.trunk.6.patch, YARN-2198.trunk.8.patch, YARN-2198.trunk.9.patch > > > YARN-1972 introduces a Secure Windows Container Executor. However this > executor requires a the process launching the container to be LocalSystem or > a member of the a local Administrators group. Since the process in question > is the NodeManager, the requirement translates to the entire NM to run as a > privileged account, a very large surface area to review and protect. > This proposal is to move the privileged operations into a dedicated NT > service. The NM can run as a low privilege account and communicate with the > privileged NT service when it needs to launch a container. This would reduce > the surface exposed to the high privileges. > There has to exist a secure, authenticated and authorized channel of > communication between the NM and the privileged NT service. Possible > alternatives are a new TCP endpoint, Java RPC etc. My proposal though would > be to use Windows LPC (Local Procedure Calls), which is a Windows platform > specific inter-process communication channel that satisfies all requirements > and is easy to deploy. The privileged NT service would register and listen on > an LPC port (NtCreatePort, NtListenPort). The NM would use JNI to interop > with libwinutils which would host the LPC client code. The client would > connect to the LPC port (NtConnectPort) and send a message requesting a > container launch (NtRequestWaitReplyPort). LPC provides authentication and > the privileged NT service can use authorization API (AuthZ) to validate the > caller. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2198) Remove the need to run NodeManager as privileged account for Windows Secure Container Executor
[ https://issues.apache.org/jira/browse/YARN-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remus Rusanu updated YARN-2198: --- Attachment: YARN-2198.trunk.10.patch > Remove the need to run NodeManager as privileged account for Windows Secure > Container Executor > -- > > Key: YARN-2198 > URL: https://issues.apache.org/jira/browse/YARN-2198 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Remus Rusanu >Assignee: Remus Rusanu > Labels: security, windows > Attachments: YARN-2198.1.patch, YARN-2198.2.patch, YARN-2198.3.patch, > YARN-2198.delta.4.patch, YARN-2198.delta.5.patch, YARN-2198.delta.6.patch, > YARN-2198.delta.7.patch, YARN-2198.separation.patch, > YARN-2198.trunk.10.patch, YARN-2198.trunk.4.patch, YARN-2198.trunk.5.patch, > YARN-2198.trunk.6.patch, YARN-2198.trunk.8.patch, YARN-2198.trunk.9.patch > > > YARN-1972 introduces a Secure Windows Container Executor. However this > executor requires a the process launching the container to be LocalSystem or > a member of the a local Administrators group. Since the process in question > is the NodeManager, the requirement translates to the entire NM to run as a > privileged account, a very large surface area to review and protect. > This proposal is to move the privileged operations into a dedicated NT > service. The NM can run as a low privilege account and communicate with the > privileged NT service when it needs to launch a container. This would reduce > the surface exposed to the high privileges. > There has to exist a secure, authenticated and authorized channel of > communication between the NM and the privileged NT service. Possible > alternatives are a new TCP endpoint, Java RPC etc. My proposal though would > be to use Windows LPC (Local Procedure Calls), which is a Windows platform > specific inter-process communication channel that satisfies all requirements > and is easy to deploy. The privileged NT service would register and listen on > an LPC port (NtCreatePort, NtListenPort). The NM would use JNI to interop > with libwinutils which would host the LPC client code. The client would > connect to the LPC port (NtConnectPort) and send a message requesting a > container launch (NtRequestWaitReplyPort). LPC provides authentication and > the privileged NT service can use authorization API (AuthZ) to validate the > caller. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2198) Remove the need to run NodeManager as privileged account for Windows Secure Container Executor
[ https://issues.apache.org/jira/browse/YARN-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remus Rusanu updated YARN-2198: --- Attachment: (was: YARN-2198.trunk.10.patch) > Remove the need to run NodeManager as privileged account for Windows Secure > Container Executor > -- > > Key: YARN-2198 > URL: https://issues.apache.org/jira/browse/YARN-2198 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Remus Rusanu >Assignee: Remus Rusanu > Labels: security, windows > Attachments: YARN-2198.1.patch, YARN-2198.2.patch, YARN-2198.3.patch, > YARN-2198.delta.4.patch, YARN-2198.delta.5.patch, YARN-2198.delta.6.patch, > YARN-2198.delta.7.patch, YARN-2198.separation.patch, YARN-2198.trunk.4.patch, > YARN-2198.trunk.5.patch, YARN-2198.trunk.6.patch, YARN-2198.trunk.8.patch, > YARN-2198.trunk.9.patch > > > YARN-1972 introduces a Secure Windows Container Executor. However this > executor requires a the process launching the container to be LocalSystem or > a member of the a local Administrators group. Since the process in question > is the NodeManager, the requirement translates to the entire NM to run as a > privileged account, a very large surface area to review and protect. > This proposal is to move the privileged operations into a dedicated NT > service. The NM can run as a low privilege account and communicate with the > privileged NT service when it needs to launch a container. This would reduce > the surface exposed to the high privileges. > There has to exist a secure, authenticated and authorized channel of > communication between the NM and the privileged NT service. Possible > alternatives are a new TCP endpoint, Java RPC etc. My proposal though would > be to use Windows LPC (Local Procedure Calls), which is a Windows platform > specific inter-process communication channel that satisfies all requirements > and is easy to deploy. The privileged NT service would register and listen on > an LPC port (NtCreatePort, NtListenPort). The NM would use JNI to interop > with libwinutils which would host the LPC client code. The client would > connect to the LPC port (NtConnectPort) and send a message requesting a > container launch (NtRequestWaitReplyPort). LPC provides authentication and > the privileged NT service can use authorization API (AuthZ) to validate the > caller. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2198) Remove the need to run NodeManager as privileged account for Windows Secure Container Executor
[
https://issues.apache.org/jira/browse/YARN-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14147952#comment-14147952
]
Remus Rusanu commented on YARN-2198:
The findbugs warning is
{code}
Inconsistent synchronization of
org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.delegationTokenSequenceNumber;
locked 71% of time
{code}
> Remove the need to run NodeManager as privileged account for Windows Secure
> Container Executor
> --
>
> Key: YARN-2198
> URL: https://issues.apache.org/jira/browse/YARN-2198
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Remus Rusanu
>Assignee: Remus Rusanu
> Labels: security, windows
> Attachments: YARN-2198.1.patch, YARN-2198.2.patch, YARN-2198.3.patch,
> YARN-2198.delta.4.patch, YARN-2198.delta.5.patch, YARN-2198.delta.6.patch,
> YARN-2198.delta.7.patch, YARN-2198.separation.patch,
> YARN-2198.trunk.10.patch, YARN-2198.trunk.4.patch, YARN-2198.trunk.5.patch,
> YARN-2198.trunk.6.patch, YARN-2198.trunk.8.patch, YARN-2198.trunk.9.patch
>
>
> YARN-1972 introduces a Secure Windows Container Executor. However this
> executor requires a the process launching the container to be LocalSystem or
> a member of the a local Administrators group. Since the process in question
> is the NodeManager, the requirement translates to the entire NM to run as a
> privileged account, a very large surface area to review and protect.
> This proposal is to move the privileged operations into a dedicated NT
> service. The NM can run as a low privilege account and communicate with the
> privileged NT service when it needs to launch a container. This would reduce
> the surface exposed to the high privileges.
> There has to exist a secure, authenticated and authorized channel of
> communication between the NM and the privileged NT service. Possible
> alternatives are a new TCP endpoint, Java RPC etc. My proposal though would
> be to use Windows LPC (Local Procedure Calls), which is a Windows platform
> specific inter-process communication channel that satisfies all requirements
> and is easy to deploy. The privileged NT service would register and listen on
> an LPC port (NtCreatePort, NtListenPort). The NM would use JNI to interop
> with libwinutils which would host the LPC client code. The client would
> connect to the LPC port (NtConnectPort) and send a message requesting a
> container launch (NtRequestWaitReplyPort). LPC provides authentication and
> the privileged NT service can use authorization API (AuthZ) to validate the
> caller.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2198) Remove the need to run NodeManager as privileged account for Windows Secure Container Executor
[
https://issues.apache.org/jira/browse/YARN-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14147946#comment-14147946
]
Remus Rusanu commented on YARN-2198:
Core test failure is:
{code}
Tests run: 7, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 120.538 sec <<<
FAILURE! - in org.apache.hadoop.crypto.random.TestOsSecureRandom
testOsSecureRandomSetConf(org.apache.hadoop.crypto.random.TestOsSecureRandom)
Time elapsed: 120.011 sec <<< ERROR!
java.lang.Exception: test timed out after 12 milliseconds
at java.io.FileInputStream.readBytes(Native Method)
at java.io.FileInputStream.read(FileInputStream.java:220)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:264)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:306)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158)
at java.io.InputStreamReader.read(InputStreamReader.java:167)
at java.io.BufferedReader.fill(BufferedReader.java:136)
at java.io.BufferedReader.read1(BufferedReader.java:187)
at java.io.BufferedReader.read(BufferedReader.java:261)
at
org.apache.hadoop.util.Shell$ShellCommandExecutor.parseExecResult(Shell.java:727)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:524)
at org.apache.hadoop.util.Shell.run(Shell.java:455)
at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:714)
at
org.apache.hadoop.crypto.random.TestOsSecureRandom.testOsSecureRandomSetConf(TestOsSecureRandom.java:149)
{code}
> Remove the need to run NodeManager as privileged account for Windows Secure
> Container Executor
> --
>
> Key: YARN-2198
> URL: https://issues.apache.org/jira/browse/YARN-2198
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Remus Rusanu
>Assignee: Remus Rusanu
> Labels: security, windows
> Attachments: YARN-2198.1.patch, YARN-2198.2.patch, YARN-2198.3.patch,
> YARN-2198.delta.4.patch, YARN-2198.delta.5.patch, YARN-2198.delta.6.patch,
> YARN-2198.delta.7.patch, YARN-2198.separation.patch,
> YARN-2198.trunk.10.patch, YARN-2198.trunk.4.patch, YARN-2198.trunk.5.patch,
> YARN-2198.trunk.6.patch, YARN-2198.trunk.8.patch, YARN-2198.trunk.9.patch
>
>
> YARN-1972 introduces a Secure Windows Container Executor. However this
> executor requires a the process launching the container to be LocalSystem or
> a member of the a local Administrators group. Since the process in question
> is the NodeManager, the requirement translates to the entire NM to run as a
> privileged account, a very large surface area to review and protect.
> This proposal is to move the privileged operations into a dedicated NT
> service. The NM can run as a low privilege account and communicate with the
> privileged NT service when it needs to launch a container. This would reduce
> the surface exposed to the high privileges.
> There has to exist a secure, authenticated and authorized channel of
> communication between the NM and the privileged NT service. Possible
> alternatives are a new TCP endpoint, Java RPC etc. My proposal though would
> be to use Windows LPC (Local Procedure Calls), which is a Windows platform
> specific inter-process communication channel that satisfies all requirements
> and is easy to deploy. The privileged NT service would register and listen on
> an LPC port (NtCreatePort, NtListenPort). The NM would use JNI to interop
> with libwinutils which would host the LPC client code. The client would
> connect to the LPC port (NtConnectPort) and send a message requesting a
> container launch (NtRequestWaitReplyPort). LPC provides authentication and
> the privileged NT service can use authorization API (AuthZ) to validate the
> caller.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-2198) Remove the need to run NodeManager as privileged account for Windows Secure Container Executor
[ https://issues.apache.org/jira/browse/YARN-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remus Rusanu updated YARN-2198: --- Attachment: YARN-2198.trunk.10.patch > Remove the need to run NodeManager as privileged account for Windows Secure > Container Executor > -- > > Key: YARN-2198 > URL: https://issues.apache.org/jira/browse/YARN-2198 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Remus Rusanu >Assignee: Remus Rusanu > Labels: security, windows > Attachments: YARN-2198.1.patch, YARN-2198.2.patch, YARN-2198.3.patch, > YARN-2198.delta.4.patch, YARN-2198.delta.5.patch, YARN-2198.delta.6.patch, > YARN-2198.delta.7.patch, YARN-2198.separation.patch, > YARN-2198.trunk.10.patch, YARN-2198.trunk.4.patch, YARN-2198.trunk.5.patch, > YARN-2198.trunk.6.patch, YARN-2198.trunk.8.patch, YARN-2198.trunk.9.patch > > > YARN-1972 introduces a Secure Windows Container Executor. However this > executor requires a the process launching the container to be LocalSystem or > a member of the a local Administrators group. Since the process in question > is the NodeManager, the requirement translates to the entire NM to run as a > privileged account, a very large surface area to review and protect. > This proposal is to move the privileged operations into a dedicated NT > service. The NM can run as a low privilege account and communicate with the > privileged NT service when it needs to launch a container. This would reduce > the surface exposed to the high privileges. > There has to exist a secure, authenticated and authorized channel of > communication between the NM and the privileged NT service. Possible > alternatives are a new TCP endpoint, Java RPC etc. My proposal though would > be to use Windows LPC (Local Procedure Calls), which is a Windows platform > specific inter-process communication channel that satisfies all requirements > and is easy to deploy. The privileged NT service would register and listen on > an LPC port (NtCreatePort, NtListenPort). The NM would use JNI to interop > with libwinutils which would host the LPC client code. The client would > connect to the LPC port (NtConnectPort) and send a message requesting a > container launch (NtRequestWaitReplyPort). LPC provides authentication and > the privileged NT service can use authorization API (AuthZ) to validate the > caller. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2198) Remove the need to run NodeManager as privileged account for Windows Secure Container Executor
[ https://issues.apache.org/jira/browse/YARN-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remus Rusanu updated YARN-2198: --- Attachment: (was: YARN-2198.trunk.10.patch) > Remove the need to run NodeManager as privileged account for Windows Secure > Container Executor > -- > > Key: YARN-2198 > URL: https://issues.apache.org/jira/browse/YARN-2198 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Remus Rusanu >Assignee: Remus Rusanu > Labels: security, windows > Attachments: YARN-2198.1.patch, YARN-2198.2.patch, YARN-2198.3.patch, > YARN-2198.delta.4.patch, YARN-2198.delta.5.patch, YARN-2198.delta.6.patch, > YARN-2198.delta.7.patch, YARN-2198.separation.patch, > YARN-2198.trunk.10.patch, YARN-2198.trunk.4.patch, YARN-2198.trunk.5.patch, > YARN-2198.trunk.6.patch, YARN-2198.trunk.8.patch, YARN-2198.trunk.9.patch > > > YARN-1972 introduces a Secure Windows Container Executor. However this > executor requires a the process launching the container to be LocalSystem or > a member of the a local Administrators group. Since the process in question > is the NodeManager, the requirement translates to the entire NM to run as a > privileged account, a very large surface area to review and protect. > This proposal is to move the privileged operations into a dedicated NT > service. The NM can run as a low privilege account and communicate with the > privileged NT service when it needs to launch a container. This would reduce > the surface exposed to the high privileges. > There has to exist a secure, authenticated and authorized channel of > communication between the NM and the privileged NT service. Possible > alternatives are a new TCP endpoint, Java RPC etc. My proposal though would > be to use Windows LPC (Local Procedure Calls), which is a Windows platform > specific inter-process communication channel that satisfies all requirements > and is easy to deploy. The privileged NT service would register and listen on > an LPC port (NtCreatePort, NtListenPort). The NM would use JNI to interop > with libwinutils which would host the LPC client code. The client would > connect to the LPC port (NtConnectPort) and send a message requesting a > container launch (NtRequestWaitReplyPort). LPC provides authentication and > the privileged NT service can use authorization API (AuthZ) to validate the > caller. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2198) Remove the need to run NodeManager as privileged account for Windows Secure Container Executor
[ https://issues.apache.org/jira/browse/YARN-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remus Rusanu updated YARN-2198: --- Attachment: YARN-2198.trunk.10.patch > Remove the need to run NodeManager as privileged account for Windows Secure > Container Executor > -- > > Key: YARN-2198 > URL: https://issues.apache.org/jira/browse/YARN-2198 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Remus Rusanu >Assignee: Remus Rusanu > Labels: security, windows > Attachments: YARN-2198.1.patch, YARN-2198.2.patch, YARN-2198.3.patch, > YARN-2198.delta.4.patch, YARN-2198.delta.5.patch, YARN-2198.delta.6.patch, > YARN-2198.delta.7.patch, YARN-2198.separation.patch, > YARN-2198.trunk.10.patch, YARN-2198.trunk.4.patch, YARN-2198.trunk.5.patch, > YARN-2198.trunk.6.patch, YARN-2198.trunk.8.patch, YARN-2198.trunk.9.patch > > > YARN-1972 introduces a Secure Windows Container Executor. However this > executor requires a the process launching the container to be LocalSystem or > a member of the a local Administrators group. Since the process in question > is the NodeManager, the requirement translates to the entire NM to run as a > privileged account, a very large surface area to review and protect. > This proposal is to move the privileged operations into a dedicated NT > service. The NM can run as a low privilege account and communicate with the > privileged NT service when it needs to launch a container. This would reduce > the surface exposed to the high privileges. > There has to exist a secure, authenticated and authorized channel of > communication between the NM and the privileged NT service. Possible > alternatives are a new TCP endpoint, Java RPC etc. My proposal though would > be to use Windows LPC (Local Procedure Calls), which is a Windows platform > specific inter-process communication channel that satisfies all requirements > and is easy to deploy. The privileged NT service would register and listen on > an LPC port (NtCreatePort, NtListenPort). The NM would use JNI to interop > with libwinutils which would host the LPC client code. The client would > connect to the LPC port (NtConnectPort) and send a message requesting a > container launch (NtRequestWaitReplyPort). LPC provides authentication and > the privileged NT service can use authorization API (AuthZ) to validate the > caller. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2198) Remove the need to run NodeManager as privileged account for Windows Secure Container Executor
[ https://issues.apache.org/jira/browse/YARN-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remus Rusanu updated YARN-2198: --- Attachment: (was: YARN-2198.trunk.10.patch) > Remove the need to run NodeManager as privileged account for Windows Secure > Container Executor > -- > > Key: YARN-2198 > URL: https://issues.apache.org/jira/browse/YARN-2198 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Remus Rusanu >Assignee: Remus Rusanu > Labels: security, windows > Attachments: YARN-2198.1.patch, YARN-2198.2.patch, YARN-2198.3.patch, > YARN-2198.delta.4.patch, YARN-2198.delta.5.patch, YARN-2198.delta.6.patch, > YARN-2198.delta.7.patch, YARN-2198.separation.patch, YARN-2198.trunk.4.patch, > YARN-2198.trunk.5.patch, YARN-2198.trunk.6.patch, YARN-2198.trunk.8.patch, > YARN-2198.trunk.9.patch > > > YARN-1972 introduces a Secure Windows Container Executor. However this > executor requires a the process launching the container to be LocalSystem or > a member of the a local Administrators group. Since the process in question > is the NodeManager, the requirement translates to the entire NM to run as a > privileged account, a very large surface area to review and protect. > This proposal is to move the privileged operations into a dedicated NT > service. The NM can run as a low privilege account and communicate with the > privileged NT service when it needs to launch a container. This would reduce > the surface exposed to the high privileges. > There has to exist a secure, authenticated and authorized channel of > communication between the NM and the privileged NT service. Possible > alternatives are a new TCP endpoint, Java RPC etc. My proposal though would > be to use Windows LPC (Local Procedure Calls), which is a Windows platform > specific inter-process communication channel that satisfies all requirements > and is easy to deploy. The privileged NT service would register and listen on > an LPC port (NtCreatePort, NtListenPort). The NM would use JNI to interop > with libwinutils which would host the LPC client code. The client would > connect to the LPC port (NtConnectPort) and send a message requesting a > container launch (NtRequestWaitReplyPort). LPC provides authentication and > the privileged NT service can use authorization API (AuthZ) to validate the > caller. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2198) Remove the need to run NodeManager as privileged account for Windows Secure Container Executor
[ https://issues.apache.org/jira/browse/YARN-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remus Rusanu updated YARN-2198: --- Attachment: (was: YARN-2198.trunk.10.patch) > Remove the need to run NodeManager as privileged account for Windows Secure > Container Executor > -- > > Key: YARN-2198 > URL: https://issues.apache.org/jira/browse/YARN-2198 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Remus Rusanu >Assignee: Remus Rusanu > Labels: security, windows > Attachments: YARN-2198.1.patch, YARN-2198.2.patch, YARN-2198.3.patch, > YARN-2198.delta.4.patch, YARN-2198.delta.5.patch, YARN-2198.delta.6.patch, > YARN-2198.delta.7.patch, YARN-2198.separation.patch, > YARN-2198.trunk.10.patch, YARN-2198.trunk.4.patch, YARN-2198.trunk.5.patch, > YARN-2198.trunk.6.patch, YARN-2198.trunk.8.patch, YARN-2198.trunk.9.patch > > > YARN-1972 introduces a Secure Windows Container Executor. However this > executor requires a the process launching the container to be LocalSystem or > a member of the a local Administrators group. Since the process in question > is the NodeManager, the requirement translates to the entire NM to run as a > privileged account, a very large surface area to review and protect. > This proposal is to move the privileged operations into a dedicated NT > service. The NM can run as a low privilege account and communicate with the > privileged NT service when it needs to launch a container. This would reduce > the surface exposed to the high privileges. > There has to exist a secure, authenticated and authorized channel of > communication between the NM and the privileged NT service. Possible > alternatives are a new TCP endpoint, Java RPC etc. My proposal though would > be to use Windows LPC (Local Procedure Calls), which is a Windows platform > specific inter-process communication channel that satisfies all requirements > and is easy to deploy. The privileged NT service would register and listen on > an LPC port (NtCreatePort, NtListenPort). The NM would use JNI to interop > with libwinutils which would host the LPC client code. The client would > connect to the LPC port (NtConnectPort) and send a message requesting a > container launch (NtRequestWaitReplyPort). LPC provides authentication and > the privileged NT service can use authorization API (AuthZ) to validate the > caller. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2198) Remove the need to run NodeManager as privileged account for Windows Secure Container Executor
[ https://issues.apache.org/jira/browse/YARN-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remus Rusanu updated YARN-2198: --- Attachment: YARN-2198.trunk.10.patch > Remove the need to run NodeManager as privileged account for Windows Secure > Container Executor > -- > > Key: YARN-2198 > URL: https://issues.apache.org/jira/browse/YARN-2198 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Remus Rusanu >Assignee: Remus Rusanu > Labels: security, windows > Attachments: YARN-2198.1.patch, YARN-2198.2.patch, YARN-2198.3.patch, > YARN-2198.delta.4.patch, YARN-2198.delta.5.patch, YARN-2198.delta.6.patch, > YARN-2198.delta.7.patch, YARN-2198.separation.patch, > YARN-2198.trunk.10.patch, YARN-2198.trunk.4.patch, YARN-2198.trunk.5.patch, > YARN-2198.trunk.6.patch, YARN-2198.trunk.8.patch, YARN-2198.trunk.9.patch > > > YARN-1972 introduces a Secure Windows Container Executor. However this > executor requires a the process launching the container to be LocalSystem or > a member of the a local Administrators group. Since the process in question > is the NodeManager, the requirement translates to the entire NM to run as a > privileged account, a very large surface area to review and protect. > This proposal is to move the privileged operations into a dedicated NT > service. The NM can run as a low privilege account and communicate with the > privileged NT service when it needs to launch a container. This would reduce > the surface exposed to the high privileges. > There has to exist a secure, authenticated and authorized channel of > communication between the NM and the privileged NT service. Possible > alternatives are a new TCP endpoint, Java RPC etc. My proposal though would > be to use Windows LPC (Local Procedure Calls), which is a Windows platform > specific inter-process communication channel that satisfies all requirements > and is easy to deploy. The privileged NT service would register and listen on > an LPC port (NtCreatePort, NtListenPort). The NM would use JNI to interop > with libwinutils which would host the LPC client code. The client would > connect to the LPC port (NtConnectPort) and send a message requesting a > container launch (NtRequestWaitReplyPort). LPC provides authentication and > the privileged NT service can use authorization API (AuthZ) to validate the > caller. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2198) Remove the need to run NodeManager as privileged account for Windows Secure Container Executor
[ https://issues.apache.org/jira/browse/YARN-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remus Rusanu updated YARN-2198: --- Attachment: YARN-2198.trunk.10.patch .trunk.10.patch contains the fixes for YARN-2391, YARN-2580, YARN-2587, YARN-2586, YARN-2551 and the renames suggested by [~cwelch]. > Remove the need to run NodeManager as privileged account for Windows Secure > Container Executor > -- > > Key: YARN-2198 > URL: https://issues.apache.org/jira/browse/YARN-2198 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Remus Rusanu >Assignee: Remus Rusanu > Labels: security, windows > Attachments: YARN-2198.1.patch, YARN-2198.2.patch, YARN-2198.3.patch, > YARN-2198.delta.4.patch, YARN-2198.delta.5.patch, YARN-2198.delta.6.patch, > YARN-2198.delta.7.patch, YARN-2198.separation.patch, > YARN-2198.trunk.10.patch, YARN-2198.trunk.4.patch, YARN-2198.trunk.5.patch, > YARN-2198.trunk.6.patch, YARN-2198.trunk.8.patch, YARN-2198.trunk.9.patch > > > YARN-1972 introduces a Secure Windows Container Executor. However this > executor requires a the process launching the container to be LocalSystem or > a member of the a local Administrators group. Since the process in question > is the NodeManager, the requirement translates to the entire NM to run as a > privileged account, a very large surface area to review and protect. > This proposal is to move the privileged operations into a dedicated NT > service. The NM can run as a low privilege account and communicate with the > privileged NT service when it needs to launch a container. This would reduce > the surface exposed to the high privileges. > There has to exist a secure, authenticated and authorized channel of > communication between the NM and the privileged NT service. Possible > alternatives are a new TCP endpoint, Java RPC etc. My proposal though would > be to use Windows LPC (Local Procedure Calls), which is a Windows platform > specific inter-process communication channel that satisfies all requirements > and is easy to deploy. The privileged NT service would register and listen on > an LPC port (NtCreatePort, NtListenPort). The NM would use JNI to interop > with libwinutils which would host the LPC client code. The client would > connect to the LPC port (NtConnectPort) and send a message requesting a > container launch (NtRequestWaitReplyPort). LPC provides authentication and > the privileged NT service can use authorization API (AuthZ) to validate the > caller. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2391) Windows Secure Container Executor helper service should assign launched process to the NM job
[
https://issues.apache.org/jira/browse/YARN-2391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14147678#comment-14147678
]
Remus Rusanu commented on YARN-2391:
The patch will be included in next YARN-2198 patch
> Windows Secure Container Executor helper service should assign launched
> process to the NM job
> -
>
> Key: YARN-2391
> URL: https://issues.apache.org/jira/browse/YARN-2391
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Remus Rusanu
>Assignee: Remus Rusanu
>Priority: Critical
> Labels: security, windows
> Attachments: YARN-2391.1.patch
>
>
> The YARN-2198 NM helper service needs to make sure the launched process is
> added to the NM job ('job' as in Windows NT job objects, not Hadoop jobs).
> This ensures that NM termination ensures launched process termination.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (YARN-2391) Windows Secure Container Executor helper service should assign launched process to the NM job
[
https://issues.apache.org/jira/browse/YARN-2391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Remus Rusanu resolved YARN-2391.
Resolution: Implemented
> Windows Secure Container Executor helper service should assign launched
> process to the NM job
> -
>
> Key: YARN-2391
> URL: https://issues.apache.org/jira/browse/YARN-2391
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Remus Rusanu
>Assignee: Remus Rusanu
>Priority: Critical
> Labels: security, windows
> Attachments: YARN-2391.1.patch
>
>
> The YARN-2198 NM helper service needs to make sure the launched process is
> added to the NM job ('job' as in Windows NT job objects, not Hadoop jobs).
> This ensures that NM termination ensures launched process termination.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-2391) Windows Secure Container Executor helper service should assign launched process to the NM job
[
https://issues.apache.org/jira/browse/YARN-2391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Remus Rusanu updated YARN-2391:
---
Attachment: YARN-2391.1.patch
Fix by adding an optional
yarn.nodemanager.windows-secure-container-executor.job-name to wsce-site.xml.
This should be configured with the desired job name.
> Windows Secure Container Executor helper service should assign launched
> process to the NM job
> -
>
> Key: YARN-2391
> URL: https://issues.apache.org/jira/browse/YARN-2391
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Remus Rusanu
>Assignee: Remus Rusanu
>Priority: Critical
> Labels: security, windows
> Attachments: YARN-2391.1.patch
>
>
> The YARN-2198 NM helper service needs to make sure the launched process is
> added to the NM job ('job' as in Windows NT job objects, not Hadoop jobs).
> This ensures that NM termination ensures launched process termination.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (YARN-2590) Windows Secure Container Executor: containerLaunch environment does not get transferred to the container process
[ https://issues.apache.org/jira/browse/YARN-2590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remus Rusanu resolved YARN-2590. Resolution: Cannot Reproduce Environment gets passed to container via the launch_container.cmd @set ... commands. There is no environment inheritance requirement between NM and the container, so the WSCE does the right thing. > Windows Secure Container Executor: containerLaunch environment does not get > transferred to the container process > > > Key: YARN-2590 > URL: https://issues.apache.org/jira/browse/YARN-2590 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Remus Rusanu >Assignee: Remus Rusanu > Labels: security, windows > > the sanitized env prepared by the container launch is ignored by the WSCE > launcher. The env has to be passwed in the createTaskAsUser call to > hadoopwinutilsvc so that is assigns it to the newly spawned process. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2590) Windows Secure Container Executor: containerLaunch environment does not get transferred to the container process
[ https://issues.apache.org/jira/browse/YARN-2590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remus Rusanu updated YARN-2590: --- Summary: Windows Secure Container Executor: containerLaunch environment does not get transferred to the container process (was: Windows Secure Container Executor: containerLaunch environment doe snot get transferred to the container process) > Windows Secure Container Executor: containerLaunch environment does not get > transferred to the container process > > > Key: YARN-2590 > URL: https://issues.apache.org/jira/browse/YARN-2590 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Remus Rusanu >Assignee: Remus Rusanu > Labels: security, windows > > the sanitized env prepared by the container launch is ignored by the WSCE > launcher. The env has to be passwed in the createTaskAsUser call to > hadoopwinutilsvc so that is assigns it to the newly spawned process. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2580) Windows Secure Container Executor: grant job query privileges to the container user
[ https://issues.apache.org/jira/browse/YARN-2580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remus Rusanu updated YARN-2580: --- Attachment: YARN-2580.1.patch This fixes the problems related to the job and process permission. The job and the spawned processes are explictly added ACEs giving access to NM, the container user and a set of hard-coded SIDs (LocalYstem, Administrators). > Windows Secure Container Executor: grant job query privileges to the > container user > --- > > Key: YARN-2580 > URL: https://issues.apache.org/jira/browse/YARN-2580 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Remus Rusanu >Assignee: Remus Rusanu > Attachments: YARN-2580.1.patch > > > mapred.MapTask.iniitalize uses WindowsBasedProcessTree which uses winutils to > query the container NT JOB. This must eb granted query permission by the > hadoopwinutilsvc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (YARN-2580) Windows Secure Container Executor: grant job query privileges to the container user
[ https://issues.apache.org/jira/browse/YARN-2580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remus Rusanu resolved YARN-2580. Resolution: Implemented Fix will be contained in next YARN-2198 patch > Windows Secure Container Executor: grant job query privileges to the > container user > --- > > Key: YARN-2580 > URL: https://issues.apache.org/jira/browse/YARN-2580 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Remus Rusanu >Assignee: Remus Rusanu > Attachments: YARN-2580.1.patch > > > mapred.MapTask.iniitalize uses WindowsBasedProcessTree which uses winutils to > query the container NT JOB. This must eb granted query permission by the > hadoopwinutilsvc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2129) Add scheduling priority to the WindowsSecureContainerExecutor
[ https://issues.apache.org/jira/browse/YARN-2129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14146275#comment-14146275 ] Remus Rusanu commented on YARN-2129: A better alternative implementation would be to sue JOB_OBJECT_LIMIT_PRIORITY_CLASS on the container job > Add scheduling priority to the WindowsSecureContainerExecutor > - > > Key: YARN-2129 > URL: https://issues.apache.org/jira/browse/YARN-2129 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 3.0.0 >Reporter: Remus Rusanu >Assignee: Remus Rusanu > Labels: security, windows > Attachments: YARN-2129.1.patch, YARN-2129.2.patch > > > The WCE (YARN-1972) could and should honor > NM_CONTAINER_EXECUTOR_SCHED_PRIORITY. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (YARN-2587) Windows Secure Container Executor: classpath in the job classpath-jar is referencing NM nmPrivate files
[
https://issues.apache.org/jira/browse/YARN-2587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Remus Rusanu resolved YARN-2587.
Resolution: Implemented
Next YARN-2198 patch will contain the fix.
> Windows Secure Container Executor: classpath in the job classpath-jar is
> referencing NM nmPrivate files
> ---
>
> Key: YARN-2587
> URL: https://issues.apache.org/jira/browse/YARN-2587
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Remus Rusanu
>Assignee: Remus Rusanu
> Labels: security, windows
> Attachments: YARN-2587.1.patch
>
>
> The classpath-jar is referencing nmPrivate files:
> {code}
> Manifest-Version: 1.0
> Class-Path: file:/c:/Hadoop/Data/Hadoop/local/usercache/HadoopUser/app
> cache/application_1411394968079_0006/container_1411394968079_0006_01_ 03
> file:/c:/hadoop/hadoop-2.4.1-SNAPSHOT/etc/hadoop/ file:/c:/had
> oop/hadoop-2.4.1-SNAPSHOT/share/hadoop/common/hadoop-common-3.0.0-SNA
> PSHOT-tests.jar file:/c:/hadoop/hadoop-2.4.1-SNAPSHOT/share/hadoop/co
> ...
> file:/c:/Hadoop/Data/Hadoop/local
> /nmPrivate/application_1411394968079_0006/container_1411394968079_000
> 6_01_03/job.jar/job.jar file:/c:/Hadoop/Data/Hadoop/local/nmPriva
> te/application_1411394968079_0006/container_1411394968079_0006_01_000
> 003/job.jar/classes/
> {code}
> This is not allowed in the WSCE enviornment, the container has no read
> privileges on the nmPrivate paths. Besides that particular jar is copied over
> into the user appcache.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-2587) Windows Secure Container Executor: classpath in the job classpath-jar is referencing NM nmPrivate files
[
https://issues.apache.org/jira/browse/YARN-2587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Remus Rusanu updated YARN-2587:
---
Attachment: YARN-2587.1.patch
1.patch fixes the issue by adding an explicit parameter for the target jar
working dir (ie. implicit dir to be added to relative paths), separated from
the jar creation working dir (ie. where the jar will be created)
> Windows Secure Container Executor: classpath in the job classpath-jar is
> referencing NM nmPrivate files
> ---
>
> Key: YARN-2587
> URL: https://issues.apache.org/jira/browse/YARN-2587
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Remus Rusanu
>Assignee: Remus Rusanu
> Labels: security, windows
> Attachments: YARN-2587.1.patch
>
>
> The classpath-jar is referencing nmPrivate files:
> {code}
> Manifest-Version: 1.0
> Class-Path: file:/c:/Hadoop/Data/Hadoop/local/usercache/HadoopUser/app
> cache/application_1411394968079_0006/container_1411394968079_0006_01_ 03
> file:/c:/hadoop/hadoop-2.4.1-SNAPSHOT/etc/hadoop/ file:/c:/had
> oop/hadoop-2.4.1-SNAPSHOT/share/hadoop/common/hadoop-common-3.0.0-SNA
> PSHOT-tests.jar file:/c:/hadoop/hadoop-2.4.1-SNAPSHOT/share/hadoop/co
> ...
> file:/c:/Hadoop/Data/Hadoop/local
> /nmPrivate/application_1411394968079_0006/container_1411394968079_000
> 6_01_03/job.jar/job.jar file:/c:/Hadoop/Data/Hadoop/local/nmPriva
> te/application_1411394968079_0006/container_1411394968079_0006_01_000
> 003/job.jar/classes/
> {code}
> This is not allowed in the WSCE enviornment, the container has no read
> privileges on the nmPrivate paths. Besides that particular jar is copied over
> into the user appcache.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (YARN-2590) Windows Secure Container Executor: containerLaunch environment doe snot get transferred to the container process
Remus Rusanu created YARN-2590: -- Summary: Windows Secure Container Executor: containerLaunch environment doe snot get transferred to the container process Key: YARN-2590 URL: https://issues.apache.org/jira/browse/YARN-2590 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Reporter: Remus Rusanu Assignee: Remus Rusanu the sanitized env prepared by the container launch is ignored by the WSCE launcher. The env has to be passwed in the createTaskAsUser call to hadoopwinutilsvc so that is assigns it to the newly spawned process. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (YARN-2129) Add scheduling priority to the WindowsSecureContainerExecutor
[ https://issues.apache.org/jira/browse/YARN-2129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remus Rusanu resolved YARN-2129. Resolution: Won't Fix Given the priority boost inheritance model in Windows, there is little incentive to do this. > Add scheduling priority to the WindowsSecureContainerExecutor > - > > Key: YARN-2129 > URL: https://issues.apache.org/jira/browse/YARN-2129 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 3.0.0 >Reporter: Remus Rusanu >Assignee: Remus Rusanu > Labels: security, windows > Attachments: YARN-2129.1.patch, YARN-2129.2.patch > > > The WCE (YARN-1972) could and should honor > NM_CONTAINER_EXECUTOR_SCHED_PRIORITY. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2587) Windows Secure Container Executor: classpath in the job classpath-jar is referencing NM nmPrivate files
[
https://issues.apache.org/jira/browse/YARN-2587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14145148#comment-14145148
]
Remus Rusanu commented on YARN-2587:
I think the root cause is this: ContainerLaunch adds relatives paths (job.jar,
job.xml and job.jar/lib/*). But the classpathJar code uses the working dir to
expand this:
{code}
if(!new Path(classPathEntry).isAbsolute()) {
fileCpEntry = new File(workingDir, classPathEntry);
}
{code}
This expands to the NM nmPrivate folders. ContianerLaunch should add
%PWD%/job.jar, %PWD%/job.xml and %PWD%/job.jar/lib/* instead.
> Windows Secure Container Executor: classpath in the job classpath-jar is
> referencing NM nmPrivate files
> ---
>
> Key: YARN-2587
> URL: https://issues.apache.org/jira/browse/YARN-2587
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Remus Rusanu
>Assignee: Remus Rusanu
> Labels: security, windows
>
> The classpath-jar is referencing nmPrivate files:
> {code}
> Manifest-Version: 1.0
> Class-Path: file:/c:/Hadoop/Data/Hadoop/local/usercache/HadoopUser/app
> cache/application_1411394968079_0006/container_1411394968079_0006_01_ 03
> file:/c:/hadoop/hadoop-2.4.1-SNAPSHOT/etc/hadoop/ file:/c:/had
> oop/hadoop-2.4.1-SNAPSHOT/share/hadoop/common/hadoop-common-3.0.0-SNA
> PSHOT-tests.jar file:/c:/hadoop/hadoop-2.4.1-SNAPSHOT/share/hadoop/co
> ...
> file:/c:/Hadoop/Data/Hadoop/local
> /nmPrivate/application_1411394968079_0006/container_1411394968079_000
> 6_01_03/job.jar/job.jar file:/c:/Hadoop/Data/Hadoop/local/nmPriva
> te/application_1411394968079_0006/container_1411394968079_0006_01_000
> 003/job.jar/classes/
> {code}
> This is not allowed in the WSCE enviornment, the container has no read
> privileges on the nmPrivate paths. Besides that particular jar is copied over
> into the user appcache.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-358) bundle container classpath in temporary jar on all platforms, not just Windows
[ https://issues.apache.org/jira/browse/YARN-358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14144555#comment-14144555 ] Remus Rusanu commented on YARN-358: --- Note that with YARN-2587 I have to fix the classpath contained within the the classpath jar. By default (DefaultContainerExecutor) the job.jar is contained in the nmPrivate path, and the classpath is referencing this path. In a secure environment this is not accessible to the container. The Windows Secure Container Executor handles this problem, not sure what you want to do about the DefaultContainerExecutor. > bundle container classpath in temporary jar on all platforms, not just Windows > -- > > Key: YARN-358 > URL: https://issues.apache.org/jira/browse/YARN-358 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: trunk-win >Reporter: Chris Nauroth > > Currently, a Windows-specific code path bundles the classpath into a > temporary jar with a manifest to work around command line length limitations. > This code path does not need to be Windows-specific. We can use the same > approach on all platforms. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (YARN-2586) Windows Secure Container Executor: use elevated file system operations to cleanup the containers
[ https://issues.apache.org/jira/browse/YARN-2586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remus Rusanu resolved YARN-2586. Resolution: Implemented The next patch of YARN-2198 will contain this fix. > Windows Secure Container Executor: use elevated file system operations to > cleanup the containers > > > Key: YARN-2586 > URL: https://issues.apache.org/jira/browse/YARN-2586 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Remus Rusanu >Assignee: Remus Rusanu > Labels: security, windows > Attachments: YARN-2586.1.patch > > > WSCE needs to use elevated filesystem operations (hadoopwinutilsvc backed) to > cleanup the container dirs. The NM does not have enough privileges to delete > these dirs. Similar to how LCE does it, using the root container-executor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2586) Windows Secure Container Executor: use elevated file system operations to cleanup the containers
[ https://issues.apache.org/jira/browse/YARN-2586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remus Rusanu updated YARN-2586: --- Attachment: YARN-2586.1.patch > Windows Secure Container Executor: use elevated file system operations to > cleanup the containers > > > Key: YARN-2586 > URL: https://issues.apache.org/jira/browse/YARN-2586 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Remus Rusanu >Assignee: Remus Rusanu > Labels: security, windows > Attachments: YARN-2586.1.patch > > > WSCE needs to use elevated filesystem operations (hadoopwinutilsvc backed) to > cleanup the container dirs. The NM does not have enough privileges to delete > these dirs. Similar to how LCE does it, using the root container-executor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-2587) Windows Secure Container Executor: classpath in the job classpath-jar is referencing NM nmPrivate files
Remus Rusanu created YARN-2587:
--
Summary: Windows Secure Container Executor: classpath in the job
classpath-jar is referencing NM nmPrivate files
Key: YARN-2587
URL: https://issues.apache.org/jira/browse/YARN-2587
Project: Hadoop YARN
Issue Type: Sub-task
Components: nodemanager
Reporter: Remus Rusanu
Assignee: Remus Rusanu
The classpath-jar is referencing nmPrivate files:
{code}
Manifest-Version: 1.0
Class-Path: file:/c:/Hadoop/Data/Hadoop/local/usercache/HadoopUser/app
cache/application_1411394968079_0006/container_1411394968079_0006_01_ 03
file:/c:/hadoop/hadoop-2.4.1-SNAPSHOT/etc/hadoop/ file:/c:/had
oop/hadoop-2.4.1-SNAPSHOT/share/hadoop/common/hadoop-common-3.0.0-SNA
PSHOT-tests.jar file:/c:/hadoop/hadoop-2.4.1-SNAPSHOT/share/hadoop/co
...
file:/c:/Hadoop/Data/Hadoop/local
/nmPrivate/application_1411394968079_0006/container_1411394968079_000
6_01_03/job.jar/job.jar file:/c:/Hadoop/Data/Hadoop/local/nmPriva
te/application_1411394968079_0006/container_1411394968079_0006_01_000
003/job.jar/classes/
{code}
This is not allowed in the WSCE enviornment, the container has no read
privileges on the nmPrivate paths. Besides that particular jar is copied over
into the user appcache.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (YARN-2586) Windows Secure Container Executor: use elevated file system operations to cleanup the containers
Remus Rusanu created YARN-2586: -- Summary: Windows Secure Container Executor: use elevated file system operations to cleanup the containers Key: YARN-2586 URL: https://issues.apache.org/jira/browse/YARN-2586 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Reporter: Remus Rusanu Assignee: Remus Rusanu WSCE needs to use elevated filesystem operations (hadoopwinutilsvc backed) to cleanup the container dirs. The NM does not have enough privileges to delete these dirs. Similar to how LCE does it, using the root container-executor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-2580) Windows Secure Container Executor: grant job query privileges to the container user
Remus Rusanu created YARN-2580: -- Summary: Windows Secure Container Executor: grant job query privileges to the container user Key: YARN-2580 URL: https://issues.apache.org/jira/browse/YARN-2580 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Reporter: Remus Rusanu mapred.MapTask.iniitalize uses WindowsBasedProcessTree which uses winutils to query the container NT JOB. This must eb granted query permission by the hadoopwinutilsvc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (YARN-2580) Windows Secure Container Executor: grant job query privileges to the container user
[ https://issues.apache.org/jira/browse/YARN-2580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remus Rusanu reassigned YARN-2580: -- Assignee: Remus Rusanu > Windows Secure Container Executor: grant job query privileges to the > container user > --- > > Key: YARN-2580 > URL: https://issues.apache.org/jira/browse/YARN-2580 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Remus Rusanu >Assignee: Remus Rusanu > > mapred.MapTask.iniitalize uses WindowsBasedProcessTree which uses winutils to > query the container NT JOB. This must eb granted query permission by the > hadoopwinutilsvc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (YARN-2551) Windows Secure Cotnainer Executor: Add checks to validate that the wsce-site.xml is write restricted to Administrators only
[ https://issues.apache.org/jira/browse/YARN-2551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remus Rusanu resolved YARN-2551. Resolution: Implemented The patch will be contained in YARN-2198 patch 10 and forward > Windows Secure Cotnainer Executor: Add checks to validate that the > wsce-site.xml is write restricted to Administrators only > --- > > Key: YARN-2551 > URL: https://issues.apache.org/jira/browse/YARN-2551 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Remus Rusanu >Assignee: Remus Rusanu > Labels: security, windows, wsce > Attachments: YARN-2551.1.patch > > > The wsce-site.xml containes the impersonate.allowed and impersonate.denied > keys that restrict/control the users that can be impersonated by the WSCE > containers. The impersonation frameworks in winutils should validate that > only Administrators have write control on this file. > This is similar to how LCE is validating that only root has write permissions > on container-executor.cfg file on secure Linux clusters. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2551) Windows Secure Cotnainer Executor: Add checks to validate that the wsce-site.xml is write restricted to Administrators only
[ https://issues.apache.org/jira/browse/YARN-2551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remus Rusanu updated YARN-2551: --- Attachment: YARN-2551.1.patch > Windows Secure Cotnainer Executor: Add checks to validate that the > wsce-site.xml is write restricted to Administrators only > --- > > Key: YARN-2551 > URL: https://issues.apache.org/jira/browse/YARN-2551 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Remus Rusanu >Assignee: Remus Rusanu > Labels: security, windows, wsce > Attachments: YARN-2551.1.patch > > > The wsce-site.xml containes the impersonate.allowed and impersonate.denied > keys that restrict/control the users that can be impersonated by the WSCE > containers. The impersonation frameworks in winutils should validate that > only Administrators have write control on this file. > This is similar to how LCE is validating that only root has write permissions > on container-executor.cfg file on secure Linux clusters. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2198) Remove the need to run NodeManager as privileged account for Windows Secure Container Executor
[ https://issues.apache.org/jira/browse/YARN-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remus Rusanu updated YARN-2198: --- Attachment: YARN-2198.trunk.9.patch .trunk.9.patch Fix javadoc, release audit and findbugs warnings. > Remove the need to run NodeManager as privileged account for Windows Secure > Container Executor > -- > > Key: YARN-2198 > URL: https://issues.apache.org/jira/browse/YARN-2198 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Remus Rusanu >Assignee: Remus Rusanu > Labels: security, windows > Attachments: YARN-2198.1.patch, YARN-2198.2.patch, YARN-2198.3.patch, > YARN-2198.delta.4.patch, YARN-2198.delta.5.patch, YARN-2198.delta.6.patch, > YARN-2198.delta.7.patch, YARN-2198.separation.patch, YARN-2198.trunk.4.patch, > YARN-2198.trunk.5.patch, YARN-2198.trunk.6.patch, YARN-2198.trunk.8.patch, > YARN-2198.trunk.9.patch > > > YARN-1972 introduces a Secure Windows Container Executor. However this > executor requires a the process launching the container to be LocalSystem or > a member of the a local Administrators group. Since the process in question > is the NodeManager, the requirement translates to the entire NM to run as a > privileged account, a very large surface area to review and protect. > This proposal is to move the privileged operations into a dedicated NT > service. The NM can run as a low privilege account and communicate with the > privileged NT service when it needs to launch a container. This would reduce > the surface exposed to the high privileges. > There has to exist a secure, authenticated and authorized channel of > communication between the NM and the privileged NT service. Possible > alternatives are a new TCP endpoint, Java RPC etc. My proposal though would > be to use Windows LPC (Local Procedure Calls), which is a Windows platform > specific inter-process communication channel that satisfies all requirements > and is easy to deploy. The privileged NT service would register and listen on > an LPC port (NtCreatePort, NtListenPort). The NM would use JNI to interop > with libwinutils which would host the LPC client code. The client would > connect to the LPC port (NtConnectPort) and send a message requesting a > container launch (NtRequestWaitReplyPort). LPC provides authentication and > the privileged NT service can use authorization API (AuthZ) to validate the > caller. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2198) Remove the need to run NodeManager as privileged account for Windows Secure Container Executor
[ https://issues.apache.org/jira/browse/YARN-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remus Rusanu updated YARN-2198: --- Attachment: YARN-2198.trunk.8.patch > Remove the need to run NodeManager as privileged account for Windows Secure > Container Executor > -- > > Key: YARN-2198 > URL: https://issues.apache.org/jira/browse/YARN-2198 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Remus Rusanu >Assignee: Remus Rusanu > Labels: security, windows > Attachments: YARN-2198.1.patch, YARN-2198.2.patch, YARN-2198.3.patch, > YARN-2198.delta.4.patch, YARN-2198.delta.5.patch, YARN-2198.delta.6.patch, > YARN-2198.delta.7.patch, YARN-2198.separation.patch, YARN-2198.trunk.4.patch, > YARN-2198.trunk.5.patch, YARN-2198.trunk.6.patch, YARN-2198.trunk.8.patch > > > YARN-1972 introduces a Secure Windows Container Executor. However this > executor requires a the process launching the container to be LocalSystem or > a member of the a local Administrators group. Since the process in question > is the NodeManager, the requirement translates to the entire NM to run as a > privileged account, a very large surface area to review and protect. > This proposal is to move the privileged operations into a dedicated NT > service. The NM can run as a low privilege account and communicate with the > privileged NT service when it needs to launch a container. This would reduce > the surface exposed to the high privileges. > There has to exist a secure, authenticated and authorized channel of > communication between the NM and the privileged NT service. Possible > alternatives are a new TCP endpoint, Java RPC etc. My proposal though would > be to use Windows LPC (Local Procedure Calls), which is a Windows platform > specific inter-process communication channel that satisfies all requirements > and is easy to deploy. The privileged NT service would register and listen on > an LPC port (NtCreatePort, NtListenPort). The NM would use JNI to interop > with libwinutils which would host the LPC client code. The client would > connect to the LPC port (NtConnectPort) and send a message requesting a > container launch (NtRequestWaitReplyPort). LPC provides authentication and > the privileged NT service can use authorization API (AuthZ) to validate the > caller. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2198) Remove the need to run NodeManager as privileged account for Windows Secure Container Executor
[ https://issues.apache.org/jira/browse/YARN-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remus Rusanu updated YARN-2198: --- Attachment: (was: YARN-2198.trunk.8.patch) > Remove the need to run NodeManager as privileged account for Windows Secure > Container Executor > -- > > Key: YARN-2198 > URL: https://issues.apache.org/jira/browse/YARN-2198 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Remus Rusanu >Assignee: Remus Rusanu > Labels: security, windows > Attachments: YARN-2198.1.patch, YARN-2198.2.patch, YARN-2198.3.patch, > YARN-2198.delta.4.patch, YARN-2198.delta.5.patch, YARN-2198.delta.6.patch, > YARN-2198.delta.7.patch, YARN-2198.separation.patch, YARN-2198.trunk.4.patch, > YARN-2198.trunk.5.patch, YARN-2198.trunk.6.patch, YARN-2198.trunk.8.patch > > > YARN-1972 introduces a Secure Windows Container Executor. However this > executor requires a the process launching the container to be LocalSystem or > a member of the a local Administrators group. Since the process in question > is the NodeManager, the requirement translates to the entire NM to run as a > privileged account, a very large surface area to review and protect. > This proposal is to move the privileged operations into a dedicated NT > service. The NM can run as a low privilege account and communicate with the > privileged NT service when it needs to launch a container. This would reduce > the surface exposed to the high privileges. > There has to exist a secure, authenticated and authorized channel of > communication between the NM and the privileged NT service. Possible > alternatives are a new TCP endpoint, Java RPC etc. My proposal though would > be to use Windows LPC (Local Procedure Calls), which is a Windows platform > specific inter-process communication channel that satisfies all requirements > and is easy to deploy. The privileged NT service would register and listen on > an LPC port (NtCreatePort, NtListenPort). The NM would use JNI to interop > with libwinutils which would host the LPC client code. The client would > connect to the LPC port (NtConnectPort) and send a message requesting a > container launch (NtRequestWaitReplyPort). LPC provides authentication and > the privileged NT service can use authorization API (AuthZ) to validate the > caller. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2198) Remove the need to run NodeManager as privileged account for Windows Secure Container Executor
[ https://issues.apache.org/jira/browse/YARN-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14142173#comment-14142173 ] Remus Rusanu commented on YARN-2198: Build error is [exec] /home/jenkins/jenkins-slave/workspace/PreCommit-YARN-Build/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/nativeio/NativeIO.c:1444:12: error: 'INVALID_HANDLE_VALUE' undeclared (first use in this function) [exec] return INVALID_HANDLE_VALUE; > Remove the need to run NodeManager as privileged account for Windows Secure > Container Executor > -- > > Key: YARN-2198 > URL: https://issues.apache.org/jira/browse/YARN-2198 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Remus Rusanu >Assignee: Remus Rusanu > Labels: security, windows > Attachments: YARN-2198.1.patch, YARN-2198.2.patch, YARN-2198.3.patch, > YARN-2198.delta.4.patch, YARN-2198.delta.5.patch, YARN-2198.delta.6.patch, > YARN-2198.delta.7.patch, YARN-2198.separation.patch, YARN-2198.trunk.4.patch, > YARN-2198.trunk.5.patch, YARN-2198.trunk.6.patch, YARN-2198.trunk.8.patch > > > YARN-1972 introduces a Secure Windows Container Executor. However this > executor requires a the process launching the container to be LocalSystem or > a member of the a local Administrators group. Since the process in question > is the NodeManager, the requirement translates to the entire NM to run as a > privileged account, a very large surface area to review and protect. > This proposal is to move the privileged operations into a dedicated NT > service. The NM can run as a low privilege account and communicate with the > privileged NT service when it needs to launch a container. This would reduce > the surface exposed to the high privileges. > There has to exist a secure, authenticated and authorized channel of > communication between the NM and the privileged NT service. Possible > alternatives are a new TCP endpoint, Java RPC etc. My proposal though would > be to use Windows LPC (Local Procedure Calls), which is a Windows platform > specific inter-process communication channel that satisfies all requirements > and is easy to deploy. The privileged NT service would register and listen on > an LPC port (NtCreatePort, NtListenPort). The NM would use JNI to interop > with libwinutils which would host the LPC client code. The client would > connect to the LPC port (NtConnectPort) and send a message requesting a > container launch (NtRequestWaitReplyPort). LPC provides authentication and > the privileged NT service can use authorization API (AuthZ) to validate the > caller. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2198) Remove the need to run NodeManager as privileged account for Windows Secure Container Executor
[ https://issues.apache.org/jira/browse/YARN-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remus Rusanu updated YARN-2198: --- Attachment: YARN-2198.trunk.8.patch Fix -^M > Remove the need to run NodeManager as privileged account for Windows Secure > Container Executor > -- > > Key: YARN-2198 > URL: https://issues.apache.org/jira/browse/YARN-2198 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Remus Rusanu >Assignee: Remus Rusanu > Labels: security, windows > Attachments: YARN-2198.1.patch, YARN-2198.2.patch, YARN-2198.3.patch, > YARN-2198.delta.4.patch, YARN-2198.delta.5.patch, YARN-2198.delta.6.patch, > YARN-2198.delta.7.patch, YARN-2198.separation.patch, YARN-2198.trunk.4.patch, > YARN-2198.trunk.5.patch, YARN-2198.trunk.6.patch, YARN-2198.trunk.8.patch > > > YARN-1972 introduces a Secure Windows Container Executor. However this > executor requires a the process launching the container to be LocalSystem or > a member of the a local Administrators group. Since the process in question > is the NodeManager, the requirement translates to the entire NM to run as a > privileged account, a very large surface area to review and protect. > This proposal is to move the privileged operations into a dedicated NT > service. The NM can run as a low privilege account and communicate with the > privileged NT service when it needs to launch a container. This would reduce > the surface exposed to the high privileges. > There has to exist a secure, authenticated and authorized channel of > communication between the NM and the privileged NT service. Possible > alternatives are a new TCP endpoint, Java RPC etc. My proposal though would > be to use Windows LPC (Local Procedure Calls), which is a Windows platform > specific inter-process communication channel that satisfies all requirements > and is easy to deploy. The privileged NT service would register and listen on > an LPC port (NtCreatePort, NtListenPort). The NM would use JNI to interop > with libwinutils which would host the LPC client code. The client would > connect to the LPC port (NtConnectPort) and send a message requesting a > container launch (NtRequestWaitReplyPort). LPC provides authentication and > the privileged NT service can use authorization API (AuthZ) to validate the > caller. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2198) Remove the need to run NodeManager as privileged account for Windows Secure Container Executor
[ https://issues.apache.org/jira/browse/YARN-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remus Rusanu updated YARN-2198: --- Attachment: (was: YARN-2198.trunk.crlf.6.patch) > Remove the need to run NodeManager as privileged account for Windows Secure > Container Executor > -- > > Key: YARN-2198 > URL: https://issues.apache.org/jira/browse/YARN-2198 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Remus Rusanu >Assignee: Remus Rusanu > Labels: security, windows > Attachments: YARN-2198.1.patch, YARN-2198.2.patch, YARN-2198.3.patch, > YARN-2198.delta.4.patch, YARN-2198.delta.5.patch, YARN-2198.delta.6.patch, > YARN-2198.delta.7.patch, YARN-2198.separation.patch, YARN-2198.trunk.4.patch, > YARN-2198.trunk.5.patch, YARN-2198.trunk.6.patch, YARN-2198.trunk.8.patch > > > YARN-1972 introduces a Secure Windows Container Executor. However this > executor requires a the process launching the container to be LocalSystem or > a member of the a local Administrators group. Since the process in question > is the NodeManager, the requirement translates to the entire NM to run as a > privileged account, a very large surface area to review and protect. > This proposal is to move the privileged operations into a dedicated NT > service. The NM can run as a low privilege account and communicate with the > privileged NT service when it needs to launch a container. This would reduce > the surface exposed to the high privileges. > There has to exist a secure, authenticated and authorized channel of > communication between the NM and the privileged NT service. Possible > alternatives are a new TCP endpoint, Java RPC etc. My proposal though would > be to use Windows LPC (Local Procedure Calls), which is a Windows platform > specific inter-process communication channel that satisfies all requirements > and is easy to deploy. The privileged NT service would register and listen on > an LPC port (NtCreatePort, NtListenPort). The NM would use JNI to interop > with libwinutils which would host the LPC client code. The client would > connect to the LPC port (NtConnectPort) and send a message requesting a > container launch (NtRequestWaitReplyPort). LPC provides authentication and > the privileged NT service can use authorization API (AuthZ) to validate the > caller. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2198) Remove the need to run NodeManager as privileged account for Windows Secure Container Executor
[ https://issues.apache.org/jira/browse/YARN-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remus Rusanu updated YARN-2198: --- Attachment: (was: YARN-2198.trunk.8.patch) > Remove the need to run NodeManager as privileged account for Windows Secure > Container Executor > -- > > Key: YARN-2198 > URL: https://issues.apache.org/jira/browse/YARN-2198 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Remus Rusanu >Assignee: Remus Rusanu > Labels: security, windows > Attachments: YARN-2198.1.patch, YARN-2198.2.patch, YARN-2198.3.patch, > YARN-2198.delta.4.patch, YARN-2198.delta.5.patch, YARN-2198.delta.6.patch, > YARN-2198.delta.7.patch, YARN-2198.separation.patch, YARN-2198.trunk.4.patch, > YARN-2198.trunk.5.patch, YARN-2198.trunk.6.patch, YARN-2198.trunk.crlf.6.patch > > > YARN-1972 introduces a Secure Windows Container Executor. However this > executor requires a the process launching the container to be LocalSystem or > a member of the a local Administrators group. Since the process in question > is the NodeManager, the requirement translates to the entire NM to run as a > privileged account, a very large surface area to review and protect. > This proposal is to move the privileged operations into a dedicated NT > service. The NM can run as a low privilege account and communicate with the > privileged NT service when it needs to launch a container. This would reduce > the surface exposed to the high privileges. > There has to exist a secure, authenticated and authorized channel of > communication between the NM and the privileged NT service. Possible > alternatives are a new TCP endpoint, Java RPC etc. My proposal though would > be to use Windows LPC (Local Procedure Calls), which is a Windows platform > specific inter-process communication channel that satisfies all requirements > and is easy to deploy. The privileged NT service would register and listen on > an LPC port (NtCreatePort, NtListenPort). The NM would use JNI to interop > with libwinutils which would host the LPC client code. The client would > connect to the LPC port (NtConnectPort) and send a message requesting a > container launch (NtRequestWaitReplyPort). LPC provides authentication and > the privileged NT service can use authorization API (AuthZ) to validate the > caller. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2198) Remove the need to run NodeManager as privileged account for Windows Secure Container Executor
[ https://issues.apache.org/jira/browse/YARN-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remus Rusanu updated YARN-2198: --- Attachment: YARN-2198.trunk.8.patch .trunk.8.patch is rebased to new repo current trunk and has the vcxproj/sln hunks manually fixed to CRLF > Remove the need to run NodeManager as privileged account for Windows Secure > Container Executor > -- > > Key: YARN-2198 > URL: https://issues.apache.org/jira/browse/YARN-2198 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Remus Rusanu >Assignee: Remus Rusanu > Labels: security, windows > Attachments: YARN-2198.1.patch, YARN-2198.2.patch, YARN-2198.3.patch, > YARN-2198.delta.4.patch, YARN-2198.delta.5.patch, YARN-2198.delta.6.patch, > YARN-2198.delta.7.patch, YARN-2198.separation.patch, YARN-2198.trunk.4.patch, > YARN-2198.trunk.5.patch, YARN-2198.trunk.6.patch, YARN-2198.trunk.8.patch, > YARN-2198.trunk.crlf.6.patch > > > YARN-1972 introduces a Secure Windows Container Executor. However this > executor requires a the process launching the container to be LocalSystem or > a member of the a local Administrators group. Since the process in question > is the NodeManager, the requirement translates to the entire NM to run as a > privileged account, a very large surface area to review and protect. > This proposal is to move the privileged operations into a dedicated NT > service. The NM can run as a low privilege account and communicate with the > privileged NT service when it needs to launch a container. This would reduce > the surface exposed to the high privileges. > There has to exist a secure, authenticated and authorized channel of > communication between the NM and the privileged NT service. Possible > alternatives are a new TCP endpoint, Java RPC etc. My proposal though would > be to use Windows LPC (Local Procedure Calls), which is a Windows platform > specific inter-process communication channel that satisfies all requirements > and is easy to deploy. The privileged NT service would register and listen on > an LPC port (NtCreatePort, NtListenPort). The NM would use JNI to interop > with libwinutils which would host the LPC client code. The client would > connect to the LPC port (NtConnectPort) and send a message requesting a > container launch (NtRequestWaitReplyPort). LPC provides authentication and > the privileged NT service can use authorization API (AuthZ) to validate the > caller. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2198) Remove the need to run NodeManager as privileged account for Windows Secure Container Executor
[ https://issues.apache.org/jira/browse/YARN-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remus Rusanu updated YARN-2198: --- Attachment: YARN-2198.trunk.crlf.6.patch This is trunk.6.patch with CRLF in .vcxproj and .sln hunks, just to see ij Mr. Jenkins is happy with it. > Remove the need to run NodeManager as privileged account for Windows Secure > Container Executor > -- > > Key: YARN-2198 > URL: https://issues.apache.org/jira/browse/YARN-2198 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Remus Rusanu >Assignee: Remus Rusanu > Labels: security, windows > Attachments: YARN-2198.1.patch, YARN-2198.2.patch, YARN-2198.3.patch, > YARN-2198.delta.4.patch, YARN-2198.delta.5.patch, YARN-2198.delta.6.patch, > YARN-2198.delta.7.patch, YARN-2198.separation.patch, YARN-2198.trunk.4.patch, > YARN-2198.trunk.5.patch, YARN-2198.trunk.6.patch, YARN-2198.trunk.crlf.6.patch > > > YARN-1972 introduces a Secure Windows Container Executor. However this > executor requires a the process launching the container to be LocalSystem or > a member of the a local Administrators group. Since the process in question > is the NodeManager, the requirement translates to the entire NM to run as a > privileged account, a very large surface area to review and protect. > This proposal is to move the privileged operations into a dedicated NT > service. The NM can run as a low privilege account and communicate with the > privileged NT service when it needs to launch a container. This would reduce > the surface exposed to the high privileges. > There has to exist a secure, authenticated and authorized channel of > communication between the NM and the privileged NT service. Possible > alternatives are a new TCP endpoint, Java RPC etc. My proposal though would > be to use Windows LPC (Local Procedure Calls), which is a Windows platform > specific inter-process communication channel that satisfies all requirements > and is easy to deploy. The privileged NT service would register and listen on > an LPC port (NtCreatePort, NtListenPort). The NM would use JNI to interop > with libwinutils which would host the LPC client code. The client would > connect to the LPC port (NtConnectPort) and send a message requesting a > container launch (NtRequestWaitReplyPort). LPC provides authentication and > the privileged NT service can use authorization API (AuthZ) to validate the > caller. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2198) Remove the need to run NodeManager as privileged account for Windows Secure Container Executor
[ https://issues.apache.org/jira/browse/YARN-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remus Rusanu updated YARN-2198: --- Attachment: YARN-2198.delta.7.patch This fixes YARN-2553 issues around KillTask. Also elevated chmod was not implemented (D'oh!) and the WSCE dir perms was wrong (0710 inherited from DCE, needs to be 0750 similar to LCE). I will not upload trunk.7.patch given that for the moment Jenkins cannot handle the CRLF mismatch in .vcxproj and .sln files. I'm following up on that issue with HW engineers. > Remove the need to run NodeManager as privileged account for Windows Secure > Container Executor > -- > > Key: YARN-2198 > URL: https://issues.apache.org/jira/browse/YARN-2198 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Remus Rusanu >Assignee: Remus Rusanu > Labels: security, windows > Attachments: YARN-2198.1.patch, YARN-2198.2.patch, YARN-2198.3.patch, > YARN-2198.delta.4.patch, YARN-2198.delta.5.patch, YARN-2198.delta.6.patch, > YARN-2198.delta.7.patch, YARN-2198.separation.patch, YARN-2198.trunk.4.patch, > YARN-2198.trunk.5.patch, YARN-2198.trunk.6.patch > > > YARN-1972 introduces a Secure Windows Container Executor. However this > executor requires a the process launching the container to be LocalSystem or > a member of the a local Administrators group. Since the process in question > is the NodeManager, the requirement translates to the entire NM to run as a > privileged account, a very large surface area to review and protect. > This proposal is to move the privileged operations into a dedicated NT > service. The NM can run as a low privilege account and communicate with the > privileged NT service when it needs to launch a container. This would reduce > the surface exposed to the high privileges. > There has to exist a secure, authenticated and authorized channel of > communication between the NM and the privileged NT service. Possible > alternatives are a new TCP endpoint, Java RPC etc. My proposal though would > be to use Windows LPC (Local Procedure Calls), which is a Windows platform > specific inter-process communication channel that satisfies all requirements > and is easy to deploy. The privileged NT service would register and listen on > an LPC port (NtCreatePort, NtListenPort). The NM would use JNI to interop > with libwinutils which would host the LPC client code. The client would > connect to the LPC port (NtConnectPort) and send a message requesting a > container launch (NtRequestWaitReplyPort). LPC provides authentication and > the privileged NT service can use authorization API (AuthZ) to validate the > caller. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (YARN-2553) Windows Secure Container Executor: assign PROCESS_TERMINATE privilege to NM on created containers
[ https://issues.apache.org/jira/browse/YARN-2553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remus Rusanu resolved YARN-2553. Resolution: Not a Problem After further investigation I concluded that there is no way to prevent the access_denied on the joc object during the container shutdown. I have moved the kill task code inside the hadoopwinutils, running as LocalSystem, with SeDebug privilege enabled, and after LocalSystem is explicitly granted JOB_OBJECT_ALL_ACCESS on the job, and still get access denied. I fixed the kill task to return success int his case and commented out the issue. The fixed code will be in the next patch of YARN-2198. > Windows Secure Container Executor: assign PROCESS_TERMINATE privilege to NM > on created containers > - > > Key: YARN-2553 > URL: https://issues.apache.org/jira/browse/YARN-2553 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Remus Rusanu >Assignee: Remus Rusanu > Labels: security, windows, wsce > > In order to open a job handle with JOB_OBJECT_TERMINATE access, the caller > must have PROCESS_TERMINATE access on the handle of each process in the job > (MSDN > http://msdn.microsoft.com/en-us/library/windows/desktop/ms686709(v=vs.85).aspx) > . > hadoopwinutilsvc process should explicitly grant PROCESS_TERMINATE access to > NM account on the newly started container process. I hope this gets > inherited... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (YARN-2552) Windows Secure Container Executor: the privileged file operations of hadoopwinutilsvc should be constrained to localdirs only
[ https://issues.apache.org/jira/browse/YARN-2552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remus Rusanu resolved YARN-2552. Resolution: Implemented Fix is included in YARN-2198.delta.7.patch > Windows Secure Container Executor: the privileged file operations of > hadoopwinutilsvc should be constrained to localdirs only > - > > Key: YARN-2552 > URL: https://issues.apache.org/jira/browse/YARN-2552 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Remus Rusanu >Assignee: Remus Rusanu > Labels: security, windows, wsce > Attachments: YARN-2552.1.patch > > > YARN-2458 added file manipulation operations executed in an elevated context > by hadoopwinutilsvc. W/o any constraint, the NM (or a hijacker that takes > over the NM) can manipulate arbitrary OS files under highest possible > privileges, an easy elevation attack vector. The service should only allow > operations on files/directories that are under the configured NM localdirs. > It should read this value from wsce-site.xml, as the yarn-site.xml cannot be > trusted, being writable by Hadoop admins (YARN-2551 ensures wsce-site.xml is > only writable by system Administrators, not Hadoop admins). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2552) Windows Secure Container Executor: the privileged file operations of hadoopwinutilsvc should be constrained to localdirs only
[ https://issues.apache.org/jira/browse/YARN-2552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remus Rusanu updated YARN-2552: --- Attachment: YARN-2552.1.patch This patch adds ValidateLocalPath function, checks all file operations for valid paths, and adds a yarn.nodemanager.windows-secure-container-executor.local-dirs key in wsce-site.xml. The SecureContainer.apt.vm is updated to describe the new config key. > Windows Secure Container Executor: the privileged file operations of > hadoopwinutilsvc should be constrained to localdirs only > - > > Key: YARN-2552 > URL: https://issues.apache.org/jira/browse/YARN-2552 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Remus Rusanu >Assignee: Remus Rusanu > Labels: security, windows, wsce > Attachments: YARN-2552.1.patch > > > YARN-2458 added file manipulation operations executed in an elevated context > by hadoopwinutilsvc. W/o any constraint, the NM (or a hijacker that takes > over the NM) can manipulate arbitrary OS files under highest possible > privileges, an easy elevation attack vector. The service should only allow > operations on files/directories that are under the configured NM localdirs. > It should read this value from wsce-site.xml, as the yarn-site.xml cannot be > trusted, being writable by Hadoop admins (YARN-2551 ensures wsce-site.xml is > only writable by system Administrators, not Hadoop admins). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (YARN-2474) document the wsce-site.xml keys in hadoop-yarn-site/src/site/apt/SecureContainer.apt.vm
[ https://issues.apache.org/jira/browse/YARN-2474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remus Rusanu resolved YARN-2474. Resolution: Implemented Included in .6.patch of YARN-2198 > document the wsce-site.xml keys in > hadoop-yarn-site/src/site/apt/SecureContainer.apt.vm > --- > > Key: YARN-2474 > URL: https://issues.apache.org/jira/browse/YARN-2474 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Remus Rusanu >Assignee: Remus Rusanu >Priority: Critical > Labels: security, windows > Attachments: YARN-2474.1.patch > > > document the keys used to configure WSCE -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2198) Remove the need to run NodeManager as privileged account for Windows Secure Container Executor
[ https://issues.apache.org/jira/browse/YARN-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remus Rusanu updated YARN-2198: --- Attachment: YARN-2198.trunk.6.patch trunk diff corresponding to delta.6 > Remove the need to run NodeManager as privileged account for Windows Secure > Container Executor > -- > > Key: YARN-2198 > URL: https://issues.apache.org/jira/browse/YARN-2198 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Remus Rusanu >Assignee: Remus Rusanu > Labels: security, windows > Attachments: YARN-2198.1.patch, YARN-2198.2.patch, YARN-2198.3.patch, > YARN-2198.delta.4.patch, YARN-2198.delta.5.patch, YARN-2198.delta.6.patch, > YARN-2198.separation.patch, YARN-2198.trunk.4.patch, YARN-2198.trunk.5.patch, > YARN-2198.trunk.6.patch > > > YARN-1972 introduces a Secure Windows Container Executor. However this > executor requires a the process launching the container to be LocalSystem or > a member of the a local Administrators group. Since the process in question > is the NodeManager, the requirement translates to the entire NM to run as a > privileged account, a very large surface area to review and protect. > This proposal is to move the privileged operations into a dedicated NT > service. The NM can run as a low privilege account and communicate with the > privileged NT service when it needs to launch a container. This would reduce > the surface exposed to the high privileges. > There has to exist a secure, authenticated and authorized channel of > communication between the NM and the privileged NT service. Possible > alternatives are a new TCP endpoint, Java RPC etc. My proposal though would > be to use Windows LPC (Local Procedure Calls), which is a Windows platform > specific inter-process communication channel that satisfies all requirements > and is easy to deploy. The privileged NT service would register and listen on > an LPC port (NtCreatePort, NtListenPort). The NM would use JNI to interop > with libwinutils which would host the LPC client code. The client would > connect to the LPC port (NtConnectPort) and send a message requesting a > container launch (NtRequestWaitReplyPort). LPC provides authentication and > the privileged NT service can use authorization API (AuthZ) to validate the > caller. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2198) Remove the need to run NodeManager as privileged account for Windows Secure Container Executor
[ https://issues.apache.org/jira/browse/YARN-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remus Rusanu updated YARN-2198: --- Attachment: YARN-2198.delta.6.patch .delta.6.patch updated with YARN-2474 and fixed the vcxproj whitespace diffs that cause patch to not apply on Linux > Remove the need to run NodeManager as privileged account for Windows Secure > Container Executor > -- > > Key: YARN-2198 > URL: https://issues.apache.org/jira/browse/YARN-2198 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Remus Rusanu >Assignee: Remus Rusanu > Labels: security, windows > Attachments: YARN-2198.1.patch, YARN-2198.2.patch, YARN-2198.3.patch, > YARN-2198.delta.4.patch, YARN-2198.delta.5.patch, YARN-2198.delta.6.patch, > YARN-2198.separation.patch, YARN-2198.trunk.4.patch, YARN-2198.trunk.5.patch, > YARN-2198.trunk.6.patch > > > YARN-1972 introduces a Secure Windows Container Executor. However this > executor requires a the process launching the container to be LocalSystem or > a member of the a local Administrators group. Since the process in question > is the NodeManager, the requirement translates to the entire NM to run as a > privileged account, a very large surface area to review and protect. > This proposal is to move the privileged operations into a dedicated NT > service. The NM can run as a low privilege account and communicate with the > privileged NT service when it needs to launch a container. This would reduce > the surface exposed to the high privileges. > There has to exist a secure, authenticated and authorized channel of > communication between the NM and the privileged NT service. Possible > alternatives are a new TCP endpoint, Java RPC etc. My proposal though would > be to use Windows LPC (Local Procedure Calls), which is a Windows platform > specific inter-process communication channel that satisfies all requirements > and is easy to deploy. The privileged NT service would register and listen on > an LPC port (NtCreatePort, NtListenPort). The NM would use JNI to interop > with libwinutils which would host the LPC client code. The client would > connect to the LPC port (NtConnectPort) and send a message requesting a > container launch (NtRequestWaitReplyPort). LPC provides authentication and > the privileged NT service can use authorization API (AuthZ) to validate the > caller. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2474) document the wsce-site.xml keys in hadoop-yarn-site/src/site/apt/SecureContainer.apt.vm
[ https://issues.apache.org/jira/browse/YARN-2474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remus Rusanu updated YARN-2474: --- Attachment: YARN-2474.1.patch > document the wsce-site.xml keys in > hadoop-yarn-site/src/site/apt/SecureContainer.apt.vm > --- > > Key: YARN-2474 > URL: https://issues.apache.org/jira/browse/YARN-2474 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Remus Rusanu >Assignee: Remus Rusanu >Priority: Critical > Labels: security, windows > Attachments: YARN-2474.1.patch > > > document the keys used to configure WSCE -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-2553) Windows Secure Container Executor: assign PROCESS_TERMINATE privilege to NM on created containers
Remus Rusanu created YARN-2553: -- Summary: Windows Secure Container Executor: assign PROCESS_TERMINATE privilege to NM on created containers Key: YARN-2553 URL: https://issues.apache.org/jira/browse/YARN-2553 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Reporter: Remus Rusanu Assignee: Remus Rusanu In order to open a job handle with JOB_OBJECT_TERMINATE access, the caller must have PROCESS_TERMINATE access on the handle of each process in the job (MSDN http://msdn.microsoft.com/en-us/library/windows/desktop/ms686709(v=vs.85).aspx) . hadoopwinutilsvc process should explicitly grant PROCESS_TERMINATE access to NM account on the newly started container process. I hope this gets inherited... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (YARN-2485) Fix WSCE folder/file/classpathJar permission/order when running as non-admin
[ https://issues.apache.org/jira/browse/YARN-2485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remus Rusanu resolved YARN-2485. Resolution: Duplicate This is fixed by YARN-2458 implementation of a en 'elevated' file system for WSCE. > Fix WSCE folder/file/classpathJar permission/order when running as non-admin > > > Key: YARN-2485 > URL: https://issues.apache.org/jira/browse/YARN-2485 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Remus Rusanu >Assignee: Remus Rusanu > Labels: security, windows > > The WSCE creates the local, usercache, filecache appcache dirs in the normal > DefaultContainerExecutor way, and then assigns ownership to the userprocess. > The WSCE configured group is added, but the permission masks used (710) do no > give write permissions on the appcache/filecache/usercache folder to the NM > itself. > The creation of these folders, as well as the creation of the temporary > classPath jar files must succeed even after thes file/dir ownership is > relinquished to the task user and the NM does not run as a local > Administrator. > LCE handles all these dirs inside the container-executor app (root). The > classpathJar issue does not exists on Linux. > The dirs can be handled by simply delaying the transfer (create all dirs and > temp files, then assign ownership in bulk) but the task classpathJar is > 'special' and needs some refactoring of the NM launch sequence. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-2552) Windows Secure Container Executor: the privileged file operations of hadoopwinutilsvc should be constrained to localdirs only
Remus Rusanu created YARN-2552: -- Summary: Windows Secure Container Executor: the privileged file operations of hadoopwinutilsvc should be constrained to localdirs only Key: YARN-2552 URL: https://issues.apache.org/jira/browse/YARN-2552 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Reporter: Remus Rusanu Assignee: Remus Rusanu YARN-2458 added file manipulation operations executed in an elevated context by hadoopwinutilsvc. W/o any constraint, the NM (or a hijacker that takes over the NM) can manipulate arbitrary OS files under highest possible privileges, an easy elevation attack vector. The service should only allow operations on files/directories that are under the configured NM localdirs. It should read this value from wsce-site.xml, as the yarn-site.xml cannot be trusted, being writable by Hadoop admins (YARN-2551 ensures wsce-site.xml is only writable by system Administrators, not Hadoop admins). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-2551) Windows Secure Cotnainer Executor: Add checks to validate that the wsce-site.xml is write restricted to Administrators only
Remus Rusanu created YARN-2551: -- Summary: Windows Secure Cotnainer Executor: Add checks to validate that the wsce-site.xml is write restricted to Administrators only Key: YARN-2551 URL: https://issues.apache.org/jira/browse/YARN-2551 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Reporter: Remus Rusanu Assignee: Remus Rusanu The wsce-site.xml containes the impersonate.allowed and impersonate.denied keys that restrict/control the users that can be impersonated by the WSCE containers. The impersonation frameworks in winutils should validate that only Administrators have write control on this file. This is similar to how LCE is validating that only root has write permissions on container-executor.cfg file on secure Linux clusters. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
