[jira] [Updated] (YARN-9834) Allow using a pool of local users to run Yarn Secure Container in secure mode
[ https://issues.apache.org/jira/browse/YARN-9834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated YARN-9834: - Labels: pull-request-available (was: ) > Allow using a pool of local users to run Yarn Secure Container in secure mode > - > > Key: YARN-9834 > URL: https://issues.apache.org/jira/browse/YARN-9834 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 3.1.2 >Reporter: shanyu zhao >Assignee: shanyu zhao >Priority: Major > Labels: pull-request-available > Attachments: YarnSecureContainerWithPoolOfLocalUsers.pdf > > Time Spent: 10m > Remaining Estimate: 0h > > Yarn Secure Container allows separation of different user's local files and > container processes running on the same node manager. This depends on an out > of band service such as SSSD/Winbind to sync all domain users to local > machine that runs Yarn node manager. *Hadoop code only works with local > users*. > Winbind/SSSD user sync has lots of overhead, especially for large > corporations. Also if running Yarn node manager inside Kubernetes cluster > (meaning node managers running inside Docker container), it doesn't make > sense for each Docker container to domain join with Active Directory and sync > a whole copy of domain users to the Docker container. > We need an optional light-weighted approach to enable Yarn Secure Container > in secure mode, as an alternative to AD domain join and SSSD/Winbind based > user-sync service. > Today, class LinuxContainerExecutor already supports running Yarn container > process as one designated local user in non-secure mode. > *We can add new configurations to Yarn, such that with LinuxContainerExecutor > we can pre-create a pool of local users on each Yarn node manager. At > runtime, Yarn node manager allocates a local user to run the container > process, for the domain user that submits the application*. When all > containers of that user are finished and all files belonging to that user are > deleted, we can release the allocation and allow other users to use the same > local user to run their Yarn containers. > Please look at attached design doc for more details. > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9834) Allow using a pool of local users to run Yarn Secure Container in secure mode
[ https://issues.apache.org/jira/browse/YARN-9834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shanyu zhao updated YARN-9834: -- Attachment: (was: YarnSecureContainerWithPoolOfLocalUsers.pdf) > Allow using a pool of local users to run Yarn Secure Container in secure mode > - > > Key: YARN-9834 > URL: https://issues.apache.org/jira/browse/YARN-9834 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 3.1.2 >Reporter: shanyu zhao >Assignee: shanyu zhao >Priority: Major > Attachments: YarnSecureContainerWithPoolOfLocalUsers.pdf > > > Yarn Secure Container allows separation of different user's local files and > container processes running on the same node manager. This depends on an out > of band service such as SSSD/Winbind to sync all domain users to local > machine that runs Yarn node manager. *Hadoop code only works with local > users*. > Winbind/SSSD user sync has lots of overhead, especially for large > corporations. Also if running Yarn node manager inside Kubernetes cluster > (meaning node managers running inside Docker container), it doesn't make > sense for each Docker container to domain join with Active Directory and sync > a whole copy of domain users to the Docker container. > We need an optional light-weighted approach to enable Yarn Secure Container > in secure mode, as an alternative to AD domain join and SSSD/Winbind based > user-sync service. > Today, class LinuxContainerExecutor already supports running Yarn container > process as one designated local user in non-secure mode. > *We can add new configurations to Yarn, such that with LinuxContainerExecutor > we can pre-create a pool of local users on each Yarn node manager. At > runtime, Yarn node manager allocates a local user to run the container > process, for the domain user that submits the application*. When all > containers of that user are finished and all files belonging to that user are > deleted, we can release the allocation and allow other users to use the same > local user to run their Yarn containers. > Please look at attached design doc for more details. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9834) Allow using a pool of local users to run Yarn Secure Container in secure mode
[ https://issues.apache.org/jira/browse/YARN-9834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shanyu zhao updated YARN-9834: -- Attachment: YarnSecureContainerWithPoolOfLocalUsers.pdf > Allow using a pool of local users to run Yarn Secure Container in secure mode > - > > Key: YARN-9834 > URL: https://issues.apache.org/jira/browse/YARN-9834 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 3.1.2 >Reporter: shanyu zhao >Assignee: shanyu zhao >Priority: Major > Attachments: YarnSecureContainerWithPoolOfLocalUsers.pdf > > > Yarn Secure Container allows separation of different user's local files and > container processes running on the same node manager. This depends on an out > of band service such as SSSD/Winbind to sync all domain users to local > machine that runs Yarn node manager. *Hadoop code only works with local > users*. > Winbind/SSSD user sync has lots of overhead, especially for large > corporations. Also if running Yarn node manager inside Kubernetes cluster > (meaning node managers running inside Docker container), it doesn't make > sense for each Docker container to domain join with Active Directory and sync > a whole copy of domain users to the Docker container. > We need an optional light-weighted approach to enable Yarn Secure Container > in secure mode, as an alternative to AD domain join and SSSD/Winbind based > user-sync service. > Today, class LinuxContainerExecutor already supports running Yarn container > process as one designated local user in non-secure mode. > *We can add new configurations to Yarn, such that with LinuxContainerExecutor > we can pre-create a pool of local users on each Yarn node manager. At > runtime, Yarn node manager allocates a local user to run the container > process, for the domain user that submits the application*. When all > containers of that user are finished and all files belonging to that user are > deleted, we can release the allocation and allow other users to use the same > local user to run their Yarn containers. > Please look at attached design doc for more details. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9834) Allow using a pool of local users to run Yarn Secure Container in secure mode
[ https://issues.apache.org/jira/browse/YARN-9834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shanyu zhao updated YARN-9834: -- Attachment: YarnSecureContainerWithPoolOfLocalUsers.pdf > Allow using a pool of local users to run Yarn Secure Container in secure mode > - > > Key: YARN-9834 > URL: https://issues.apache.org/jira/browse/YARN-9834 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 3.1.2 >Reporter: shanyu zhao >Assignee: shanyu zhao >Priority: Major > Attachments: YarnSecureContainerWithPoolOfLocalUsers.pdf > > > Yarn Secure Container allows separation of different user's local files and > container processes running on the same node manager. This depends on an out > of band service such as SSSD/Winbind to sync all domain users to local > machine that runs Yarn node manager. *Hadoop code only works with local > users*. > Winbind/SSSD user sync has lots of overhead, especially for large > corporations. Also if running Yarn node manager inside Kubernetes cluster > (meaning node managers running inside Docker container), it doesn't make > sense for each Docker container to domain join with Active Directory and sync > a whole copy of domain users to the Docker container. > We need an optional light-weighted approach to enable Yarn Secure Container > in secure mode, as an alternative to AD domain join and SSSD/Winbind based > user-sync service. > Today, class LinuxContainerExecutor already supports running Yarn container > process as one designated local user in non-secure mode. > *We can add new configurations to Yarn, such that with LinuxContainerExecutor > we can pre-create a pool of local users on each Yarn node manager. At > runtime, Yarn node manager allocates a local user to run the container > process, for the domain user that submits the application*. When all > containers of that user are finished and all files belonging to that user are > deleted, we can release the allocation and allow other users to use the same > local user to run their Yarn containers. > Please look at attached design doc for more details. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9834) Allow using a pool of local users to run Yarn Secure Container in secure mode
[ https://issues.apache.org/jira/browse/YARN-9834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shanyu zhao updated YARN-9834: -- Description: Yarn Secure Container allows separation of different user's local files and container processes running on the same node manager. This depends on an out of band service such as SSSD/Winbind to sync all domain users to local machine that runs Yarn node manager. * Hadoop code only works with local users*. Winbind/SSSD user sync has lots of overhead, especially for large corporations. Also if running Yarn node manager inside Kubernetes cluster (meaning node managers running inside Docker container), it doesn't make sense for each Docker container to domain join with Active Directory and sync a whole copy of domain users to the Docker container. We need an optional light-weighted approach to enable Yarn Secure Container in secure mode, as an alternative to AD domain join and SSSD/Winbind based user-sync service. Today, class LinuxContainerExecutor already supports running Yarn container process as one designated local user in non-secure mode. *We can add new configurations to Yarn, such that with LinuxContainerExecutor we can pre-create a pool of local users on each Yarn node manager. At runtime, Yarn node manager allocates a local user to run the container process, for the domain user that submits the application*. When all containers of that user are finished and all files belonging to that user are deleted, we can release the allocation and allow other users to use the same local user to run their Yarn containers. Please look at attached design doc for more details. was: Yarn Secure Container allows separation of different user's local files and container processes running on the same node manager. This depends on an out of band service such as SSSD/Winbind to sync all domain users to local machine that runs Yarn node manager.* Hadoop code only works with local users*. Winbind/SSSD user sync has lots of overhead, especially for large corporations. Also if running Yarn node manager inside Kubernetes cluster (meaning node managers running inside Docker container), it doesn't make sense for each Docker container to domain join with Active Directory and sync a whole copy of domain users to the Docker container. We need an optional light-weighted approach to enable Yarn Secure Container in secure mode, as an alternative to AD domain join and SSSD/Winbind based user-sync service. Today, class LinuxContainerExecutor already supports running Yarn container process as one designated local user in non-secure mode. *We can add new configurations to Yarn, such that with LinuxContainerExecutor we can pre-create a pool of local users on each Yarn node manager. At runtime, Yarn node manager allocates a local user to run the container process, for the domain user that submits the application*. When all containers of that user are finished and all files belonging to that user are deleted, we can release the allocation and allow other users to use the same local user to run their Yarn containers. Please look at attached design doc for more details. > Allow using a pool of local users to run Yarn Secure Container in secure mode > - > > Key: YARN-9834 > URL: https://issues.apache.org/jira/browse/YARN-9834 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 3.1.2 >Reporter: shanyu zhao >Assignee: shanyu zhao >Priority: Major > > Yarn Secure Container allows separation of different user's local files and > container processes running on the same node manager. This depends on an out > of band service such as SSSD/Winbind to sync all domain users to local > machine that runs Yarn node manager. * Hadoop code only works with local > users*. > Winbind/SSSD user sync has lots of overhead, especially for large > corporations. Also if running Yarn node manager inside Kubernetes cluster > (meaning node managers running inside Docker container), it doesn't make > sense for each Docker container to domain join with Active Directory and sync > a whole copy of domain users to the Docker container. > We need an optional light-weighted approach to enable Yarn Secure Container > in secure mode, as an alternative to AD domain join and SSSD/Winbind based > user-sync service. > Today, class LinuxContainerExecutor already supports running Yarn container > process as one designated local user in non-secure mode. > *We can add new configurations to Yarn, such that with LinuxContainerExecutor > we can pre-create a pool of local users on each Yarn node manager. At > runtime, Yarn node manager allocates a local user to run the container > process, for the domain user that
[jira] [Updated] (YARN-9834) Allow using a pool of local users to run Yarn Secure Container in secure mode
[ https://issues.apache.org/jira/browse/YARN-9834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shanyu zhao updated YARN-9834: -- Description: Yarn Secure Container allows separation of different user's local files and container processes running on the same node manager. This depends on an out of band service such as SSSD/Winbind to sync all domain users to local machine that runs Yarn node manager. *Hadoop code only works with local users*. Winbind/SSSD user sync has lots of overhead, especially for large corporations. Also if running Yarn node manager inside Kubernetes cluster (meaning node managers running inside Docker container), it doesn't make sense for each Docker container to domain join with Active Directory and sync a whole copy of domain users to the Docker container. We need an optional light-weighted approach to enable Yarn Secure Container in secure mode, as an alternative to AD domain join and SSSD/Winbind based user-sync service. Today, class LinuxContainerExecutor already supports running Yarn container process as one designated local user in non-secure mode. *We can add new configurations to Yarn, such that with LinuxContainerExecutor we can pre-create a pool of local users on each Yarn node manager. At runtime, Yarn node manager allocates a local user to run the container process, for the domain user that submits the application*. When all containers of that user are finished and all files belonging to that user are deleted, we can release the allocation and allow other users to use the same local user to run their Yarn containers. Please look at attached design doc for more details. was: Yarn Secure Container allows separation of different user's local files and container processes running on the same node manager. This depends on an out of band service such as SSSD/Winbind to sync all domain users to local machine that runs Yarn node manager. * Hadoop code only works with local users*. Winbind/SSSD user sync has lots of overhead, especially for large corporations. Also if running Yarn node manager inside Kubernetes cluster (meaning node managers running inside Docker container), it doesn't make sense for each Docker container to domain join with Active Directory and sync a whole copy of domain users to the Docker container. We need an optional light-weighted approach to enable Yarn Secure Container in secure mode, as an alternative to AD domain join and SSSD/Winbind based user-sync service. Today, class LinuxContainerExecutor already supports running Yarn container process as one designated local user in non-secure mode. *We can add new configurations to Yarn, such that with LinuxContainerExecutor we can pre-create a pool of local users on each Yarn node manager. At runtime, Yarn node manager allocates a local user to run the container process, for the domain user that submits the application*. When all containers of that user are finished and all files belonging to that user are deleted, we can release the allocation and allow other users to use the same local user to run their Yarn containers. Please look at attached design doc for more details. > Allow using a pool of local users to run Yarn Secure Container in secure mode > - > > Key: YARN-9834 > URL: https://issues.apache.org/jira/browse/YARN-9834 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 3.1.2 >Reporter: shanyu zhao >Assignee: shanyu zhao >Priority: Major > > Yarn Secure Container allows separation of different user's local files and > container processes running on the same node manager. This depends on an out > of band service such as SSSD/Winbind to sync all domain users to local > machine that runs Yarn node manager. *Hadoop code only works with local > users*. > Winbind/SSSD user sync has lots of overhead, especially for large > corporations. Also if running Yarn node manager inside Kubernetes cluster > (meaning node managers running inside Docker container), it doesn't make > sense for each Docker container to domain join with Active Directory and sync > a whole copy of domain users to the Docker container. > We need an optional light-weighted approach to enable Yarn Secure Container > in secure mode, as an alternative to AD domain join and SSSD/Winbind based > user-sync service. > Today, class LinuxContainerExecutor already supports running Yarn container > process as one designated local user in non-secure mode. > *We can add new configurations to Yarn, such that with LinuxContainerExecutor > we can pre-create a pool of local users on each Yarn node manager. At > runtime, Yarn node manager allocates a local user to run the container > process, for the domain user that
[jira] [Updated] (YARN-9834) Allow using a pool of local users to run Yarn Secure Container in secure mode
[
https://issues.apache.org/jira/browse/YARN-9834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
shanyu zhao updated YARN-9834:
--
Description:
Yarn Secure Container allows separation of different user's local files and
container processes running on the same node manager. This depends on an out of
band service such as SSSD/Winbind to sync all domain users to local machine
that runs Yarn node manager.* Hadoop code only works with local users*.
Winbind/SSSD user sync has lots of overhead, especially for large corporations.
Also if running Yarn node manager inside Kubernetes cluster (meaning node
managers running inside Docker container), it doesn't make sense for each
Docker container to domain join with Active Directory and sync a whole copy of
domain users to the Docker container.
We need an optional light-weighted approach to enable Yarn Secure Container in
secure mode, as an alternative to AD domain join and SSSD/Winbind based
user-sync service.
Today, class LinuxContainerExecutor already supports running Yarn container
process as one designated local user in non-secure mode.
*We can add new configurations to Yarn, such that with LinuxContainerExecutor
we can pre-create a pool of local users on each Yarn node manager. At runtime,
Yarn node manager allocates a local user to run the container process, for the
domain user that submits the application*. When all containers of that user are
finished and all files belonging to that user are deleted, we can release the
allocation and allow other users to use the same local user to run their Yarn
containers.
Please look at attached design doc for more details.
was:
Yarn Secure Container in secure mode allows separation of different user's
local files and container processes running on the same node manager. This
depends on an out of band service such as SSSD/Winbind to sync all domain users
to local machine.
Winbind user sync has lots of overhead, especially for large corporations. Also
if running Yarn inside Kubernetes cluster (meaning node managers running inside
Docker container), it doesn't make sense for each container (running node
manager inside) to domain join with Active Directory and sync a whole copy of
domain users.
We need a light-weighted config to enable Yarn Secure Container as an
alternative to AD domain join, SSSD/Winbind.
We should allow a new configuration to Yarn, such that we can pre-create a pool
of users on each machine/Docker container. And at runtime, Yarn allocates a
local user to the domain user that submits the application. When all containers
of that user are finished and all files belonging to that user are deleted, we
can release the allocation and allow other users to use the same local user to
run their Yarn containers.
h2. Design
We propose to extend LinuxContainerExecutor to support pool-user in secure
mode. LinuxContainerExecutor is the main class and accompanying classes and the
container-executor binary to implement the Yarn Secure Container feature.
There are existing configurations like
"yarn.nodemanager.linux-container-executor.nonsecure-mode.xxx", we propose to
add these new configurations for secure mode:
{code:java}
yarn.nodemanager.linux-container-executor.secure-mode.use-pool-user, defaults
to false
yarn.nodemanager.linux-container-executor.secure-mode.pool-user-prefix,
defaults to "user"
yarn.nodemanager.linux-container-executor.secure-mode.pool-user-count, defaults
to -1, meaning the value of yarn.nodemanager.resource.cpu-vcores are used.
{code}
By default this feature is turned off. If we enable it, with pool-user-prefix
set to "user", then we expect there are pre-created local users user0 - usern,
where the total number of local users equals to pool-user-count. The default
pool-user-count equals to cpu-vcores, because in theory that's the maximum
number of concurrent containers running on a given yarn node manager.
We use an in-memory allocator to keep the domain user to local user mapping.
Now when to add the mapping and when to remove it?
In node manager, ApplicationImpl implements the state machine for a Yarn app
life cycle, only if the app has at least 1 container running on that node
manager. We can hook up the code to add the mapping during application
initialization.
For removing the mapping, we need to wait for 3 things:
1) All applications of the same user is completed;
2) All log handling of the applications (log aggregation or non-aggregated
handling) is done;
3) All pending FileDeletionTask that use the user's identity is finished.
Note that all operation to these reference counting should be synchronized
operation.
If all of our local users in the pool are allocated, we'll return
"nonexistuser" as runas user, this will cause the container to fail to execute
and Yarn will relaunch it in other nodes.
What about node manager restarts? During
[jira] [Updated] (YARN-9834) Allow using a pool of local users to run Yarn Secure Container in secure mode
[
https://issues.apache.org/jira/browse/YARN-9834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
shanyu zhao updated YARN-9834:
--
Description:
Yarn Secure Container in secure mode allows separation of different user's
local files and container processes running on the same node manager. This
depends on an out of band service such as SSSD/Winbind to sync all domain users
to local machine.
Winbind user sync has lots of overhead, especially for large corporations. Also
if running Yarn inside Kubernetes cluster (meaning node managers running inside
Docker container), it doesn't make sense for each container (running node
manager inside) to domain join with Active Directory and sync a whole copy of
domain users.
We need a light-weighted config to enable Yarn Secure Container as an
alternative to AD domain join, SSSD/Winbind.
We should allow a new configuration to Yarn, such that we can pre-create a pool
of users on each machine/Docker container. And at runtime, Yarn allocates a
local user to the domain user that submits the application. When all containers
of that user are finished and all files belonging to that user are deleted, we
can release the allocation and allow other users to use the same local user to
run their Yarn containers.
h2. Design
We propose to extend LinuxContainerExecutor to support pool-user in secure
mode. LinuxContainerExecutor is the main class and accompanying classes and the
container-executor binary to implement the Yarn Secure Container feature.
There are existing configurations like
"yarn.nodemanager.linux-container-executor.nonsecure-mode.xxx", we propose to
add these new configurations for secure mode:
{code:java}
yarn.nodemanager.linux-container-executor.secure-mode.use-pool-user, defaults
to false
yarn.nodemanager.linux-container-executor.secure-mode.pool-user-prefix,
defaults to "user"
yarn.nodemanager.linux-container-executor.secure-mode.pool-user-count, defaults
to -1, meaning the value of yarn.nodemanager.resource.cpu-vcores are used.
{code}
By default this feature is turned off. If we enable it, with pool-user-prefix
set to "user", then we expect there are pre-created local users user0 - usern,
where the total number of local users equals to pool-user-count. The default
pool-user-count equals to cpu-vcores, because in theory that's the maximum
number of concurrent containers running on a given yarn node manager.
We use an in-memory allocator to keep the domain user to local user mapping.
Now when to add the mapping and when to remove it?
In node manager, ApplicationImpl implements the state machine for a Yarn app
life cycle, only if the app has at least 1 container running on that node
manager. We can hook up the code to add the mapping during application
initialization.
For removing the mapping, we need to wait for 3 things:
1) All applications of the same user is completed;
2) All log handling of the applications (log aggregation or non-aggregated
handling) is done;
3) All pending FileDeletionTask that use the user's identity is finished.
Note that all operation to these reference counting should be synchronized
operation.
If all of our local users in the pool are allocated, we'll return
"nonexistuser" as runas user, this will cause the container to fail to execute
and Yarn will relaunch it in other nodes.
What about node manager restarts? During ResourceLocalizationService init, it
renames the root folders used by the node manager and schedules
FileDeletionTask to delete the content of these files as the owner (local pool
users) of these local files. To prevent the newly launched Yarn containers to
be able to peek into the yet-to-be-deleted old application folders right after
node manager restart, we can allocate these local pool users to the requested
user in FileDeletionTask, which results in a call to incrementFileOpCount().
Therefore we allow local pool user allocation during the call to
incrementFileOpCount(appUser) and if appUser matches with a local pool user, we
allocate that user to the same named appUser, preventing new containers to
reuse the same local pool user, until all the FileDeletionTask belonging to
that user are done.
h2. Limitations
1) This feature does not support PRIVATE visibility type of resource
allocation. Because PRIVATE type of resources are potentially cached in the
node manager for a very long time, supporting it will be a security problem
that a user might be able to peek into previous user's PRIVATE resources. We
can modify code to treat all PRIVATE type of resource as APPLICATION type.
2) It is recommended to enable DominantResourceCalculator so that no more than
"cpu-vcores" number of concurrent containers running on a node manager:
{code:java}
yarn.scheduler.capacity.resource-calculator
= org.apache.hadoop.yarn.util.resource.DominantResourceCalculator {code}
3) Currently this feature does not work with
[jira] [Updated] (YARN-9834) Allow using a pool of local users to run Yarn Secure Container in secure mode
[
https://issues.apache.org/jira/browse/YARN-9834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
shanyu zhao updated YARN-9834:
--
Description:
Yarn Secure Container in secure mode allows separation of different user's
local files and container processes running on the same node manager. This
depends on an out of band service such as SSSD/Winbind to sync all domain users
to local machine.
Winbind user sync has lots of overhead, especially for large corporations. Also
if running Yarn inside Kubernetes cluster (meaning node managers running inside
Docker container), it doesn't make sense for each container to domain join with
Active Directory and sync a whole copy of domain users.
We should allow a new configuration to Yarn, such that we can pre-create a pool
of users on each machine/Docker container. And at runtime, Yarn allocates a
local user to the domain user that submits the application. When all containers
of that user are finished and all files belonging to that user are deleted, we
can release the allocation and allow other users to use the same local user to
run their Yarn containers.
h2. Design
We propose to add these new configurations:
{code:java}
yarn.nodemanager.linux-container-executor.secure-mode.use-pool-user, defaults
to false
yarn.nodemanager.linux-container-executor.secure-mode.pool-user-prefix,
defaults to "user"
yarn.nodemanager.linux-container-executor.secure-mode.pool-user-count, defaults
to -1, meaning the value of yarn.nodemanager.resource.cpu-vcores are used.
{code}
By default this feature is turned off. If we enable it, with pool-user-prefix
set to "user", then we expect there are pre-created local users user0 - usern,
where the total number of local users equals to pool-user-count. The default
pool-user-count equals to cpu-vcores, because in theory that's the maximum
number of concurrent containers running on a given yarn node manager.
We use an in-memory allocator to keep the domain user to local user mapping.
Now when to add the mapping and when to remove it?
In node manager, ApplicationImpl implements the state machine for a Yarn app
life cycle, only if the app has at least 1 container running on that node
manager. We can hook up the code to add the mapping during application
initialization.
For removing the mapping, we need to wait for 3 things:
1) All applications of the same user is completed;
2) All log handling of the applications (log aggregation or non-aggregated
handling) is done;
3) All pending FileDeletionTask that use the user's identity is finished.
Note that all operation to these reference counting should be synchronized
operation.
If all of our local users in the pool are allocated, we'll return
"nonexistuser" as runas user, this will cause the container to fail to execute
and Yarn will relaunch it in other nodes.
What about node manager restarts? During ResourceLocalizationService init, it
renames the root folders used by the node manager and schedules
FileDeletionTask to delete the content of these files as the owner (local pool
users) of these local files. To prevent the newly launched Yarn containers to
be able to peek into the yet-to-be-deleted old application folders right after
node manager restart, we can allocate these local pool users to the requested
user in FileDeletionTask, which results in a call to incrementFileOpCount().
Therefore we allow local pool user allocation during the call to
incrementFileOpCount(appUser) and if appUser matches with a local pool user, we
allocate that user to the same named appUser, preventing new containers to
reuse the same local pool user, until all the FileDeletionTask belonging to
that user are done.
h2. Limitations
1) This feature does not support PRIVATE visibility type of resource
allocation. Because PRIVATE type of resources are potentially cached in the
node manager for a very long time, supporting it will be a security problem
that a user might be able to peek into previous user's PRIVATE resources. We
can modify code to treat all PRIVATE type of resource as APPLICATION type.
2) It is recommended to enable DominantResourceCalculator so that no more than
"cpu-vcores" number of concurrent containers running on a node manager:
{code:java}
yarn.scheduler.capacity.resource-calculator
= org.apache.hadoop.yarn.util.resource.DominantResourceCalculator {code}
3) Currently this feature does not work with Yarn Node Manager recovery. This
is because the mappings are kept in memory, it cannot be recovered after node
manager restart.
was:
Yarn Secure Container in secure mode allows separation of different user's
local files and container processes running on the same node manager. This
depends on an out of band service such as SSSD/Winbind to sync all domain users
to local machine.
Winbind user sync has lots of overhead, especially for large corporations. Also
if running Yarn inside
[jira] [Updated] (YARN-9834) Allow using a pool of local users to run Yarn Secure Container in secure mode
[
https://issues.apache.org/jira/browse/YARN-9834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
shanyu zhao updated YARN-9834:
--
Description:
Yarn Secure Container in secure mode allows separation of different user's
local files and container processes running on the same node manager. This
depends on an out of band service such as SSSD/Winbind to sync all domain users
to local machine.
Winbind user sync has lots of overhead, especially for large corporations. Also
if running Yarn inside Kubernetes cluster (meaning node managers running inside
Docker container), it doesn't make sense for each container to domain join with
Active Directory and sync a whole copy of domain users.
We should allow a new configuration to Yarn, such that we can pre-create a pool
of users on each machine/Docker container. And at runtime, Yarn allocates a
local user to the domain user that submits the application. When all containers
of that user are finished and all files belonging to that user are deleted, we
can release the allocation and allow other users to use the same local user to
run their Yarn containers.
h2. Design
We propose to add these new configurations:
{code:java}
yarn.nodemanager.linux-container-executor.secure-mode.use-local-user, defaults
to false
yarn.nodemanager.linux-container-executor.secure-mode.local-user-prefix,
defaults to "user"{code}
By default this feature is turned off. If we enable it, with local-user-prefix
set to "user", then we expect there are pre-created local users user0 - usern,
where the total number of local users equals to:
{code:java}
yarn.nodemanager.resource.cpu-vcores {code}
We can use an in-memory allocator to keep the domain user to local user
mapping.
Now when to add the mapping and when to remove it?
In node manager, ApplicationImpl implements the state machine for a Yarn app
life cycle, only if the app has at least 1 container running on that node
manager. We can hook up the code to add the mapping during application
initialization.
For removing the mapping, we need to wait for 3 things:
1) All applications of the same user is completed;
2) All log handling of the applications (log aggregation or non-aggregated
handling) is done;
3) All pending FileDeletionTask that use the user's identity is finished.
Note that all operation to these reference counting should be synchronized
operation.
If all of our local users in the pool are allocated, we'll return
"nonexistuser" as runas user, this will cause the container to fail to execute
and Yarn will relaunch it in other nodes.
What about node manager restarts? During ResourceLocalizationService init, it
renames the root folders used by the node manager and schedules
FileDeletionTask to delete the content of these files. To prevent the newly
launched Yarn containers to be able to peek into the yet-to-be-deleted old
application folders right after node manager restart, we can chmod the root
folders to 700 right after rename.
h2. Limitations
1) This feature does not support PRIVATE visibility type of resource
allocation. Because PRIVATE type of resources are potentially cached in the
node manager for a very long time, supporting it will be a security problem
that a user might be able to peek into previous user's PRIVATE resources. We
can modify code to treat all PRIVATE type of resource as APPLICATION type.
2) It is recommended to enable DominantResourceCalculator so that no more than
"cpu-vcores" number of concurrent containers running on a node manager:
{code:java}
yarn.scheduler.capacity.resource-calculator
= org.apache.hadoop.yarn.util.resource.DominantResourceCalculator {code}
3) Currently this feature does not work with Yarn Node Manager recovery. This
is because the mappings are kept in memory, it cannot be recovered after node
manager restart.
was:
Yarn Secure Container in secure mode allows separation of different user's
local files and container processes running on the same node manager. This
depends on an out of band service such as SSSD/Winbind to sync all domain users
to local machine.
Winbind user sync has lots of overhead, especially for large corporations. Also
if running Yarn inside Kubernetes cluster (meaning node managers running inside
Docker container), it doesn't make sense for each container to domain join with
Active Directory and sync a whole copy of domain users.
We should allow a new configuration to Yarn, such that we can pre-create a pool
of users on each machine/Docker container. And at runtime, Yarn allocates a
local user to the domain user that submits the application. When all containers
of that user are finished and all files belonging to that user are deleted, we
can release the allocation and allow other users to use the same local user to
run their Yarn containers.
h2. Design
We propose to add these new configurations:
{code:java}
[jira] [Updated] (YARN-9834) Allow using a pool of local users to run Yarn Secure Container in secure mode
[
https://issues.apache.org/jira/browse/YARN-9834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
shanyu zhao updated YARN-9834:
--
Description:
Yarn Secure Container in secure mode allows separation of different user's
local files and container processes running on the same node manager. This
depends on an out of band service such as SSSD/Winbind to sync all domain users
to local machine.
Winbind user sync has lots of overhead, especially for large corporations. Also
if running Yarn inside Kubernetes cluster (meaning node managers running inside
Docker container), it doesn't make sense for each container to domain join with
Active Directory and sync a whole copy of domain users.
We should allow a new configuration to Yarn, such that we can pre-create a pool
of users on each machine/Docker container. And at runtime, Yarn allocates a
local user to the domain user that submits the application. When all containers
of that user are finished and all files belonging to that user are deleted, we
can release the allocation and allow other users to use the same local user to
run their Yarn containers.
h2. Design
We propose to add these new configurations:
{code:java}
yarn.nodemanager.linux-container-executor.secure-mode.use-local-user, defaults
to false
yarn.nodemanager.linux-container-executor.secure-mode.local-user-prefix,
defaults to "user"{code}
By default this feature is turned off. If we enable it, with local-user-prefix
set to "user", then we expect there are pre-created local users user0 - usern,
where the total number of local users equals to:
{code:java}
yarn.nodemanager.resource.cpu-vcores {code}
We can use an in-memory allocator to keep the domain user to local user
mapping.
Now when to add the mapping and when to remove it?
In node manager, ApplicationImpl implements the state machine for a Yarn app
life cycle, only if the app has at least 1 container running on that node
manager. We can hook up the code to add the mapping during application
initialization.
For removing the mapping, we need to wait for 3 things:
1) All applications of the same user is completed;
2) All log handling of the applications (log aggregation or non-aggregated
handling) is done;
3) All pending FileDeletionTask that use the user's identity is finished.
Note that all operation to these reference counting should be synchronized
operation.
If all of our local users in the pool are allocated, we'll return
"nonexistuser" as runas user, this will cause the container to fail to execute
and Yarn will relaunch it in other nodes.
h2. Limitations
1) This feature does not support PRIVATE visibility type of resource
allocation. Because PRIVATE type of resources are potentially cached in the
node manager for a very long time, supporting it will be a security problem
that a user might be able to peek into previous user's PRIVATE resources. We
can modify code to treat all PRIVATE type of resource as APPLICATION type.
2) It is recommended to enable DominantResourceCalculator so that no more than
"cpu-vcores" number of concurrent containers running on a node manager:
{code:java}
yarn.scheduler.capacity.resource-calculator
= org.apache.hadoop.yarn.util.resource.DominantResourceCalculator {code}
3) Currently this feature does not work with Yarn Node Manager recovery. This
is because the mappings are kept in memory, it cannot be recovered after node
manager restart.
was:
Yarn Secure Container in secure mode allows separation of different user's
local files and container processes running on the same node manager. This
depends on an out of band service such as SSSD/Winbind to sync all domain users
to local machine.
SSSD/Winbind user sync has lots of overhead, especially for large corporations.
Also if running Yarn inside Kubernetes cluster (meaning node managers running
inside Docker container), it doesn't make sense for each container to sync a
whole copy of domain users.
We should allow a new configuration to Yarn, such that we can pre-create a pool
of users on each machine/Docker container. And at runtime, Yarn allocates a
local user to the domain user that submits the application. When all containers
of that user are finished and all files belonging to that user are deleted, we
can release the allocation and allow other users to use the same local user to
run their Yarn containers.
h2. Design
We propose to add these new configurations:
{code:java}
yarn.nodemanager.linux-container-executor.secure-mode.use-local-user, defaults
to false
yarn.nodemanager.linux-container-executor.secure-mode.local-user-prefix,
defaults to "user"{code}
By default this feature is turned off. If we enable it, with local-user-prefix
set to "user", then we expect there are pre-created local users user0 - usern,
where the total number of local users equals to:
{code:java}
yarn.nodemanager.resource.cpu-vcores {code}
We can use an in-memory
[jira] [Updated] (YARN-9834) Allow using a pool of local users to run Yarn Secure Container in secure mode
[
https://issues.apache.org/jira/browse/YARN-9834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
shanyu zhao updated YARN-9834:
--
Description:
Yarn Secure Container in secure mode allows separation of different user's
local files and container processes running on the same node manager. This
depends on an out of band service such as SSSD/Winbind to sync all domain users
to local machine.
SSSD/Winbind user sync has lots of overhead, especially for large corporations.
Also if running Yarn inside Kubernetes cluster (meaning node managers running
inside Docker container), it doesn't make sense for each container to sync a
whole copy of domain users.
We should allow a new configuration to Yarn, such that we can pre-create a pool
of users on each machine/Docker container. And at runtime, Yarn allocates a
local user to the domain user that submits the application. When all containers
of that user are finished and all files belonging to that user are deleted, we
can release the allocation and allow other users to use the same local user to
run their Yarn containers.
h2. Design
We propose to add these new configurations:
{code:java}
yarn.nodemanager.linux-container-executor.secure-mode.use-local-user, defaults
to false
yarn.nodemanager.linux-container-executor.secure-mode.local-user-prefix,
defaults to "user"{code}
By default this feature is turned off. If we enable it, with local-user-prefix
set to "user", then we expect there are pre-created local users user0 - usern,
where the total number of local users equals to:
{code:java}
yarn.nodemanager.resource.cpu-vcores {code}
We can use an in-memory allocator to keep the domain user to local user
mapping.
Now when to add the mapping and when to remove it?
In node manager, ApplicationImpl implements the state machine for a Yarn app
life cycle, only if the app has at least 1 container running on that node
manager. We can hook up the code to add the mapping during application
initialization.
For removing the mapping, we need to wait for 3 things:
1) All applications of the same user is completed;
2) All log handling of the applications (log aggregation or non-aggregated
handling) is done;
3) All pending FileDeletionTask that use the user's identity is finished.
If all of our local users in the pool are allocated, we'll return
"nonexistuser" as runas user, this will cause the container to fail to execute
and Yarn will relaunch it in other nodes.
h2. Limitations
1) This feature does not support PRIVATE visibility type of resource
allocation. Because PRIVATE type of resources are potentially cached in the
node manager for a very long time, supporting it will be a security problem
that a user might be able to peek into previous user's PRIVATE resources. We
can modify code to treat all PRIVATE type of resource as APPLICATION type.
2) It is recommended to enable DominantResourceCalculator so that no more than
"cpu-vcores" number of concurrent containers running on a node manager:
{code:java}
yarn.scheduler.capacity.resource-calculator
= org.apache.hadoop.yarn.util.resource.DominantResourceCalculator {code}
3) Currently this feature does not work with Yarn Node Manager recovery. This
is because the mappings are kept in memory, it cannot be recovered after node
manager restart.
was:
Yarn Secure Container in secure mode allows separation of different user's
local files and container processes running on the same node manager. This
depends on an out of band service such as SSSD/Winbind to sync all domain users
to local machine.
SSSD/Winbind user sync has lots of overhead, especially for large corporations.
Also if running Yarn inside Kubernetes cluster (meaning node managers running
inside Docker container), it doesn't make sense for each container to sync a
whole copy of domain users.
We should allow a new configuration to Yarn, such that we can pre-create a pool
of users on each machine/Docker container. And at runtime, Yarn allocates a
local user to the domain user that submits the application. When all containers
of that user and all files belonging to that user are deleted, we can release
the allocation and allow other users to use the same local user to run their
Yarn containers.
h2. Design
We propose to add these new configurations:
{code:java}
yarn.nodemanager.linux-container-executor.secure-mode.use-local-user, defaults
to false
yarn.nodemanager.linux-container-executor.secure-mode.local-user-prefix,
defaults to "user"{code}
By default this feature is turned off. If we enable it, with local-user-prefix
set to "user", then we expect there are pre-created local users user0 - usern,
where n equals to:
{code:java}
yarn.nodemanager.resource.cpu-vcores {code}
We can use an in-memory allocator to keep the domain user to local user
mapping. When to add the mapping and when to remove it?
In node manager, ApplicationImpl implements the state machine
[jira] [Updated] (YARN-9834) Allow using a pool of local users to run Yarn Secure Container in secure mode
[
https://issues.apache.org/jira/browse/YARN-9834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
shanyu zhao updated YARN-9834:
--
Description:
Yarn Secure Container in secure mode allows separation of different user's
local files and container processes running on the same node manager. This
depends on an out of band service such as SSSD/Winbind to sync all domain users
to local machine.
SSSD/Winbind user sync has lots of overhead, especially for large corporations.
Also if running Yarn inside Kubernetes cluster (meaning node managers running
inside Docker container), it doesn't make sense for each container to sync a
whole copy of domain users.
We should allow a new configuration to Yarn, such that we can pre-create a pool
of users on each machine/Docker container. And at runtime, Yarn allocates a
local user to the domain user that submits the application. When all containers
of that user and all files belonging to that user are deleted, we can release
the allocation and allow other users to use the same local user to run their
Yarn containers.
h2. Design
We propose to add these new configurations:
{code:java}
yarn.nodemanager.linux-container-executor.secure-mode.use-local-user, defaults
to false
yarn.nodemanager.linux-container-executor.secure-mode.local-user-prefix,
defaults to "user"{code}
By default this feature is turned off. If we enable it, with local-user-prefix
set to "user", then we expect there are pre-created local users user0 - usern,
where n equals to:
{code:java}
yarn.nodemanager.resource.cpu-vcores {code}
We can use an in-memory allocator to keep the domain user to local user
mapping. When to add the mapping and when to remove it?
In node manager, ApplicationImpl implements the state machine for a Yarn app
life cycle, only if the app has at least 1 container running on that node
manager. We can hook up the code to add the mapping during application
initialization.
For removing the mapping, we need to wait for 3 things:
1) All applications of the same user is completed;
2) All log handling of the applications (log aggregation or non-aggregated
handling) is done;
3) All pending FileDeletionTask that use the user's identity is finished.
If all of our local users in the pool are allocated, we'll return
"nonexistuser" as runas user, this will cause the container to fail to execute
and Yarn will relaunch it in other nodes.
h2. Limitations
1) This feature does not support PRIVATE visibility type of resource
allocation. Because PRIVATE type of resources are potentially cached in the
node manager for a very long time, supporting it will be a security problem
that a user might be able to peek into previous user's PRIVATE resources. We
can modify code to treat all PRIVATE type of resource as APPLICATION type.
2) It is recommended to enable DominantResourceCalculator so that no more than
"cpu-vcores" number of concurrent containers running on a node manager:
{code:java}
yarn.scheduler.capacity.resource-calculator
= org.apache.hadoop.yarn.util.resource.DominantResourceCalculator {code}
3) Currently this feature does not work with Yarn Node Manager recovery. This
is because the mappings are kept in memory, it cannot be recovered after node
manager restart.
was:
Yarn Secure Container in secure mode allows separation of different user's
local files and container processes running on the same node manager. This
depends on an out of band service such as SSSD/Winbind to sync all domain users
to local machine.
SSSD/Winbind user sync has lots of overhead, especially for large corporations.
Also if running Yarn inside Kubernetes cluster (meaning node managers running
inside Docker container), it doesn't make sense for each container to sync a
whole copy of domain users.
We should allow a new configuration to Yarn, such that we can pre-create a pool
of users on each machine/Docker container. And at runtime, Yarn allocates a
local user to the domain user that submits the application. When all containers
of that user and all files belonging to that user are deleted, we can release
the allocation and allow other users to use the same local user to run their
Yarn containers.
h2. Design
We propose to add these new configurations:
{code:java}
yarn.nodemanager.linux-container-executor.secure-mode.use-local-user, defaults
to false
yarn.nodemanager.linux-container-executor.secure-mode.local-user-prefix,
defaults to "user"{code}
By default this feature is turned off. If we enable it, with local-user-prefix
set to "user", then we expect there are pre-created local users user0 - usern,
where n equals to:
{code:java}
yarn.nodemanager.resource.cpu-vcores {code}
We can use an in-memory allocator to keep the domain user to local user
mapping. When to add the mapping and when to remove it?
In node manager, ApplicationImpl implements the state machine for a Yarn app
life cycle, only if the app has
[jira] [Updated] (YARN-9834) Allow using a pool of local users to run Yarn Secure Container in secure mode
[
https://issues.apache.org/jira/browse/YARN-9834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
shanyu zhao updated YARN-9834:
--
Description:
Yarn Secure Container in secure mode allows separation of different user's
local files and container processes running on the same node manager. This
depends on an out of band service such as SSSD/Winbind to sync all domain users
to local machine.
SSSD/Winbind user sync has lots of overhead, especially for large corporations.
Also if running Yarn inside Kubernetes cluster (meaning node managers running
inside Docker container), it doesn't make sense for each container to sync a
whole copy of domain users.
We should allow a new configuration to Yarn, such that we can pre-create a pool
of users on each machine/Docker container. And at runtime, Yarn allocates a
local user to the domain user that submits the application. When all containers
of that user and all files belonging to that user are deleted, we can release
the allocation and allow other users to use the same local user to run their
Yarn containers.
h2. Design
We propose to add these new configurations:
{code:java}
yarn.nodemanager.linux-container-executor.secure-mode.use-local-user, defaults
to false
yarn.nodemanager.linux-container-executor.secure-mode.local-user-prefix,
defaults to "user"{code}
By default this feature is turned off. If we enable it, with local-user-prefix
set to "user", then we expect there are pre-created local users user0 - usern,
where n equals to:
{code:java}
yarn.nodemanager.resource.cpu-vcores {code}
We can use an in-memory allocator to keep the domain user to local user
mapping. When to add the mapping and when to remove it?
In node manager, ApplicationImpl implements the state machine for a Yarn app
life cycle, only if the app has at least 1 container running on that node
manager. We can hook up the code to add the mapping during application
initialization.
For removing the mapping, we need to wait for 3 things:
1) All applications of the same user is completed;
2) All log handling of the applications (log aggregation or non-aggregated
handling) is done;
3) All pending FileDeletionTask that use the user's identity is finished.
h2. Limitations
1) This feature does not support PRIVATE visibility type of resource
allocation. Because PRIVATE type of resources are potentially cached in the
node manager for a very long time, supporting it will be a security problem
that a user might be able to peek into previous user's PRIVATE resources. We
can modify code to treat all PRIVATE type of resource as APPLICATION type.
2) It is recommended to enable DominantResourceCalculator so that no more than
"cpu-vcores" number of concurrent containers running on a node manager:
{code:java}
yarn.scheduler.capacity.resource-calculator
= org.apache.hadoop.yarn.util.resource.DominantResourceCalculator {code}
3) Currently this feature does not work with Yarn Node Manager recovery. We may
add recovery support in the future when we hook up with the right calls in the
recovery flow.
was:
Yarn Secure Container in secure mode allows separation of different user's
local files and container processes running on the same node manager. This
depends on an out of band service such as SSSD/Winbind to sync all domain users
to local machine.
SSSD/Winbind user sync has lots of overhead, especially for large corporations.
Also if running Yarn inside Kubernetes cluster (meaning node managers running
inside Docker container), it doesn't make sense for each container to sync a
whole copy of domain users.
We should allow a new configuration to Yarn, such that we can pre-create a pool
of users on each machine/Docker container. And at runtime, Yarn allocates a
local user to the domain user that submits the application. When all containers
of that user and all files belonging to that user are deleted, we can release
the allocation and allow other users to use the same local user to run their
Yarn containers.
We propose to add these new configurations:
{code:java}
yarn.nodemanager.linux-container-executor.secure-mode.use-local-user, defaults
to false
yarn.nodemanager.linux-container-executor.secure-mode.local-user-prefix,
defaults to "user"{code}
If we enable this feature, with local-user-prefix set to "user", then we expect
there are pre-created local users user0 - usern, where n equals to:
{code:java}
yarn.nodemanager.resource.cpu-vcores {code}
We can use an in-memory allocator to keep the domain user to local user mapping.
Limitations:
1) This feature does not support PRIVATE type of resource allocation. Because
PRIVATE type of resources are potentially cached in the node manager for a very
long time, supporting it will be a security problem that a user might be able
to peek into previous user's PRIVATE resources. We can modify code to treat all
PRIVATE type of resource as APPLICATION.
2) It is
