[jira] [Commented] (YARN-9834) Allow using a pool of local users to run Yarn Secure Container in secure mode
[
https://issues.apache.org/jira/browse/YARN-9834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942361#comment-16942361
]
shanyu zhao commented on YARN-9834:
---
[~eyang] Please see my response inline:
{quote}The current patch does not work for recovery.{quote}
Yes this is listed in as limitations in the attached design doc. It does not
work for Yarn node manager recovery, which by default is turned off. If trying
to enable both local pool users and Yarn node manager recovery, Yarn node
manager will exit with error message.
{quote}Past, present, and future usage of the same pool user may have
conflicts. Detached user process can monitor data generated by future
user.{quote}
Like I already explained in our meeting. There is no concrete example of the
"conflict". The design will restrict pool users from creating folders outside
of Yarn node manager usercache folder. The design guarantees the local files
are all deleted before the local pool user can be reused.
{quote}YARN container does not have full isolation of working directories, path
traversal to user home directory is possible. Docker and runc container can
provide better isolation to prevent accidental leakage of file owned by pool
users.{quote}
As I explained above. There will be no user directory (/home/), and one
local pool user cannot access other local pool user's usercache folder. We have
e2e test cases to cover all kinds of scenarios like this.
{quote}The patch is vulnerable to user job to play tricks with config flag to
trigger code path designed for pool user.{quote}
How could user job play tricks to modify Yarn node manager's configuration? If
they could, they could just steal node manager's keytab file and talk to Yarn
resource manager directly.
{quote}Group membership association. Whether the file should be written with
primary group, or the current directory group owner. This should be handled
with care by application.{quote}
What files are you talking about? Local files does not have any domain group
membership association. This is expected because we are not sharing files for
the same domain user across applications. Please see one of the limitation
section in the design doc. We do not support PRIVATE visibility resources and
will treat it as APPLICATION visibility.
As for your recommendations:
{quote}Implement the new incompatible security model in a separate container
executor to prevent adding security holes to Linux container executor.{quote}
I already mentioned the whole idea of this work is based on extending
LinuxContainerExecutor. This feature is protected by a configuration and turned
off by default, just like an existing configuration
"yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users". I do
not see how can this add security holes to Linux container executor.
{quote}Use Docker container/runc or chroot to provide isolation and remove path
traversal.{quote}
We cannot run Docker container inside Docker container. We are relying on
running Yarn container process as different local user for the isolation, which
is based on the existing Yarn Secure Container implementation in
LinuxContainerExecutor.
{quote}Wangda Tan recommended to implement Auxiliary service to reactively
change the execution model rather than direct modification to
LinuxContainerExecutor to prevent security bugs leaks into
LinuxContainerExecutor through a series of if else statements.{quote}
We can look into Aux services, but as I said, my understanding of Aux services
is that it can only provide service to Yarn node manager, e.g. maybe creating
users or something, but in the end, we need modification to
LinuxContainerExecutor to launch Yarn container process so that we can use Yarn
Secure Container in a light-weight setup.
Finally, I want to stress here, that whoever might be interested in running
Yarn on Kubernetes with security setup, this JIRA provides a secure alternative
way to enable Yarn Secure Container without the need for container domain join
and winbind/SSSD service inside Docker containers.
{quote}
> Allow using a pool of local users to run Yarn Secure Container in secure mode
> -
>
> Key: YARN-9834
> URL: https://issues.apache.org/jira/browse/YARN-9834
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 3.1.2
>Reporter: shanyu zhao
>Assignee: shanyu zhao
>Priority: Major
> Attachments: YarnSecureContainerWithPoolOfLocalUsers.pdf
>
>
> Yarn Secure Container allows separation of different user's local files and
> container processes running on the same node manager. This depends on an out
> of band service such as SSSD/Winbind to sync all domain users to local
> machine that runs Yarn node
[jira] [Updated] (YARN-9834) Allow using a pool of local users to run Yarn Secure Container in secure mode
[ https://issues.apache.org/jira/browse/YARN-9834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shanyu zhao updated YARN-9834: -- Attachment: (was: YarnSecureContainerWithPoolOfLocalUsers.pdf) > Allow using a pool of local users to run Yarn Secure Container in secure mode > - > > Key: YARN-9834 > URL: https://issues.apache.org/jira/browse/YARN-9834 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 3.1.2 >Reporter: shanyu zhao >Assignee: shanyu zhao >Priority: Major > Attachments: YarnSecureContainerWithPoolOfLocalUsers.pdf > > > Yarn Secure Container allows separation of different user's local files and > container processes running on the same node manager. This depends on an out > of band service such as SSSD/Winbind to sync all domain users to local > machine that runs Yarn node manager. *Hadoop code only works with local > users*. > Winbind/SSSD user sync has lots of overhead, especially for large > corporations. Also if running Yarn node manager inside Kubernetes cluster > (meaning node managers running inside Docker container), it doesn't make > sense for each Docker container to domain join with Active Directory and sync > a whole copy of domain users to the Docker container. > We need an optional light-weighted approach to enable Yarn Secure Container > in secure mode, as an alternative to AD domain join and SSSD/Winbind based > user-sync service. > Today, class LinuxContainerExecutor already supports running Yarn container > process as one designated local user in non-secure mode. > *We can add new configurations to Yarn, such that with LinuxContainerExecutor > we can pre-create a pool of local users on each Yarn node manager. At > runtime, Yarn node manager allocates a local user to run the container > process, for the domain user that submits the application*. When all > containers of that user are finished and all files belonging to that user are > deleted, we can release the allocation and allow other users to use the same > local user to run their Yarn containers. > Please look at attached design doc for more details. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9834) Allow using a pool of local users to run Yarn Secure Container in secure mode
[ https://issues.apache.org/jira/browse/YARN-9834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shanyu zhao updated YARN-9834: -- Attachment: YarnSecureContainerWithPoolOfLocalUsers.pdf > Allow using a pool of local users to run Yarn Secure Container in secure mode > - > > Key: YARN-9834 > URL: https://issues.apache.org/jira/browse/YARN-9834 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 3.1.2 >Reporter: shanyu zhao >Assignee: shanyu zhao >Priority: Major > Attachments: YarnSecureContainerWithPoolOfLocalUsers.pdf > > > Yarn Secure Container allows separation of different user's local files and > container processes running on the same node manager. This depends on an out > of band service such as SSSD/Winbind to sync all domain users to local > machine that runs Yarn node manager. *Hadoop code only works with local > users*. > Winbind/SSSD user sync has lots of overhead, especially for large > corporations. Also if running Yarn node manager inside Kubernetes cluster > (meaning node managers running inside Docker container), it doesn't make > sense for each Docker container to domain join with Active Directory and sync > a whole copy of domain users to the Docker container. > We need an optional light-weighted approach to enable Yarn Secure Container > in secure mode, as an alternative to AD domain join and SSSD/Winbind based > user-sync service. > Today, class LinuxContainerExecutor already supports running Yarn container > process as one designated local user in non-secure mode. > *We can add new configurations to Yarn, such that with LinuxContainerExecutor > we can pre-create a pool of local users on each Yarn node manager. At > runtime, Yarn node manager allocates a local user to run the container > process, for the domain user that submits the application*. When all > containers of that user are finished and all files belonging to that user are > deleted, we can release the allocation and allow other users to use the same > local user to run their Yarn containers. > Please look at attached design doc for more details. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9834) Allow using a pool of local users to run Yarn Secure Container in secure mode
[ https://issues.apache.org/jira/browse/YARN-9834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shanyu zhao updated YARN-9834: -- Attachment: YarnSecureContainerWithPoolOfLocalUsers.pdf > Allow using a pool of local users to run Yarn Secure Container in secure mode > - > > Key: YARN-9834 > URL: https://issues.apache.org/jira/browse/YARN-9834 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 3.1.2 >Reporter: shanyu zhao >Assignee: shanyu zhao >Priority: Major > Attachments: YarnSecureContainerWithPoolOfLocalUsers.pdf > > > Yarn Secure Container allows separation of different user's local files and > container processes running on the same node manager. This depends on an out > of band service such as SSSD/Winbind to sync all domain users to local > machine that runs Yarn node manager. *Hadoop code only works with local > users*. > Winbind/SSSD user sync has lots of overhead, especially for large > corporations. Also if running Yarn node manager inside Kubernetes cluster > (meaning node managers running inside Docker container), it doesn't make > sense for each Docker container to domain join with Active Directory and sync > a whole copy of domain users to the Docker container. > We need an optional light-weighted approach to enable Yarn Secure Container > in secure mode, as an alternative to AD domain join and SSSD/Winbind based > user-sync service. > Today, class LinuxContainerExecutor already supports running Yarn container > process as one designated local user in non-secure mode. > *We can add new configurations to Yarn, such that with LinuxContainerExecutor > we can pre-create a pool of local users on each Yarn node manager. At > runtime, Yarn node manager allocates a local user to run the container > process, for the domain user that submits the application*. When all > containers of that user are finished and all files belonging to that user are > deleted, we can release the allocation and allow other users to use the same > local user to run their Yarn containers. > Please look at attached design doc for more details. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9834) Allow using a pool of local users to run Yarn Secure Container in secure mode
[ https://issues.apache.org/jira/browse/YARN-9834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shanyu zhao updated YARN-9834: -- Description: Yarn Secure Container allows separation of different user's local files and container processes running on the same node manager. This depends on an out of band service such as SSSD/Winbind to sync all domain users to local machine that runs Yarn node manager. * Hadoop code only works with local users*. Winbind/SSSD user sync has lots of overhead, especially for large corporations. Also if running Yarn node manager inside Kubernetes cluster (meaning node managers running inside Docker container), it doesn't make sense for each Docker container to domain join with Active Directory and sync a whole copy of domain users to the Docker container. We need an optional light-weighted approach to enable Yarn Secure Container in secure mode, as an alternative to AD domain join and SSSD/Winbind based user-sync service. Today, class LinuxContainerExecutor already supports running Yarn container process as one designated local user in non-secure mode. *We can add new configurations to Yarn, such that with LinuxContainerExecutor we can pre-create a pool of local users on each Yarn node manager. At runtime, Yarn node manager allocates a local user to run the container process, for the domain user that submits the application*. When all containers of that user are finished and all files belonging to that user are deleted, we can release the allocation and allow other users to use the same local user to run their Yarn containers. Please look at attached design doc for more details. was: Yarn Secure Container allows separation of different user's local files and container processes running on the same node manager. This depends on an out of band service such as SSSD/Winbind to sync all domain users to local machine that runs Yarn node manager.* Hadoop code only works with local users*. Winbind/SSSD user sync has lots of overhead, especially for large corporations. Also if running Yarn node manager inside Kubernetes cluster (meaning node managers running inside Docker container), it doesn't make sense for each Docker container to domain join with Active Directory and sync a whole copy of domain users to the Docker container. We need an optional light-weighted approach to enable Yarn Secure Container in secure mode, as an alternative to AD domain join and SSSD/Winbind based user-sync service. Today, class LinuxContainerExecutor already supports running Yarn container process as one designated local user in non-secure mode. *We can add new configurations to Yarn, such that with LinuxContainerExecutor we can pre-create a pool of local users on each Yarn node manager. At runtime, Yarn node manager allocates a local user to run the container process, for the domain user that submits the application*. When all containers of that user are finished and all files belonging to that user are deleted, we can release the allocation and allow other users to use the same local user to run their Yarn containers. Please look at attached design doc for more details. > Allow using a pool of local users to run Yarn Secure Container in secure mode > - > > Key: YARN-9834 > URL: https://issues.apache.org/jira/browse/YARN-9834 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 3.1.2 >Reporter: shanyu zhao >Assignee: shanyu zhao >Priority: Major > > Yarn Secure Container allows separation of different user's local files and > container processes running on the same node manager. This depends on an out > of band service such as SSSD/Winbind to sync all domain users to local > machine that runs Yarn node manager. * Hadoop code only works with local > users*. > Winbind/SSSD user sync has lots of overhead, especially for large > corporations. Also if running Yarn node manager inside Kubernetes cluster > (meaning node managers running inside Docker container), it doesn't make > sense for each Docker container to domain join with Active Directory and sync > a whole copy of domain users to the Docker container. > We need an optional light-weighted approach to enable Yarn Secure Container > in secure mode, as an alternative to AD domain join and SSSD/Winbind based > user-sync service. > Today, class LinuxContainerExecutor already supports running Yarn container > process as one designated local user in non-secure mode. > *We can add new configurations to Yarn, such that with LinuxContainerExecutor > we can pre-create a pool of local users on each Yarn node manager. At > runtime, Yarn node manager allocates a local user to run the container > process, for the domain user that
[jira] [Updated] (YARN-9834) Allow using a pool of local users to run Yarn Secure Container in secure mode
[ https://issues.apache.org/jira/browse/YARN-9834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shanyu zhao updated YARN-9834: -- Description: Yarn Secure Container allows separation of different user's local files and container processes running on the same node manager. This depends on an out of band service such as SSSD/Winbind to sync all domain users to local machine that runs Yarn node manager. *Hadoop code only works with local users*. Winbind/SSSD user sync has lots of overhead, especially for large corporations. Also if running Yarn node manager inside Kubernetes cluster (meaning node managers running inside Docker container), it doesn't make sense for each Docker container to domain join with Active Directory and sync a whole copy of domain users to the Docker container. We need an optional light-weighted approach to enable Yarn Secure Container in secure mode, as an alternative to AD domain join and SSSD/Winbind based user-sync service. Today, class LinuxContainerExecutor already supports running Yarn container process as one designated local user in non-secure mode. *We can add new configurations to Yarn, such that with LinuxContainerExecutor we can pre-create a pool of local users on each Yarn node manager. At runtime, Yarn node manager allocates a local user to run the container process, for the domain user that submits the application*. When all containers of that user are finished and all files belonging to that user are deleted, we can release the allocation and allow other users to use the same local user to run their Yarn containers. Please look at attached design doc for more details. was: Yarn Secure Container allows separation of different user's local files and container processes running on the same node manager. This depends on an out of band service such as SSSD/Winbind to sync all domain users to local machine that runs Yarn node manager. * Hadoop code only works with local users*. Winbind/SSSD user sync has lots of overhead, especially for large corporations. Also if running Yarn node manager inside Kubernetes cluster (meaning node managers running inside Docker container), it doesn't make sense for each Docker container to domain join with Active Directory and sync a whole copy of domain users to the Docker container. We need an optional light-weighted approach to enable Yarn Secure Container in secure mode, as an alternative to AD domain join and SSSD/Winbind based user-sync service. Today, class LinuxContainerExecutor already supports running Yarn container process as one designated local user in non-secure mode. *We can add new configurations to Yarn, such that with LinuxContainerExecutor we can pre-create a pool of local users on each Yarn node manager. At runtime, Yarn node manager allocates a local user to run the container process, for the domain user that submits the application*. When all containers of that user are finished and all files belonging to that user are deleted, we can release the allocation and allow other users to use the same local user to run their Yarn containers. Please look at attached design doc for more details. > Allow using a pool of local users to run Yarn Secure Container in secure mode > - > > Key: YARN-9834 > URL: https://issues.apache.org/jira/browse/YARN-9834 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 3.1.2 >Reporter: shanyu zhao >Assignee: shanyu zhao >Priority: Major > > Yarn Secure Container allows separation of different user's local files and > container processes running on the same node manager. This depends on an out > of band service such as SSSD/Winbind to sync all domain users to local > machine that runs Yarn node manager. *Hadoop code only works with local > users*. > Winbind/SSSD user sync has lots of overhead, especially for large > corporations. Also if running Yarn node manager inside Kubernetes cluster > (meaning node managers running inside Docker container), it doesn't make > sense for each Docker container to domain join with Active Directory and sync > a whole copy of domain users to the Docker container. > We need an optional light-weighted approach to enable Yarn Secure Container > in secure mode, as an alternative to AD domain join and SSSD/Winbind based > user-sync service. > Today, class LinuxContainerExecutor already supports running Yarn container > process as one designated local user in non-secure mode. > *We can add new configurations to Yarn, such that with LinuxContainerExecutor > we can pre-create a pool of local users on each Yarn node manager. At > runtime, Yarn node manager allocates a local user to run the container > process, for the domain user that
[jira] [Updated] (YARN-9834) Allow using a pool of local users to run Yarn Secure Container in secure mode
[
https://issues.apache.org/jira/browse/YARN-9834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
shanyu zhao updated YARN-9834:
--
Description:
Yarn Secure Container allows separation of different user's local files and
container processes running on the same node manager. This depends on an out of
band service such as SSSD/Winbind to sync all domain users to local machine
that runs Yarn node manager.* Hadoop code only works with local users*.
Winbind/SSSD user sync has lots of overhead, especially for large corporations.
Also if running Yarn node manager inside Kubernetes cluster (meaning node
managers running inside Docker container), it doesn't make sense for each
Docker container to domain join with Active Directory and sync a whole copy of
domain users to the Docker container.
We need an optional light-weighted approach to enable Yarn Secure Container in
secure mode, as an alternative to AD domain join and SSSD/Winbind based
user-sync service.
Today, class LinuxContainerExecutor already supports running Yarn container
process as one designated local user in non-secure mode.
*We can add new configurations to Yarn, such that with LinuxContainerExecutor
we can pre-create a pool of local users on each Yarn node manager. At runtime,
Yarn node manager allocates a local user to run the container process, for the
domain user that submits the application*. When all containers of that user are
finished and all files belonging to that user are deleted, we can release the
allocation and allow other users to use the same local user to run their Yarn
containers.
Please look at attached design doc for more details.
was:
Yarn Secure Container in secure mode allows separation of different user's
local files and container processes running on the same node manager. This
depends on an out of band service such as SSSD/Winbind to sync all domain users
to local machine.
Winbind user sync has lots of overhead, especially for large corporations. Also
if running Yarn inside Kubernetes cluster (meaning node managers running inside
Docker container), it doesn't make sense for each container (running node
manager inside) to domain join with Active Directory and sync a whole copy of
domain users.
We need a light-weighted config to enable Yarn Secure Container as an
alternative to AD domain join, SSSD/Winbind.
We should allow a new configuration to Yarn, such that we can pre-create a pool
of users on each machine/Docker container. And at runtime, Yarn allocates a
local user to the domain user that submits the application. When all containers
of that user are finished and all files belonging to that user are deleted, we
can release the allocation and allow other users to use the same local user to
run their Yarn containers.
h2. Design
We propose to extend LinuxContainerExecutor to support pool-user in secure
mode. LinuxContainerExecutor is the main class and accompanying classes and the
container-executor binary to implement the Yarn Secure Container feature.
There are existing configurations like
"yarn.nodemanager.linux-container-executor.nonsecure-mode.xxx", we propose to
add these new configurations for secure mode:
{code:java}
yarn.nodemanager.linux-container-executor.secure-mode.use-pool-user, defaults
to false
yarn.nodemanager.linux-container-executor.secure-mode.pool-user-prefix,
defaults to "user"
yarn.nodemanager.linux-container-executor.secure-mode.pool-user-count, defaults
to -1, meaning the value of yarn.nodemanager.resource.cpu-vcores are used.
{code}
By default this feature is turned off. If we enable it, with pool-user-prefix
set to "user", then we expect there are pre-created local users user0 - usern,
where the total number of local users equals to pool-user-count. The default
pool-user-count equals to cpu-vcores, because in theory that's the maximum
number of concurrent containers running on a given yarn node manager.
We use an in-memory allocator to keep the domain user to local user mapping.
Now when to add the mapping and when to remove it?
In node manager, ApplicationImpl implements the state machine for a Yarn app
life cycle, only if the app has at least 1 container running on that node
manager. We can hook up the code to add the mapping during application
initialization.
For removing the mapping, we need to wait for 3 things:
1) All applications of the same user is completed;
2) All log handling of the applications (log aggregation or non-aggregated
handling) is done;
3) All pending FileDeletionTask that use the user's identity is finished.
Note that all operation to these reference counting should be synchronized
operation.
If all of our local users in the pool are allocated, we'll return
"nonexistuser" as runas user, this will cause the container to fail to execute
and Yarn will relaunch it in other nodes.
What about node manager restarts? During
[jira] [Updated] (YARN-9834) Allow using a pool of local users to run Yarn Secure Container in secure mode
[
https://issues.apache.org/jira/browse/YARN-9834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
shanyu zhao updated YARN-9834:
--
Description:
Yarn Secure Container in secure mode allows separation of different user's
local files and container processes running on the same node manager. This
depends on an out of band service such as SSSD/Winbind to sync all domain users
to local machine.
Winbind user sync has lots of overhead, especially for large corporations. Also
if running Yarn inside Kubernetes cluster (meaning node managers running inside
Docker container), it doesn't make sense for each container (running node
manager inside) to domain join with Active Directory and sync a whole copy of
domain users.
We need a light-weighted config to enable Yarn Secure Container as an
alternative to AD domain join, SSSD/Winbind.
We should allow a new configuration to Yarn, such that we can pre-create a pool
of users on each machine/Docker container. And at runtime, Yarn allocates a
local user to the domain user that submits the application. When all containers
of that user are finished and all files belonging to that user are deleted, we
can release the allocation and allow other users to use the same local user to
run their Yarn containers.
h2. Design
We propose to extend LinuxContainerExecutor to support pool-user in secure
mode. LinuxContainerExecutor is the main class and accompanying classes and the
container-executor binary to implement the Yarn Secure Container feature.
There are existing configurations like
"yarn.nodemanager.linux-container-executor.nonsecure-mode.xxx", we propose to
add these new configurations for secure mode:
{code:java}
yarn.nodemanager.linux-container-executor.secure-mode.use-pool-user, defaults
to false
yarn.nodemanager.linux-container-executor.secure-mode.pool-user-prefix,
defaults to "user"
yarn.nodemanager.linux-container-executor.secure-mode.pool-user-count, defaults
to -1, meaning the value of yarn.nodemanager.resource.cpu-vcores are used.
{code}
By default this feature is turned off. If we enable it, with pool-user-prefix
set to "user", then we expect there are pre-created local users user0 - usern,
where the total number of local users equals to pool-user-count. The default
pool-user-count equals to cpu-vcores, because in theory that's the maximum
number of concurrent containers running on a given yarn node manager.
We use an in-memory allocator to keep the domain user to local user mapping.
Now when to add the mapping and when to remove it?
In node manager, ApplicationImpl implements the state machine for a Yarn app
life cycle, only if the app has at least 1 container running on that node
manager. We can hook up the code to add the mapping during application
initialization.
For removing the mapping, we need to wait for 3 things:
1) All applications of the same user is completed;
2) All log handling of the applications (log aggregation or non-aggregated
handling) is done;
3) All pending FileDeletionTask that use the user's identity is finished.
Note that all operation to these reference counting should be synchronized
operation.
If all of our local users in the pool are allocated, we'll return
"nonexistuser" as runas user, this will cause the container to fail to execute
and Yarn will relaunch it in other nodes.
What about node manager restarts? During ResourceLocalizationService init, it
renames the root folders used by the node manager and schedules
FileDeletionTask to delete the content of these files as the owner (local pool
users) of these local files. To prevent the newly launched Yarn containers to
be able to peek into the yet-to-be-deleted old application folders right after
node manager restart, we can allocate these local pool users to the requested
user in FileDeletionTask, which results in a call to incrementFileOpCount().
Therefore we allow local pool user allocation during the call to
incrementFileOpCount(appUser) and if appUser matches with a local pool user, we
allocate that user to the same named appUser, preventing new containers to
reuse the same local pool user, until all the FileDeletionTask belonging to
that user are done.
h2. Limitations
1) This feature does not support PRIVATE visibility type of resource
allocation. Because PRIVATE type of resources are potentially cached in the
node manager for a very long time, supporting it will be a security problem
that a user might be able to peek into previous user's PRIVATE resources. We
can modify code to treat all PRIVATE type of resource as APPLICATION type.
2) It is recommended to enable DominantResourceCalculator so that no more than
"cpu-vcores" number of concurrent containers running on a node manager:
{code:java}
yarn.scheduler.capacity.resource-calculator
= org.apache.hadoop.yarn.util.resource.DominantResourceCalculator {code}
3) Currently this feature does not work with
[jira] [Updated] (YARN-9834) Allow using a pool of local users to run Yarn Secure Container in secure mode
[
https://issues.apache.org/jira/browse/YARN-9834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
shanyu zhao updated YARN-9834:
--
Description:
Yarn Secure Container in secure mode allows separation of different user's
local files and container processes running on the same node manager. This
depends on an out of band service such as SSSD/Winbind to sync all domain users
to local machine.
Winbind user sync has lots of overhead, especially for large corporations. Also
if running Yarn inside Kubernetes cluster (meaning node managers running inside
Docker container), it doesn't make sense for each container to domain join with
Active Directory and sync a whole copy of domain users.
We should allow a new configuration to Yarn, such that we can pre-create a pool
of users on each machine/Docker container. And at runtime, Yarn allocates a
local user to the domain user that submits the application. When all containers
of that user are finished and all files belonging to that user are deleted, we
can release the allocation and allow other users to use the same local user to
run their Yarn containers.
h2. Design
We propose to add these new configurations:
{code:java}
yarn.nodemanager.linux-container-executor.secure-mode.use-pool-user, defaults
to false
yarn.nodemanager.linux-container-executor.secure-mode.pool-user-prefix,
defaults to "user"
yarn.nodemanager.linux-container-executor.secure-mode.pool-user-count, defaults
to -1, meaning the value of yarn.nodemanager.resource.cpu-vcores are used.
{code}
By default this feature is turned off. If we enable it, with pool-user-prefix
set to "user", then we expect there are pre-created local users user0 - usern,
where the total number of local users equals to pool-user-count. The default
pool-user-count equals to cpu-vcores, because in theory that's the maximum
number of concurrent containers running on a given yarn node manager.
We use an in-memory allocator to keep the domain user to local user mapping.
Now when to add the mapping and when to remove it?
In node manager, ApplicationImpl implements the state machine for a Yarn app
life cycle, only if the app has at least 1 container running on that node
manager. We can hook up the code to add the mapping during application
initialization.
For removing the mapping, we need to wait for 3 things:
1) All applications of the same user is completed;
2) All log handling of the applications (log aggregation or non-aggregated
handling) is done;
3) All pending FileDeletionTask that use the user's identity is finished.
Note that all operation to these reference counting should be synchronized
operation.
If all of our local users in the pool are allocated, we'll return
"nonexistuser" as runas user, this will cause the container to fail to execute
and Yarn will relaunch it in other nodes.
What about node manager restarts? During ResourceLocalizationService init, it
renames the root folders used by the node manager and schedules
FileDeletionTask to delete the content of these files as the owner (local pool
users) of these local files. To prevent the newly launched Yarn containers to
be able to peek into the yet-to-be-deleted old application folders right after
node manager restart, we can allocate these local pool users to the requested
user in FileDeletionTask, which results in a call to incrementFileOpCount().
Therefore we allow local pool user allocation during the call to
incrementFileOpCount(appUser) and if appUser matches with a local pool user, we
allocate that user to the same named appUser, preventing new containers to
reuse the same local pool user, until all the FileDeletionTask belonging to
that user are done.
h2. Limitations
1) This feature does not support PRIVATE visibility type of resource
allocation. Because PRIVATE type of resources are potentially cached in the
node manager for a very long time, supporting it will be a security problem
that a user might be able to peek into previous user's PRIVATE resources. We
can modify code to treat all PRIVATE type of resource as APPLICATION type.
2) It is recommended to enable DominantResourceCalculator so that no more than
"cpu-vcores" number of concurrent containers running on a node manager:
{code:java}
yarn.scheduler.capacity.resource-calculator
= org.apache.hadoop.yarn.util.resource.DominantResourceCalculator {code}
3) Currently this feature does not work with Yarn Node Manager recovery. This
is because the mappings are kept in memory, it cannot be recovered after node
manager restart.
was:
Yarn Secure Container in secure mode allows separation of different user's
local files and container processes running on the same node manager. This
depends on an out of band service such as SSSD/Winbind to sync all domain users
to local machine.
Winbind user sync has lots of overhead, especially for large corporations. Also
if running Yarn inside
[jira] [Commented] (YARN-9834) Allow using a pool of local users to run Yarn Secure Container in secure mode
[
https://issues.apache.org/jira/browse/YARN-9834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932748#comment-16932748
]
shanyu zhao commented on YARN-9834:
---
[~eyang]
{quote}User's home directory may be used by multiple parties. Some application
may install ssh key or user preference into user home directory. User home
directory becomes a public space in this design. This breaks user home
directory privacy for existing applications.{quote}
There is no home directory configured for "domain users" and Yarn container are
not supposed to use home directory. The working directory of a container is
managed by Yarn node manager and will be deleted when the container is
finished. Any additional credentials (other than hadoop delegation token) is
also localized to the container working directory and will be cleaned up.
{quote}Application can not rely on any group membership lookup because the user
does not have group membership in the container. To access external posix like
file system in application, group authorization needs to be custom code like
going through HDFS client rather than standard file interfaces. This is
incompatible with most file systems options today.{quote}
Again for HDFS access hadoop delegation token is used. For any external
resource access such as JDBC, the credentials (e.g. keytab file) should be
localized to the container as a file resource. The only thing I can think of is
this scenario: we want to allow sharing some content in local file system of
the node manager and configure a certain group of user can read it. I don't
believe this is an important usage scenario, and we can always put that shared
content to HDFS and rely on resource localization and manage the ACLs on HDFS.
{quote}User can not access raw devices like GPU. Many of the sudo or kernel
capabilities are only granted permissively to users with sudo rights or proper
group permission. Node manager does not have the logic to grant user sudo
rights. If continue on this path, this means node manager becomes root to
delegate rights of the user. This makes node manager more powerful and more
dangerous.{quote}
GPU scheduling and isolation rely on CGroup. The "domain user" cannot be
sudoers otherwise they can peek into each other's working directory.
I'm trying to provide an alternative to SSSD (the domain user sync) to provide
a light-weight secure mode for LinuxContainerExecutor to provide the benefit of
Yarn Secure Containers (user isolation inside node manager). I believe it works
for most of the scenarios. If there are certain scenario this cannot support,
we can call it out. It's not a bad idea to offer an additional choice. But I
don't want to have any security holes in any situation.
{quote}
> Allow using a pool of local users to run Yarn Secure Container in secure mode
> -
>
> Key: YARN-9834
> URL: https://issues.apache.org/jira/browse/YARN-9834
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 3.1.2
>Reporter: shanyu zhao
>Assignee: shanyu zhao
>Priority: Major
>
> Yarn Secure Container in secure mode allows separation of different user's
> local files and container processes running on the same node manager. This
> depends on an out of band service such as SSSD/Winbind to sync all domain
> users to local machine.
> Winbind user sync has lots of overhead, especially for large corporations.
> Also if running Yarn inside Kubernetes cluster (meaning node managers running
> inside Docker container), it doesn't make sense for each container to domain
> join with Active Directory and sync a whole copy of domain users.
> We should allow a new configuration to Yarn, such that we can pre-create a
> pool of users on each machine/Docker container. And at runtime, Yarn
> allocates a local user to the domain user that submits the application. When
> all containers of that user are finished and all files belonging to that user
> are deleted, we can release the allocation and allow other users to use the
> same local user to run their Yarn containers.
> h2. Design
> We propose to add these new configurations:
> {code:java}
> yarn.nodemanager.linux-container-executor.secure-mode.use-local-user,
> defaults to false
> yarn.nodemanager.linux-container-executor.secure-mode.local-user-prefix,
> defaults to "user"{code}
> By default this feature is turned off. If we enable it, with
> local-user-prefix set to "user", then we expect there are pre-created local
> users user0 - usern, where the total number of local users equals to:
> {code:java}
> yarn.nodemanager.resource.cpu-vcores {code}
> We can use an in-memory allocator to keep the domain user to local user
> mapping.
> Now when to add the mapping and when
[jira] [Updated] (YARN-9834) Allow using a pool of local users to run Yarn Secure Container in secure mode
[
https://issues.apache.org/jira/browse/YARN-9834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
shanyu zhao updated YARN-9834:
--
Description:
Yarn Secure Container in secure mode allows separation of different user's
local files and container processes running on the same node manager. This
depends on an out of band service such as SSSD/Winbind to sync all domain users
to local machine.
Winbind user sync has lots of overhead, especially for large corporations. Also
if running Yarn inside Kubernetes cluster (meaning node managers running inside
Docker container), it doesn't make sense for each container to domain join with
Active Directory and sync a whole copy of domain users.
We should allow a new configuration to Yarn, such that we can pre-create a pool
of users on each machine/Docker container. And at runtime, Yarn allocates a
local user to the domain user that submits the application. When all containers
of that user are finished and all files belonging to that user are deleted, we
can release the allocation and allow other users to use the same local user to
run their Yarn containers.
h2. Design
We propose to add these new configurations:
{code:java}
yarn.nodemanager.linux-container-executor.secure-mode.use-local-user, defaults
to false
yarn.nodemanager.linux-container-executor.secure-mode.local-user-prefix,
defaults to "user"{code}
By default this feature is turned off. If we enable it, with local-user-prefix
set to "user", then we expect there are pre-created local users user0 - usern,
where the total number of local users equals to:
{code:java}
yarn.nodemanager.resource.cpu-vcores {code}
We can use an in-memory allocator to keep the domain user to local user
mapping.
Now when to add the mapping and when to remove it?
In node manager, ApplicationImpl implements the state machine for a Yarn app
life cycle, only if the app has at least 1 container running on that node
manager. We can hook up the code to add the mapping during application
initialization.
For removing the mapping, we need to wait for 3 things:
1) All applications of the same user is completed;
2) All log handling of the applications (log aggregation or non-aggregated
handling) is done;
3) All pending FileDeletionTask that use the user's identity is finished.
Note that all operation to these reference counting should be synchronized
operation.
If all of our local users in the pool are allocated, we'll return
"nonexistuser" as runas user, this will cause the container to fail to execute
and Yarn will relaunch it in other nodes.
What about node manager restarts? During ResourceLocalizationService init, it
renames the root folders used by the node manager and schedules
FileDeletionTask to delete the content of these files. To prevent the newly
launched Yarn containers to be able to peek into the yet-to-be-deleted old
application folders right after node manager restart, we can chmod the root
folders to 700 right after rename.
h2. Limitations
1) This feature does not support PRIVATE visibility type of resource
allocation. Because PRIVATE type of resources are potentially cached in the
node manager for a very long time, supporting it will be a security problem
that a user might be able to peek into previous user's PRIVATE resources. We
can modify code to treat all PRIVATE type of resource as APPLICATION type.
2) It is recommended to enable DominantResourceCalculator so that no more than
"cpu-vcores" number of concurrent containers running on a node manager:
{code:java}
yarn.scheduler.capacity.resource-calculator
= org.apache.hadoop.yarn.util.resource.DominantResourceCalculator {code}
3) Currently this feature does not work with Yarn Node Manager recovery. This
is because the mappings are kept in memory, it cannot be recovered after node
manager restart.
was:
Yarn Secure Container in secure mode allows separation of different user's
local files and container processes running on the same node manager. This
depends on an out of band service such as SSSD/Winbind to sync all domain users
to local machine.
Winbind user sync has lots of overhead, especially for large corporations. Also
if running Yarn inside Kubernetes cluster (meaning node managers running inside
Docker container), it doesn't make sense for each container to domain join with
Active Directory and sync a whole copy of domain users.
We should allow a new configuration to Yarn, such that we can pre-create a pool
of users on each machine/Docker container. And at runtime, Yarn allocates a
local user to the domain user that submits the application. When all containers
of that user are finished and all files belonging to that user are deleted, we
can release the allocation and allow other users to use the same local user to
run their Yarn containers.
h2. Design
We propose to add these new configurations:
{code:java}
[jira] [Commented] (YARN-9834) Allow using a pool of local users to run Yarn Secure Container in secure mode
[
https://issues.apache.org/jira/browse/YARN-9834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932085#comment-16932085
]
shanyu zhao commented on YARN-9834:
---
Thanks for the review [~eyang]. If you see any security hole in this design,
please let me know.
> Allow using a pool of local users to run Yarn Secure Container in secure mode
> -
>
> Key: YARN-9834
> URL: https://issues.apache.org/jira/browse/YARN-9834
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 3.1.2
>Reporter: shanyu zhao
>Assignee: shanyu zhao
>Priority: Major
>
> Yarn Secure Container in secure mode allows separation of different user's
> local files and container processes running on the same node manager. This
> depends on an out of band service such as SSSD/Winbind to sync all domain
> users to local machine.
> Winbind user sync has lots of overhead, especially for large corporations.
> Also if running Yarn inside Kubernetes cluster (meaning node managers running
> inside Docker container), it doesn't make sense for each container to domain
> join with Active Directory and sync a whole copy of domain users.
> We should allow a new configuration to Yarn, such that we can pre-create a
> pool of users on each machine/Docker container. And at runtime, Yarn
> allocates a local user to the domain user that submits the application. When
> all containers of that user are finished and all files belonging to that user
> are deleted, we can release the allocation and allow other users to use the
> same local user to run their Yarn containers.
> h2. Design
> We propose to add these new configurations:
> {code:java}
> yarn.nodemanager.linux-container-executor.secure-mode.use-local-user,
> defaults to false
> yarn.nodemanager.linux-container-executor.secure-mode.local-user-prefix,
> defaults to "user"{code}
> By default this feature is turned off. If we enable it, with
> local-user-prefix set to "user", then we expect there are pre-created local
> users user0 - usern, where the total number of local users equals to:
> {code:java}
> yarn.nodemanager.resource.cpu-vcores {code}
> We can use an in-memory allocator to keep the domain user to local user
> mapping.
> Now when to add the mapping and when to remove it?
> In node manager, ApplicationImpl implements the state machine for a Yarn app
> life cycle, only if the app has at least 1 container running on that node
> manager. We can hook up the code to add the mapping during application
> initialization.
> For removing the mapping, we need to wait for 3 things:
> 1) All applications of the same user is completed;
> 2) All log handling of the applications (log aggregation or non-aggregated
> handling) is done;
> 3) All pending FileDeletionTask that use the user's identity is finished.
> Note that all operation to these reference counting should be synchronized
> operation.
> If all of our local users in the pool are allocated, we'll return
> "nonexistuser" as runas user, this will cause the container to fail to
> execute and Yarn will relaunch it in other nodes.
> h2. Limitations
> 1) This feature does not support PRIVATE visibility type of resource
> allocation. Because PRIVATE type of resources are potentially cached in the
> node manager for a very long time, supporting it will be a security problem
> that a user might be able to peek into previous user's PRIVATE resources. We
> can modify code to treat all PRIVATE type of resource as APPLICATION type.
> 2) It is recommended to enable DominantResourceCalculator so that no more
> than "cpu-vcores" number of concurrent containers running on a node manager:
> {code:java}
> yarn.scheduler.capacity.resource-calculator
> = org.apache.hadoop.yarn.util.resource.DominantResourceCalculator {code}
> 3) Currently this feature does not work with Yarn Node Manager recovery. This
> is because the mappings are kept in memory, it cannot be recovered after node
> manager restart.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9834) Allow using a pool of local users to run Yarn Secure Container in secure mode
[
https://issues.apache.org/jira/browse/YARN-9834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932009#comment-16932009
]
shanyu zhao edited comment on YARN-9834 at 9/18/19 5:45 AM:
{quote}Given the reasoning of node manager running in Docker container, and
node manager isn't really authenticating with Kerberos for the host credential.
The proposal drops the basic security of trusted hosts. This means replay
attack is possible. Wouldn't it be easier to run the cluster in simple security
instead of breaking secure cluster to work like simple security cluster?{quote}
Node manager actually authenticate with Resource manager via Kerberos, we
configured a keytab file for node manager to use. All the local pool users do
not have permission to access this keytab file. It is still a secure Hadoop
cluster. The only reason for Winbind/SSSD to sync domain user to local user, is
for LinuxCotnainerExecutor to start the Yarn container process as the synced
"domain user" name, without any implicit permission associated with that domain
user. What we did here is to skip the domain user sync part, and dynamically
allocate local users to Yarn containers to achieve files and processes
isolation.
{quote}What happen if the node manager restarted? Will this cause Joe's
delegation token to leak?{quote}
You raised a good point here. Node manager process restart will cause
re-initialization of ResourceLocalizationService, which will rename the local
directories for the node manager (including all existing application folders as
sub folders), then schedule delete task. Joe's delegation token was among the
files to be deleted. However, this is async process, so in theory there is a
short window that if the scheduled FileDeletionTask has not been executed yet.
A simple fix is in addition to the rename, change permission to 700 on these
folders before scheduling FileDeletionTask:
{code}
renameLocalDir(lfs, localDir, ContainerLocalizer.USERCACHE,
currentTimeStamp);
renameLocalDir(lfs, localDir, ContainerLocalizer.FILECACHE,
currentTimeStamp);
renameLocalDir(lfs, localDir, ResourceLocalizationService.NM_PRIVATE_DIR,
currentTimeStamp);
{code}
was (Author: shanyu):
{quote}Given the reasoning of node manager running in Docker container, and
node manager isn't really authenticating with Kerberos for the host credential.
The proposal drops the basic security of trusted hosts. This means replay
attack is possible. Wouldn't it be easier to run the cluster in simple security
instead of breaking secure cluster to work like simple security cluster?{quote}
Node manager actually authenticate with Resource manager via Kerberos, we
configured a keytab file for node manager to use. All the local pool users do
not have permission to access this keytab file. It is still a secure Hadoop
cluster. The only reason for Winbind/SSSD to sync domain user to local user, is
for LinuxCotnainerExecutor to start the Yarn container process as the synced
"domain user" name, without any implicit permission associated with that domain
user. What we did here is to skip the domain user sync part, and dynamically
allocate local users to Yarn containers to achieve files and processes
isolation.
{quote}What happen if the node manager restarted? Will this cause Joe's
delegation token to leak?{quote}
You raised a good point here. Node manager process restart will cause
re-initialization of ResourceLocalizationService, which will rename the local
directories for the node manager (including all existing application folders as
sub folders), then schedule delete task. Joe's delegation token was among the
files to be deleted. However, this is async process, so in theory there is a
short window that if the scheduled FileDeletionTask has not been executed yet.
A simple fix is in additional to the rename, change permission to 700 on these
folders:
{code}
renameLocalDir(lfs, localDir, ContainerLocalizer.USERCACHE,
currentTimeStamp);
renameLocalDir(lfs, localDir, ContainerLocalizer.FILECACHE,
currentTimeStamp);
renameLocalDir(lfs, localDir, ResourceLocalizationService.NM_PRIVATE_DIR,
currentTimeStamp);
{code}
> Allow using a pool of local users to run Yarn Secure Container in secure mode
> -
>
> Key: YARN-9834
> URL: https://issues.apache.org/jira/browse/YARN-9834
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 3.1.2
>Reporter: shanyu zhao
>Assignee: shanyu zhao
>Priority: Major
>
> Yarn Secure Container in secure mode allows separation of different user's
> local files and container processes running on the same node manager. This
> depends on an out of band service such
[jira] [Comment Edited] (YARN-9834) Allow using a pool of local users to run Yarn Secure Container in secure mode
[
https://issues.apache.org/jira/browse/YARN-9834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932009#comment-16932009
]
shanyu zhao edited comment on YARN-9834 at 9/18/19 3:23 AM:
{quote}Given the reasoning of node manager running in Docker container, and
node manager isn't really authenticating with Kerberos for the host credential.
The proposal drops the basic security of trusted hosts. This means replay
attack is possible. Wouldn't it be easier to run the cluster in simple security
instead of breaking secure cluster to work like simple security cluster?{quote}
Node manager actually authenticate with Resource manager via Kerberos, we
configured a keytab file for node manager to use. All the local pool users do
not have permission to access this keytab file. It is still a secure Hadoop
cluster. The only reason for Winbind/SSSD to sync domain user to local user, is
for LinuxCotnainerExecutor to start the Yarn container process as the synced
"domain user" name, without any implicit permission associated with that domain
user. What we did here is to skip the domain user sync part, and dynamically
allocate local users to Yarn containers to achieve files and processes
isolation.
{quote}What happen if the node manager restarted? Will this cause Joe's
delegation token to leak?{quote}
You raised a good point here. Node manager process restart will cause
re-initialization of ResourceLocalizationService, which will rename the local
directories for the node manager (including all existing application folders as
sub folders), then schedule delete task. Joe's delegation token was among the
files to be deleted. However, this is async process, so in theory there is a
short window that if the scheduled FileDeletionTask has not been executed yet.
A simple fix is in additional to the rename, change permission to 700 on these
folders:
{code}
renameLocalDir(lfs, localDir, ContainerLocalizer.USERCACHE,
currentTimeStamp);
renameLocalDir(lfs, localDir, ContainerLocalizer.FILECACHE,
currentTimeStamp);
renameLocalDir(lfs, localDir, ResourceLocalizationService.NM_PRIVATE_DIR,
currentTimeStamp);
{code}
was (Author: shanyu):
{quote}Given the reasoning of node manager running in Docker container, and
node manager isn't really authenticating with Kerberos for the host credential.
The proposal drops the basic security of trusted hosts. This means replay
attack is possible. Wouldn't it be easier to run the cluster in simple security
instead of breaking secure cluster to work like simple security cluster?{quote}
Node manager actually authenticate with Resource manager via Kerberos, we
configured a keytab file for node manager to use. All the local pool users do
not have permission to access this keytab file. It is still a secure Hadoop
cluster. The only reason for Winbind/SSSD to sync domain user to local user, is
for LinuxCotnainerExecutor to start the Yarn container process as the synced
"domain user" name, without any implicit permission associated with that domain
user. What we did here is to skip the domain user sync part, and dynamically
allocate local users to Yarn containers to achieve files and processes
isolation.
{quote}What happen if the node manager restarted? Will this cause Joe's
delegation token to leak?{quote}
You raised a good point here. Node manager process restart will cause
re-initialization of ResourceLocalizationService, which will rename the local
directories for the node manager (including all existing application folders as
sub folders), then schedule delete task. Joe's delegation token was among the
files to be deleted. However, this is async process, so in theory there is a
short window that if the scheduled FileDeletionTask has not been executed yet.
A simple fix is in additional to the rename, change permission to 700 on these
folders:
{code}
renameLocalDir(lfs, localDir, ContainerLocalizer.USERCACHE,
currentTimeStamp);
renameLocalDir(lfs, localDir, ContainerLocalizer.FILECACHE,
currentTimeStamp);
renameLocalDir(lfs, localDir, ResourceLocalizationService.NM_PRIVATE_DIR,
currentTimeStamp);
{code}
> Allow using a pool of local users to run Yarn Secure Container in secure mode
> -
>
> Key: YARN-9834
> URL: https://issues.apache.org/jira/browse/YARN-9834
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 3.1.2
>Reporter: shanyu zhao
>Assignee: shanyu zhao
>Priority: Major
>
> Yarn Secure Container in secure mode allows separation of different user's
> local files and container processes running on the same node manager. This
> depends on an out of band service such as SSSD/Winbind to sync all domain
>
[jira] [Commented] (YARN-9834) Allow using a pool of local users to run Yarn Secure Container in secure mode
[
https://issues.apache.org/jira/browse/YARN-9834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932009#comment-16932009
]
shanyu zhao commented on YARN-9834:
---
{quote}Given the reasoning of node manager running in Docker container, and
node manager isn't really authenticating with Kerberos for the host credential.
The proposal drops the basic security of trusted hosts. This means replay
attack is possible. Wouldn't it be easier to run the cluster in simple security
instead of breaking secure cluster to work like simple security cluster?{quote}
Node manager actually authenticate with Resource manager via Kerberos, we
configured a keytab file for node manager to use. All the local pool users do
not have permission to access this keytab file. It is still a secure Hadoop
cluster. The only reason for Winbind/SSSD to sync domain user to local user, is
for LinuxCotnainerExecutor to start the Yarn container process as the synced
"domain user" name, without any implicit permission associated with that domain
user. What we did here is to skip the domain user sync part, and dynamically
allocate local users to Yarn containers to achieve files and processes
isolation.
{quote}What happen if the node manager restarted? Will this cause Joe's
delegation token to leak?{quote}
You raised a good point here. Node manager process restart will cause
re-initialization of ResourceLocalizationService, which will rename the local
directories for the node manager (including all existing application folders as
sub folders), then schedule delete task. Joe's delegation token was among the
files to be deleted. However, this is async process, so in theory there is a
short window that if the scheduled FileDeletionTask has not been executed yet.
A simple fix is in additional to the rename, change permission to 700 on these
folders:
{code}
renameLocalDir(lfs, localDir, ContainerLocalizer.USERCACHE,
currentTimeStamp);
renameLocalDir(lfs, localDir, ContainerLocalizer.FILECACHE,
currentTimeStamp);
renameLocalDir(lfs, localDir, ResourceLocalizationService.NM_PRIVATE_DIR,
currentTimeStamp);
{code}
> Allow using a pool of local users to run Yarn Secure Container in secure mode
> -
>
> Key: YARN-9834
> URL: https://issues.apache.org/jira/browse/YARN-9834
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 3.1.2
>Reporter: shanyu zhao
>Assignee: shanyu zhao
>Priority: Major
>
> Yarn Secure Container in secure mode allows separation of different user's
> local files and container processes running on the same node manager. This
> depends on an out of band service such as SSSD/Winbind to sync all domain
> users to local machine.
> Winbind user sync has lots of overhead, especially for large corporations.
> Also if running Yarn inside Kubernetes cluster (meaning node managers running
> inside Docker container), it doesn't make sense for each container to domain
> join with Active Directory and sync a whole copy of domain users.
> We should allow a new configuration to Yarn, such that we can pre-create a
> pool of users on each machine/Docker container. And at runtime, Yarn
> allocates a local user to the domain user that submits the application. When
> all containers of that user are finished and all files belonging to that user
> are deleted, we can release the allocation and allow other users to use the
> same local user to run their Yarn containers.
> h2. Design
> We propose to add these new configurations:
> {code:java}
> yarn.nodemanager.linux-container-executor.secure-mode.use-local-user,
> defaults to false
> yarn.nodemanager.linux-container-executor.secure-mode.local-user-prefix,
> defaults to "user"{code}
> By default this feature is turned off. If we enable it, with
> local-user-prefix set to "user", then we expect there are pre-created local
> users user0 - usern, where the total number of local users equals to:
> {code:java}
> yarn.nodemanager.resource.cpu-vcores {code}
> We can use an in-memory allocator to keep the domain user to local user
> mapping.
> Now when to add the mapping and when to remove it?
> In node manager, ApplicationImpl implements the state machine for a Yarn app
> life cycle, only if the app has at least 1 container running on that node
> manager. We can hook up the code to add the mapping during application
> initialization.
> For removing the mapping, we need to wait for 3 things:
> 1) All applications of the same user is completed;
> 2) All log handling of the applications (log aggregation or non-aggregated
> handling) is done;
> 3) All pending FileDeletionTask that use the user's identity is finished.
> Note that all operation to these reference counting should be
[jira] [Commented] (YARN-9834) Allow using a pool of local users to run Yarn Secure Container in secure mode
[
https://issues.apache.org/jira/browse/YARN-9834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931953#comment-16931953
]
shanyu zhao commented on YARN-9834:
---
{quote}I think host joining AD is required to keep authorized servers to access
LDAP. Without AD authenticate the host, it would be wide open, no?{quote}
"host" in this scenario is the Docker container running node manager. That's
why I said it doesn't make sense for that container to domain join just to sync
domain user to local Docker container. With this PR there is no need to sync
domain users to "host", we precreate pool of users and dynamically assign to
Yarn container process to run.
{quote}I am still not clear on how the pool users would be used. It sounds like
user1 can be given to Joe, and Fred at different time. Wouldn't that give Fred
access to hdfs data stored by Joe as user1?{quote}
Yes local user "user1" can be given to Joe and Fred at different time. But
before we reassign "user1" from Joe to Fred, we make sure all local files as
the result or running the Yarn container (data and logs) are deleted. Note that
to access HDFS from the Yarn container process, it always need to use the
Hadoop delegation token, which is just a APPLICATION type of resource localized
to the working directory of that container. When the application is finished,
this file is deleted so when Freq's container is running it won't be able to
talk to HDFS with Joe's Hadoop delegation token.
{quote}How about the same user? Let's consider:
1. Joe run as user1, debug delay enabled, running:
application_1568407112772_0007
2. Fred run as user1, running: application_1568407112772_0008{quote}
In this case, Fred won't get allocation of "user1" because there are pending
FileDeletionTask, the reference count for FileOpCount of LocalUserInfo is not
zero therefore Joe still holds user1, until all the pending FileDeletionTask is
executed and the reference count reaches zero. Then "user1" is up for reuse.
> Allow using a pool of local users to run Yarn Secure Container in secure mode
> -
>
> Key: YARN-9834
> URL: https://issues.apache.org/jira/browse/YARN-9834
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 3.1.2
>Reporter: shanyu zhao
>Assignee: shanyu zhao
>Priority: Major
>
> Yarn Secure Container in secure mode allows separation of different user's
> local files and container processes running on the same node manager. This
> depends on an out of band service such as SSSD/Winbind to sync all domain
> users to local machine.
> Winbind user sync has lots of overhead, especially for large corporations.
> Also if running Yarn inside Kubernetes cluster (meaning node managers running
> inside Docker container), it doesn't make sense for each container to domain
> join with Active Directory and sync a whole copy of domain users.
> We should allow a new configuration to Yarn, such that we can pre-create a
> pool of users on each machine/Docker container. And at runtime, Yarn
> allocates a local user to the domain user that submits the application. When
> all containers of that user are finished and all files belonging to that user
> are deleted, we can release the allocation and allow other users to use the
> same local user to run their Yarn containers.
> h2. Design
> We propose to add these new configurations:
> {code:java}
> yarn.nodemanager.linux-container-executor.secure-mode.use-local-user,
> defaults to false
> yarn.nodemanager.linux-container-executor.secure-mode.local-user-prefix,
> defaults to "user"{code}
> By default this feature is turned off. If we enable it, with
> local-user-prefix set to "user", then we expect there are pre-created local
> users user0 - usern, where the total number of local users equals to:
> {code:java}
> yarn.nodemanager.resource.cpu-vcores {code}
> We can use an in-memory allocator to keep the domain user to local user
> mapping.
> Now when to add the mapping and when to remove it?
> In node manager, ApplicationImpl implements the state machine for a Yarn app
> life cycle, only if the app has at least 1 container running on that node
> manager. We can hook up the code to add the mapping during application
> initialization.
> For removing the mapping, we need to wait for 3 things:
> 1) All applications of the same user is completed;
> 2) All log handling of the applications (log aggregation or non-aggregated
> handling) is done;
> 3) All pending FileDeletionTask that use the user's identity is finished.
> Note that all operation to these reference counting should be synchronized
> operation.
> If all of our local users in the pool are allocated, we'll return
> "nonexistuser" as runas user, this will cause the
[jira] [Commented] (YARN-9834) Allow using a pool of local users to run Yarn Secure Container in secure mode
[
https://issues.apache.org/jira/browse/YARN-9834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931725#comment-16931725
]
shanyu zhao commented on YARN-9834:
---
[~eyang]
{quote}The mechanism doesn't need to be different between both solutions.{quote}
For SSSD to work it needs to have credentials to talk to LDAP, the recommended
way is to use "realm join". This means the node manager running inside Docker
container needs to have a "computer account" on AD.
{quote}Unless I am mistaken, the container user can access external storage
like HDFS. Pool user can be reused, and data stored on HDFS will become someone
else's file. What mechanism is there to protect future user from accessing past
user data on external storage?{quote}
This is not true. The Yarn container user does not have any special
permissions, it is using the Hadoop delegation token to talk to HDFS. The
Hadoop delegation token is an APPLICATION visibility resource "localized" to
the current Yarn container working directory. Once the container is finished
the Hadoop delegation token is deleted.
{quote}The new ContainerExecutor can extend LinuxContainerExecutor to provide
the one line change. Please keep this separated from LinuxContainerExecutor for
other people who are not comfortable with pool users idea.{quote}
Ok, I can create a LinuxPoolUserContainerExecutor then.
{quote}This change will cause private data to be accessible by multiple users
via path traversal. This does not sound good from security point of view. More
thoughts is required on resource permission.{quote}
It is not possible to access PRIVATE visibility resources using a different
pool user. Note that APPLICATION visibility is more restrictive then PRIVATE
visibility. After the change to treat PRIVATE visibility as APPLICATION
visibility, the private resources are located in:
/usercache//appcache//filecache
This folder is owned by pool local user, and its permission is 710. Trying to
access it from a different pool user results in "permission denied"
{code}
ls
/mnt/tmp/nm-local-dir/usercache/admin/appcache/application_1568407112772_0007/filecache/
-al
ls: cannot open directory
'/mnt/tmp/nm-local-dir/usercache/admin/appcache/application_1568407112772_0007/filecache/':
Permission denied
{code}
> Allow using a pool of local users to run Yarn Secure Container in secure mode
> -
>
> Key: YARN-9834
> URL: https://issues.apache.org/jira/browse/YARN-9834
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 3.1.2
>Reporter: shanyu zhao
>Assignee: shanyu zhao
>Priority: Major
>
> Yarn Secure Container in secure mode allows separation of different user's
> local files and container processes running on the same node manager. This
> depends on an out of band service such as SSSD/Winbind to sync all domain
> users to local machine.
> Winbind user sync has lots of overhead, especially for large corporations.
> Also if running Yarn inside Kubernetes cluster (meaning node managers running
> inside Docker container), it doesn't make sense for each container to domain
> join with Active Directory and sync a whole copy of domain users.
> We should allow a new configuration to Yarn, such that we can pre-create a
> pool of users on each machine/Docker container. And at runtime, Yarn
> allocates a local user to the domain user that submits the application. When
> all containers of that user are finished and all files belonging to that user
> are deleted, we can release the allocation and allow other users to use the
> same local user to run their Yarn containers.
> h2. Design
> We propose to add these new configurations:
> {code:java}
> yarn.nodemanager.linux-container-executor.secure-mode.use-local-user,
> defaults to false
> yarn.nodemanager.linux-container-executor.secure-mode.local-user-prefix,
> defaults to "user"{code}
> By default this feature is turned off. If we enable it, with
> local-user-prefix set to "user", then we expect there are pre-created local
> users user0 - usern, where the total number of local users equals to:
> {code:java}
> yarn.nodemanager.resource.cpu-vcores {code}
> We can use an in-memory allocator to keep the domain user to local user
> mapping.
> Now when to add the mapping and when to remove it?
> In node manager, ApplicationImpl implements the state machine for a Yarn app
> life cycle, only if the app has at least 1 container running on that node
> manager. We can hook up the code to add the mapping during application
> initialization.
> For removing the mapping, we need to wait for 3 things:
> 1) All applications of the same user is completed;
> 2) All log handling of the applications (log aggregation or non-aggregated
> handling) is done;
>
[jira] [Commented] (YARN-9834) Allow using a pool of local users to run Yarn Secure Container in secure mode
[
https://issues.apache.org/jira/browse/YARN-9834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930998#comment-16930998
]
shanyu zhao commented on YARN-9834:
---
[~eyang], You are talking about Docker container executor. What I meant is
running Yarn node manager itself inside Docker container (managed by
Kubernetes). The node manager (that runs SSSD) container needs to be domain
joined to be able to sync domain users.
This change will honor the fundamental security design of process owner and
file owner, that's why it uses reference counting for all these local files,
the local user is not released to reallocate until all FileDeletionTask are
done, so a Yarn container won't be able to access any other previous
container's local files/logs.
Currently LinuxContainerExecutor already supports non-secure mode and secure
mode, and this change is just modifying the getRunAsUser() behavior for secure
mode. I could inherit from LinuxContainerExecutor and override this single file
with one line change. But in the end, it is still LinuxContainerExecutor, not a
drastically different container executor.
As for ResourceLocalization modification, basically we are treating "PRIVATE"
type of resources as "APPLICATION" without banning "PRIVATE" resources. The end
result is that the PRIVATE resources are kept within application folder and
will be removed after the application is done.
> Allow using a pool of local users to run Yarn Secure Container in secure mode
> -
>
> Key: YARN-9834
> URL: https://issues.apache.org/jira/browse/YARN-9834
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 3.1.2
>Reporter: shanyu zhao
>Assignee: shanyu zhao
>Priority: Major
>
> Yarn Secure Container in secure mode allows separation of different user's
> local files and container processes running on the same node manager. This
> depends on an out of band service such as SSSD/Winbind to sync all domain
> users to local machine.
> Winbind user sync has lots of overhead, especially for large corporations.
> Also if running Yarn inside Kubernetes cluster (meaning node managers running
> inside Docker container), it doesn't make sense for each container to domain
> join with Active Directory and sync a whole copy of domain users.
> We should allow a new configuration to Yarn, such that we can pre-create a
> pool of users on each machine/Docker container. And at runtime, Yarn
> allocates a local user to the domain user that submits the application. When
> all containers of that user are finished and all files belonging to that user
> are deleted, we can release the allocation and allow other users to use the
> same local user to run their Yarn containers.
> h2. Design
> We propose to add these new configurations:
> {code:java}
> yarn.nodemanager.linux-container-executor.secure-mode.use-local-user,
> defaults to false
> yarn.nodemanager.linux-container-executor.secure-mode.local-user-prefix,
> defaults to "user"{code}
> By default this feature is turned off. If we enable it, with
> local-user-prefix set to "user", then we expect there are pre-created local
> users user0 - usern, where the total number of local users equals to:
> {code:java}
> yarn.nodemanager.resource.cpu-vcores {code}
> We can use an in-memory allocator to keep the domain user to local user
> mapping.
> Now when to add the mapping and when to remove it?
> In node manager, ApplicationImpl implements the state machine for a Yarn app
> life cycle, only if the app has at least 1 container running on that node
> manager. We can hook up the code to add the mapping during application
> initialization.
> For removing the mapping, we need to wait for 3 things:
> 1) All applications of the same user is completed;
> 2) All log handling of the applications (log aggregation or non-aggregated
> handling) is done;
> 3) All pending FileDeletionTask that use the user's identity is finished.
> Note that all operation to these reference counting should be synchronized
> operation.
> If all of our local users in the pool are allocated, we'll return
> "nonexistuser" as runas user, this will cause the container to fail to
> execute and Yarn will relaunch it in other nodes.
> h2. Limitations
> 1) This feature does not support PRIVATE visibility type of resource
> allocation. Because PRIVATE type of resources are potentially cached in the
> node manager for a very long time, supporting it will be a security problem
> that a user might be able to peek into previous user's PRIVATE resources. We
> can modify code to treat all PRIVATE type of resource as APPLICATION type.
> 2) It is recommended to enable DominantResourceCalculator so that no more
> than "cpu-vcores" number of
[jira] [Commented] (YARN-9834) Allow using a pool of local users to run Yarn Secure Container in secure mode
[
https://issues.apache.org/jira/browse/YARN-9834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930945#comment-16930945
]
shanyu zhao commented on YARN-9834:
---
Thanks [~eyang]! I forgot to mention that for Winbind/SSSD to work the
container needs to be domain joined to Active Directory, which doesn't work and
doesn't seem to be efficient.
I pushed a change to rename "use-local-user" to "use-pool-user", and
"local-user-prefix" to "pool-user-prefix".
> Allow using a pool of local users to run Yarn Secure Container in secure mode
> -
>
> Key: YARN-9834
> URL: https://issues.apache.org/jira/browse/YARN-9834
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 3.1.2
>Reporter: shanyu zhao
>Assignee: shanyu zhao
>Priority: Major
>
> Yarn Secure Container in secure mode allows separation of different user's
> local files and container processes running on the same node manager. This
> depends on an out of band service such as SSSD/Winbind to sync all domain
> users to local machine.
> Winbind user sync has lots of overhead, especially for large corporations.
> Also if running Yarn inside Kubernetes cluster (meaning node managers running
> inside Docker container), it doesn't make sense for each container to domain
> join with Active Directory and sync a whole copy of domain users.
> We should allow a new configuration to Yarn, such that we can pre-create a
> pool of users on each machine/Docker container. And at runtime, Yarn
> allocates a local user to the domain user that submits the application. When
> all containers of that user are finished and all files belonging to that user
> are deleted, we can release the allocation and allow other users to use the
> same local user to run their Yarn containers.
> h2. Design
> We propose to add these new configurations:
> {code:java}
> yarn.nodemanager.linux-container-executor.secure-mode.use-local-user,
> defaults to false
> yarn.nodemanager.linux-container-executor.secure-mode.local-user-prefix,
> defaults to "user"{code}
> By default this feature is turned off. If we enable it, with
> local-user-prefix set to "user", then we expect there are pre-created local
> users user0 - usern, where the total number of local users equals to:
> {code:java}
> yarn.nodemanager.resource.cpu-vcores {code}
> We can use an in-memory allocator to keep the domain user to local user
> mapping.
> Now when to add the mapping and when to remove it?
> In node manager, ApplicationImpl implements the state machine for a Yarn app
> life cycle, only if the app has at least 1 container running on that node
> manager. We can hook up the code to add the mapping during application
> initialization.
> For removing the mapping, we need to wait for 3 things:
> 1) All applications of the same user is completed;
> 2) All log handling of the applications (log aggregation or non-aggregated
> handling) is done;
> 3) All pending FileDeletionTask that use the user's identity is finished.
> Note that all operation to these reference counting should be synchronized
> operation.
> If all of our local users in the pool are allocated, we'll return
> "nonexistuser" as runas user, this will cause the container to fail to
> execute and Yarn will relaunch it in other nodes.
> h2. Limitations
> 1) This feature does not support PRIVATE visibility type of resource
> allocation. Because PRIVATE type of resources are potentially cached in the
> node manager for a very long time, supporting it will be a security problem
> that a user might be able to peek into previous user's PRIVATE resources. We
> can modify code to treat all PRIVATE type of resource as APPLICATION type.
> 2) It is recommended to enable DominantResourceCalculator so that no more
> than "cpu-vcores" number of concurrent containers running on a node manager:
> {code:java}
> yarn.scheduler.capacity.resource-calculator
> = org.apache.hadoop.yarn.util.resource.DominantResourceCalculator {code}
> 3) Currently this feature does not work with Yarn Node Manager recovery. This
> is because the mappings are kept in memory, it cannot be recovered after node
> manager restart.
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9834) Allow using a pool of local users to run Yarn Secure Container in secure mode
[
https://issues.apache.org/jira/browse/YARN-9834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
shanyu zhao updated YARN-9834:
--
Description:
Yarn Secure Container in secure mode allows separation of different user's
local files and container processes running on the same node manager. This
depends on an out of band service such as SSSD/Winbind to sync all domain users
to local machine.
Winbind user sync has lots of overhead, especially for large corporations. Also
if running Yarn inside Kubernetes cluster (meaning node managers running inside
Docker container), it doesn't make sense for each container to domain join with
Active Directory and sync a whole copy of domain users.
We should allow a new configuration to Yarn, such that we can pre-create a pool
of users on each machine/Docker container. And at runtime, Yarn allocates a
local user to the domain user that submits the application. When all containers
of that user are finished and all files belonging to that user are deleted, we
can release the allocation and allow other users to use the same local user to
run their Yarn containers.
h2. Design
We propose to add these new configurations:
{code:java}
yarn.nodemanager.linux-container-executor.secure-mode.use-local-user, defaults
to false
yarn.nodemanager.linux-container-executor.secure-mode.local-user-prefix,
defaults to "user"{code}
By default this feature is turned off. If we enable it, with local-user-prefix
set to "user", then we expect there are pre-created local users user0 - usern,
where the total number of local users equals to:
{code:java}
yarn.nodemanager.resource.cpu-vcores {code}
We can use an in-memory allocator to keep the domain user to local user
mapping.
Now when to add the mapping and when to remove it?
In node manager, ApplicationImpl implements the state machine for a Yarn app
life cycle, only if the app has at least 1 container running on that node
manager. We can hook up the code to add the mapping during application
initialization.
For removing the mapping, we need to wait for 3 things:
1) All applications of the same user is completed;
2) All log handling of the applications (log aggregation or non-aggregated
handling) is done;
3) All pending FileDeletionTask that use the user's identity is finished.
Note that all operation to these reference counting should be synchronized
operation.
If all of our local users in the pool are allocated, we'll return
"nonexistuser" as runas user, this will cause the container to fail to execute
and Yarn will relaunch it in other nodes.
h2. Limitations
1) This feature does not support PRIVATE visibility type of resource
allocation. Because PRIVATE type of resources are potentially cached in the
node manager for a very long time, supporting it will be a security problem
that a user might be able to peek into previous user's PRIVATE resources. We
can modify code to treat all PRIVATE type of resource as APPLICATION type.
2) It is recommended to enable DominantResourceCalculator so that no more than
"cpu-vcores" number of concurrent containers running on a node manager:
{code:java}
yarn.scheduler.capacity.resource-calculator
= org.apache.hadoop.yarn.util.resource.DominantResourceCalculator {code}
3) Currently this feature does not work with Yarn Node Manager recovery. This
is because the mappings are kept in memory, it cannot be recovered after node
manager restart.
was:
Yarn Secure Container in secure mode allows separation of different user's
local files and container processes running on the same node manager. This
depends on an out of band service such as SSSD/Winbind to sync all domain users
to local machine.
SSSD/Winbind user sync has lots of overhead, especially for large corporations.
Also if running Yarn inside Kubernetes cluster (meaning node managers running
inside Docker container), it doesn't make sense for each container to sync a
whole copy of domain users.
We should allow a new configuration to Yarn, such that we can pre-create a pool
of users on each machine/Docker container. And at runtime, Yarn allocates a
local user to the domain user that submits the application. When all containers
of that user are finished and all files belonging to that user are deleted, we
can release the allocation and allow other users to use the same local user to
run their Yarn containers.
h2. Design
We propose to add these new configurations:
{code:java}
yarn.nodemanager.linux-container-executor.secure-mode.use-local-user, defaults
to false
yarn.nodemanager.linux-container-executor.secure-mode.local-user-prefix,
defaults to "user"{code}
By default this feature is turned off. If we enable it, with local-user-prefix
set to "user", then we expect there are pre-created local users user0 - usern,
where the total number of local users equals to:
{code:java}
yarn.nodemanager.resource.cpu-vcores {code}
We can use an in-memory
[jira] [Updated] (YARN-9834) Allow using a pool of local users to run Yarn Secure Container in secure mode
[
https://issues.apache.org/jira/browse/YARN-9834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
shanyu zhao updated YARN-9834:
--
Description:
Yarn Secure Container in secure mode allows separation of different user's
local files and container processes running on the same node manager. This
depends on an out of band service such as SSSD/Winbind to sync all domain users
to local machine.
SSSD/Winbind user sync has lots of overhead, especially for large corporations.
Also if running Yarn inside Kubernetes cluster (meaning node managers running
inside Docker container), it doesn't make sense for each container to sync a
whole copy of domain users.
We should allow a new configuration to Yarn, such that we can pre-create a pool
of users on each machine/Docker container. And at runtime, Yarn allocates a
local user to the domain user that submits the application. When all containers
of that user are finished and all files belonging to that user are deleted, we
can release the allocation and allow other users to use the same local user to
run their Yarn containers.
h2. Design
We propose to add these new configurations:
{code:java}
yarn.nodemanager.linux-container-executor.secure-mode.use-local-user, defaults
to false
yarn.nodemanager.linux-container-executor.secure-mode.local-user-prefix,
defaults to "user"{code}
By default this feature is turned off. If we enable it, with local-user-prefix
set to "user", then we expect there are pre-created local users user0 - usern,
where the total number of local users equals to:
{code:java}
yarn.nodemanager.resource.cpu-vcores {code}
We can use an in-memory allocator to keep the domain user to local user
mapping.
Now when to add the mapping and when to remove it?
In node manager, ApplicationImpl implements the state machine for a Yarn app
life cycle, only if the app has at least 1 container running on that node
manager. We can hook up the code to add the mapping during application
initialization.
For removing the mapping, we need to wait for 3 things:
1) All applications of the same user is completed;
2) All log handling of the applications (log aggregation or non-aggregated
handling) is done;
3) All pending FileDeletionTask that use the user's identity is finished.
If all of our local users in the pool are allocated, we'll return
"nonexistuser" as runas user, this will cause the container to fail to execute
and Yarn will relaunch it in other nodes.
h2. Limitations
1) This feature does not support PRIVATE visibility type of resource
allocation. Because PRIVATE type of resources are potentially cached in the
node manager for a very long time, supporting it will be a security problem
that a user might be able to peek into previous user's PRIVATE resources. We
can modify code to treat all PRIVATE type of resource as APPLICATION type.
2) It is recommended to enable DominantResourceCalculator so that no more than
"cpu-vcores" number of concurrent containers running on a node manager:
{code:java}
yarn.scheduler.capacity.resource-calculator
= org.apache.hadoop.yarn.util.resource.DominantResourceCalculator {code}
3) Currently this feature does not work with Yarn Node Manager recovery. This
is because the mappings are kept in memory, it cannot be recovered after node
manager restart.
was:
Yarn Secure Container in secure mode allows separation of different user's
local files and container processes running on the same node manager. This
depends on an out of band service such as SSSD/Winbind to sync all domain users
to local machine.
SSSD/Winbind user sync has lots of overhead, especially for large corporations.
Also if running Yarn inside Kubernetes cluster (meaning node managers running
inside Docker container), it doesn't make sense for each container to sync a
whole copy of domain users.
We should allow a new configuration to Yarn, such that we can pre-create a pool
of users on each machine/Docker container. And at runtime, Yarn allocates a
local user to the domain user that submits the application. When all containers
of that user and all files belonging to that user are deleted, we can release
the allocation and allow other users to use the same local user to run their
Yarn containers.
h2. Design
We propose to add these new configurations:
{code:java}
yarn.nodemanager.linux-container-executor.secure-mode.use-local-user, defaults
to false
yarn.nodemanager.linux-container-executor.secure-mode.local-user-prefix,
defaults to "user"{code}
By default this feature is turned off. If we enable it, with local-user-prefix
set to "user", then we expect there are pre-created local users user0 - usern,
where n equals to:
{code:java}
yarn.nodemanager.resource.cpu-vcores {code}
We can use an in-memory allocator to keep the domain user to local user
mapping. When to add the mapping and when to remove it?
In node manager, ApplicationImpl implements the state machine
[jira] [Updated] (YARN-9834) Allow using a pool of local users to run Yarn Secure Container in secure mode
[
https://issues.apache.org/jira/browse/YARN-9834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
shanyu zhao updated YARN-9834:
--
Description:
Yarn Secure Container in secure mode allows separation of different user's
local files and container processes running on the same node manager. This
depends on an out of band service such as SSSD/Winbind to sync all domain users
to local machine.
SSSD/Winbind user sync has lots of overhead, especially for large corporations.
Also if running Yarn inside Kubernetes cluster (meaning node managers running
inside Docker container), it doesn't make sense for each container to sync a
whole copy of domain users.
We should allow a new configuration to Yarn, such that we can pre-create a pool
of users on each machine/Docker container. And at runtime, Yarn allocates a
local user to the domain user that submits the application. When all containers
of that user and all files belonging to that user are deleted, we can release
the allocation and allow other users to use the same local user to run their
Yarn containers.
h2. Design
We propose to add these new configurations:
{code:java}
yarn.nodemanager.linux-container-executor.secure-mode.use-local-user, defaults
to false
yarn.nodemanager.linux-container-executor.secure-mode.local-user-prefix,
defaults to "user"{code}
By default this feature is turned off. If we enable it, with local-user-prefix
set to "user", then we expect there are pre-created local users user0 - usern,
where n equals to:
{code:java}
yarn.nodemanager.resource.cpu-vcores {code}
We can use an in-memory allocator to keep the domain user to local user
mapping. When to add the mapping and when to remove it?
In node manager, ApplicationImpl implements the state machine for a Yarn app
life cycle, only if the app has at least 1 container running on that node
manager. We can hook up the code to add the mapping during application
initialization.
For removing the mapping, we need to wait for 3 things:
1) All applications of the same user is completed;
2) All log handling of the applications (log aggregation or non-aggregated
handling) is done;
3) All pending FileDeletionTask that use the user's identity is finished.
If all of our local users in the pool are allocated, we'll return
"nonexistuser" as runas user, this will cause the container to fail to execute
and Yarn will relaunch it in other nodes.
h2. Limitations
1) This feature does not support PRIVATE visibility type of resource
allocation. Because PRIVATE type of resources are potentially cached in the
node manager for a very long time, supporting it will be a security problem
that a user might be able to peek into previous user's PRIVATE resources. We
can modify code to treat all PRIVATE type of resource as APPLICATION type.
2) It is recommended to enable DominantResourceCalculator so that no more than
"cpu-vcores" number of concurrent containers running on a node manager:
{code:java}
yarn.scheduler.capacity.resource-calculator
= org.apache.hadoop.yarn.util.resource.DominantResourceCalculator {code}
3) Currently this feature does not work with Yarn Node Manager recovery. This
is because the mappings are kept in memory, it cannot be recovered after node
manager restart.
was:
Yarn Secure Container in secure mode allows separation of different user's
local files and container processes running on the same node manager. This
depends on an out of band service such as SSSD/Winbind to sync all domain users
to local machine.
SSSD/Winbind user sync has lots of overhead, especially for large corporations.
Also if running Yarn inside Kubernetes cluster (meaning node managers running
inside Docker container), it doesn't make sense for each container to sync a
whole copy of domain users.
We should allow a new configuration to Yarn, such that we can pre-create a pool
of users on each machine/Docker container. And at runtime, Yarn allocates a
local user to the domain user that submits the application. When all containers
of that user and all files belonging to that user are deleted, we can release
the allocation and allow other users to use the same local user to run their
Yarn containers.
h2. Design
We propose to add these new configurations:
{code:java}
yarn.nodemanager.linux-container-executor.secure-mode.use-local-user, defaults
to false
yarn.nodemanager.linux-container-executor.secure-mode.local-user-prefix,
defaults to "user"{code}
By default this feature is turned off. If we enable it, with local-user-prefix
set to "user", then we expect there are pre-created local users user0 - usern,
where n equals to:
{code:java}
yarn.nodemanager.resource.cpu-vcores {code}
We can use an in-memory allocator to keep the domain user to local user
mapping. When to add the mapping and when to remove it?
In node manager, ApplicationImpl implements the state machine for a Yarn app
life cycle, only if the app has
[jira] [Commented] (YARN-9834) Allow using a pool of local users to run Yarn Secure Container in secure mode
[
https://issues.apache.org/jira/browse/YARN-9834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16929804#comment-16929804
]
shanyu zhao commented on YARN-9834:
---
[~ashvin] I missed that file during git push, it is added now. Thanks!
> Allow using a pool of local users to run Yarn Secure Container in secure mode
> -
>
> Key: YARN-9834
> URL: https://issues.apache.org/jira/browse/YARN-9834
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 3.1.2
>Reporter: shanyu zhao
>Assignee: shanyu zhao
>Priority: Major
>
> Yarn Secure Container in secure mode allows separation of different user's
> local files and container processes running on the same node manager. This
> depends on an out of band service such as SSSD/Winbind to sync all domain
> users to local machine.
> SSSD/Winbind user sync has lots of overhead, especially for large
> corporations. Also if running Yarn inside Kubernetes cluster (meaning node
> managers running inside Docker container), it doesn't make sense for each
> container to sync a whole copy of domain users.
> We should allow a new configuration to Yarn, such that we can pre-create a
> pool of users on each machine/Docker container. And at runtime, Yarn
> allocates a local user to the domain user that submits the application. When
> all containers of that user and all files belonging to that user are deleted,
> we can release the allocation and allow other users to use the same local
> user to run their Yarn containers.
> h2. Design
> We propose to add these new configurations:
> {code:java}
> yarn.nodemanager.linux-container-executor.secure-mode.use-local-user,
> defaults to false
> yarn.nodemanager.linux-container-executor.secure-mode.local-user-prefix,
> defaults to "user"{code}
> By default this feature is turned off. If we enable it, with
> local-user-prefix set to "user", then we expect there are pre-created local
> users user0 - usern, where n equals to:
> {code:java}
> yarn.nodemanager.resource.cpu-vcores {code}
> We can use an in-memory allocator to keep the domain user to local user
> mapping. When to add the mapping and when to remove it?
> In node manager, ApplicationImpl implements the state machine for a Yarn app
> life cycle, only if the app has at least 1 container running on that node
> manager. We can hook up the code to add the mapping during application
> initialization.
> For removing the mapping, we need to wait for 3 things:
> 1) All applications of the same user is completed;
> 2) All log handling of the applications (log aggregation or non-aggregated
> handling) is done;
> 3) All pending FileDeletionTask that use the user's identity is finished.
> h2. Limitations
> 1) This feature does not support PRIVATE visibility type of resource
> allocation. Because PRIVATE type of resources are potentially cached in the
> node manager for a very long time, supporting it will be a security problem
> that a user might be able to peek into previous user's PRIVATE resources. We
> can modify code to treat all PRIVATE type of resource as APPLICATION type.
> 2) It is recommended to enable DominantResourceCalculator so that no more
> than "cpu-vcores" number of concurrent containers running on a node manager:
> {code:java}
> yarn.scheduler.capacity.resource-calculator
> = org.apache.hadoop.yarn.util.resource.DominantResourceCalculator {code}
> 3) Currently this feature does not work with Yarn Node Manager recovery. We
> may add recovery support in the future when we hook up with the right calls
> in the recovery flow.
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9834) Allow using a pool of local users to run Yarn Secure Container in secure mode
[
https://issues.apache.org/jira/browse/YARN-9834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
shanyu zhao updated YARN-9834:
--
Description:
Yarn Secure Container in secure mode allows separation of different user's
local files and container processes running on the same node manager. This
depends on an out of band service such as SSSD/Winbind to sync all domain users
to local machine.
SSSD/Winbind user sync has lots of overhead, especially for large corporations.
Also if running Yarn inside Kubernetes cluster (meaning node managers running
inside Docker container), it doesn't make sense for each container to sync a
whole copy of domain users.
We should allow a new configuration to Yarn, such that we can pre-create a pool
of users on each machine/Docker container. And at runtime, Yarn allocates a
local user to the domain user that submits the application. When all containers
of that user and all files belonging to that user are deleted, we can release
the allocation and allow other users to use the same local user to run their
Yarn containers.
h2. Design
We propose to add these new configurations:
{code:java}
yarn.nodemanager.linux-container-executor.secure-mode.use-local-user, defaults
to false
yarn.nodemanager.linux-container-executor.secure-mode.local-user-prefix,
defaults to "user"{code}
By default this feature is turned off. If we enable it, with local-user-prefix
set to "user", then we expect there are pre-created local users user0 - usern,
where n equals to:
{code:java}
yarn.nodemanager.resource.cpu-vcores {code}
We can use an in-memory allocator to keep the domain user to local user
mapping. When to add the mapping and when to remove it?
In node manager, ApplicationImpl implements the state machine for a Yarn app
life cycle, only if the app has at least 1 container running on that node
manager. We can hook up the code to add the mapping during application
initialization.
For removing the mapping, we need to wait for 3 things:
1) All applications of the same user is completed;
2) All log handling of the applications (log aggregation or non-aggregated
handling) is done;
3) All pending FileDeletionTask that use the user's identity is finished.
h2. Limitations
1) This feature does not support PRIVATE visibility type of resource
allocation. Because PRIVATE type of resources are potentially cached in the
node manager for a very long time, supporting it will be a security problem
that a user might be able to peek into previous user's PRIVATE resources. We
can modify code to treat all PRIVATE type of resource as APPLICATION type.
2) It is recommended to enable DominantResourceCalculator so that no more than
"cpu-vcores" number of concurrent containers running on a node manager:
{code:java}
yarn.scheduler.capacity.resource-calculator
= org.apache.hadoop.yarn.util.resource.DominantResourceCalculator {code}
3) Currently this feature does not work with Yarn Node Manager recovery. We may
add recovery support in the future when we hook up with the right calls in the
recovery flow.
was:
Yarn Secure Container in secure mode allows separation of different user's
local files and container processes running on the same node manager. This
depends on an out of band service such as SSSD/Winbind to sync all domain users
to local machine.
SSSD/Winbind user sync has lots of overhead, especially for large corporations.
Also if running Yarn inside Kubernetes cluster (meaning node managers running
inside Docker container), it doesn't make sense for each container to sync a
whole copy of domain users.
We should allow a new configuration to Yarn, such that we can pre-create a pool
of users on each machine/Docker container. And at runtime, Yarn allocates a
local user to the domain user that submits the application. When all containers
of that user and all files belonging to that user are deleted, we can release
the allocation and allow other users to use the same local user to run their
Yarn containers.
We propose to add these new configurations:
{code:java}
yarn.nodemanager.linux-container-executor.secure-mode.use-local-user, defaults
to false
yarn.nodemanager.linux-container-executor.secure-mode.local-user-prefix,
defaults to "user"{code}
If we enable this feature, with local-user-prefix set to "user", then we expect
there are pre-created local users user0 - usern, where n equals to:
{code:java}
yarn.nodemanager.resource.cpu-vcores {code}
We can use an in-memory allocator to keep the domain user to local user mapping.
Limitations:
1) This feature does not support PRIVATE type of resource allocation. Because
PRIVATE type of resources are potentially cached in the node manager for a very
long time, supporting it will be a security problem that a user might be able
to peek into previous user's PRIVATE resources. We can modify code to treat all
PRIVATE type of resource as APPLICATION.
2) It is
[jira] [Assigned] (YARN-9834) Allow using a pool of local users to run Yarn Secure Container in secure mode
[
https://issues.apache.org/jira/browse/YARN-9834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
shanyu zhao reassigned YARN-9834:
-
Assignee: shanyu zhao
> Allow using a pool of local users to run Yarn Secure Container in secure mode
> -
>
> Key: YARN-9834
> URL: https://issues.apache.org/jira/browse/YARN-9834
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 3.1.2
>Reporter: shanyu zhao
>Assignee: shanyu zhao
>Priority: Major
>
> Yarn Secure Container in secure mode allows separation of different user's
> local files and container processes running on the same node manager. This
> depends on an out of band service such as SSSD/Winbind to sync all domain
> users to local machine.
> SSSD/Winbind user sync has lots of overhead, especially for large
> corporations. Also if running Yarn inside Kubernetes cluster (meaning node
> managers running inside Docker container), it doesn't make sense for each
> container to sync a whole copy of domain users.
> We should allow a new configuration to Yarn, such that we can pre-create a
> pool of users on each machine/Docker container. And at runtime, Yarn
> allocates a local user to the domain user that submits the application. When
> all containers of that user and all files belonging to that user are deleted,
> we can release the allocation and allow other users to use the same local
> user to run their Yarn containers.
> We propose to add these new configurations:
> {code:java}
> yarn.nodemanager.linux-container-executor.secure-mode.use-local-user,
> defaults to false
> yarn.nodemanager.linux-container-executor.secure-mode.local-user-prefix,
> defaults to "user"{code}
> If we enable this feature, with local-user-prefix set to "user", then we
> expect there are pre-created local users user0 - usern, where n equals to:
> {code:java}
> yarn.nodemanager.resource.cpu-vcores {code}
> We can use an in-memory allocator to keep the domain user to local user
> mapping.
> Limitations:
> 1) This feature does not support PRIVATE type of resource allocation. Because
> PRIVATE type of resources are potentially cached in the node manager for a
> very long time, supporting it will be a security problem that a user might be
> able to peek into previous user's PRIVATE resources. We can modify code to
> treat all PRIVATE type of resource as APPLICATION.
> 2) It is recommended to enable DominantResourceCalculator so that no more
> than "cpu-vcores" number of concurrent containers running on a node manager:
> {code:java}
> yarn.scheduler.capacity.resource-calculator
> = org.apache.hadoop.yarn.util.resource.DominantResourceCalculator {code}
> 3) Currently this feature does not work with Yarn Node Manager recovery. We
> may add recovery support in the future when we hook up with the right calls
> in the recovery flow.
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9834) Allow using a pool of local users to run Yarn Secure Container in secure mode
shanyu zhao created YARN-9834:
-
Summary: Allow using a pool of local users to run Yarn Secure
Container in secure mode
Key: YARN-9834
URL: https://issues.apache.org/jira/browse/YARN-9834
Project: Hadoop YARN
Issue Type: Bug
Components: nodemanager
Affects Versions: 3.1.2
Reporter: shanyu zhao
Yarn Secure Container in secure mode allows separation of different user's
local files and container processes running on the same node manager. This
depends on an out of band service such as SSSD/Winbind to sync all domain users
to local machine.
SSSD/Winbind user sync has lots of overhead, especially for large corporations.
Also if running Yarn inside Kubernetes cluster (meaning node managers running
inside Docker container), it doesn't make sense for each container to sync a
whole copy of domain users.
We should allow a new configuration to Yarn, such that we can pre-create a pool
of users on each machine/Docker container. And at runtime, Yarn allocates a
local user to the domain user that submits the application. When all containers
of that user and all files belonging to that user are deleted, we can release
the allocation and allow other users to use the same local user to run their
Yarn containers.
We propose to add these new configurations:
{code:java}
yarn.nodemanager.linux-container-executor.secure-mode.use-local-user, defaults
to false
yarn.nodemanager.linux-container-executor.secure-mode.local-user-prefix,
defaults to "user"{code}
If we enable this feature, with local-user-prefix set to "user", then we expect
there are pre-created local users user0 - usern, where n equals to:
{code:java}
yarn.nodemanager.resource.cpu-vcores {code}
We can use an in-memory allocator to keep the domain user to local user mapping.
Limitations:
1) This feature does not support PRIVATE type of resource allocation. Because
PRIVATE type of resources are potentially cached in the node manager for a very
long time, supporting it will be a security problem that a user might be able
to peek into previous user's PRIVATE resources. We can modify code to treat all
PRIVATE type of resource as APPLICATION.
2) It is recommended to enable DominantResourceCalculator so that no more than
"cpu-vcores" number of concurrent containers running on a node manager:
{code:java}
yarn.scheduler.capacity.resource-calculator
= org.apache.hadoop.yarn.util.resource.DominantResourceCalculator {code}
3) Currently this feature does not work with Yarn Node Manager recovery. We may
add recovery support in the future when we hook up with the right calls in the
recovery flow.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8001) Newly created Yarn application ID lost after RM failover
[ https://issues.apache.org/jira/browse/YARN-8001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389881#comment-16389881 ] shanyu zhao commented on YARN-8001: --- The failure is a pig job. After a client submitted an application successfully, later when it tries to query the status of the app then failed. Who is going to re-submit the application? Are you saying when using the Yarn API to get application it will automatically resubmit the application? > Newly created Yarn application ID lost after RM failover > > > Key: YARN-8001 > URL: https://issues.apache.org/jira/browse/YARN-8001 > Project: Hadoop YARN > Issue Type: Bug > Components: RM >Affects Versions: 2.7.3, 2.9.0 >Reporter: shanyu zhao >Priority: Major > > I’ve seen a problem in Hadoop 2.7.3 where the newly submitted yarn > application was lost after a RM failover. It looks like when handling > Application submission, RM does not write it to the state-store (We are using > zookeeper based state store) immediately before it respond to the client. But > later it failed over to another RM and all write call to the state store > failed. The new RM recovers state from the state-store, and this app is lost. > > The symptom is error message at client side claiming a previously submitted > application ID does not exist: > 2018-02-22 14:54:50,258 [JobControl] WARN > org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider - > Invocation returned exception on [rm1] : > org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException: Application > with id 'application_1519310222933_0160' doesn't exist in RM. Please check > that the job submission was successful. > > This is a timeline excerpted from the resource manager logs: > 2018-02-22 14:54:06.7685260 headnode1 Storing application with id > application_1519310222933_0160 > 2018-02-22 14:54:06.7685660 headnode1 > application_1519310222933_0160 State change from NEW to NEW_SAVING > 2018-02-22 14:54:17.8924760 headnode1 Transitioning to standby state > 2018-02-22 14:54:30.3951160 headnode0 Transitioning to active state -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-8001) Newly created Yarn application ID lost after RM failover
shanyu zhao created YARN-8001: - Summary: Newly created Yarn application ID lost after RM failover Key: YARN-8001 URL: https://issues.apache.org/jira/browse/YARN-8001 Project: Hadoop YARN Issue Type: Bug Components: RM Affects Versions: 2.9.0, 2.7.3 Reporter: shanyu zhao I’ve seen a problem in Hadoop 2.7.3 where the newly submitted yarn application was lost after a RM failover. It looks like when handling Application submission, RM does not write it to the state-store (We are using zookeeper based state store) immediately before it respond to the client. But later it failed over to another RM and all write call to the state store failed. The new RM recovers state from the state-store, and this app is lost. The symptom is error message at client side claiming a previously submitted application ID does not exist: 2018-02-22 14:54:50,258 [JobControl] WARN org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider - Invocation returned exception on [rm1] : org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException: Application with id 'application_1519310222933_0160' doesn't exist in RM. Please check that the job submission was successful. This is a timeline excerpted from the resource manager logs: 2018-02-22 14:54:06.7685260 headnode1 Storing application with id application_1519310222933_0160 2018-02-22 14:54:06.7685660 headnode1 application_1519310222933_0160 State change from NEW to NEW_SAVING 2018-02-22 14:54:17.8924760 headnode1 Transitioning to standby state 2018-02-22 14:54:30.3951160 headnode0 Transitioning to active state -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6868) Add test scope to certain entries in hadoop-yarn-server-resourcemanager pom.xml
[
https://issues.apache.org/jira/browse/YARN-6868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351489#comment-16351489
]
shanyu zhao commented on YARN-6868:
---
Can we also port this patch to branch-2.9? In branch-2.9 the
hadoop-resourceestimator in hadoop-tools depends on
hadoop-yarn-server-resourcemanager which will bring zookeeper and curator tests
jar into the distribution of hadoop-tools.
> Add test scope to certain entries in hadoop-yarn-server-resourcemanager
> pom.xml
> ---
>
> Key: YARN-6868
> URL: https://issues.apache.org/jira/browse/YARN-6868
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.0.0-beta1
>Reporter: Ray Chiang
>Assignee: Ray Chiang
>Priority: Major
> Fix For: 3.0.0-beta1
>
> Attachments: YARN-6868.001.patch
>
>
> The tag
> {noformat}
> test
> {noformat}
> is missing from a few entries in the pom.xml for
> hadoop-yarn-server-resourcemanager.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-1713) Implement getnewapplication and submitapp as part of RM web service
[
https://issues.apache.org/jira/browse/YARN-1713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13987099#comment-13987099
]
shanyu zhao commented on YARN-1713:
---
[~vvasudev] I tested this patch and it works great!
I thought about your implementation of reflecting how RPC structures the API
and it sounds good to me.
The only feedback I have is the style of the REST API, does the following API
change make sense for you?
POST /app/id -- POST /appids
POST /app/{appid} -- POST /apps/{appid}
This way the submit app REST API is more consistent with app status and kill
app API.
Sorry I missed the part that your implementation actually calls doAs() on
behalf of the remote user. That should work just fine.
Thanks!
Implement getnewapplication and submitapp as part of RM web service
---
Key: YARN-1713
URL: https://issues.apache.org/jira/browse/YARN-1713
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Varun Vasudev
Assignee: Varun Vasudev
Attachments: apache-yarn-1713.3.patch, apache-yarn-1713.4.patch,
apache-yarn-1713.5.patch, apache-yarn-1713.cumulative.2.patch,
apache-yarn-1713.cumulative.3.patch, apache-yarn-1713.cumulative.4.patch,
apache-yarn-1713.cumulative.patch, apache-yarn-1713.patch
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1215) Yarn URL should include userinfo
[
https://issues.apache.org/jira/browse/YARN-1215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13780637#comment-13780637
]
shanyu zhao commented on YARN-1215:
---
+1
Looks good to me.
Yarn URL should include userinfo
Key: YARN-1215
URL: https://issues.apache.org/jira/browse/YARN-1215
Project: Hadoop YARN
Issue Type: Bug
Components: api
Affects Versions: 3.0.0
Reporter: Chuan Liu
Assignee: Chuan Liu
Attachments: YARN-1215-trunk.2.patch, YARN-1215-trunk.patch
In the {{org.apache.hadoop.yarn.api.records.URL}} class, we don't have an
userinfo as part of the URL. When converting a {{java.net.URI}} object into
the YARN URL object in {{ConverterUtils.getYarnUrlFromURI()}} method, we will
set uri host as the url host. If the uri has a userinfo part, the userinfo is
discarded. This will lead to information loss if the original uri has the
userinfo, e.g. foo://username:passw...@example.com will be converted to
foo://example.com and username/password information is lost during the
conversion.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-1219) FSDownload changes file suffix making FileUtil.unTar() throw exception
[
https://issues.apache.org/jira/browse/YARN-1219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
shanyu zhao updated YARN-1219:
--
Fix Version/s: 2.1.2-beta
FSDownload changes file suffix making FileUtil.unTar() throw exception
--
Key: YARN-1219
URL: https://issues.apache.org/jira/browse/YARN-1219
Project: Hadoop YARN
Issue Type: Bug
Components: nodemanager
Affects Versions: 3.0.0, 2.1.1-beta, 2.1.2-beta
Reporter: shanyu zhao
Assignee: shanyu zhao
Fix For: 2.1.2-beta
Attachments: YARN-1219.patch
While running a Hive join operation on Yarn, I saw exception as described
below. This is caused by FSDownload copy the files into a temp file and
change the suffix into .tmp before unpacking it. In unpack(), it uses
FileUtil.unTar() which will determine if the file is gzipped by looking at
the file suffix:
{code}
boolean gzipped = inFile.toString().endsWith(gz);
{code}
To fix this problem, we can remove the .tmp in the temp file name.
Here is the detailed exception:
org.apache.commons.compress.archivers.tar.TarArchiveInputStream.getNextTarEntry(TarArchiveInputStream.java:240)
at org.apache.hadoop.fs.FileUtil.unTarUsingJava(FileUtil.java:676)
at org.apache.hadoop.fs.FileUtil.unTar(FileUtil.java:625)
at org.apache.hadoop.yarn.util.FSDownload.unpack(FSDownload.java:203)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:287)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:50)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1219) FSDownload changes file suffix making FileUtil.unTar() throw exception
[
https://issues.apache.org/jira/browse/YARN-1219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13778330#comment-13778330
]
shanyu zhao commented on YARN-1219:
---
The impact of this bug is that Yarn will fail to download any .tar.gz or .tgz
files as resources on Windows platform.
[~ojoshi] Thank you for reviewing this patch. I think the .tmp file name
suffix is only used to make sure the temporary archive file use a different
name than the archive file name itself. So the file name of dCopy has to be
different from sCopy. Because the final unarchived files need to be put in a
folder whose name is the exactly the original archive file name.
Here is what happens when a container downloads an archived resource file,
let's say the file name is stage-3.tar.gz and the destination dir is
filecache/10.
a. Create destDirPath folder filecache/10
b. Create dst_work folder filecache/10_tmp
c. Copy archive file to dTmp filecache/10_tmp/stage-3.tar.gz.tmp
d. Unpack dTmp filecache/10_tmp/stage-3.tar.gz.tmp into dFinal
filecache/10_tmp/stage-3.tar.gz/
e. Rename folder dst_work filecache/10_tmp to destDirPath filecache/10
So after all these steps, the unzipped unarchived files can be found in
filecache/10/stage-3.tar.gz/*
I also checked in Hadoop 1.x the file name being downloaded from HDFS does not
have the .tmp suffix. It is using the original file name as the file name to
be downloaded.
I didn't see anywhere in code to treat the .tmp file differently. If you know
please let me know. If the original author only used a suffix to make sure the
name is different than the original file name, it doesn't seem to be worth it
to add an unnecessary and error-prone rename operations just to keep the
temporary file name suffix.
FSDownload changes file suffix making FileUtil.unTar() throw exception
--
Key: YARN-1219
URL: https://issues.apache.org/jira/browse/YARN-1219
Project: Hadoop YARN
Issue Type: Bug
Components: nodemanager
Affects Versions: 3.0.0, 2.1.1-beta, 2.1.2-beta
Reporter: shanyu zhao
Assignee: shanyu zhao
Fix For: 2.1.2-beta
Attachments: YARN-1219.patch
While running a Hive join operation on Yarn, I saw exception as described
below. This is caused by FSDownload copy the files into a temp file and
change the suffix into .tmp before unpacking it. In unpack(), it uses
FileUtil.unTar() which will determine if the file is gzipped by looking at
the file suffix:
{code}
boolean gzipped = inFile.toString().endsWith(gz);
{code}
To fix this problem, we can remove the .tmp in the temp file name.
Here is the detailed exception:
org.apache.commons.compress.archivers.tar.TarArchiveInputStream.getNextTarEntry(TarArchiveInputStream.java:240)
at org.apache.hadoop.fs.FileUtil.unTarUsingJava(FileUtil.java:676)
at org.apache.hadoop.fs.FileUtil.unTar(FileUtil.java:625)
at org.apache.hadoop.yarn.util.FSDownload.unpack(FSDownload.java:203)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:287)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:50)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1215) Yarn URL should include userinfo
[
https://issues.apache.org/jira/browse/YARN-1215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13775718#comment-13775718
]
shanyu zhao commented on YARN-1215:
---
Looks good to me overall. But I think the best fix is to add a userInfo field
for org.apache.hadoop.yarn.api.records.URL.
Yarn URL should include userinfo
Key: YARN-1215
URL: https://issues.apache.org/jira/browse/YARN-1215
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Chuan Liu
Assignee: Chuan Liu
Attachments: YARN-1215-trunk.patch
In the {{org.apache.hadoop.yarn.api.records.URL}} class, we don't have an
userinfo as part of the URL. When converting a {{java.net.URI}} object into
the YARN URL object in {{ConverterUtils.getYarnUrlFromURI()}} method, we will
set uri host as the url host. If the uri has a userinfo part, the userinfo is
discarded. This will lead to information loss if the original uri has the
userinfo, e.g. foo://username:passw...@example.com will be converted to
foo://example.com and username/password information is lost during the
conversion.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-1219) FSDownload changes file suffix making FileUtil.unTar() throw exception
[
https://issues.apache.org/jira/browse/YARN-1219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
shanyu zhao updated YARN-1219:
--
Attachment: YARN-1219.patch
Patch attached. There's only one line change in source code. Added a new unit
test case to test .tgz file download. Also refactored the testing code to avoid
code redundancy.
FSDownload changes file suffix making FileUtil.unTar() throw exception
--
Key: YARN-1219
URL: https://issues.apache.org/jira/browse/YARN-1219
Project: Hadoop YARN
Issue Type: Bug
Components: nodemanager
Affects Versions: 3.0.0, 2.1.1-beta
Reporter: shanyu zhao
Assignee: shanyu zhao
Attachments: YARN-1219.patch
While running a Hive join operation on Yarn, I saw exception as described
below. This is caused by FSDownload copy the files into a temp file and
change the suffix into .tmp before unpacking it. In unpack(), it uses
FileUtil.unTar() which will determine if the file is gzipped by looking at
the file suffix:
{code}
boolean gzipped = inFile.toString().endsWith(gz);
{code}
To fix this problem, we can remove the .tmp in the temp file name.
Here is the detailed exception:
org.apache.commons.compress.archivers.tar.TarArchiveInputStream.getNextTarEntry(TarArchiveInputStream.java:240)
at org.apache.hadoop.fs.FileUtil.unTarUsingJava(FileUtil.java:676)
at org.apache.hadoop.fs.FileUtil.unTar(FileUtil.java:625)
at org.apache.hadoop.yarn.util.FSDownload.unpack(FSDownload.java:203)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:287)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:50)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-1219) FSDownload changes file suffix making FileUtil.unTar() throw exception
shanyu zhao created YARN-1219:
-
Summary: FSDownload changes file suffix making FileUtil.unTar()
throw exception
Key: YARN-1219
URL: https://issues.apache.org/jira/browse/YARN-1219
Project: Hadoop YARN
Issue Type: Bug
Components: nodemanager
Affects Versions: 3.0.0, 2.1.1-beta
Reporter: shanyu zhao
While running a Hive join operation on Yarn, I saw exception as described
below. This is caused by FSDownload copy the files into a temp file and change
the suffix into .tmp before unpacking it. In unpack(), it uses
FileUtil.unTar() which will determine if the file is gzipped by looking at
the file suffix:
{code}
boolean gzipped = inFile.toString().endsWith(gz);
{code}
To fix this problem, we can remove the .tmp in the temp file name.
Here is the detailed exception:
org.apache.commons.compress.archivers.tar.TarArchiveInputStream.getNextTarEntry(TarArchiveInputStream.java:240)
at org.apache.hadoop.fs.FileUtil.unTarUsingJava(FileUtil.java:676)
at org.apache.hadoop.fs.FileUtil.unTar(FileUtil.java:625)
at org.apache.hadoop.yarn.util.FSDownload.unpack(FSDownload.java:203)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:287)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:50)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-1219) FSDownload changes file suffix making FileUtil.unTar() throw exception
[
https://issues.apache.org/jira/browse/YARN-1219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
shanyu zhao updated YARN-1219:
--
Assignee: shanyu zhao
FSDownload changes file suffix making FileUtil.unTar() throw exception
--
Key: YARN-1219
URL: https://issues.apache.org/jira/browse/YARN-1219
Project: Hadoop YARN
Issue Type: Bug
Components: nodemanager
Affects Versions: 3.0.0, 2.1.1-beta
Reporter: shanyu zhao
Assignee: shanyu zhao
While running a Hive join operation on Yarn, I saw exception as described
below. This is caused by FSDownload copy the files into a temp file and
change the suffix into .tmp before unpacking it. In unpack(), it uses
FileUtil.unTar() which will determine if the file is gzipped by looking at
the file suffix:
{code}
boolean gzipped = inFile.toString().endsWith(gz);
{code}
To fix this problem, we can remove the .tmp in the temp file name.
Here is the detailed exception:
org.apache.commons.compress.archivers.tar.TarArchiveInputStream.getNextTarEntry(TarArchiveInputStream.java:240)
at org.apache.hadoop.fs.FileUtil.unTarUsingJava(FileUtil.java:676)
at org.apache.hadoop.fs.FileUtil.unTar(FileUtil.java:625)
at org.apache.hadoop.yarn.util.FSDownload.unpack(FSDownload.java:203)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:287)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:50)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
