[jira] [Commented] (YARN-5670) Add support for Docker image clean up
[ https://issues.apache.org/jira/browse/YARN-5670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820449#comment-16820449 ] Eric Badger commented on YARN-5670: --- I think that we need to be able to remove the things that we have pulled. So if we pull an image, we have the right to delete it. In environments where the images are pulled outside of YARN, this deletion feature can be disabled and the admin will have control over how they want to delete the images from each node. Then in environments where docker is just running side by side with YARN, image deletion will mean at worst that the node has to redownload the image. If the image is in use, the deletion will fail. So it should be safe in this regard. > Add support for Docker image clean up > - > > Key: YARN-5670 > URL: https://issues.apache.org/jira/browse/YARN-5670 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Zhankun Tang >Priority: Major > Labels: Docker > Attachments: Localization Support For Docker Images_002.pdf > > > Regarding to Docker image localization, we also need a way to clean up the > old/stale Docker image to save storage space. We may extend deletion service > to utilize "docker rm" to do this. > This is related to YARN-3854 and may depend on its implementation. Please > refer to YARN-3854 for Docker image localization details. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5670) Add support for Docker image clean up
[ https://issues.apache.org/jira/browse/YARN-5670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16809338#comment-16809338 ] Eric Yang commented on YARN-5670: - In today's YARN Docker meeting, there is consensus on using Node Manager track LRU by digest ID, and apply mark and sweep algorithm to prune images seem by node manager. Open concern is still around corner case where locally tagged system admin images can get deleted when the same image is used by a job. Kubernetes tackles docker image pruning problem by making assumption that [system not require human operators to work reliably|https://thenewstack.io/deletion-garbage-collection-kubernetes-objects/]. I think this is a safe assumption, and wait for [~shaneku...@gmail.com] and [~ebadger] to process this information. > Add support for Docker image clean up > - > > Key: YARN-5670 > URL: https://issues.apache.org/jira/browse/YARN-5670 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Zhankun Tang >Priority: Major > Labels: Docker > Attachments: Localization Support For Docker Images_002.pdf > > > Regarding to Docker image localization, we also need a way to clean up the > old/stale Docker image to save storage space. We may extend deletion service > to utilize "docker rm" to do this. > This is related to YARN-3854 and may depend on its implementation. Please > refer to YARN-3854 for Docker image localization details. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5670) Add support for Docker image clean up
[ https://issues.apache.org/jira/browse/YARN-5670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16805436#comment-16805436 ] Eric Yang commented on YARN-5670: - Looks like option 2 is a no go in current docker implementation. [~billie.rinaldi] identified that docker only stored CreatedSince timestamp. Hence, docker image prune will delete images, if the images were created in distance past rather than based on last recently used date. > Add support for Docker image clean up > - > > Key: YARN-5670 > URL: https://issues.apache.org/jira/browse/YARN-5670 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Zhankun Tang >Assignee: Eric Yang >Priority: Major > Labels: Docker > Attachments: Localization Support For Docker Images_002.pdf > > > Regarding to Docker image localization, we also need a way to clean up the > old/stale Docker image to save storage space. We may extend deletion service > to utilize "docker rm" to do this. > This is related to YARN-3854 and may depend on its implementation. Please > refer to YARN-3854 for Docker image localization details. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5670) Add support for Docker image clean up
[ https://issues.apache.org/jira/browse/YARN-5670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16798409#comment-16798409 ] Eric Yang commented on YARN-5670: - [~ebadger] How to solve the corner cases in option 1? I don't see a path forward that can separate admin images from node manager pulled image in existing system. > Add support for Docker image clean up > - > > Key: YARN-5670 > URL: https://issues.apache.org/jira/browse/YARN-5670 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Zhankun Tang >Assignee: Eric Yang >Priority: Major > Labels: Docker > Attachments: Localization Support For Docker Images_002.pdf > > > Regarding to Docker image localization, we also need a way to clean up the > old/stale Docker image to save storage space. We may extend deletion service > to utilize "docker rm" to do this. > This is related to YARN-3854 and may depend on its implementation. Please > refer to YARN-3854 for Docker image localization details. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5670) Add support for Docker image clean up
[ https://issues.apache.org/jira/browse/YARN-5670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16798391#comment-16798391 ] Eric Badger commented on YARN-5670: --- As I said in [this comment|https://issues.apache.org/jira/browse/YARN-7848?focusedCommentId=16798387=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16798387] on YARN-7848, I don't believe we should be managing docker on the node with the assumption that we can completely control it. I am not comfortable with the nodemanager removing images that it did not put there. In option 2, we would be deleting any image regardless of its origin. > Add support for Docker image clean up > - > > Key: YARN-5670 > URL: https://issues.apache.org/jira/browse/YARN-5670 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Zhankun Tang >Assignee: Eric Yang >Priority: Major > Labels: Docker > Attachments: Localization Support For Docker Images_002.pdf > > > Regarding to Docker image localization, we also need a way to clean up the > old/stale Docker image to save storage space. We may extend deletion service > to utilize "docker rm" to do this. > This is related to YARN-3854 and may depend on its implementation. Please > refer to YARN-3854 for Docker image localization details. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5670) Add support for Docker image clean up
[
https://issues.apache.org/jira/browse/YARN-5670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16798241#comment-16798241
]
Eric Yang commented on YARN-5670:
-
{quote}The cache can be backed by NMStateStore so even when the NM comes back
or is restarted, it will know what images it localized.{quote}
The problem is not related to LRU is persisted by NMStateStore. The problem is
related to docker image tags are moving targets. Let's consider that node
manager tracks images by name and tags combo. centos:latest has digest id:
123. This image was used yesterday. The image is updated to digest id: 234
today. When NM delete centos:latest tomorrow because it has not been in use
for 24 hours. Image with digest id: 123 will not be deleted because it is no
longer associated with the same name from two days ago.
Let's take another view, if image is tracked by digest id by NM. System admin
tagged centos:latest (digest id: 123) to private_image:my_version. He is
hoping that no one will delete his image. A job started with centos:latest,
and resolved to digest id: 123. Centos:latest updated to digest id: 234 by
another job a few hours later. 24 hours later, private_image:my_version is
deleted by digest id 123 clean up job because there is only
private_image:my_version referenced with digest id 123.
Hadoop 3.1.x and 3.2.x don't have clean up ability. Therefore, dangling images
are already accumulating in production systems. There is no way to identify
images that were pulled by the system or placed by admin, this makes option 1
less attractive to implement because it can not reach the desired clean state
without undesired side effects above.
Option 2 is safer to implement in Hadoop because giving system admin an option
to turn on. They can be more prepared in their internal infrastructure setup
to be less of a one off and reach the same definition of clean state that
[Docker swarm uses with system prune|https://github.com/moby/moby/issues/31254].
> Add support for Docker image clean up
> -
>
> Key: YARN-5670
> URL: https://issues.apache.org/jira/browse/YARN-5670
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Zhankun Tang
>Assignee: Eric Yang
>Priority: Major
> Labels: Docker
> Attachments: Localization Support For Docker Images_002.pdf
>
>
> Regarding to Docker image localization, we also need a way to clean up the
> old/stale Docker image to save storage space. We may extend deletion service
> to utilize "docker rm" to do this.
> This is related to YARN-3854 and may depend on its implementation. Please
> refer to YARN-3854 for Docker image localization details.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5670) Add support for Docker image clean up
[
https://issues.apache.org/jira/browse/YARN-5670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16797855#comment-16797855
]
Chandni Singh commented on YARN-5670:
-
{quote}
it is possible that some images tracking are lost from LRU and result in
dangling images over time.
{quote}
The cache can be backed by NMStateStore so even when the NM comes back or is
restarted, it will know what images it localized.
The reason I am against using {{docker image prune}} is because there can be
multiple images on that node which an admin may have pulled explicitly or some
other process may have downloaded. Now, even if they don't use it within the
last {{24 h}} or whatever time we have configured for the NM, the NM should not
be the one deciding to remove that image. It is surprising for the admin/other
process that the image they pulled is being mysteriously deleted.
> Add support for Docker image clean up
> -
>
> Key: YARN-5670
> URL: https://issues.apache.org/jira/browse/YARN-5670
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Zhankun Tang
>Assignee: Eric Yang
>Priority: Major
> Labels: Docker
> Attachments: Localization Support For Docker Images_002.pdf
>
>
> Regarding to Docker image localization, we also need a way to clean up the
> old/stale Docker image to save storage space. We may extend deletion service
> to utilize "docker rm" to do this.
> This is related to YARN-3854 and may depend on its implementation. Please
> refer to YARN-3854 for Docker image localization details.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5670) Add support for Docker image clean up
[ https://issues.apache.org/jira/browse/YARN-5670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16797768#comment-16797768 ] Eric Yang commented on YARN-5670: - [~csingh] If node manager is offline for whatever reason, it is possible that some images tracking are lost from LRU and result in dangling images over time. In production environment, it is best to automate as much as possible to keep system running without manual intervention. I think option 2 will keep system cleaner over long period of service. With proper deployment of Docker private registry, there is no good reason to use local image as permanent storage of docker images in production environment because populating local docker with consistent images can be painful at scale. [~ebadger]'s team moved onto using squashFS with runc, so his team does not depend on local docker to store images. Hence, his opinion may change over time. > Add support for Docker image clean up > - > > Key: YARN-5670 > URL: https://issues.apache.org/jira/browse/YARN-5670 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Zhankun Tang >Assignee: Eric Yang >Priority: Major > Labels: Docker > Attachments: Localization Support For Docker Images_002.pdf > > > Regarding to Docker image localization, we also need a way to clean up the > old/stale Docker image to save storage space. We may extend deletion service > to utilize "docker rm" to do this. > This is related to YARN-3854 and may depend on its implementation. Please > refer to YARN-3854 for Docker image localization details. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5670) Add support for Docker image clean up
[
https://issues.apache.org/jira/browse/YARN-5670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16797675#comment-16797675
]
Chandni Singh commented on YARN-5670:
-
[~eyang] I have attached the doc from YARN-3854 here as well which talks about
the 2 options
1. NM maintains an LRU cache which holds meta-info of docker images and it
only deletes the images that it pulled/localized
2. Using docker image prune
IMO Option 1 is better because {{docker image prune}} will remove all the
images on the node, even the ones which the NM never localized.
Also [~ebadger] had a comment in favor of this option. The link to it is:
https://issues.apache.org/jira/browse/YARN-3854?focusedCommentId=16645496=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16645496
> Add support for Docker image clean up
> -
>
> Key: YARN-5670
> URL: https://issues.apache.org/jira/browse/YARN-5670
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Zhankun Tang
>Assignee: Eric Yang
>Priority: Major
> Labels: Docker
> Attachments: Localization Support For Docker Images_002.pdf
>
>
> Regarding to Docker image localization, we also need a way to clean up the
> old/stale Docker image to save storage space. We may extend deletion service
> to utilize "docker rm" to do this.
> This is related to YARN-3854 and may depend on its implementation. Please
> refer to YARN-3854 for Docker image localization details.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5670) Add support for Docker image clean up
[
https://issues.apache.org/jira/browse/YARN-5670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16797658#comment-16797658
]
Eric Yang commented on YARN-5670:
-
The current proposal is to have a configuration flag to enable a background
image clean up thread. The image clean up thread wakes up every 4 hours to
remove docker images that has not been used in the last 24 hours. Docker
command to run is:
{code}
docker image prune -a --filter "until=24h" -f
{code}
> Add support for Docker image clean up
> -
>
> Key: YARN-5670
> URL: https://issues.apache.org/jira/browse/YARN-5670
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Zhankun Tang
>Assignee: Eric Yang
>Priority: Major
> Labels: Docker
>
> Regarding to Docker image localization, we also need a way to clean up the
> old/stale Docker image to save storage space. We may extend deletion service
> to utilize "docker rm" to do this.
> This is related to YARN-3854 and may depend on its implementation. Please
> refer to YARN-3854 for Docker image localization details.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5670) Add support for Docker image clean up
[ https://issues.apache.org/jira/browse/YARN-5670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16611072#comment-16611072 ] Chandni Singh commented on YARN-5670: - discussed with [~shaneku...@gmail.com] and [~eyang] offline. Assigning it to myself. > Add support for Docker image clean up > - > > Key: YARN-5670 > URL: https://issues.apache.org/jira/browse/YARN-5670 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Zhankun Tang >Assignee: Shane Kumpf >Priority: Major > Labels: Docker > > Regarding to Docker image localization, we also need a way to clean up the > old/stale Docker image to save storage space. We may extend deletion service > to utilize "docker rm" to do this. > This is related to YARN-3854 and may depend on its implementation. Please > refer to YARN-3854 for Docker image localization details. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5670) Add support for Docker image clean up
[ https://issues.apache.org/jira/browse/YARN-5670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15669111#comment-15669111 ] luhuichun commented on YARN-5670: - [~sidharta-s] Hi Sidharta, we have some issue for this JIRA 1. When to clean these docker images ? yarn deletion service will do clean up service for distributed cache , but as a framework, I think it should not to consider about the external service (docker), it should be the docker’s responsibility 2. Which image should be clean up? > Add support for Docker image clean up > - > > Key: YARN-5670 > URL: https://issues.apache.org/jira/browse/YARN-5670 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Zhankun Tang >Assignee: luhuichun > > Regarding to Docker image localization, we also need a way to clean up the > old/stale Docker image to save storage space. We may extend deletion service > to utilize "docker rm" to do this. > This is related to YARN-3854 and may depend on its implementation. Please > refer to YARN-3854 for Docker image localization details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
