[Bug 1879473] Re: hundreds of snapshots created by docker zfs lead to zsys timeout
This bug was fixed in the package docker.io - 19.03.8-0ubuntu1.20.04 --- docker.io (19.03.8-0ubuntu1.20.04) focal; urgency=medium [ Jean-Baptiste Lallement ] [ Didier Roche ] * Fix use with ZFS on root: - docker creates one dataset for any layer of containers that were created. Create now a /var/lib/docker for creating them in the persistent namespace. - don’t migrate existing ones but display a message to the end user on ZSys wiki with instrutions. The migration only impacts the ubuntu desktop installation with experimental ZFS on root. (LP: #1879473) -- Didier Roche Thu, 18 Jun 2020 10:26:54 +0200 ** Changed in: docker.io (Ubuntu Focal) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1879473 Title: hundreds of snapshots created by docker zfs lead to zsys timeout To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/docker.io/+bug/1879473/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1879473] Re: hundreds of snapshots created by docker zfs lead to zsys timeout
SRU verification for Focal: I have reproduced the problem with docker.io 19.03.8-0ubuntu1 in focal and have verified that the version of docker.io 19.03.8-0ubuntu1.20.04 in -proposed fixes the issue. Marking as verification-done ** Tags removed: verification-needed verification-needed-focal ** Tags added: verification-done verification-done-focal -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1879473 Title: hundreds of snapshots created by docker zfs lead to zsys timeout To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/docker.io/+bug/1879473/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1879473] Re: hundreds of snapshots created by docker zfs lead to zsys timeout
It seems to me like the wiki page referenced in the SRU should mention the version of the package which is being SRU'ed not the version of the package which is in Groovy. https://github.com/ubuntu/zsys/wiki/Performance-issue-with-docker-on- ubuntu-20.04-LTS ** Changed in: docker.io (Ubuntu Focal) Status: Triaged => Fix Committed ** Tags added: verification-needed verification-needed-focal -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1879473 Title: hundreds of snapshots created by docker zfs lead to zsys timeout To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/docker.io/+bug/1879473/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1879473] Re: hundreds of snapshots created by docker zfs lead to zsys timeout
The new postinst failed for me in groovy, here is the bug I filed: https://bugs.launchpad.net/ubuntu/+source/docker.io/+bug/1882942 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1879473 Title: hundreds of snapshots created by docker zfs lead to zsys timeout To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/docker.io/+bug/1879473/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1879473] Re: hundreds of snapshots created by docker zfs lead to zsys timeout
Here we go: https://github.com/moby/moby/issues/41055 ** Bug watch added: github.com/moby/moby/issues #41055 https://github.com/moby/moby/issues/41055 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1879473 Title: hundreds of snapshots created by docker zfs lead to zsys timeout To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/docker.io/+bug/1879473/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1879473] Re: hundreds of snapshots created by docker zfs lead to zsys timeout
The "zfs" graph driver is part of Docker itself (https://github.com/moby/moby/tree/89382f2f20745b9e63bed6c066f104980dff4396/daemon/graphdriver/zfs), so https://github.com/moby/moby/issues would be the appropriate place to file an issue (or enhancement request) against it. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1879473 Title: hundreds of snapshots created by docker zfs lead to zsys timeout To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/docker.io/+bug/1879473/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1879473] Re: hundreds of snapshots created by docker zfs lead to zsys timeout
If you have the upstream bug report for the ZFS driver (not docker itself), that would be welcome. I didn’t find anything in the upstream wiki for the driver which is shipped by default in debian/ubuntu. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1879473 Title: hundreds of snapshots created by docker zfs lead to zsys timeout To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/docker.io/+bug/1879473/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1879473] Re: hundreds of snapshots created by docker zfs lead to zsys timeout
So yeah, the driver should ideally do the same (create directly under rpool/), but it really depends on how people layed out their system. I think in general upstream should clear up their datasets (for stopped containers), which is more generic. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1879473 Title: hundreds of snapshots created by docker zfs lead to zsys timeout To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/docker.io/+bug/1879473/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1879473] Re: hundreds of snapshots created by docker zfs lead to zsys timeout
Has this been reported upstream? The fix (in postinst) is large enough that it seems like something upstream should at least be aware of, and is probably something they should implement a fix for as well. Do you think it's something that could/should be fixed in "dockerd" itself? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1879473 Title: hundreds of snapshots created by docker zfs lead to zsys timeout To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/docker.io/+bug/1879473/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1879473] Re: hundreds of snapshots created by docker zfs lead to zsys timeout
This bug was fixed in the package docker.io - 19.03.8-0ubuntu2 --- docker.io (19.03.8-0ubuntu2) groovy; urgency=medium [ Jean-Baptiste Lallement ] [ Didier Roche ] * Fix use with ZFS on root: - docker creates one dataset for any layer of containers that were created. Create now a /var/lib/docker for creating them in the persistent namespace and migrate existing one here. - purge the automated historic that was created. The migration only impacts the ubuntu desktop installation with experimental ZFS on root, and we have thus to stop and start the daemon to migrate data. (LP: #1879473) -- Didier Roche Tue, 19 May 2020 11:01:22 +0200 ** Changed in: docker.io (Ubuntu) Status: Triaged => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1879473 Title: hundreds of snapshots created by docker zfs lead to zsys timeout To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/docker.io/+bug/1879473/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1879473] Re: hundreds of snapshots created by docker zfs lead to zsys timeout
** Also affects: docker.io (Ubuntu Focal) Importance: Undecided Status: New ** Changed in: docker.io (Ubuntu Focal) Status: New => Triaged ** Changed in: docker.io (Ubuntu Focal) Importance: Undecided => High ** Changed in: docker.io (Ubuntu Focal) Assignee: (unassigned) => Didier Roche (didrocks) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1879473 Title: hundreds of snapshots created by docker zfs lead to zsys timeout To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/docker.io/+bug/1879473/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1879473] Re: hundreds of snapshots created by docker zfs lead to zsys timeout
** Description changed: [Rationale] ZFS module of docker creates many snapshots under /ROOT/machineid/var/lib/. These snapshots are processed by zsys and lead to timeout. This patch migrates the snapshots created by docker under the persistent dataset /var/lib/docker/ so they are scanned and then ignored by zsys. [Test Case] 1. On a machine with docker installed, create a docker container: $ docker run ubuntu 2. Verify that docker created snapshots under /ROOT/machineid/var/lib/ with zfs list 3. Install the new version 4. Verify that the snapshots have been migrated to /var/lib/docker/ 5. Verify that docker is still working: $ docker run ubuntu [Regression Potential] The change only affects the postinst. The risk is that dataset are not migrated or partially migrated. Then new snapshots are created on a different location and containers created or run from an existing image won't start. + + It impacts only Ubuntu Desktop installation with zfs on root support. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1879473 Title: hundreds of snapshots created by docker zfs lead to zsys timeout To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/docker.io/+bug/1879473/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1879473] Re: hundreds of snapshots created by docker zfs lead to zsys timeout
** Description changed: [Rationale] - ZFS module of docker creates many snapshots under /var/lib/. These snapshots are processed by zsys and lead to timeout. + ZFS module of docker creates many snapshots under /ROOT/machineid/var/lib/. These snapshots are processed by zsys and lead to timeout. This patch migrates the snapshots created by docker under the persistent - dataset /var/lib/docker/ so they are scanned and then ignored by zsys. + dataset /var/lib/docker/ so they are scanned and then ignored by + zsys. [Test Case] 1. On a machine with docker installed, create a docker container: $ docker run ubuntu - 2. Verify that docker created snapshots under /var/lib/ with zfs list + 2. Verify that docker created snapshots under /ROOT/machineid/var/lib/ with zfs list 3. Install the new version - 4. Verify that the snapshots have been migrated to /var/lib/docker/ + 4. Verify that the snapshots have been migrated to /var/lib/docker/ 5. Verify that docker is still working: $ docker run ubuntu [Regression Potential] - The change only affects the postinst. + The change only affects the postinst. The risk is that dataset are not migrated or partially migrated. Then new snapshots are created on a different location and containers created or run from an existing image won't start. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1879473 Title: hundreds of snapshots created by docker zfs lead to zsys timeout To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/docker.io/+bug/1879473/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
