[Kernel-packages] [Bug 1548009] Re: ZFS pools should be automatically scrubbed
This bug was fixed in the package zfs-linux - 0.6.5.6-0ubuntu12 --- zfs-linux (0.6.5.6-0ubuntu12) xenial; urgency=medium * Change to include some important tools in zfsutils-linux build. - Add tools : arcstat.py, arc_summary.py & dbufstat.py in /usr/sbin. - Change utilities path (bindir) to /usr/sbin per https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1574342/comments/6 - Add python related dependencies (LP: #1574342) zfs-linux (0.6.5.6-0ubuntu11) xenial; urgency=medium [ Richard Laager ] * Set PATH in cron.d job to fix monthly scrubs. (LP: #1548009) -- Eric Desrochers Thu, 28 Jul 2016 16:18:16 -0400 ** Changed in: zfs-linux (Ubuntu Xenial) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to zfs-linux in Ubuntu. https://bugs.launchpad.net/bugs/1548009 Title: ZFS pools should be automatically scrubbed Status in zfs-linux package in Ubuntu: Fix Released Status in zfs-linux source package in Xenial: Fix Released Bug description: [Impact] Xenial shipped with a cron job to automatically scrub ZFS pools, as desired by many users and as implemented by mdadm for traditional Linux software RAID. Unfortunately, this cron job does not work, because it needs a PATH line for /sbin, where the zpool utility lives. Given the existence of the cron job and various discussions on IRC, etc., users expect that scrubs are happening, when they are not. This means ZFS is not pre-emptively checking for (and correcting) corruption. The odds of disk corruption are admittedly very low, but violating users' expectations of data safety, especially when they've gone out of their way to use a filesystem which touts data safety, is bad. [Test Case] $ truncate -s 1G test.img $ sudo zpool create test `pwd`/test.img $ sudo zpool status test $ sudo vi /etc/cron.d/zfsutils-linux Modify /etc/cron.d/zfsutils-linux to run the cron job in a few minutes (modifying the date range if it's not currently the 8th through the 14th and the "-eq 0" check if it's not currently a Sunday). $ grep zfs /var/log/cron.log Verify in /var/log/cron.log that the job ran. $ sudo zpool status test Expected results: scan: scrub repaired 0 in ... on Actual results: scan: none requested Then, add the PATH line, update the time rules in the cron job, and repeat the test. Now it will work. - OR - The best test case is to leave the cron job file untouched, install the patched package, wait for the second Sunday of the month, and verify with zpool status that a scrub ran. I did this, on Xenial, with the package I built. The debdiff is in comment #11 and was accepted to Yakkety. If someone can get this in -proposed before the 14th, I'll gladly install the actual package from -proposed and make sure it runs correctly on the 14th. [Regression Potential] The patch only touches the cron.d file, which has only one cron job in it. This cron job is completely broken (inoperative) at the moment, so the regression potential is very low. ORIGINAL, PRE-SRU, DESCRIPTION: mdadm automatically checks MD arrays. ZFS should automatically scrub pools too. Scrubbing a pool allows ZFS to detect on-disk corruption and (when the pool has redundancy) correct it. Note that ZFS does not blindly assume the other copy is correct; it will only overwrite bad data with data that is known to be good (i.e. it passes the checksum). I've attached a debdiff which accomplishes this. It builds and installs cleanly. The meat of it is the scrub script I've been using on production systems, both servers and laptops, and recommending in my Ubuntu root- on-ZFS HOWTO, for years, which scrubs all *healthy* pools. If a pool is not healthy, scrubbing it is bad for two reasons: 1) It adds a lot of disk load which could theoretically lead to another failure. We should save that disk load for resilvering. 2) Performance is already less on a degraded pool and scrubbing can make that worse, even though scrubs are throttled. Arguably, I might be being too conservative here, but the marginal benefit of scrubbing a *degraded* pool is pretty minimal as pools should not be left degraded for very long. The cron.d in this patch scrubs on the second Sunday of the month. mdadm scrubs on the first Sunday of the month. This way, if a system has both MD and ZFS pools, the load doesn't all happen at the same time. If the system doesn't have both types, it shouldn't really matter which week. If you'd rather make it the same week as MD, I see no problem with that. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1548009/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~
[Kernel-packages] [Bug 1548009] Re: ZFS pools should be automatically scrubbed
This works for me "in the wild" (i.e. waiting until today, which was the second Sunday of the month). I had two servers, one with 0.6.5.6-0ubuntu11 and one with 0.6.5.6-0ubuntu12. ** Tags added: verification-done -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to zfs-linux in Ubuntu. https://bugs.launchpad.net/bugs/1548009 Title: ZFS pools should be automatically scrubbed Status in zfs-linux package in Ubuntu: Fix Released Status in zfs-linux source package in Xenial: Fix Committed Bug description: [Impact] Xenial shipped with a cron job to automatically scrub ZFS pools, as desired by many users and as implemented by mdadm for traditional Linux software RAID. Unfortunately, this cron job does not work, because it needs a PATH line for /sbin, where the zpool utility lives. Given the existence of the cron job and various discussions on IRC, etc., users expect that scrubs are happening, when they are not. This means ZFS is not pre-emptively checking for (and correcting) corruption. The odds of disk corruption are admittedly very low, but violating users' expectations of data safety, especially when they've gone out of their way to use a filesystem which touts data safety, is bad. [Test Case] $ truncate -s 1G test.img $ sudo zpool create test `pwd`/test.img $ sudo zpool status test $ sudo vi /etc/cron.d/zfsutils-linux Modify /etc/cron.d/zfsutils-linux to run the cron job in a few minutes (modifying the date range if it's not currently the 8th through the 14th and the "-eq 0" check if it's not currently a Sunday). $ grep zfs /var/log/cron.log Verify in /var/log/cron.log that the job ran. $ sudo zpool status test Expected results: scan: scrub repaired 0 in ... on Actual results: scan: none requested Then, add the PATH line, update the time rules in the cron job, and repeat the test. Now it will work. - OR - The best test case is to leave the cron job file untouched, install the patched package, wait for the second Sunday of the month, and verify with zpool status that a scrub ran. I did this, on Xenial, with the package I built. The debdiff is in comment #11 and was accepted to Yakkety. If someone can get this in -proposed before the 14th, I'll gladly install the actual package from -proposed and make sure it runs correctly on the 14th. [Regression Potential] The patch only touches the cron.d file, which has only one cron job in it. This cron job is completely broken (inoperative) at the moment, so the regression potential is very low. ORIGINAL, PRE-SRU, DESCRIPTION: mdadm automatically checks MD arrays. ZFS should automatically scrub pools too. Scrubbing a pool allows ZFS to detect on-disk corruption and (when the pool has redundancy) correct it. Note that ZFS does not blindly assume the other copy is correct; it will only overwrite bad data with data that is known to be good (i.e. it passes the checksum). I've attached a debdiff which accomplishes this. It builds and installs cleanly. The meat of it is the scrub script I've been using on production systems, both servers and laptops, and recommending in my Ubuntu root- on-ZFS HOWTO, for years, which scrubs all *healthy* pools. If a pool is not healthy, scrubbing it is bad for two reasons: 1) It adds a lot of disk load which could theoretically lead to another failure. We should save that disk load for resilvering. 2) Performance is already less on a degraded pool and scrubbing can make that worse, even though scrubs are throttled. Arguably, I might be being too conservative here, but the marginal benefit of scrubbing a *degraded* pool is pretty minimal as pools should not be left degraded for very long. The cron.d in this patch scrubs on the second Sunday of the month. mdadm scrubs on the first Sunday of the month. This way, if a system has both MD and ZFS pools, the load doesn't all happen at the same time. If the system doesn't have both types, it shouldn't really matter which week. If you'd rather make it the same week as MD, I see no problem with that. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1548009/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1548009] Re: ZFS pools should be automatically scrubbed
This works for me, using the first test case. If this feels good enough, feel free to change the tag yourself. Otherwise, I'll do so after verifying the unmodified cron configuration works on the 14th. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to zfs-linux in Ubuntu. https://bugs.launchpad.net/bugs/1548009 Title: ZFS pools should be automatically scrubbed Status in zfs-linux package in Ubuntu: Fix Released Status in zfs-linux source package in Xenial: Fix Committed Bug description: [Impact] Xenial shipped with a cron job to automatically scrub ZFS pools, as desired by many users and as implemented by mdadm for traditional Linux software RAID. Unfortunately, this cron job does not work, because it needs a PATH line for /sbin, where the zpool utility lives. Given the existence of the cron job and various discussions on IRC, etc., users expect that scrubs are happening, when they are not. This means ZFS is not pre-emptively checking for (and correcting) corruption. The odds of disk corruption are admittedly very low, but violating users' expectations of data safety, especially when they've gone out of their way to use a filesystem which touts data safety, is bad. [Test Case] $ truncate -s 1G test.img $ sudo zpool create test `pwd`/test.img $ sudo zpool status test $ sudo vi /etc/cron.d/zfsutils-linux Modify /etc/cron.d/zfsutils-linux to run the cron job in a few minutes (modifying the date range if it's not currently the 8th through the 14th and the "-eq 0" check if it's not currently a Sunday). $ grep zfs /var/log/cron.log Verify in /var/log/cron.log that the job ran. $ sudo zpool status test Expected results: scan: scrub repaired 0 in ... on Actual results: scan: none requested Then, add the PATH line, update the time rules in the cron job, and repeat the test. Now it will work. - OR - The best test case is to leave the cron job file untouched, install the patched package, wait for the second Sunday of the month, and verify with zpool status that a scrub ran. I did this, on Xenial, with the package I built. The debdiff is in comment #11 and was accepted to Yakkety. If someone can get this in -proposed before the 14th, I'll gladly install the actual package from -proposed and make sure it runs correctly on the 14th. [Regression Potential] The patch only touches the cron.d file, which has only one cron job in it. This cron job is completely broken (inoperative) at the moment, so the regression potential is very low. ORIGINAL, PRE-SRU, DESCRIPTION: mdadm automatically checks MD arrays. ZFS should automatically scrub pools too. Scrubbing a pool allows ZFS to detect on-disk corruption and (when the pool has redundancy) correct it. Note that ZFS does not blindly assume the other copy is correct; it will only overwrite bad data with data that is known to be good (i.e. it passes the checksum). I've attached a debdiff which accomplishes this. It builds and installs cleanly. The meat of it is the scrub script I've been using on production systems, both servers and laptops, and recommending in my Ubuntu root- on-ZFS HOWTO, for years, which scrubs all *healthy* pools. If a pool is not healthy, scrubbing it is bad for two reasons: 1) It adds a lot of disk load which could theoretically lead to another failure. We should save that disk load for resilvering. 2) Performance is already less on a degraded pool and scrubbing can make that worse, even though scrubs are throttled. Arguably, I might be being too conservative here, but the marginal benefit of scrubbing a *degraded* pool is pretty minimal as pools should not be left degraded for very long. The cron.d in this patch scrubs on the second Sunday of the month. mdadm scrubs on the first Sunday of the month. This way, if a system has both MD and ZFS pools, the load doesn't all happen at the same time. If the system doesn't have both types, it shouldn't really matter which week. If you'd rather make it the same week as MD, I see no problem with that. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1548009/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1548009] Re: ZFS pools should be automatically scrubbed
Also note that this is the exact PATH from /etc/crontab, which is thus
also the PATH under which /etc/cron.{hourly,daily,weekly,monthly} run.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to zfs-linux in Ubuntu.
https://bugs.launchpad.net/bugs/1548009
Title:
ZFS pools should be automatically scrubbed
Status in zfs-linux package in Ubuntu:
Fix Released
Status in zfs-linux source package in Xenial:
Fix Committed
Bug description:
[Impact]
Xenial shipped with a cron job to automatically scrub ZFS pools, as
desired by many users and as implemented by mdadm for traditional Linux
software RAID. Unfortunately, this cron job does not work, because it needs
a PATH line for /sbin, where the zpool utility lives.
Given the existence of the cron job and various discussions on IRC, etc.,
users expect that scrubs are happening, when they are not. This means ZFS
is not pre-emptively checking for (and correcting) corruption. The odds of
disk corruption are admittedly very low, but violating users' expectations
of data safety, especially when they've gone out of their way to use a
filesystem which touts data safety, is bad.
[Test Case]
$ truncate -s 1G test.img
$ sudo zpool create test `pwd`/test.img
$ sudo zpool status test
$ sudo vi /etc/cron.d/zfsutils-linux
Modify /etc/cron.d/zfsutils-linux to run the cron job in a few minutes
(modifying the date range if it's not currently the 8th through the 14th
and the "-eq 0" check if it's not currently a Sunday).
$ grep zfs /var/log/cron.log
Verify in /var/log/cron.log that the job ran.
$ sudo zpool status test
Expected results:
scan: scrub repaired 0 in ... on
Actual results:
scan: none requested
Then, add the PATH line, update the time rules in the cron job, and repeat
the test. Now it will work.
- OR -
The best test case is to leave the cron job file untouched, install the
patched package, wait for the second Sunday of the month, and verify with
zpool status that a scrub ran. I did this, on Xenial, with the package I
built. The debdiff is in comment #11 and was accepted to Yakkety.
If someone can get this in -proposed before the 14th, I'll gladly install
the actual package from -proposed and make sure it runs correctly on the
14th.
[Regression Potential]
The patch only touches the cron.d file, which has only one cron job in it.
This cron job is completely broken (inoperative) at the moment, so the
regression potential is very low.
ORIGINAL, PRE-SRU, DESCRIPTION:
mdadm automatically checks MD arrays. ZFS should automatically scrub
pools too. Scrubbing a pool allows ZFS to detect on-disk corruption
and (when the pool has redundancy) correct it. Note that ZFS does not
blindly assume the other copy is correct; it will only overwrite bad
data with data that is known to be good (i.e. it passes the checksum).
I've attached a debdiff which accomplishes this. It builds and
installs cleanly.
The meat of it is the scrub script I've been using on production
systems, both servers and laptops, and recommending in my Ubuntu root-
on-ZFS HOWTO, for years, which scrubs all *healthy* pools. If a pool
is not healthy, scrubbing it is bad for two reasons: 1) It adds a lot
of disk load which could theoretically lead to another failure. We
should save that disk load for resilvering. 2) Performance is already
less on a degraded pool and scrubbing can make that worse, even though
scrubs are throttled. Arguably, I might be being too conservative
here, but the marginal benefit of scrubbing a *degraded* pool is
pretty minimal as pools should not be left degraded for very long.
The cron.d in this patch scrubs on the second Sunday of the month.
mdadm scrubs on the first Sunday of the month. This way, if a system
has both MD and ZFS pools, the load doesn't all happen at the same
time. If the system doesn't have both types, it shouldn't really
matter which week. If you'd rather make it the same week as MD, I see
no problem with that.
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1548009/+subscriptions
--
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1548009] Re: ZFS pools should be automatically scrubbed
** Changed in: zfs-linux (Ubuntu Xenial) Importance: Undecided => Medium -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to zfs-linux in Ubuntu. https://bugs.launchpad.net/bugs/1548009 Title: ZFS pools should be automatically scrubbed Status in zfs-linux package in Ubuntu: Fix Released Status in zfs-linux source package in Xenial: Fix Committed Bug description: [Impact] Xenial shipped with a cron job to automatically scrub ZFS pools, as desired by many users and as implemented by mdadm for traditional Linux software RAID. Unfortunately, this cron job does not work, because it needs a PATH line for /sbin, where the zpool utility lives. Given the existence of the cron job and various discussions on IRC, etc., users expect that scrubs are happening, when they are not. This means ZFS is not pre-emptively checking for (and correcting) corruption. The odds of disk corruption are admittedly very low, but violating users' expectations of data safety, especially when they've gone out of their way to use a filesystem which touts data safety, is bad. [Test Case] $ truncate -s 1G test.img $ sudo zpool create test `pwd`/test.img $ sudo zpool status test $ sudo vi /etc/cron.d/zfsutils-linux Modify /etc/cron.d/zfsutils-linux to run the cron job in a few minutes (modifying the date range if it's not currently the 8th through the 14th and the "-eq 0" check if it's not currently a Sunday). $ grep zfs /var/log/cron.log Verify in /var/log/cron.log that the job ran. $ sudo zpool status test Expected results: scan: scrub repaired 0 in ... on Actual results: scan: none requested Then, add the PATH line, update the time rules in the cron job, and repeat the test. Now it will work. - OR - The best test case is to leave the cron job file untouched, install the patched package, wait for the second Sunday of the month, and verify with zpool status that a scrub ran. I did this, on Xenial, with the package I built. The debdiff is in comment #11 and was accepted to Yakkety. If someone can get this in -proposed before the 14th, I'll gladly install the actual package from -proposed and make sure it runs correctly on the 14th. [Regression Potential] The patch only touches the cron.d file, which has only one cron job in it. This cron job is completely broken (inoperative) at the moment, so the regression potential is very low. ORIGINAL, PRE-SRU, DESCRIPTION: mdadm automatically checks MD arrays. ZFS should automatically scrub pools too. Scrubbing a pool allows ZFS to detect on-disk corruption and (when the pool has redundancy) correct it. Note that ZFS does not blindly assume the other copy is correct; it will only overwrite bad data with data that is known to be good (i.e. it passes the checksum). I've attached a debdiff which accomplishes this. It builds and installs cleanly. The meat of it is the scrub script I've been using on production systems, both servers and laptops, and recommending in my Ubuntu root- on-ZFS HOWTO, for years, which scrubs all *healthy* pools. If a pool is not healthy, scrubbing it is bad for two reasons: 1) It adds a lot of disk load which could theoretically lead to another failure. We should save that disk load for resilvering. 2) Performance is already less on a degraded pool and scrubbing can make that worse, even though scrubs are throttled. Arguably, I might be being too conservative here, but the marginal benefit of scrubbing a *degraded* pool is pretty minimal as pools should not be left degraded for very long. The cron.d in this patch scrubs on the second Sunday of the month. mdadm scrubs on the first Sunday of the month. This way, if a system has both MD and ZFS pools, the load doesn't all happen at the same time. If the system doesn't have both types, it shouldn't really matter which week. If you'd rather make it the same week as MD, I see no problem with that. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1548009/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1548009] Re: ZFS pools should be automatically scrubbed
Could /usr/lib/zfs-linux/scrub not just give the full path to zpool instead? Then PATH wouldn't need modifying at all. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to zfs-linux in Ubuntu. https://bugs.launchpad.net/bugs/1548009 Title: ZFS pools should be automatically scrubbed Status in zfs-linux package in Ubuntu: Fix Released Status in zfs-linux source package in Xenial: Fix Committed Bug description: [Impact] Xenial shipped with a cron job to automatically scrub ZFS pools, as desired by many users and as implemented by mdadm for traditional Linux software RAID. Unfortunately, this cron job does not work, because it needs a PATH line for /sbin, where the zpool utility lives. Given the existence of the cron job and various discussions on IRC, etc., users expect that scrubs are happening, when they are not. This means ZFS is not pre-emptively checking for (and correcting) corruption. The odds of disk corruption are admittedly very low, but violating users' expectations of data safety, especially when they've gone out of their way to use a filesystem which touts data safety, is bad. [Test Case] $ truncate -s 1G test.img $ sudo zpool create test `pwd`/test.img $ sudo zpool status test $ sudo vi /etc/cron.d/zfsutils-linux Modify /etc/cron.d/zfsutils-linux to run the cron job in a few minutes (modifying the date range if it's not currently the 8th through the 14th and the "-eq 0" check if it's not currently a Sunday). $ grep zfs /var/log/cron.log Verify in /var/log/cron.log that the job ran. $ sudo zpool status test Expected results: scan: scrub repaired 0 in ... on Actual results: scan: none requested Then, add the PATH line, update the time rules in the cron job, and repeat the test. Now it will work. - OR - The best test case is to leave the cron job file untouched, install the patched package, wait for the second Sunday of the month, and verify with zpool status that a scrub ran. I did this, on Xenial, with the package I built. The debdiff is in comment #11 and was accepted to Yakkety. If someone can get this in -proposed before the 14th, I'll gladly install the actual package from -proposed and make sure it runs correctly on the 14th. [Regression Potential] The patch only touches the cron.d file, which has only one cron job in it. This cron job is completely broken (inoperative) at the moment, so the regression potential is very low. ORIGINAL, PRE-SRU, DESCRIPTION: mdadm automatically checks MD arrays. ZFS should automatically scrub pools too. Scrubbing a pool allows ZFS to detect on-disk corruption and (when the pool has redundancy) correct it. Note that ZFS does not blindly assume the other copy is correct; it will only overwrite bad data with data that is known to be good (i.e. it passes the checksum). I've attached a debdiff which accomplishes this. It builds and installs cleanly. The meat of it is the scrub script I've been using on production systems, both servers and laptops, and recommending in my Ubuntu root- on-ZFS HOWTO, for years, which scrubs all *healthy* pools. If a pool is not healthy, scrubbing it is bad for two reasons: 1) It adds a lot of disk load which could theoretically lead to another failure. We should save that disk load for resilvering. 2) Performance is already less on a degraded pool and scrubbing can make that worse, even though scrubs are throttled. Arguably, I might be being too conservative here, but the marginal benefit of scrubbing a *degraded* pool is pretty minimal as pools should not be left degraded for very long. The cron.d in this patch scrubs on the second Sunday of the month. mdadm scrubs on the first Sunday of the month. This way, if a system has both MD and ZFS pools, the load doesn't all happen at the same time. If the system doesn't have both types, it shouldn't really matter which week. If you'd rather make it the same week as MD, I see no problem with that. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1548009/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1548009] Re: ZFS pools should be automatically scrubbed
Hello Richard, or anyone else affected, Accepted zfs-linux into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/zfs- linux/0.6.5.6-0ubuntu11 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance! ** Changed in: zfs-linux (Ubuntu Xenial) Status: In Progress => Fix Committed -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to zfs-linux in Ubuntu. https://bugs.launchpad.net/bugs/1548009 Title: ZFS pools should be automatically scrubbed Status in zfs-linux package in Ubuntu: Fix Released Status in zfs-linux source package in Xenial: Fix Committed Bug description: [Impact] Xenial shipped with a cron job to automatically scrub ZFS pools, as desired by many users and as implemented by mdadm for traditional Linux software RAID. Unfortunately, this cron job does not work, because it needs a PATH line for /sbin, where the zpool utility lives. Given the existence of the cron job and various discussions on IRC, etc., users expect that scrubs are happening, when they are not. This means ZFS is not pre-emptively checking for (and correcting) corruption. The odds of disk corruption are admittedly very low, but violating users' expectations of data safety, especially when they've gone out of their way to use a filesystem which touts data safety, is bad. [Test Case] $ truncate -s 1G test.img $ sudo zpool create test `pwd`/test.img $ sudo zpool status test $ sudo vi /etc/cron.d/zfsutils-linux Modify /etc/cron.d/zfsutils-linux to run the cron job in a few minutes (modifying the date range if it's not currently the 8th through the 14th and the "-eq 0" check if it's not currently a Sunday). $ grep zfs /var/log/cron.log Verify in /var/log/cron.log that the job ran. $ sudo zpool status test Expected results: scan: scrub repaired 0 in ... on Actual results: scan: none requested Then, add the PATH line, update the time rules in the cron job, and repeat the test. Now it will work. - OR - The best test case is to leave the cron job file untouched, install the patched package, wait for the second Sunday of the month, and verify with zpool status that a scrub ran. I did this, on Xenial, with the package I built. The debdiff is in comment #11 and was accepted to Yakkety. If someone can get this in -proposed before the 14th, I'll gladly install the actual package from -proposed and make sure it runs correctly on the 14th. [Regression Potential] The patch only touches the cron.d file, which has only one cron job in it. This cron job is completely broken (inoperative) at the moment, so the regression potential is very low. ORIGINAL, PRE-SRU, DESCRIPTION: mdadm automatically checks MD arrays. ZFS should automatically scrub pools too. Scrubbing a pool allows ZFS to detect on-disk corruption and (when the pool has redundancy) correct it. Note that ZFS does not blindly assume the other copy is correct; it will only overwrite bad data with data that is known to be good (i.e. it passes the checksum). I've attached a debdiff which accomplishes this. It builds and installs cleanly. The meat of it is the scrub script I've been using on production systems, both servers and laptops, and recommending in my Ubuntu root- on-ZFS HOWTO, for years, which scrubs all *healthy* pools. If a pool is not healthy, scrubbing it is bad for two reasons: 1) It adds a lot of disk load which could theoretically lead to another failure. We should save that disk load for resilvering. 2) Performance is already less on a degraded pool and scrubbing can make that worse, even though scrubs are throttled. Arguably, I might be being too conservative here, but the marginal benefit of scrubbing a *degraded* pool is pretty minimal as pools should not be left degraded for very long. The cron.d in this patch scrubs on the second Sunday of the month. mdadm scrubs on the first Sunday of the month. This way, if a system has both MD and ZFS pools, the load doesn't all happen at the same time. If the system doesn't have both types, it shouldn't really matter which
[Kernel-packages] [Bug 1548009] Re: ZFS pools should be automatically scrubbed
Ok, comparing this to other users of /etc/cron.d it does seem to be a common idiom used there. To update the path to exactly the above form. So this is not making things any worse. I am inclined to let this go on that basis given it is already in yakkety. /etc/cron.d/anacron:PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin /etc/cron.d/certbot:PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin /etc/cron.d/popularity-contest:PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin ** Changed in: zfs-linux (Ubuntu Xenial) Status: Incomplete => In Progress -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to zfs-linux in Ubuntu. https://bugs.launchpad.net/bugs/1548009 Title: ZFS pools should be automatically scrubbed Status in zfs-linux package in Ubuntu: Fix Released Status in zfs-linux source package in Xenial: Fix Committed Bug description: [Impact] Xenial shipped with a cron job to automatically scrub ZFS pools, as desired by many users and as implemented by mdadm for traditional Linux software RAID. Unfortunately, this cron job does not work, because it needs a PATH line for /sbin, where the zpool utility lives. Given the existence of the cron job and various discussions on IRC, etc., users expect that scrubs are happening, when they are not. This means ZFS is not pre-emptively checking for (and correcting) corruption. The odds of disk corruption are admittedly very low, but violating users' expectations of data safety, especially when they've gone out of their way to use a filesystem which touts data safety, is bad. [Test Case] $ truncate -s 1G test.img $ sudo zpool create test `pwd`/test.img $ sudo zpool status test $ sudo vi /etc/cron.d/zfsutils-linux Modify /etc/cron.d/zfsutils-linux to run the cron job in a few minutes (modifying the date range if it's not currently the 8th through the 14th and the "-eq 0" check if it's not currently a Sunday). $ grep zfs /var/log/cron.log Verify in /var/log/cron.log that the job ran. $ sudo zpool status test Expected results: scan: scrub repaired 0 in ... on Actual results: scan: none requested Then, add the PATH line, update the time rules in the cron job, and repeat the test. Now it will work. - OR - The best test case is to leave the cron job file untouched, install the patched package, wait for the second Sunday of the month, and verify with zpool status that a scrub ran. I did this, on Xenial, with the package I built. The debdiff is in comment #11 and was accepted to Yakkety. If someone can get this in -proposed before the 14th, I'll gladly install the actual package from -proposed and make sure it runs correctly on the 14th. [Regression Potential] The patch only touches the cron.d file, which has only one cron job in it. This cron job is completely broken (inoperative) at the moment, so the regression potential is very low. ORIGINAL, PRE-SRU, DESCRIPTION: mdadm automatically checks MD arrays. ZFS should automatically scrub pools too. Scrubbing a pool allows ZFS to detect on-disk corruption and (when the pool has redundancy) correct it. Note that ZFS does not blindly assume the other copy is correct; it will only overwrite bad data with data that is known to be good (i.e. it passes the checksum). I've attached a debdiff which accomplishes this. It builds and installs cleanly. The meat of it is the scrub script I've been using on production systems, both servers and laptops, and recommending in my Ubuntu root- on-ZFS HOWTO, for years, which scrubs all *healthy* pools. If a pool is not healthy, scrubbing it is bad for two reasons: 1) It adds a lot of disk load which could theoretically lead to another failure. We should save that disk load for resilvering. 2) Performance is already less on a degraded pool and scrubbing can make that worse, even though scrubs are throttled. Arguably, I might be being too conservative here, but the marginal benefit of scrubbing a *degraded* pool is pretty minimal as pools should not be left degraded for very long. The cron.d in this patch scrubs on the second Sunday of the month. mdadm scrubs on the first Sunday of the month. This way, if a system has both MD and ZFS pools, the load doesn't all happen at the same time. If the system doesn't have both types, it shouldn't really matter which week. If you'd rather make it the same week as MD, I see no problem with that. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1548009/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1548009] Re: ZFS pools should be automatically scrubbed
I get not wanting unnecessary PATHs... But is /usr/local really considered a security issue? It's in /usr... It's part of the standard PATH for the root user. And if an admin has installed tools in /usr/local, wouldn't they expect them to be used? -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to zfs-linux in Ubuntu. https://bugs.launchpad.net/bugs/1548009 Title: ZFS pools should be automatically scrubbed Status in zfs-linux package in Ubuntu: Fix Released Status in zfs-linux source package in Xenial: Incomplete Bug description: [Impact] Xenial shipped with a cron job to automatically scrub ZFS pools, as desired by many users and as implemented by mdadm for traditional Linux software RAID. Unfortunately, this cron job does not work, because it needs a PATH line for /sbin, where the zpool utility lives. Given the existence of the cron job and various discussions on IRC, etc., users expect that scrubs are happening, when they are not. This means ZFS is not pre-emptively checking for (and correcting) corruption. The odds of disk corruption are admittedly very low, but violating users' expectations of data safety, especially when they've gone out of their way to use a filesystem which touts data safety, is bad. [Test Case] $ truncate -s 1G test.img $ sudo zpool create test `pwd`/test.img $ sudo zpool status test $ sudo vi /etc/cron.d/zfsutils-linux Modify /etc/cron.d/zfsutils-linux to run the cron job in a few minutes (modifying the date range if it's not currently the 8th through the 14th and the "-eq 0" check if it's not currently a Sunday). $ grep zfs /var/log/cron.log Verify in /var/log/cron.log that the job ran. $ sudo zpool status test Expected results: scan: scrub repaired 0 in ... on Actual results: scan: none requested Then, add the PATH line, update the time rules in the cron job, and repeat the test. Now it will work. - OR - The best test case is to leave the cron job file untouched, install the patched package, wait for the second Sunday of the month, and verify with zpool status that a scrub ran. I did this, on Xenial, with the package I built. The debdiff is in comment #11 and was accepted to Yakkety. If someone can get this in -proposed before the 14th, I'll gladly install the actual package from -proposed and make sure it runs correctly on the 14th. [Regression Potential] The patch only touches the cron.d file, which has only one cron job in it. This cron job is completely broken (inoperative) at the moment, so the regression potential is very low. ORIGINAL, PRE-SRU, DESCRIPTION: mdadm automatically checks MD arrays. ZFS should automatically scrub pools too. Scrubbing a pool allows ZFS to detect on-disk corruption and (when the pool has redundancy) correct it. Note that ZFS does not blindly assume the other copy is correct; it will only overwrite bad data with data that is known to be good (i.e. it passes the checksum). I've attached a debdiff which accomplishes this. It builds and installs cleanly. The meat of it is the scrub script I've been using on production systems, both servers and laptops, and recommending in my Ubuntu root- on-ZFS HOWTO, for years, which scrubs all *healthy* pools. If a pool is not healthy, scrubbing it is bad for two reasons: 1) It adds a lot of disk load which could theoretically lead to another failure. We should save that disk load for resilvering. 2) Performance is already less on a degraded pool and scrubbing can make that worse, even though scrubs are throttled. Arguably, I might be being too conservative here, but the marginal benefit of scrubbing a *degraded* pool is pretty minimal as pools should not be left degraded for very long. The cron.d in this patch scrubs on the second Sunday of the month. mdadm scrubs on the first Sunday of the month. This way, if a system has both MD and ZFS pools, the load doesn't all happen at the same time. If the system doesn't have both types, it shouldn't really matter which week. If you'd rather make it the same week as MD, I see no problem with that. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1548009/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1548009] Re: ZFS pools should be automatically scrubbed
Would it not make more sense to have /usr/lib/zfs-linux/scrub set the PATH that it requires in order to find the zpool command? -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to zfs-linux in Ubuntu. https://bugs.launchpad.net/bugs/1548009 Title: ZFS pools should be automatically scrubbed Status in zfs-linux package in Ubuntu: Fix Released Status in zfs-linux source package in Xenial: Incomplete Bug description: [Impact] Xenial shipped with a cron job to automatically scrub ZFS pools, as desired by many users and as implemented by mdadm for traditional Linux software RAID. Unfortunately, this cron job does not work, because it needs a PATH line for /sbin, where the zpool utility lives. Given the existence of the cron job and various discussions on IRC, etc., users expect that scrubs are happening, when they are not. This means ZFS is not pre-emptively checking for (and correcting) corruption. The odds of disk corruption are admittedly very low, but violating users' expectations of data safety, especially when they've gone out of their way to use a filesystem which touts data safety, is bad. [Test Case] $ truncate -s 1G test.img $ sudo zpool create test `pwd`/test.img $ sudo zpool status test $ sudo vi /etc/cron.d/zfsutils-linux Modify /etc/cron.d/zfsutils-linux to run the cron job in a few minutes (modifying the date range if it's not currently the 8th through the 14th and the "-eq 0" check if it's not currently a Sunday). $ grep zfs /var/log/cron.log Verify in /var/log/cron.log that the job ran. $ sudo zpool status test Expected results: scan: scrub repaired 0 in ... on Actual results: scan: none requested Then, add the PATH line, update the time rules in the cron job, and repeat the test. Now it will work. - OR - The best test case is to leave the cron job file untouched, install the patched package, wait for the second Sunday of the month, and verify with zpool status that a scrub ran. I did this, on Xenial, with the package I built. The debdiff is in comment #11 and was accepted to Yakkety. If someone can get this in -proposed before the 14th, I'll gladly install the actual package from -proposed and make sure it runs correctly on the 14th. [Regression Potential] The patch only touches the cron.d file, which has only one cron job in it. This cron job is completely broken (inoperative) at the moment, so the regression potential is very low. ORIGINAL, PRE-SRU, DESCRIPTION: mdadm automatically checks MD arrays. ZFS should automatically scrub pools too. Scrubbing a pool allows ZFS to detect on-disk corruption and (when the pool has redundancy) correct it. Note that ZFS does not blindly assume the other copy is correct; it will only overwrite bad data with data that is known to be good (i.e. it passes the checksum). I've attached a debdiff which accomplishes this. It builds and installs cleanly. The meat of it is the scrub script I've been using on production systems, both servers and laptops, and recommending in my Ubuntu root- on-ZFS HOWTO, for years, which scrubs all *healthy* pools. If a pool is not healthy, scrubbing it is bad for two reasons: 1) It adds a lot of disk load which could theoretically lead to another failure. We should save that disk load for resilvering. 2) Performance is already less on a degraded pool and scrubbing can make that worse, even though scrubs are throttled. Arguably, I might be being too conservative here, but the marginal benefit of scrubbing a *degraded* pool is pretty minimal as pools should not be left degraded for very long. The cron.d in this patch scrubs on the second Sunday of the month. mdadm scrubs on the first Sunday of the month. This way, if a system has both MD and ZFS pools, the load doesn't all happen at the same time. If the system doesn't have both types, it shouldn't really matter which week. If you'd rather make it the same week as MD, I see no problem with that. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1548009/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1548009] Re: ZFS pools should be automatically scrubbed
/usr/local/bin and /usr/local/sbin are not required for anything zfs specific, so given that they are redundant they should not be added to the PATH. Plus the security issue as mentioned above. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to zfs-linux in Ubuntu. https://bugs.launchpad.net/bugs/1548009 Title: ZFS pools should be automatically scrubbed Status in zfs-linux package in Ubuntu: Fix Released Status in zfs-linux source package in Xenial: Incomplete Bug description: [Impact] Xenial shipped with a cron job to automatically scrub ZFS pools, as desired by many users and as implemented by mdadm for traditional Linux software RAID. Unfortunately, this cron job does not work, because it needs a PATH line for /sbin, where the zpool utility lives. Given the existence of the cron job and various discussions on IRC, etc., users expect that scrubs are happening, when they are not. This means ZFS is not pre-emptively checking for (and correcting) corruption. The odds of disk corruption are admittedly very low, but violating users' expectations of data safety, especially when they've gone out of their way to use a filesystem which touts data safety, is bad. [Test Case] $ truncate -s 1G test.img $ sudo zpool create test `pwd`/test.img $ sudo zpool status test $ sudo vi /etc/cron.d/zfsutils-linux Modify /etc/cron.d/zfsutils-linux to run the cron job in a few minutes (modifying the date range if it's not currently the 8th through the 14th and the "-eq 0" check if it's not currently a Sunday). $ grep zfs /var/log/cron.log Verify in /var/log/cron.log that the job ran. $ sudo zpool status test Expected results: scan: scrub repaired 0 in ... on Actual results: scan: none requested Then, add the PATH line, update the time rules in the cron job, and repeat the test. Now it will work. - OR - The best test case is to leave the cron job file untouched, install the patched package, wait for the second Sunday of the month, and verify with zpool status that a scrub ran. I did this, on Xenial, with the package I built. The debdiff is in comment #11 and was accepted to Yakkety. If someone can get this in -proposed before the 14th, I'll gladly install the actual package from -proposed and make sure it runs correctly on the 14th. [Regression Potential] The patch only touches the cron.d file, which has only one cron job in it. This cron job is completely broken (inoperative) at the moment, so the regression potential is very low. ORIGINAL, PRE-SRU, DESCRIPTION: mdadm automatically checks MD arrays. ZFS should automatically scrub pools too. Scrubbing a pool allows ZFS to detect on-disk corruption and (when the pool has redundancy) correct it. Note that ZFS does not blindly assume the other copy is correct; it will only overwrite bad data with data that is known to be good (i.e. it passes the checksum). I've attached a debdiff which accomplishes this. It builds and installs cleanly. The meat of it is the scrub script I've been using on production systems, both servers and laptops, and recommending in my Ubuntu root- on-ZFS HOWTO, for years, which scrubs all *healthy* pools. If a pool is not healthy, scrubbing it is bad for two reasons: 1) It adds a lot of disk load which could theoretically lead to another failure. We should save that disk load for resilvering. 2) Performance is already less on a degraded pool and scrubbing can make that worse, even though scrubs are throttled. Arguably, I might be being too conservative here, but the marginal benefit of scrubbing a *degraded* pool is pretty minimal as pools should not be left degraded for very long. The cron.d in this patch scrubs on the second Sunday of the month. mdadm scrubs on the first Sunday of the month. This way, if a system has both MD and ZFS pools, the load doesn't all happen at the same time. If the system doesn't have both types, it shouldn't really matter which week. If you'd rather make it the same week as MD, I see no problem with that. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1548009/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1548009] Re: ZFS pools should be automatically scrubbed
This should be rejected for Xenial and reverted for Yakkety and a more minimal path used. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to zfs-linux in Ubuntu. https://bugs.launchpad.net/bugs/1548009 Title: ZFS pools should be automatically scrubbed Status in zfs-linux package in Ubuntu: Fix Released Status in zfs-linux source package in Xenial: Incomplete Bug description: [Impact] Xenial shipped with a cron job to automatically scrub ZFS pools, as desired by many users and as implemented by mdadm for traditional Linux software RAID. Unfortunately, this cron job does not work, because it needs a PATH line for /sbin, where the zpool utility lives. Given the existence of the cron job and various discussions on IRC, etc., users expect that scrubs are happening, when they are not. This means ZFS is not pre-emptively checking for (and correcting) corruption. The odds of disk corruption are admittedly very low, but violating users' expectations of data safety, especially when they've gone out of their way to use a filesystem which touts data safety, is bad. [Test Case] $ truncate -s 1G test.img $ sudo zpool create test `pwd`/test.img $ sudo zpool status test $ sudo vi /etc/cron.d/zfsutils-linux Modify /etc/cron.d/zfsutils-linux to run the cron job in a few minutes (modifying the date range if it's not currently the 8th through the 14th and the "-eq 0" check if it's not currently a Sunday). $ grep zfs /var/log/cron.log Verify in /var/log/cron.log that the job ran. $ sudo zpool status test Expected results: scan: scrub repaired 0 in ... on Actual results: scan: none requested Then, add the PATH line, update the time rules in the cron job, and repeat the test. Now it will work. - OR - The best test case is to leave the cron job file untouched, install the patched package, wait for the second Sunday of the month, and verify with zpool status that a scrub ran. I did this, on Xenial, with the package I built. The debdiff is in comment #11 and was accepted to Yakkety. If someone can get this in -proposed before the 14th, I'll gladly install the actual package from -proposed and make sure it runs correctly on the 14th. [Regression Potential] The patch only touches the cron.d file, which has only one cron job in it. This cron job is completely broken (inoperative) at the moment, so the regression potential is very low. ORIGINAL, PRE-SRU, DESCRIPTION: mdadm automatically checks MD arrays. ZFS should automatically scrub pools too. Scrubbing a pool allows ZFS to detect on-disk corruption and (when the pool has redundancy) correct it. Note that ZFS does not blindly assume the other copy is correct; it will only overwrite bad data with data that is known to be good (i.e. it passes the checksum). I've attached a debdiff which accomplishes this. It builds and installs cleanly. The meat of it is the scrub script I've been using on production systems, both servers and laptops, and recommending in my Ubuntu root- on-ZFS HOWTO, for years, which scrubs all *healthy* pools. If a pool is not healthy, scrubbing it is bad for two reasons: 1) It adds a lot of disk load which could theoretically lead to another failure. We should save that disk load for resilvering. 2) Performance is already less on a degraded pool and scrubbing can make that worse, even though scrubs are throttled. Arguably, I might be being too conservative here, but the marginal benefit of scrubbing a *degraded* pool is pretty minimal as pools should not be left degraded for very long. The cron.d in this patch scrubs on the second Sunday of the month. mdadm scrubs on the first Sunday of the month. This way, if a system has both MD and ZFS pools, the load doesn't all happen at the same time. If the system doesn't have both types, it shouldn't really matter which week. If you'd rather make it the same week as MD, I see no problem with that. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1548009/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1548009] Re: ZFS pools should be automatically scrubbed
This patch adds /usr/local/bin and /usr/local/sbin to the path. Given the utilities in use are all official binaries this seems unexpected and a potential security risk. ** Changed in: zfs-linux (Ubuntu Xenial) Status: New => Incomplete -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to zfs-linux in Ubuntu. https://bugs.launchpad.net/bugs/1548009 Title: ZFS pools should be automatically scrubbed Status in zfs-linux package in Ubuntu: Fix Released Status in zfs-linux source package in Xenial: Incomplete Bug description: [Impact] Xenial shipped with a cron job to automatically scrub ZFS pools, as desired by many users and as implemented by mdadm for traditional Linux software RAID. Unfortunately, this cron job does not work, because it needs a PATH line for /sbin, where the zpool utility lives. Given the existence of the cron job and various discussions on IRC, etc., users expect that scrubs are happening, when they are not. This means ZFS is not pre-emptively checking for (and correcting) corruption. The odds of disk corruption are admittedly very low, but violating users' expectations of data safety, especially when they've gone out of their way to use a filesystem which touts data safety, is bad. [Test Case] $ truncate -s 1G test.img $ sudo zpool create test `pwd`/test.img $ sudo zpool status test $ sudo vi /etc/cron.d/zfsutils-linux Modify /etc/cron.d/zfsutils-linux to run the cron job in a few minutes (modifying the date range if it's not currently the 8th through the 14th and the "-eq 0" check if it's not currently a Sunday). $ grep zfs /var/log/cron.log Verify in /var/log/cron.log that the job ran. $ sudo zpool status test Expected results: scan: scrub repaired 0 in ... on Actual results: scan: none requested Then, add the PATH line, update the time rules in the cron job, and repeat the test. Now it will work. - OR - The best test case is to leave the cron job file untouched, install the patched package, wait for the second Sunday of the month, and verify with zpool status that a scrub ran. I did this, on Xenial, with the package I built. The debdiff is in comment #11 and was accepted to Yakkety. If someone can get this in -proposed before the 14th, I'll gladly install the actual package from -proposed and make sure it runs correctly on the 14th. [Regression Potential] The patch only touches the cron.d file, which has only one cron job in it. This cron job is completely broken (inoperative) at the moment, so the regression potential is very low. ORIGINAL, PRE-SRU, DESCRIPTION: mdadm automatically checks MD arrays. ZFS should automatically scrub pools too. Scrubbing a pool allows ZFS to detect on-disk corruption and (when the pool has redundancy) correct it. Note that ZFS does not blindly assume the other copy is correct; it will only overwrite bad data with data that is known to be good (i.e. it passes the checksum). I've attached a debdiff which accomplishes this. It builds and installs cleanly. The meat of it is the scrub script I've been using on production systems, both servers and laptops, and recommending in my Ubuntu root- on-ZFS HOWTO, for years, which scrubs all *healthy* pools. If a pool is not healthy, scrubbing it is bad for two reasons: 1) It adds a lot of disk load which could theoretically lead to another failure. We should save that disk load for resilvering. 2) Performance is already less on a degraded pool and scrubbing can make that worse, even though scrubs are throttled. Arguably, I might be being too conservative here, but the marginal benefit of scrubbing a *degraded* pool is pretty minimal as pools should not be left degraded for very long. The cron.d in this patch scrubs on the second Sunday of the month. mdadm scrubs on the first Sunday of the month. This way, if a system has both MD and ZFS pools, the load doesn't all happen at the same time. If the system doesn't have both types, it shouldn't really matter which week. If you'd rather make it the same week as MD, I see no problem with that. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1548009/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1548009] Re: ZFS pools should be automatically scrubbed
I've uploaded the patch to xenial-proposed too. Thanks! -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to zfs-linux in Ubuntu. https://bugs.launchpad.net/bugs/1548009 Title: ZFS pools should be automatically scrubbed Status in zfs-linux package in Ubuntu: Fix Released Status in zfs-linux source package in Xenial: New Bug description: [Impact] Xenial shipped with a cron job to automatically scrub ZFS pools, as desired by many users and as implemented by mdadm for traditional Linux software RAID. Unfortunately, this cron job does not work, because it needs a PATH line for /sbin, where the zpool utility lives. Given the existence of the cron job and various discussions on IRC, etc., users expect that scrubs are happening, when they are not. This means ZFS is not pre-emptively checking for (and correcting) corruption. The odds of disk corruption are admittedly very low, but violating users' expectations of data safety, especially when they've gone out of their way to use a filesystem which touts data safety, is bad. [Test Case] $ truncate -s 1G test.img $ sudo zpool create test `pwd`/test.img $ sudo zpool status test $ sudo vi /etc/cron.d/zfsutils-linux Modify /etc/cron.d/zfsutils-linux to run the cron job in a few minutes (modifying the date range if it's not currently the 8th through the 14th and the "-eq 0" check if it's not currently a Sunday). $ grep zfs /var/log/cron.log Verify in /var/log/cron.log that the job ran. $ sudo zpool status test Expected results: scan: scrub repaired 0 in ... on Actual results: scan: none requested Then, add the PATH line, update the time rules in the cron job, and repeat the test. Now it will work. - OR - The best test case is to leave the cron job file untouched, install the patched package, wait for the second Sunday of the month, and verify with zpool status that a scrub ran. I did this, on Xenial, with the package I built. The debdiff is in comment #11 and was accepted to Yakkety. If someone can get this in -proposed before the 14th, I'll gladly install the actual package from -proposed and make sure it runs correctly on the 14th. [Regression Potential] The patch only touches the cron.d file, which has only one cron job in it. This cron job is completely broken (inoperative) at the moment, so the regression potential is very low. ORIGINAL, PRE-SRU, DESCRIPTION: mdadm automatically checks MD arrays. ZFS should automatically scrub pools too. Scrubbing a pool allows ZFS to detect on-disk corruption and (when the pool has redundancy) correct it. Note that ZFS does not blindly assume the other copy is correct; it will only overwrite bad data with data that is known to be good (i.e. it passes the checksum). I've attached a debdiff which accomplishes this. It builds and installs cleanly. The meat of it is the scrub script I've been using on production systems, both servers and laptops, and recommending in my Ubuntu root- on-ZFS HOWTO, for years, which scrubs all *healthy* pools. If a pool is not healthy, scrubbing it is bad for two reasons: 1) It adds a lot of disk load which could theoretically lead to another failure. We should save that disk load for resilvering. 2) Performance is already less on a degraded pool and scrubbing can make that worse, even though scrubs are throttled. Arguably, I might be being too conservative here, but the marginal benefit of scrubbing a *degraded* pool is pretty minimal as pools should not be left degraded for very long. The cron.d in this patch scrubs on the second Sunday of the month. mdadm scrubs on the first Sunday of the month. This way, if a system has both MD and ZFS pools, the load doesn't all happen at the same time. If the system doesn't have both types, it shouldn't really matter which week. If you'd rather make it the same week as MD, I see no problem with that. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1548009/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1548009] Re: ZFS pools should be automatically scrubbed
** Also affects: zfs-linux (Ubuntu Xenial) Importance: Undecided Status: New -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to zfs-linux in Ubuntu. https://bugs.launchpad.net/bugs/1548009 Title: ZFS pools should be automatically scrubbed Status in zfs-linux package in Ubuntu: Fix Released Status in zfs-linux source package in Xenial: New Bug description: [Impact] Xenial shipped with a cron job to automatically scrub ZFS pools, as desired by many users and as implemented by mdadm for traditional Linux software RAID. Unfortunately, this cron job does not work, because it needs a PATH line for /sbin, where the zpool utility lives. Given the existence of the cron job and various discussions on IRC, etc., users expect that scrubs are happening, when they are not. This means ZFS is not pre-emptively checking for (and correcting) corruption. The odds of disk corruption are admittedly very low, but violating users' expectations of data safety, especially when they've gone out of their way to use a filesystem which touts data safety, is bad. [Test Case] $ truncate -s 1G test.img $ sudo zpool create test `pwd`/test.img $ sudo zpool status test $ sudo vi /etc/cron.d/zfsutils-linux Modify /etc/cron.d/zfsutils-linux to run the cron job in a few minutes (modifying the date range if it's not currently the 8th through the 14th and the "-eq 0" check if it's not currently a Sunday). $ grep zfs /var/log/cron.log Verify in /var/log/cron.log that the job ran. $ sudo zpool status test Expected results: scan: scrub repaired 0 in ... on Actual results: scan: none requested Then, add the PATH line, update the time rules in the cron job, and repeat the test. Now it will work. - OR - The best test case is to leave the cron job file untouched, install the patched package, wait for the second Sunday of the month, and verify with zpool status that a scrub ran. I did this, on Xenial, with the package I built. The debdiff is in comment #11 and was accepted to Yakkety. If someone can get this in -proposed before the 14th, I'll gladly install the actual package from -proposed and make sure it runs correctly on the 14th. [Regression Potential] The patch only touches the cron.d file, which has only one cron job in it. This cron job is completely broken (inoperative) at the moment, so the regression potential is very low. ORIGINAL, PRE-SRU, DESCRIPTION: mdadm automatically checks MD arrays. ZFS should automatically scrub pools too. Scrubbing a pool allows ZFS to detect on-disk corruption and (when the pool has redundancy) correct it. Note that ZFS does not blindly assume the other copy is correct; it will only overwrite bad data with data that is known to be good (i.e. it passes the checksum). I've attached a debdiff which accomplishes this. It builds and installs cleanly. The meat of it is the scrub script I've been using on production systems, both servers and laptops, and recommending in my Ubuntu root- on-ZFS HOWTO, for years, which scrubs all *healthy* pools. If a pool is not healthy, scrubbing it is bad for two reasons: 1) It adds a lot of disk load which could theoretically lead to another failure. We should save that disk load for resilvering. 2) Performance is already less on a degraded pool and scrubbing can make that worse, even though scrubs are throttled. Arguably, I might be being too conservative here, but the marginal benefit of scrubbing a *degraded* pool is pretty minimal as pools should not be left degraded for very long. The cron.d in this patch scrubs on the second Sunday of the month. mdadm scrubs on the first Sunday of the month. This way, if a system has both MD and ZFS pools, the load doesn't all happen at the same time. If the system doesn't have both types, it shouldn't really matter which week. If you'd rather make it the same week as MD, I see no problem with that. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1548009/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1548009] Re: ZFS pools should be automatically scrubbed
** Description changed: + [Impact] + + Xenial shipped with a cron job to automatically scrub ZFS pools, as + desired by many users and as implemented by mdadm for traditional Linux + software RAID. Unfortunately, this cron job does not work, because it needs a PATH line for /sbin, where the zpool utility lives. + + Given the existence of the cron job and various discussions on IRC, etc., + users expect that scrubs are happening, when they are not. This means ZFS + is not pre-emptively checking for (and correcting) corruption. The odds of + disk corruption are admittedly very low, but violating users' expectations + of data safety, especially when they've gone out of their way to use a + filesystem which touts data safety, is bad. + + [Test Case] + + $ truncate -s 1G test.img + $ sudo zpool create test `pwd`/test.img + $ sudo zpool status test + + $ sudo vi /etc/cron.d/zfsutils-linux + Modify /etc/cron.d/zfsutils-linux to run the cron job in a few minutes + (modifying the date range if it's not currently the 8th through the 14th + and the "-eq 0" check if it's not currently a Sunday). + + $ grep zfs /var/log/cron.log + Verify in /var/log/cron.log that the job ran. + + $ sudo zpool status test + + Expected results: + scan: scrub repaired 0 in ... on + + Actual results: + scan: none requested + + Then, add the PATH line, update the time rules in the cron job, and repeat + the test. Now it will work. + + - OR - + + The best test case is to leave the cron job file untouched, install the + patched package, wait for the second Sunday of the month, and verify with + zpool status that a scrub ran. I did this, on Xenial, with the package I + built. The debdiff is in comment #11 and was accepted to Yakkety. + + If someone can get this in -proposed before the 14th, I'll gladly install + the actual package from -proposed and make sure it runs correctly on the + 14th. + + [Regression Potential] + + The patch only touches the cron.d file, which has only one cron job in it. + This cron job is completely broken (inoperative) at the moment, so the + regression potential is very low. + + + + ORIGINAL, PRE-SRU, DESCRIPTION: + mdadm automatically checks MD arrays. ZFS should automatically scrub pools too. Scrubbing a pool allows ZFS to detect on-disk corruption and (when the pool has redundancy) correct it. Note that ZFS does not blindly assume the other copy is correct; it will only overwrite bad data with data that is known to be good (i.e. it passes the checksum). I've attached a debdiff which accomplishes this. It builds and installs cleanly. The meat of it is the scrub script I've been using on production systems, both servers and laptops, and recommending in my Ubuntu root- on-ZFS HOWTO, for years, which scrubs all *healthy* pools. If a pool is not healthy, scrubbing it is bad for two reasons: 1) It adds a lot of disk load which could theoretically lead to another failure. We should save that disk load for resilvering. 2) Performance is already less on a degraded pool and scrubbing can make that worse, even though scrubs are throttled. Arguably, I might be being too conservative here, but the marginal benefit of scrubbing a *degraded* pool is pretty minimal as pools should not be left degraded for very long. The cron.d in this patch scrubs on the second Sunday of the month. mdadm scrubs on the first Sunday of the month. This way, if a system has both MD and ZFS pools, the load doesn't all happen at the same time. If the system doesn't have both types, it shouldn't really matter which week. If you'd rather make it the same week as MD, I see no problem with that. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to zfs-linux in Ubuntu. https://bugs.launchpad.net/bugs/1548009 Title: ZFS pools should be automatically scrubbed Status in zfs-linux package in Ubuntu: Fix Released Bug description: [Impact] Xenial shipped with a cron job to automatically scrub ZFS pools, as desired by many users and as implemented by mdadm for traditional Linux software RAID. Unfortunately, this cron job does not work, because it needs a PATH line for /sbin, where the zpool utility lives. Given the existence of the cron job and various discussions on IRC, etc., users expect that scrubs are happening, when they are not. This means ZFS is not pre-emptively checking for (and correcting) corruption. The odds of disk corruption are admittedly very low, but violating users' expectations of data safety, especially when they've gone out of their way to use a filesystem which touts data safety, is bad. [Test Case] $ truncate -s 1G test.img $ sudo zpool create test `pwd`/test.img $ sudo zpool status test $ sudo vi /etc/cron.d/zfsutils-linux Modify /etc/cron.d/zfsutils-linux to run the cron job in a few minutes (modifying the date range if it's not currently the 8th through
[Kernel-packages] [Bug 1548009] Re: ZFS pools should be automatically scrubbed
This bug was fixed in the package zfs-linux - 0.6.5.7-0ubuntu4 --- zfs-linux (0.6.5.7-0ubuntu4) yakkety; urgency=medium * Set PATH in cron.d job to fix monthly scrubs. (LP: #1548009) -- Richard Laager Wed, 06 Jul 2016 22:37:47 -0500 ** Changed in: zfs-linux (Ubuntu) Status: In Progress => Fix Released -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to zfs-linux in Ubuntu. https://bugs.launchpad.net/bugs/1548009 Title: ZFS pools should be automatically scrubbed Status in zfs-linux package in Ubuntu: Fix Released Bug description: mdadm automatically checks MD arrays. ZFS should automatically scrub pools too. Scrubbing a pool allows ZFS to detect on-disk corruption and (when the pool has redundancy) correct it. Note that ZFS does not blindly assume the other copy is correct; it will only overwrite bad data with data that is known to be good (i.e. it passes the checksum). I've attached a debdiff which accomplishes this. It builds and installs cleanly. The meat of it is the scrub script I've been using on production systems, both servers and laptops, and recommending in my Ubuntu root- on-ZFS HOWTO, for years, which scrubs all *healthy* pools. If a pool is not healthy, scrubbing it is bad for two reasons: 1) It adds a lot of disk load which could theoretically lead to another failure. We should save that disk load for resilvering. 2) Performance is already less on a degraded pool and scrubbing can make that worse, even though scrubs are throttled. Arguably, I might be being too conservative here, but the marginal benefit of scrubbing a *degraded* pool is pretty minimal as pools should not be left degraded for very long. The cron.d in this patch scrubs on the second Sunday of the month. mdadm scrubs on the first Sunday of the month. This way, if a system has both MD and ZFS pools, the load doesn't all happen at the same time. If the system doesn't have both types, it shouldn't really matter which week. If you'd rather make it the same week as MD, I see no problem with that. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1548009/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1548009] Re: ZFS pools should be automatically scrubbed
Thanks for the patch! I've uploaded to yakkety, this bug should autoclose. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to zfs-linux in Ubuntu. https://bugs.launchpad.net/bugs/1548009 Title: ZFS pools should be automatically scrubbed Status in zfs-linux package in Ubuntu: In Progress Bug description: mdadm automatically checks MD arrays. ZFS should automatically scrub pools too. Scrubbing a pool allows ZFS to detect on-disk corruption and (when the pool has redundancy) correct it. Note that ZFS does not blindly assume the other copy is correct; it will only overwrite bad data with data that is known to be good (i.e. it passes the checksum). I've attached a debdiff which accomplishes this. It builds and installs cleanly. The meat of it is the scrub script I've been using on production systems, both servers and laptops, and recommending in my Ubuntu root- on-ZFS HOWTO, for years, which scrubs all *healthy* pools. If a pool is not healthy, scrubbing it is bad for two reasons: 1) It adds a lot of disk load which could theoretically lead to another failure. We should save that disk load for resilvering. 2) Performance is already less on a degraded pool and scrubbing can make that worse, even though scrubs are throttled. Arguably, I might be being too conservative here, but the marginal benefit of scrubbing a *degraded* pool is pretty minimal as pools should not be left degraded for very long. The cron.d in this patch scrubs on the second Sunday of the month. mdadm scrubs on the first Sunday of the month. This way, if a system has both MD and ZFS pools, the load doesn't all happen at the same time. If the system doesn't have both types, it shouldn't really matter which week. If you'd rather make it the same week as MD, I see no problem with that. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1548009/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1548009] Re: ZFS pools should be automatically scrubbed
** Changed in: zfs-linux (Ubuntu) Importance: Wishlist => Medium ** Changed in: zfs-linux (Ubuntu) Status: Confirmed => In Progress -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to zfs-linux in Ubuntu. https://bugs.launchpad.net/bugs/1548009 Title: ZFS pools should be automatically scrubbed Status in zfs-linux package in Ubuntu: In Progress Bug description: mdadm automatically checks MD arrays. ZFS should automatically scrub pools too. Scrubbing a pool allows ZFS to detect on-disk corruption and (when the pool has redundancy) correct it. Note that ZFS does not blindly assume the other copy is correct; it will only overwrite bad data with data that is known to be good (i.e. it passes the checksum). I've attached a debdiff which accomplishes this. It builds and installs cleanly. The meat of it is the scrub script I've been using on production systems, both servers and laptops, and recommending in my Ubuntu root- on-ZFS HOWTO, for years, which scrubs all *healthy* pools. If a pool is not healthy, scrubbing it is bad for two reasons: 1) It adds a lot of disk load which could theoretically lead to another failure. We should save that disk load for resilvering. 2) Performance is already less on a degraded pool and scrubbing can make that worse, even though scrubs are throttled. Arguably, I might be being too conservative here, but the marginal benefit of scrubbing a *degraded* pool is pretty minimal as pools should not be left degraded for very long. The cron.d in this patch scrubs on the second Sunday of the month. mdadm scrubs on the first Sunday of the month. This way, if a system has both MD and ZFS pools, the load doesn't all happen at the same time. If the system doesn't have both types, it shouldn't really matter which week. If you'd rather make it the same week as MD, I see no problem with that. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1548009/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1548009] Re: ZFS pools should be automatically scrubbed
I can confirm that the .debdiff I attached in comment #11 fixes the problem. My system ran the scrub today. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to zfs-linux in Ubuntu. https://bugs.launchpad.net/bugs/1548009 Title: ZFS pools should be automatically scrubbed Status in zfs-linux package in Ubuntu: Confirmed Bug description: mdadm automatically checks MD arrays. ZFS should automatically scrub pools too. Scrubbing a pool allows ZFS to detect on-disk corruption and (when the pool has redundancy) correct it. Note that ZFS does not blindly assume the other copy is correct; it will only overwrite bad data with data that is known to be good (i.e. it passes the checksum). I've attached a debdiff which accomplishes this. It builds and installs cleanly. The meat of it is the scrub script I've been using on production systems, both servers and laptops, and recommending in my Ubuntu root- on-ZFS HOWTO, for years, which scrubs all *healthy* pools. If a pool is not healthy, scrubbing it is bad for two reasons: 1) It adds a lot of disk load which could theoretically lead to another failure. We should save that disk load for resilvering. 2) Performance is already less on a degraded pool and scrubbing can make that worse, even though scrubs are throttled. Arguably, I might be being too conservative here, but the marginal benefit of scrubbing a *degraded* pool is pretty minimal as pools should not be left degraded for very long. The cron.d in this patch scrubs on the second Sunday of the month. mdadm scrubs on the first Sunday of the month. This way, if a system has both MD and ZFS pools, the load doesn't all happen at the same time. If the system doesn't have both types, it shouldn't really matter which week. If you'd rather make it the same week as MD, I see no problem with that. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1548009/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1548009] Re: ZFS pools should be automatically scrubbed
** Summary changed: - [FFe] ZFS pools should be automatically scrubbed + ZFS pools should be automatically scrubbed -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to zfs-linux in Ubuntu. https://bugs.launchpad.net/bugs/1548009 Title: ZFS pools should be automatically scrubbed Status in zfs-linux package in Ubuntu: Confirmed Bug description: mdadm automatically checks MD arrays. ZFS should automatically scrub pools too. Scrubbing a pool allows ZFS to detect on-disk corruption and (when the pool has redundancy) correct it. Note that ZFS does not blindly assume the other copy is correct; it will only overwrite bad data with data that is known to be good (i.e. it passes the checksum). I've attached a debdiff which accomplishes this. It builds and installs cleanly. The meat of it is the scrub script I've been using on production systems, both servers and laptops, and recommending in my Ubuntu root- on-ZFS HOWTO, for years, which scrubs all *healthy* pools. If a pool is not healthy, scrubbing it is bad for two reasons: 1) It adds a lot of disk load which could theoretically lead to another failure. We should save that disk load for resilvering. 2) Performance is already less on a degraded pool and scrubbing can make that worse, even though scrubs are throttled. Arguably, I might be being too conservative here, but the marginal benefit of scrubbing a *degraded* pool is pretty minimal as pools should not be left degraded for very long. The cron.d in this patch scrubs on the second Sunday of the month. mdadm scrubs on the first Sunday of the month. This way, if a system has both MD and ZFS pools, the load doesn't all happen at the same time. If the system doesn't have both types, it shouldn't really matter which week. If you'd rather make it the same week as MD, I see no problem with that. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1548009/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
