Re: [CentOS] How to stagger fsck executions
From: Warren Young Sent: April 22, 2015 20:46 On Apr 22, 2015, at 11:56 AM, Hugh E Cruickshank h...@forsoft.com wrote: I have done some what if testing. Using which tool? My simulator, or something you cooked up yourself? If the latter, would you care to share? I cobbled something together in OpenEdge ABL. I have uploaded it to http://pastebin.ca/2979494 This was intended only for my use so, while the code is relatively clean, it is not documented. Regards, Hugh -- Hugh E Cruickshank, Forward Software, www.forward-software.com ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] How to stagger fsck executions
On Apr 23, 2015, at 8:49 AM, Hugh E Cruickshank h...@forsoft.com wrote: From: Warren Young Sent: April 22, 2015 20:46 On Apr 22, 2015, at 11:56 AM, Hugh E Cruickshank h...@forsoft.com wrote: I have done some what if testing. Using which tool? My simulator, or something you cooked up yourself? If the latter, would you care to share? I cobbled something together in OpenEdge ABL. I have uploaded it to http://pastebin.ca/2979494 Thanks! I like how it saves work by taking real world conditions — service life of the machine and reboot frequency — into account. Mine just computes all of the possibilities, even if in practice it would require a machine a million years old to hit them all. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] How to stagger fsck executions
From: Warren Young Sent: April 21, 2015 14:13 On Apr 21, 2015, at 9:50 AM, Hugh E Cruickshank wrote: From: Kay Diederichs Sent: April 21, 2015 03:43 instead of having 20 for all of them, set the first filesystem to 17, the second to 19, the third to 23, and the fourth to 29. Thanks but that is not much different then my second idea and does not fully avoid the problem. You may be missing a key fact of how prime numbers work. You are right and I stand corrected. I have done some what if testing. Assuming 8 filesystems and weekly reboots over a ten year period... The random numbers would result in as many as 40 weeks per year where 2, 3 or 4 fscks would be run in the same week depending on the random numbers selected. Using the prime numbers 7, 11, 13, 17, 19, 23, 29 and 31 the is a maximum of 7 incidents per year of 2 fscks per week and none for 3 or more. Clearly the prime numbers are better. Thank, Hugh -- Hugh E Cruickshank, Forward Software, www.forward-software.com ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] How to stagger fsck executions
On Apr 22, 2015, at 11:56 AM, Hugh E Cruickshank h...@forsoft.com wrote: I have done some what if testing. Using which tool? My simulator, or something you cooked up yourself? If the latter, would you care to share? I’ve updated mine to break out the stats for 3+ volumes instead of just reporting all multi-volume fscks together: https://gist.github.com/wyoung/7c94967bb635de48d058 Then I rewrote that in C++, since these 8-volume simulations were literally going to take days with the Perl version: https://gist.github.com/wyoung/966383b4efbb63aafc71 (The Perl version is about 1/5 the speed of the C++ one. This actually isn’t all that bad, considering that it’s an interpreted dynamically-typed language going up against a statically-typed native-code compiled language.) Using the prime numbers 7, 11, 13, 17, 19, 23, 29 and 31 the is a maximum of 7 incidents per year of 2 fscks per week and none for 3 or more. This is why I pointed out that you only need *relatively* prime numbers: so that if you decide the largest max-mount-count can’t be over 31, you don’t have to go clear down to 7 in order to find the last prime for 8 volumes. Using relatively-prime numbers, you can skew the set upwards quite a bit without increasing the largest value. The most efficient set I’ve been able to come up with is: 17, 19, 23, 25, 27, 28, 29, 31 The three composite values (25, 27, and 28) do not share any common factors: 25 uses 5 twice, 27 uses 3 thrice, and 28 uses 7 plus twice 2. My newer simulators give these results for the chances of a multi-volume fsck with your prime set: period: ~6.7 billion 2-volume: 8.12% 3-volume: 1.08% 4-volume: 0.09% 5:0.01% chance total:9.3% My relatively prime improved set gives these results because the set’s median is higher while keeping the same maximum, while also avoiding any reuse of prime factors: period: ~126.2 billion 2-volume: 0.37% 3-volume: 0.33% 4-volume: 0.02% 5:0.01% chance total:0.7% See? Number theory *is* useful in real life. :) ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] How to stagger fsck executions
On 4/20/2015 9:08 PM, Hugh E Cruickshank wrote: The second idea was to set each filesystem to a different random count value. This would run the risk of having two or more executions at the same time but it would probably not be very frequent. Does anyone have a suggestion for a better way of doing this? use XFS, no fsck's until/unless something catastrophic happens to the file system. -- john r pierce, recycling bits in santa cruz ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] How to stagger fsck executions
On Tue, Apr 21, 2015 at 9:38 AM, Hugh E Cruickshank h...@forsoft.com wrote: CentOS 6 My first idea was to manually run fsck on each filesystem, one every couple of weeks. That way they will not all come due at the same time if we reboot on a regular basis. The second idea was to set each filesystem to a different random count value. This would run the risk of having two or more executions at the same time but it would probably not be very frequent. Does anyone have a suggestion for a better way of doing this? Take a look at 'man tune2fs' and 'man fstab' for modifying the fsck order in your system. HTH, -- Arun Khan ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] How to stagger fsck executions
On 04/21/2015 09:40 AM, Hugh E Cruickshank wrote: I accept that fscks are required on a periodic basis and I am willing to reboot more often to achieve these but I would like to minimize downtime (during the reboot) where possible. Why do you accept that? The default behavior for filesystems set up by Red Hat tools (anaconda) is not to fsck. Not by mount count, nor by time. The default behavior for e2fsprogs was changed to disable periodic fsck in Feb 2011. CentOS 6 includes a version of e2fsprogs from before that change, but the filesystem is considered very stable, and the periodic fsck is not generally considered necessary. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] How to stagger fsck executions
From: Les Mikesell Sent: April 21, 2015 09:54 On Tue, Apr 21, 2015 at 11:40 AM, Hugh E Cruickshank wrote: I am trying to avoid running them at the same time in an effort to avoid 70 minute boot times (which is what happened on the weekend). How many filesystems do you have? It varies from system to system but is typically 8-10. If you look at ./etc.fstab, everything where the final number is '1' (normally just the root filesystem) should complete first, then everything with a 2 will run at once. If the other mounts are each on different drive/spindles they won't conflict with each other and will complete in the same time as running just the largest one of them. If you are running fscks of partitions on the same drive in parallel it will obviously go slower. I am aware of that. With the exception of /, /boot and /home which are on one spindle (actually a hardware mirrored pair) the remaining filesystems are on separate drives (actually hardware mirrored pairs or RAID 10 arrays). The largest of the filesystems (four of them) share a common SAS controller, data channel and external disk array hardware (HP D2600) so running these in parallel might not be as effective as they could be. Regards, Hugh -- Hugh E Cruickshank, Forward Software, www.forward-software.com ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] How to stagger fsck executions
From: Gordon Messmer Sent: April 21, 2015 10:30 On 04/21/2015 09:40 AM, Hugh E Cruickshank wrote: I accept that fscks are required on a periodic basis and I am willing to reboot more often to achieve these but I would like to minimize downtime (during the reboot) where possible. Why do you accept that? Every article I have read on the subject has recommended this a good practice. The default behavior for filesystems set up by Red Hat tools (anaconda) is not to fsck. Not by mount count, nor by time. The default behavior for e2fsprogs was changed to disable periodic fsck in Feb 2011. CentOS 6 includes a version of e2fsprogs from before that change, but the filesystem is considered very stable, and the periodic fsck is not generally considered necessary. I have confirmed that filesystems setup by anaconda on both CentOS 6 and RHEL 6 have both boot count and interval disabled however they are not disabled for any manually created filesystems (they are set to 24 and 6 months, respectively). I find it interesting that as late as 2014 Red Hat is recommending: . If automatic filesystem checks are inconvenient, then it is recommended to disable the automated filesystem check as discussed in the following article: How to turn off forced/automatic fsck in Red Hat Enterprise Linux? . Once disabled, it is recommended to schedule regular human controlled/monitored filsystem checks, when it is convenient to do so. These checks should not be ignored, or scheduled too far apart. This is from https://access.redhat.com/solutions/70531 Regards, Hugh -- Hugh E Cruickshank, Forward Software, www.forward-software.com ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] How to stagger fsck executions
From: John R Pierce Sent: April 20, 2015 23:58 On 4/20/2015 9:08 PM, Hugh E Cruickshank wrote: The second idea was to set each filesystem to a different random count value. This would run the risk of having two or more executions at the same time but it would probably not be very frequent. Does anyone have a suggestion for a better way of doing this? use XFS, no fsck's until/unless something catastrophic happens to the file system. Thanks. That would avoid the problem on future systems. Regards, Hugh -- Hugh E Cruickshank, Forward Software, www.forward-software.com ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] How to stagger fsck executions
From: Kay Diederichs Sent: April 21, 2015 03:43 On 04/21/2015 06:08 AM, Hugh E Cruickshank wrote: The second idea was to set each filesystem to a different random count value. This would run the risk of having two or more executions at the same time but it would probably not be very frequent. Using tune2fs -c, set the max-mount-counts to a different prime number for each filesystem. So e.g. instead of having 20 for all of them, set the first filesystem to 17, the second to 19, the third to 23, and the fourth to 29. This way, three or more fscks on the same boot are quite unlikely. Thanks but that is not much different then my second idea and does not fully avoid the problem. Regards, Hugh -- Hugh E Cruickshank, Forward Software, www.forward-software.com ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] How to stagger fsck executions
From: Arun Khan Sent: April 20, 2015 23:49 Take a look at 'man tune2fs' and 'man fstab' for modifying the fsck order in your system. Thanks but I did look at those and I was not able to find anything that would limit the fsck executions to one per reboot. Changing the order of execution will not address my concern. Regards, Hugh -- Hugh E Cruickshank, Forward Software, www.forward-software.com ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] How to stagger fsck executions
From: Mark Milhollan Sent: April 21, 2015 05:35 On Mon, 20 Apr 2015, Hugh E Cruickshank wrote: CentOS 6 From ''man fstab'' ... The sixth field, (fs_passno), is used by the fsck(8) program to determine the order in which filesystem checks are done at reboot time. The root filesystem should be specified with a fs_passno of 1, and other filesystems should have a fs_passno of 2. Filesystems within a drive will be checked sequentially, but filesystems on different drives will be checked at the same time to utilize parallelism available in the hardware. If the sixth field is not present or zero, a value of zero is returned and fsck will assume that the filesystem does not need to be checked. Thanks but changing the order of execution or executing them in parallel does not help with executing them one per reboot. Regards, Hugh -- Hugh E Cruickshank, Forward Software, www.forward-software.com ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] How to stagger fsck executions
On Apr 21, 2015, at 3:12 PM, Warren Young w...@etr-usa.com wrote: With the four values that Kay provided, I calculate a 1.2% chance on average that two or more volumes will need to be checked on the same reboot. Ooops, forgot to mention one other minor detail: This calculator gives the chance or a 2+ volume simultaneous check, which is actually a much more stringent case than you ran into. If you have four volumes and are using a relatively prime max-mount-count set, you won’t get a simultaneous check of all four volumes until your reboot count equals the *product* of all values in the set. With Kay’s prime set, that means it won’t happen until you have rebooted 215,441 times. If your machine lives 10 years, that’s enough to allow for about 60 reboots a day while still having *zero* chance of a 4-volume fsck. I think it’s fair to say that Kay’s solution (or a relatively-prime set) does actually solve the problem. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] How to stagger fsck executions
On Tue, April 21, 2015 2:13 pm, Hugh E Cruickshank wrote: From: Gordon Messmer Sent: April 21, 2015 10:30 On 04/21/2015 09:40 AM, Hugh E Cruickshank wrote: I accept that fscks are required on a periodic basis and I am willing to reboot more often to achieve these but I would like to minimize downtime (during the reboot) where possible. Why do you accept that? Every article I have read on the subject has recommended this a good practice. The default behavior for filesystems set up by Red Hat tools (anaconda) is not to fsck. Not by mount count, nor by time. The default behavior for e2fsprogs was changed to disable periodic fsck in Feb 2011. CentOS 6 includes a version of e2fsprogs from before that change, but the filesystem is considered very stable, and the periodic fsck is not generally considered necessary. I have confirmed that filesystems setup by anaconda on both CentOS 6 and RHEL 6 have both boot count and interval disabled however they are not disabled for any manually created filesystems (they are set to 24 and 6 months, respectively). I find it interesting that as late as 2014 Red Hat is recommending: . If automatic filesystem checks are inconvenient, then it is recommended to disable the automated filesystem check as discussed in the following article: How to turn off forced/automatic fsck in Red Hat Enterprise Linux? . Once disabled, it is recommended to schedule regular human controlled/monitored filsystem checks, when it is convenient to do so. These checks should not be ignored, or scheduled too far apart. Thus converting Enterprise into more or less home user system - for user with plenty of time to spare for manual system maintenance... ;-( Valeri This is from https://access.redhat.com/solutions/70531 Regards, Hugh -- Hugh E Cruickshank, Forward Software, www.forward-software.com ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos Valeri Galtsev Sr System Administrator Department of Astronomy and Astrophysics Kavli Institute for Cosmological Physics University of Chicago Phone: 773-702-4247 ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] How to stagger fsck executions
On Apr 21, 2015, at 9:50 AM, Hugh E Cruickshank h...@forsoft.com wrote:
From: Kay Diederichs Sent: April 21, 2015 03:43
instead of having 20 for all of them, set
the first filesystem to 17, the second to 19, the third to 23, and the
fourth to 29.
Thanks but that is not much different then my second idea and does not
fully avoid the problem.
You may be missing a key fact of how prime numbers work.
You can only get two or more fscks on a single reboot when the mount count is a
multiple of two or more of the max-mount-count values. When those numbers are
all prime, the frequency of such occurrences is much lower than when you use
purely random values.
With the four values that Kay provided, I calculate a 1.2% chance on average
that two or more volumes will need to be checked on the same reboot. If you
reboot on average once a month, that means it only happens once every 7 years
or so. That means the machine may well be retired before it happens even once,
and then only if you reboot that often in the first place.
If you take the same set of values and add one to them to make them all even,
and thus composite numbers, the chance rises to 3.3%, or about once every 2.5
years for monthly reboots. Thus, Kay’s solution is actually more than twice as
good as your second solution.
Interestingly, the numbers don’t actually have to be prime to take advantage of
this property of integers. They just have to be *relatively* prime:
http://goo.gl/bQbu5Z
For example, the set {15, 19, 22, 23} isn’t all prime, but is *relatively*
prime, since there is no integer other than 1 that evenly divides any pair
among the set. This set gives about a 1.5% chance of a 2+ volume collision,
nearly as good as Kay’s prime set.
I’ve put the calculator I built to test this up here:
https://gist.github.com/wyoung/7c94967bb635de48d058
___
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] How to stagger fsck executions
On 04/21/2015 12:13 PM, Hugh E Cruickshank wrote: From: Gordon Messmer Sent: April 21, 2015 10:30 Why do you accept that? Every article I have read on the subject has recommended this a good practice. Not every source is equal. The maintainers turned that behavior off by default sometime around the release of RHEL 6 (before the release, but after the package set was finalized). Among at least any group I am aware of, ext had been considered very stable for quite a long time before that, and common practice was to disable periodic checks. (As Anaconda has since who-even-knows when. WAY back.) I have confirmed that filesystems setup by anaconda on both CentOS 6 and RHEL 6 have both boot count and interval disabled however they are not disabled for any manually created filesystems (they are set to 24 and 6 months, respectively). Right. Red Hat's policy was to disable the checks long before that default changed in e2fsprogs (in version 1.42). . Once disabled, it is recommended to schedule regular human controlled/monitored filsystem checks, when it is convenient to do so. These checks should not be ignored, or scheduled too far apart. CYA? I'm not aware of such a recommendation in any of their publicly available documentation. This is from https://access.redhat.com/solutions/70531 Can't access it, myself. My university has a site license for RHEL, but it doesn't give me access to the KB. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] How to stagger fsck executions
On 04/21/2015 06:08 AM, Hugh E Cruickshank wrote: CentOS 6 Hi All: Over the weekend I had to reboot one of my systems and got hit with fsck runs on all of the filesystems. I would not mind so much except doing them all at once took over an hour. I would like to be able to stagger these, ideally only execute one fsck per reboot. I have been able to think of two possible solutions but neither is terrific. My first idea was to manually run fsck on each filesystem, one every couple of weeks. That way they will not all come due at the same time if we reboot on a regular basis. The second idea was to set each filesystem to a different random count value. This would run the risk of having two or more executions at the same time but it would probably not be very frequent. Does anyone have a suggestion for a better way of doing this? TIA Regards, Hugh Using tune2fs -c, set the max-mount-counts to a different prime number for each filesystem. So e.g. instead of having 20 for all of them, set the first filesystem to 17, the second to 19, the third to 23, and the fourth to 29. This way, three or more fscks on the same boot are quite unlikely. HTH, Kay ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] How to stagger fsck executions
On Tue, Apr 21, 2015 at 11:40 AM, Hugh E Cruickshank h...@forsoft.com wrote: From: Les Mikesell Sent: April 21, 2015 09:19 Why do you care about running them at the same time when it doesn't take longer to run them all in parallel? Except I think the root filesystem normally runs first. So you might want to stagger it vs. everything else. I am trying to avoid running them at the same time in an effort to avoid 70 minute boot times (which is what happened on the weekend). How many filesystems do you have? If you look at ./etc.fstab, everything where the final number is '1' (normally just the root filesystem) should complete first, then everything with a 2 will run at once. If the other mounts are each on different drive/spindles they won't conflict with each other and will complete in the same time as running just the largest one of them. If you are running fscks of partitions on the same drive in parallel it will obviously go slower. -- Les Mikesell lesmikes...@gmail.com ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] How to stagger fsck executions
On Tue, April 21, 2015 11:19 am, Les Mikesell wrote: On Tue, Apr 21, 2015 at 11:01 AM, Hugh E Cruickshank h...@forsoft.com wrote: Thanks but changing the order of execution or executing them in parallel does not help with executing them one per reboot. Why do you care about running them at the same time when it doesn't take longer to run them all in parallel? Except I think the root filesystem normally runs first. So you might want to stagger it vs. everything else. And unless you reboot frequently you are probably hitting the time setting, not the mount count. How frequently does one reboot (CentOS) Linux? Well, my observation is: every 30-45 days there is either kernel or glibc update so you have to reboot. This makes it about 10 reboots a year, so you are pretty much close to hitting mount count as much as time from last fsck for ext[2,3,4]. As it was already mentioned: XFS is marvellous. I use it forever for huge filesystems on Linux boxes. I remember howto by Russel Ingram was titled Linux + XFS HOWTO. Linux on Steroids... Valeri Valeri Galtsev Sr System Administrator Department of Astronomy and Astrophysics Kavli Institute for Cosmological Physics University of Chicago Phone: 773-702-4247 ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] How to stagger fsck executions
From: Les Mikesell Sent: April 21, 2015 09:19 Why do you care about running them at the same time when it doesn't take longer to run them all in parallel? Except I think the root filesystem normally runs first. So you might want to stagger it vs. everything else. I am trying to avoid running them at the same time in an effort to avoid 70 minute boot times (which is what happened on the weekend). I accept that fscks are required on a periodic basis and I am willing to reboot more often to achieve these but I would like to minimize downtime (during the reboot) where possible. And unless you reboot frequently you are probably hitting the time setting, not the mount count. This is in fact what transpired on the weekend and I would leave this in place as a protective measure. Regards, Hugh -- Hugh E Cruickshank, Forward Software, www.forward-software.com ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] How to stagger fsck executions
From: Hugh E Cruickshank Sent: April 20, 2015 21:09 Over the weekend I had to reboot one of my systems and got hit with fsck runs on all of the filesystems. I would not mind so much except doing them all at once took over an hour. I would like to be able to stagger these, ideally only execute one fsck per reboot. I have been able to think of two possible solutions but neither is terrific. I have come up with a third idea that would seem to address what I am looking for... 1. Create a file with the list of filesystems in desired order of execution. 2. Create an init.d script that: a. Sets tune2fs -c 0 on all filesystems. b. Extracts the first filesystem from the file, c. Sets tune2fs -c 1 on it, and d. moves it to the end of the file. The result is that on each reboot (after the first) only one filesystem will be checked on each boot. The down side is that an fsck will be run on every reboot however this could be mitigated by adding a boot count that would be maintained and checked in the file. It would appear that I have the beginnings of a workable solution. Thanks for everyone's suggestions. Regards, Hugh -- Hugh E Cruickshank, Forward Software, www.forward-software.com ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] How to stagger fsck executions
On Tue, Apr 21, 2015 at 11:01 AM, Hugh E Cruickshank h...@forsoft.com wrote: Thanks but changing the order of execution or executing them in parallel does not help with executing them one per reboot. Why do you care about running them at the same time when it doesn't take longer to run them all in parallel? Except I think the root filesystem normally runs first. So you might want to stagger it vs. everything else. And unless you reboot frequently you are probably hitting the time setting, not the mount count. -- Les Mikesell lesmikes...@gmail.com ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
[CentOS] How to stagger fsck executions
CentOS 6 Hi All: Over the weekend I had to reboot one of my systems and got hit with fsck runs on all of the filesystems. I would not mind so much except doing them all at once took over an hour. I would like to be able to stagger these, ideally only execute one fsck per reboot. I have been able to think of two possible solutions but neither is terrific. My first idea was to manually run fsck on each filesystem, one every couple of weeks. That way they will not all come due at the same time if we reboot on a regular basis. The second idea was to set each filesystem to a different random count value. This would run the risk of having two or more executions at the same time but it would probably not be very frequent. Does anyone have a suggestion for a better way of doing this? TIA Regards, Hugh -- Hugh E Cruickshank, Forward Software, www.forward-software.com ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
