Re: Improvements to fsck performance in -current ...?
Kevin Oberman wrote: To get to UFS2, you must newfs the partition. I don't know of nay other way. The basic improvement in UFS2 is the expansion of many fields to 64 bits to allow for much larger structures. While newfs in V5.1 and CURRENT defaults to UFS2, there are no problems continuing to run UFS1 file systems. Extended attributes, ACL and MAC do not work well on UFS1. I've tried, a decided it was just not worth the pain (particularly since that was rwatson's general feeling about it too). -- Daniel C. Sobral Gerência de Operações Divisão de Comunicação de Dados Coordenação de Segurança VIVO Centro Oeste Norte Fones: 55-61-313-7654/Cel: 55-61-9618-0904 E-mail: [EMAIL PROTECTED] [EMAIL PROTECTED] [EMAIL PROTECTED] ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Improvements to fsck performance in -current ...?
Terry Lambert wrote: Unfortunately, IDE disks do not permit disconnected writes, due to a bug in the original IDE implementation, Therefore IDE disks almost universally lie to the driver any time write caching is enabled on an IDE drive. I understand that SATA has fixed a number of problems in the command set over PATA. Does anyone know if SATA handles this issue correctly? Tim ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Improvements to fsck performance in -current ...?
Don Lewis wrote:
On 2 Oct, Terry Lambert wrote:
[...]
Actually, write caching is not so much the problem, as the disk
reporting that the write has completed before the contents of
the transaction saved in the write cache have actually been
committed to stable storage.
Unfortunately, IDE disks do not permit disconnected writes, due
to a bug in the original IDE implementation, which has been
carried forward for [insert no good reason here].
Therefore IDE disks almost universally lie to the driver any
time write caching is enabled on an IDE drive.
In most cases, if you use SCSI, the problem will go away.
Nope, they lie as well unless you turn of the WCE bit. Fortunately
with tagged command queuing there is very little performance penalty for
doing this in most cases. The main exception to this is when you run
newfs which talks to the raw partition and only has one command
outstanding at a time.
Back in the days when our SCSI implementation would spam the console
whenever it reduced the number of tagged openings because the drive
indicated that its queue was full, I'd see the number of tagged openings
stay at 63 if write caching was disabled, but the number would drop
significantly under load (50%?) if write caching was enabled. I always
suspected that the drive's cache was full of data for write commands
that it had indicated to the host as being complete even though the data
hadn't been written to stable storage.
Unfortunately SCSI drives all seem to ship with the WCE bit set,
probably for benchmarking reasons, so I always have to remember to
turn this bit off whenever I install a new drive.
A message from this morning ('file system (UFS2) consistancy
after -current crash?') to this list describes exactly the
situation on my fileserver a few month ago, except my machine
runs with FreeBSD 4-STABLE and has an ICP-Vortex 6528RD controller.
I think, disk's or controllers (short hardware) write cache
is a problem. Maybe it shouldn't be in theory, but it is in
real world :-)
Best regards,
Jens
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Improvements to fsck performance in -current ...?
Jens Rehsack wrote:
Don Lewis wrote:
On 2 Oct, Terry Lambert wrote:
[...]
Actually, write caching is not so much the problem, as the disk
reporting that the write has completed before the contents of
the transaction saved in the write cache have actually been
committed to stable storage.
Unfortunately, IDE disks do not permit disconnected writes, due
to a bug in the original IDE implementation, which has been
carried forward for [insert no good reason here].
Therefore IDE disks almost universally lie to the driver any
time write caching is enabled on an IDE drive.
In most cases, if you use SCSI, the problem will go away.
Nope, they lie as well unless you turn of the WCE bit. Fortunately
with tagged command queuing there is very little performance penalty for
doing this in most cases. The main exception to this is when you run
newfs which talks to the raw partition and only has one command
outstanding at a time.
Back in the days when our SCSI implementation would spam the console
whenever it reduced the number of tagged openings because the drive
indicated that its queue was full, I'd see the number of tagged openings
stay at 63 if write caching was disabled, but the number would drop
significantly under load (50%?) if write caching was enabled. I always
suspected that the drive's cache was full of data for write commands
that it had indicated to the host as being complete even though the data
hadn't been written to stable storage.
Unfortunately SCSI drives all seem to ship with the WCE bit set,
probably for benchmarking reasons, so I always have to remember to
turn this bit off whenever I install a new drive.
A message from this morning ('file system (UFS2) consistancy
after -current crash?') to this list describes exactly the
situation on my fileserver a few month ago, except my machine
runs with FreeBSD 4-STABLE and has an ICP-Vortex 6528RD controller.
I think, disk's or controllers (short hardware) write cache
is a problem. Maybe it shouldn't be in theory, but it is in
real world :-)
This is somewhat relevent to a discussion occurring this week on the
PostgreSQL performance mailing list.
A fellow was testing a number of caching options for disk drives, in
conjunction with the performance impact it had on Postgre. Near the
end of the discussion and his testing, he decided to do a plug test
(i.e., pull the power plug out of the wall while Postgre was running a
benchmark and see if the database was recoverable on reboot).
The tests don't 100% apply, since he was testing with Linux and XFS,
but I think the results speak VOLUMES!
Every single plug test with WC enabled on the IDE drives resulted in
an unrecoverable database - every time, even with XFS' journalling,
and no matter what sync options he had enabled in Postgres.
Every single plug test with WC disabled on the IDE drives resulted
in a filesystem and database that was recoverable, even when sync
was turned totally off in Postgres.
Additionally, he noticed that turning WC on resulted in something
like 40x performance improvement.
To me, this means:
a) if you want reliable, don't use IDE with WC
b) if you want reliable and fast, don't use IDE, period, use SCSI.
--
Bill Moran
Potential Technologies
http://www.potentialtech.com
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Improvements to fsck performance in -current ...?
Bill Moran wrote: [...] To me, this means: a) if you want reliable, don't use IDE with WC Reducable of 'don't use IDE' :-) b) if you want reliable and fast, don't use IDE, period, use SCSI. If you look at the recent postings, SCSI didn't help you out everytime. I use the fileserver in current configuration for nearly 5 years, and only once this happend, each other case all filesystems including hold data was recoverable. Jens ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Improvements to fsck performance in -current ...?
Bill Moran wrote:
Jens Rehsack wrote:
Don Lewis wrote:
On 2 Oct, Terry Lambert wrote:
[...]
Actually, write caching is not so much the problem, as the disk
reporting that the write has completed before the contents of
the transaction saved in the write cache have actually been
committed to stable storage.
Unfortunately, IDE disks do not permit disconnected writes, due
to a bug in the original IDE implementation, which has been
carried forward for [insert no good reason here].
Therefore IDE disks almost universally lie to the driver any
time write caching is enabled on an IDE drive.
In most cases, if you use SCSI, the problem will go away.
Nope, they lie as well unless you turn of the WCE bit. Fortunately
with tagged command queuing there is very little performance penalty for
doing this in most cases. The main exception to this is when you run
newfs which talks to the raw partition and only has one command
outstanding at a time.
Back in the days when our SCSI implementation would spam the console
whenever it reduced the number of tagged openings because the drive
indicated that its queue was full, I'd see the number of tagged openings
stay at 63 if write caching was disabled, but the number would drop
significantly under load (50%?) if write caching was enabled. I always
suspected that the drive's cache was full of data for write commands
that it had indicated to the host as being complete even though the data
hadn't been written to stable storage.
Unfortunately SCSI drives all seem to ship with the WCE bit set,
probably for benchmarking reasons, so I always have to remember to
turn this bit off whenever I install a new drive.
A message from this morning ('file system (UFS2) consistancy
after -current crash?') to this list describes exactly the
situation on my fileserver a few month ago, except my machine
runs with FreeBSD 4-STABLE and has an ICP-Vortex 6528RD controller.
I think, disk's or controllers (short hardware) write cache
is a problem. Maybe it shouldn't be in theory, but it is in
real world :-)
This is somewhat relevent to a discussion occurring this week on the
PostgreSQL performance mailing list.
A fellow was testing a number of caching options for disk drives, in
conjunction with the performance impact it had on Postgre. Near the
end of the discussion and his testing, he decided to do a plug test
(i.e., pull the power plug out of the wall while Postgre was running a
benchmark and see if the database was recoverable on reboot).
The tests don't 100% apply, since he was testing with Linux and XFS,
but I think the results speak VOLUMES!
You realize the sync() and fsync() commandos are severely BROKEN
under Linux already on VFS level? OK. kernel 2.6 will get it
right somehow. But until then one should not even think about
using Linux in a trully sensitive environment as a DB server.
I doubt seriously that it is the disk caching which is to be blamed here, since
otherwise crashing on journaling filesystems would be almost for sure
desasterous every time you do it... The cache sized on disks
are so thinny in comparision to the sector size that you will almost
immediately have the caches flushed anyway by a margin of well below
one second.
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Improvements to fsck performance in -current ...?
According to Brooks Davis: I believe this problem has been fixed. At least that's what I got out It has been fixed for a few months now. That fix could be backported to stable but it requires careful testing as many files are touched by the change. -- Ollivier ROBERT -=- FreeBSD: The Power to Serve! -=- [EMAIL PROTECTED] Darwin snuadh.freenix.org Kernel Version 6.6: Thu May 1 21:48:54 PDT 2003 ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Improvements to fsck performance in -current ...?
Jens Rehsack wrote: Kevin Oberman wrote: Current has two major changes re speeding up fsck. The most significant is the background operation of fsck on file system with soft updates enabled. Because of the way softupdates works, you are assured of metadata consistency on reboot, so the file systems can be mounted and used immediately with fsck started up in the background about a minute after the system comes up. Be careful what you promise :-) Most new disks have an own disk cache and some of them have a write cache enabled. In case of a hardware failure (or power failure) this data may get lost and the disk's metadata isn't consistent. It's only when no write cache below the system is active. Actually, write caching is not so much the problem, as the disk reporting that the write has completed before the contents of the transaction saved in the write cache have actually been committed to stable storage. Unfortunately, IDE disks do not permit disconnected writes, due to a bug in the original IDE implementation, which has been carried forward for [insert no good reason here]. Therefore IDE disks almost universally lie to the driver any time write caching is enabled on an IDE drive. In most cases, if you use SCSI, the problem will go away. -- Terry ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Improvements to fsck performance in -current ...?
On 2 Oct, Terry Lambert wrote: Jens Rehsack wrote: Kevin Oberman wrote: Current has two major changes re speeding up fsck. The most significant is the background operation of fsck on file system with soft updates enabled. Because of the way softupdates works, you are assured of metadata consistency on reboot, so the file systems can be mounted and used immediately with fsck started up in the background about a minute after the system comes up. Be careful what you promise :-) Most new disks have an own disk cache and some of them have a write cache enabled. In case of a hardware failure (or power failure) this data may get lost and the disk's metadata isn't consistent. It's only when no write cache below the system is active. Actually, write caching is not so much the problem, as the disk reporting that the write has completed before the contents of the transaction saved in the write cache have actually been committed to stable storage. Unfortunately, IDE disks do not permit disconnected writes, due to a bug in the original IDE implementation, which has been carried forward for [insert no good reason here]. Therefore IDE disks almost universally lie to the driver any time write caching is enabled on an IDE drive. In most cases, if you use SCSI, the problem will go away. Nope, they lie as well unless you turn of the WCE bit. Fortunately with tagged command queuing there is very little performance penalty for doing this in most cases. The main exception to this is when you run newfs which talks to the raw partition and only has one command outstanding at a time. Back in the days when our SCSI implementation would spam the console whenever it reduced the number of tagged openings because the drive indicated that its queue was full, I'd see the number of tagged openings stay at 63 if write caching was disabled, but the number would drop significantly under load (50%?) if write caching was enabled. I always suspected that the drive's cache was full of data for write commands that it had indicated to the host as being complete even though the data hadn't been written to stable storage. Unfortunately SCSI drives all seem to ship with the WCE bit set, probably for benchmarking reasons, so I always have to remember to turn this bit off whenever I install a new drive. ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Improvements to fsck performance in -current ...?
On Tue, 30 Sep 2003, Steve Kargl wrote: On Wed, Oct 01, 2003 at 01:04:51AM +0200, Lukas Ertl wrote: are either of these enhancements back-patchable to the 4.x fsck, or do they require some non-4.x compatible changes to work? It's not just the fsck application itself, background fsck basically needs file system snapshots, which are only available on UFS2, and I'm not sure if they can be backported to UFS1 at all. As soon as Kirk committed the snapshot capability, snapshot became available on UFS1. The only requirement is softupdates and softupdates pre-dates UFS2. Oh, yeah, sorry, I think I got that wrong :-/. regards, le -- Lukas Ertl eMail: [EMAIL PROTECTED] UNIX Systemadministrator Tel.: (+43 1) 4277-14073 Vienna University Computer Center Fax.: (+43 1) 4277-9140 University of Vienna http://mailbox.univie.ac.at/~le/ ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Improvements to fsck performance in -current ...?
Kevin Oberman wrote: [...] Current has two major changes re speeding up fsck. The most significant is the background operation of fsck on file system with soft updates enabled. Because of the way softupdates works, you are assured of metadata consistency on reboot, so the file systems can be mounted and used immediately with fsck started up in the background about a minute after the system comes up. Be careful what you promise :-) Most new disks have an own disk cache and some of them have a write cache enabled. In case of a hardware failure (or power failure) this data may get lost and the disk's metadata isn't consistent. It's only when no write cache below the system is active. Jens ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Improvements to fsck performance in -current ...?
On Tue, 30 Sep 2003 16:25:06 -0700 Steve Kargl [EMAIL PROTECTED] wrote: As soon as Kirk committed the snapshot capability, snapshot became available on UFS1. The only requirement is softupdates and softupdates pre-dates UFS2. Snapshots are available in 4.9? I thought it's not only about the FS structure, it's about the code in -current (which is much different from the 4.x code). Bye, Alexander. -- One world, one web, one program -- Microsoft promotional ad Ein Volk, ein Reich, ein Fuehrer -- Adolf Hitler http://www.Leidinger.net Alexander @ Leidinger.net GPG fingerprint = C518 BC70 E67F 143F BE91 3365 79E2 9C60 B006 3FE7 ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Improvements to fsck performance in -current ...?
On Tue, 30 Sep 2003 19:49:33 -0300 (ADT) Marc G. Fournier [EMAIL PROTECTED] wrote: Now,I don't/wouldn't have softupdates enabled on / .. does the 'background fsck' know to not background if softupdates are not enabled? I'm going to switch back to -p and look a bit closer the next time it happens (if it happens) to see if it is/was a softupdate file system that failed, now that I have a better idea of what I'm looking for ... I can only repeat what Robert already told you, bg-fsck is much better now. I suspect that these enhancements may both require that soft updates be enabled for the file systems. are either of these enhancements back-patchable to the 4.x fsck, or do they require some non-4.x compatible changes to work? ... I'm at 3.5hrs and counting right now ... any speedup would be great ... The second enhancement isn't that much magic... just newfs with a large value for -c (a recent 4.x-newfs may do it by default, as it does in -current). Together with a larger block size (-b 16384 if it isn't already the case) and a suitable fragment size (-f 2048) this will reduce the time fsck will need. Bye, Alexander. -- Loose bits sink chips. http://www.Leidinger.net Alexander @ Leidinger.net GPG fingerprint = C518 BC70 E67F 143F BE91 3365 79E2 9C60 B006 3FE7 ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Improvements to fsck performance in -current ...?
On Wed, Oct 01, 2003 at 01:19:26PM +0200, Alexander Leidinger wrote: On Tue, 30 Sep 2003 16:25:06 -0700 Steve Kargl [EMAIL PROTECTED] wrote: As soon as Kirk committed the snapshot capability, snapshot became available on UFS1. The only requirement is softupdates and softupdates pre-dates UFS2. Snapshots are available in 4.9? I thought it's not only about the FS structure, it's about the code in -current (which is much different from the 4.x code). No - snapshots are only available under 5.x, but for UFS1 and UFS2. -- B.Walter BWCThttp://www.bwct.de [EMAIL PROTECTED] [EMAIL PROTECTED] ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Improvements to fsck performance in -current ...?
On Wed, Oct 01, 2003 at 01:19:26PM +0200, Alexander Leidinger wrote: On Tue, 30 Sep 2003 16:25:06 -0700 Steve Kargl [EMAIL PROTECTED] wrote: As soon as Kirk committed the snapshot capability, snapshot became available on UFS1. The only requirement is softupdates and softupdates pre-dates UFS2. Snapshots are available in 4.9? I thought it's not only about the FS structure, it's about the code in -current (which is much different from the 4.x code). I wrote snapshots require softupdates. How you jumped to the conclusion that 4.x has snapshots is beyond me. My truck has a 4-speed manual transmission, therefore all trucks have 4-speed manual transmissions. -- Steve ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Improvements to fsck performance in -current ...?
On Wed, 1 Oct 2003 07:22:58 -0700 Steve Kargl [EMAIL PROTECTED] wrote: On Wed, Oct 01, 2003 at 01:19:26PM +0200, Alexander Leidinger wrote: On Tue, 30 Sep 2003 16:25:06 -0700 Steve Kargl [EMAIL PROTECTED] wrote: As soon as Kirk committed the snapshot capability, snapshot became available on UFS1. The only requirement is softupdates and softupdates pre-dates UFS2. Snapshots are available in 4.9? I thought it's not only about the FS structure, it's about the code in -current (which is much different from the 4.x code). I wrote snapshots require softupdates. How you jumped to the conclusion that 4.x has snapshots is beyond me. My truck has a 4-speed manual transmission, therefore all trucks have 4-speed manual transmissions. I've read a committed ... to RELENG_4, snapshot because I had the impression we talked about RELENG_4. Sorry, Alexander. -- Loose bits sink chips. http://www.Leidinger.net Alexander @ Leidinger.net GPG fingerprint = C518 BC70 E67F 143F BE91 3365 79E2 9C60 B006 3FE7 ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Improvements to fsck performance in -current ...?
Date: Wed, 01 Oct 2003 06:39:47 + From: Jens Rehsack [EMAIL PROTECTED] Kevin Oberman wrote: [...] Current has two major changes re speeding up fsck. The most significant is the background operation of fsck on file system with soft updates enabled. Because of the way softupdates works, you are assured of metadata consistency on reboot, so the file systems can be mounted and used immediately with fsck started up in the background about a minute after the system comes up. Be careful what you promise :-) Most new disks have an own disk cache and some of them have a write cache enabled. In case of a hardware failure (or power failure) this data may get lost and the disk's metadata isn't consistent. It's only when no write cache below the system is active. Yes. This is fairly well known with many, many messages in the archives. If you want serious stability, you need to turn off the disk write cache. I have it off on my office system here and on on my laptop. But thanks for bringing this up as it is important. And, yes, it has burned me, although it required a confluence of things all going wrong at exactly the right timing to catch a bunch of metadata in cache. (This could only have happened on a CURRENT system back in the 5.0 time frame.) It could only happen when the file system had been very active with an installworld. But it did happen. The trade-off is a big performance hit. With disk cache on, I can copy my entire FreeBSD partition to another disk in about 15 minutes. With disk cache off, it took a few HOURS. This was a worst case example with dd on my laptop (slow disks). -- R. Kevin Oberman, Network Engineer Energy Sciences Network (ESnet) Ernest O. Lawrence Berkeley National Laboratory (Berkeley Lab) E-mail: [EMAIL PROTECTED] Phone: +1 510 486-8634 ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Improvements to fsck performance in -current ...?
Kevin Oberman wrote: Date: Wed, 01 Oct 2003 06:39:47 + From: Jens Rehsack [EMAIL PROTECTED] Kevin Oberman wrote: [...] Current has two major changes re speeding up fsck. The most significant is the background operation of fsck on file system with soft updates enabled. Because of the way softupdates works, you are assured of metadata consistency on reboot, so the file systems can be mounted and used immediately with fsck started up in the background about a minute after the system comes up. Be careful what you promise :-) Most new disks have an own disk cache and some of them have a write cache enabled. In case of a hardware failure (or power failure) this data may get lost and the disk's metadata isn't consistent. It's only when no write cache below the system is active. [...] But thanks for bringing this up as it is important. And, yes, it has burned me, although it required a confluence of things all going wrong at exactly the right timing to catch a bunch of metadata in cache. (This could only have happened on a CURRENT system back in the 5.0 time frame.) It could only happen when the file system had been very active with an installworld. But it did happen. It happens to me in another circumstance. On my fileserver runs a portupgrade and during that something nasty fails. I couldn't determine what, but nearly the complete /usr/, /var/ and some of the /export/ data was lost+found :-) /var and /usr could be restored, and the rest came from backup or restored from lost+found, but it showed me the end of disk caching. The trade-off is a big performance hit. With disk cache on, I can copy my entire FreeBSD partition to another disk in about 15 minutes. With disk cache off, it took a few HOURS. This was a worst case example with dd on my laptop (slow disks). Here you should try to use other block sizes. Try it with smaller ranges, eg. 100MB. The result may be best somewhere between 8192 and 65536 bytes per block. Best regards, Jens ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
Improvements to fsck performance in -current ...?
Due to an electrician flipping the wrong circuit breaker this morning, I had my servers go down hard ... they are all -STABLE, with one of the four taking a *very* long time to fsck: jupiter# ps aux | grep fsck root 361 99.0 2.3 95572 95508 p0 R+4:21PM 121:13.21 fsck -y /dev/da0s1h jupiter# date Tue Sep 30 18:37:02 ADT 2003 jupiter# Now, CPU time is rising, so I figure its still working away, and fsck shows: jupiter# fsck -y /dev/da0s1h ** /dev/da0s1h ** Last Mounted on /vm ** Phase 1 - Check Blocks and Sizes ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts so it isn't finding any errors ... A friend of mine asked if we had a journalling file system, which I told him know, as I don't believe we do ... but are/have there been any improvements to fsck in -CURRENT to improve performance on large file systems (this is a 6x36G RAID5 system)? Does UFS2 address any of this? I've actually had a 6x18gig RAID5 file system once take 11+hrs to fsck ... and when it was completed, everything seemed fine, with no reports of any file or directory corruption ... it obviously did a good job of checking the file system, just hate the lengthy downtime ... ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Improvements to fsck performance in -current ...?
Date: Tue, 30 Sep 2003 18:42:21 -0300 (ADT) From: Marc G. Fournier [EMAIL PROTECTED] Sender: [EMAIL PROTECTED] Due to an electrician flipping the wrong circuit breaker this morning, I had my servers go down hard ... they are all -STABLE, with one of the four taking a *very* long time to fsck: jupiter# ps aux | grep fsck root 361 99.0 2.3 95572 95508 p0 R+4:21PM 121:13.21 fsck -y /dev/da0s1h jupiter# date Tue Sep 30 18:37:02 ADT 2003 jupiter# Now, CPU time is rising, so I figure its still working away, and fsck shows: jupiter# fsck -y /dev/da0s1h ** /dev/da0s1h ** Last Mounted on /vm ** Phase 1 - Check Blocks and Sizes ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts so it isn't finding any errors ... A friend of mine asked if we had a journalling file system, which I told him know, as I don't believe we do ... but are/have there been any improvements to fsck in -CURRENT to improve performance on large file systems (this is a 6x36G RAID5 system)? Does UFS2 address any of this? I've actually had a 6x18gig RAID5 file system once take 11+hrs to fsck ... and when it was completed, everything seemed fine, with no reports of any file or directory corruption ... it obviously did a good job of checking the file system, just hate the lengthy downtime ... Current has two major changes re speeding up fsck. The most significant is the background operation of fsck on file system with soft updates enabled. Because of the way softupdates works, you are assured of metadata consistency on reboot, so the file systems can be mounted and used immediately with fsck started up in the background about a minute after the system comes up. Until fsck runs it is possible (likely) that some free blocks on the filesystem amy be unavailable until the fsck completes, but that should be the only issue. The other issue is significant speedup in the time fsck takes to run. On my little 30 MB /usr/partition it now takes only seconds to fsck vs. about 2 minutes when I was running V4 on the system. On huge system, I suspect the speedup is even more significant, but don't know for sure. I suspect that these enhancements may both require that soft updates be enabled for the file systems. -- R. Kevin Oberman, Network Engineer Energy Sciences Network (ESnet) Ernest O. Lawrence Berkeley National Laboratory (Berkeley Lab) E-mail: [EMAIL PROTECTED] Phone: +1 510 486-8634 ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Improvements to fsck performance in -current ...?
On Tue, 30 Sep 2003, Kevin Oberman wrote: Current has two major changes re speeding up fsck. The most significant is the background operation of fsck on file system with soft updates enabled. Because of the way softupdates works, you are assured of metadata consistency on reboot, so the file systems can be mounted and used immediately with fsck started up in the background about a minute after the system comes up. Actually, I had this blow up on my -CURRENT desktop once ... didn't have a clue on how to debug it, so I switched from fsck -p to fsck -y to prevent it from happening again :( Now,I don't/wouldn't have softupdates enabled on / .. does the 'background fsck' know to not background if softupdates are not enabled? I'm going to switch back to -p and look a bit closer the next time it happens (if it happens) to see if it is/was a softupdate file system that failed, now that I have a better idea of what I'm looking for ... I suspect that these enhancements may both require that soft updates be enabled for the file systems. are either of these enhancements back-patchable to the 4.x fsck, or do they require some non-4.x compatible changes to work? ... I'm at 3.5hrs and counting right now ... any speedup would be great ... ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Improvements to fsck performance in -current ...?
On Tue, 30 Sep 2003, Marc G. Fournier wrote: Now,I don't/wouldn't have softupdates enabled on / .. does the 'background fsck' know to not background if softupdates are not enabled? Yes, this is no problem, if the FS doesn't have SU, it just checks it the old way. Since / is usually rather small, this is acceptable. I suspect that these enhancements may both require that soft updates be enabled for the file systems. are either of these enhancements back-patchable to the 4.x fsck, or do they require some non-4.x compatible changes to work? It's not just the fsck application itself, background fsck basically needs file system snapshots, which are only available on UFS2, and I'm not sure if they can be backported to UFS1 at all. regards, le -- Lukas Ertl eMail: [EMAIL PROTECTED] UNIX Systemadministrator Tel.: (+43 1) 4277-14073 Vienna University Computer Center Fax.: (+43 1) 4277-9140 University of Vienna http://mailbox.univie.ac.at/~le/ ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Improvements to fsck performance in -current ...?
On Wed, Oct 01, 2003 at 01:04:51AM +0200, Lukas Ertl wrote: are either of these enhancements back-patchable to the 4.x fsck, or do they require some non-4.x compatible changes to work? It's not just the fsck application itself, background fsck basically needs file system snapshots, which are only available on UFS2, and I'm not sure if they can be backported to UFS1 at all. As soon as Kirk committed the snapshot capability, snapshot became available on UFS1. The only requirement is softupdates and softupdates pre-dates UFS2. -- Steve ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Improvements to fsck performance in -current ...?
On Wed, 1 Oct 2003, Lukas Ertl wrote: are either of these enhancements back-patchable to the 4.x fsck, or do they require some non-4.x compatible changes to work? It's not just the fsck application itself, background fsck basically needs file system snapshots, which are only available on UFS2, and I'm not sure if they can be backported to UFS1 at all. Ah, okay, so when I move my servers to 5.x, then I'm going to need to reformat the systems from scratch, else I lose some serious improvements ... is there a list somewhere of what UFS2 has over UFS1? file system snapshots, is that similar to journalling? ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Improvements to fsck performance in -current ...?
On Tue, Sep 30, 2003 at 04:44:30PM -0700, Kevin Oberman wrote: Date: Tue, 30 Sep 2003 20:28:20 -0300 (ADT) From: Marc G. Fournier [EMAIL PROTECTED] On Wed, 1 Oct 2003, Lukas Ertl wrote: are either of these enhancements back-patchable to the 4.x fsck, or do they require some non-4.x compatible changes to work? It's not just the fsck application itself, background fsck basically needs file system snapshots, which are only available on UFS2, and I'm not sure if they can be backported to UFS1 at all. Ah, okay, so when I move my servers to 5.x, then I'm going to need to reformat the systems from scratch, else I lose some serious improvements ... is there a list somewhere of what UFS2 has over UFS1? file system snapshots, is that similar to journalling? I think this is wrong. I have seen no issue in snapshots or background fsck with UFS1 volumes. And, if you mean jounalling as in jfs, no. As I understand it, snapshot is an atomic capture of the file system metadata which allows fsck to analyze and repair it on an active file system. (Note: This only applies to softupdate enabled systems as softupdates assure a level of consistency in the metadata that assures that fsck will not make changes to active file space on the file system. To get to UFS2, you must newfs the partition. I don't know of nay other way. The basic improvement in UFS2 is the expansion of many fields to 64 bits to allow for much larger structures. While newfs in V5.1 and CURRENT defaults to UFS2, there are no problems continuing to run UFS1 file systems. UFS2 also allocates less inodes at newfs time and instead adds them on demand so you have less of them to check when you don't have any files using them. Finally, I don't think there is any issue any longer with using soft updates on /. The only reason that root did not default to SU under V4 is that SU disks only periodically update free space information and a small root partition can fill up during an installworld when all of the files in (especially) /sbin are replaces which requires enough space to hold two copies of the directory. Under the default partition sizes in V5, this is really not an issue and all partitions created by sysinstall under V5 will default to SU, including /. I believe this problem has been fixed. At least that's what I got out of Kirk's talk as BSDCon. If you have a USENIX membership, you can read about UFS2 at: http://www.usenix.org/publications/library/proceedings/bsdcon03/tech/mckusick.html -- Brooks -- Any statement of the form X is the one, true Y is FALSE. PGP fingerprint 655D 519C 26A7 82E7 2529 9BF0 5D8E 8BE9 F238 1AD4 pgp0.pgp Description: PGP signature
Re: Improvements to fsck performance in -current ...?
Lukas Ertl [EMAIL PROTECTED] wrote: It's not just the fsck application itself, background fsck basically needs file system snapshots, which are only available on UFS2, and I'm not sure if they can be backported to UFS1 at all. Huh? Snapshots are available for both UFS1 and UFS2, but only on 5.x. -- Christian naddy Weisgerber [EMAIL PROTECTED] ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Improvements to fsck performance in -current ...?
On Tue, 30 Sep 2003, Marc G. Fournier wrote: Current has two major changes re speeding up fsck. The most significant is the background operation of fsck on file system with soft updates enabled. Because of the way softupdates works, you are assured of metadata consistency on reboot, so the file systems can be mounted and used immediately with fsck started up in the background about a minute after the system comes up. Actually, I had this blow up on my -CURRENT desktop once ... didn't have a clue on how to debug it, so I switched from fsck -p to fsck -y to prevent it from happening again :( No idea when this happened to you, but background fsck/snapshots have become dramatically more stable since about half way between 5.0-release and 5.1-release. Kirk chased down a lot of serious bugs and issues with hangs. So experience from before that time may not be characteristic of current behavior. Now,I don't/wouldn't have softupdates enabled on / .. does the 'background fsck' know to not background if softupdates are not enabled? I'm going to switch back to -p and look a bit closer the next time it happens (if it happens) to see if it is/was a softupdate file system that failed, now that I have a better idea of what I'm looking for ... sysinstall doesn't enable soft updates on / by default, as for small partitions you increase the chance of running into space concerns. Many of the space concerns have been resolved by some more recent behavioral changes in UFS. The issue in question is that soft updates trickles out changes in a write-behind, and in the event of a large delete followed by an immediate large allocation, the deleted storage may not have been reclaimed when the allocation comes along. For example, if you had a really small / and did an installkernel. In more recent 5.x, UFS now tracks the space that will be freed and reports it as freed, and includes some support for waiting for space to become available (which it inevitably will in that situation, once the soft updates have been processed). So the picture may have improved a lot since you last used it, depending on when that was. Robert N M Watson FreeBSD Core Team, TrustedBSD Projects [EMAIL PROTECTED] Network Associates Laboratories ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
