Re: UFS+J panics on HEAD
On Thu, May 24, 2012 at 09:58:52PM +0200, Matthew Seaman wrote: On 24/05/2012 00:05, Mark Linimon wrote: On Wed, May 23, 2012 at 10:58:48PM +0100, Steven Hartland wrote: While it might be a shame to see FFS go by the wayside are there any big reasons why you would rather stick with FFS instead of moving to ZFS with all the benefits that brings? - ZFS eats bytes for breakfast. It is completely inappropriate for anything with less than 4GB RAM. - ZFS performs poorly under disk-nearly-full conditions. - ZFS is not optimal for situations where there are a lot of small, randomly dispersed IOs around the disk space. Like in any sort of RDBMS. This is very true for reads, not for writes because it is a COW filesystem so writes are usually sequencial disk-wise. -- Jeremie Le Hen Men are born free and equal. Later on, they're on their own. Jean Yanne ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: UFS+J panics on HEAD
Hello, Steven. You wrote 24 мая 2012 г., 1:58:48: SH While it might be a shame to see FFS go by the wayside are there any SH big reasons why you would rather stick with FFS instead of moving SH to ZFS with all the benefits that brings? I afraid, that after real hardware failure (like real HDD death, not these pseudo-broken-hardware situations, when HDDs is perfectly alive and in good condition), all data will be lost. I could restore data from remains of FFS by hands (format is straightforward and well-known), but ZFS is different story... And all these messages about panics after pool import... Yes, it could be result of REAL hardware problems, but I want to have some hope to restore some of the data after such crashes. Yes, backups is solution, but I don't have money to buy (reliable) hardware to backup 4Tb of data :( I attended Solaris internals 5-days training four years ago (when I worked for Sun Microsystems), and instructor says same words... -- // Black Lion AKA Lev Serebryakov l...@freebsd.org ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: UFS+J panics on HEAD
If all you are doing is reading, the ZFS on-disk format is well documented and fairly easy to work with. Take a look at the ZFS bootloader code - that implements a ZFS reader in not too many lines of code and could easily be re-purposed for a recovery tool. On 24 May 2012 09:04, Lev Serebryakov l...@freebsd.org wrote: Hello, Steven. You wrote 24 мая 2012 г., 1:58:48: SH While it might be a shame to see FFS go by the wayside are there any SH big reasons why you would rather stick with FFS instead of moving SH to ZFS with all the benefits that brings? I afraid, that after real hardware failure (like real HDD death, not these pseudo-broken-hardware situations, when HDDs is perfectly alive and in good condition), all data will be lost. I could restore data from remains of FFS by hands (format is straightforward and well-known), but ZFS is different story... And all these messages about panics after pool import... Yes, it could be result of REAL hardware problems, but I want to have some hope to restore some of the data after such crashes. Yes, backups is solution, but I don't have money to buy (reliable) hardware to backup 4Tb of data :( I attended Solaris internals 5-days training four years ago (when I worked for Sun Microsystems), and instructor says same words... -- // Black Lion AKA Lev Serebryakov l...@freebsd.org ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: UFS+J panics on HEAD
On 23. May 2012, at 21:38 , Lev Serebryakov wrote: Hello, Konstantin. You wrote 23 мая 2012 г., 17:10:46: KB This panic is another protective panic caused by on-disk inconsistent KB structures. The bitmap indicated that an inode was free, but actual inode KB context suggested that the inode is in use. KB I would not worry much about ffs code until known hardware problem on KB the machine are fixed. Konstantin, it is very sad, that official position of one of FFS maintainers (according to mailing list activity), is to blame hardware on every FFS/SU/SUJ inconsistency and do nothing with code. Well, I had talked to him offline as well, and the machine, as I had indicated, has well known bad memory, though ECC claims to correct still, so it's not unreasonable. I am however still not sure it is indeed hardware but hope will know early next week if not the next days. We'll see. /bz -- Bjoern A. Zeeb You have to have visions! It does not matter how good you are. It matters what good you do! ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: UFS+J panics on HEAD
On 24/05/2012 00:05, Mark Linimon wrote: On Wed, May 23, 2012 at 10:58:48PM +0100, Steven Hartland wrote: While it might be a shame to see FFS go by the wayside are there any big reasons why you would rather stick with FFS instead of moving to ZFS with all the benefits that brings? - ZFS eats bytes for breakfast. It is completely inappropriate for anything with less than 4GB RAM. - ZFS performs poorly under disk-nearly-full conditions. - ZFS is not optimal for situations where there are a lot of small, randomly dispersed IOs around the disk space. Like in any sort of RDBMS. Even so, ZFS is certainly my personal default nowadays. On a machine of any size, the question is not should I use ZFS? but are there any good reasons why I shouldn't use ZFS? (And if so, what could I do to make it possible to use ZFS anyhow...) With Andriy's recent patches to zfboot to extend support for Boot Environments, it's all starting to look particularly sexy. Cheers, Matthew -- Dr Matthew J Seaman MA, D.Phil. PGP: http://www.infracaninophile.co.uk/pgpkey signature.asc Description: OpenPGP digital signature
Re: UFS+J panics on HEAD
On 24/05/2012 00:05, Mark Linimon wrote: On Wed, May 23, 2012 at 10:58:48PM +0100, Steven Hartland wrote: While it might be a shame to see FFS go by the wayside are there any big reasons why you would rather stick with FFS instead of moving to ZFS with all the benefits that brings? - ZFS eats bytes for breakfast. It is completely inappropriate for anything with less than 4GB RAM. - ZFS performs poorly under disk-nearly-full conditions. - ZFS is not optimal for situations where there are a lot of small, randomly dispersed IOs around the disk space. Like in any sort of RDBMS. Even so, ZFS is certainly my personal default nowadays. On a machine of any size, the question is not should I use ZFS? but are there any good reasons why I shouldn't use ZFS? (And if so, what could I do to make it possible to use ZFS anyhow...) With Andriy's recent patches to zfsboot to extend support for Boot Environments, it's all starting to look particularly sexy. Cheers, Matthew -- Dr Matthew J Seaman MA, D.Phil. PGP: http://www.infracaninophile.co.uk/pgpkey signature.asc Description: OpenPGP digital signature
Re: UFS+J panics on HEAD
On 24/05/2012 00:05, Mark Linimon wrote: On Wed, May 23, 2012 at 10:58:48PM +0100, Steven Hartland wrote: While it might be a shame to see FFS go by the wayside are there any big reasons why you would rather stick with FFS instead of moving to ZFS with all the benefits that brings? - ZFS eats bytes for breakfast. It is completely inappropriate for anything with less than 4GB RAM. - ZFS performs poorly under disk-nearly-full conditions. - ZFS is not optimal for situations where there are a lot of small, randomly dispersed IOs around the disk space. Like in any sort of RDBMS. We actually use it in very random high IOP's applications with small requests, so high standard disk's aren't even an option so SDD's all the way and we see nice performance. I can't say we've compared it to say FFS as that simply doesn't provide the management tools we needed so wasn't even considered, but its far from shabby in our environment :) Regards Steve This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmas...@multiplay.co.uk. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: UFS+J panics on HEAD
On 2012-May-24 12:04:21 +0400, Lev Serebryakov l...@freebsd.org wrote: I afraid, that after real hardware failure (like real HDD death, not these pseudo-broken-hardware situations, when HDDs is perfectly alive and in good condition), all data will be lost. I could restore data from remains of FFS by hands (format is straightforward and well-known), but ZFS is different story... If your disk dies then you need a redundant copy of your data - either via backups or via RAID. Normally, you'd run ZFS with some level of redundancy so that disk failures did not result in data loss. That said, ZFS is touchier about data - if it can't verify the checksums in your data, it will refuse to give it to you - whereas UFS will hand you back a pile of bytes that may or may the same as what you gave it to store. And you can't necessarily get _any_ data off a failed disk. Yes, backups is solution, but I don't have money to buy (reliable) hardware to backup 4Tb of data :( 4TB disks are available but not really economical at present. 2TB disks still seem to be the happy medium. If your data will compress down to 2TB then save it to a disk, otherwise split your backups across a pair of disks. A 2TB disk with enclosure is USD150. If you don't trust that, buy a second set. (And if you value your data, get a trusted friend to store one copy at their house in case anything happens at your house). I attended Solaris internals 5-days training four years ago (when I worked for Sun Microsystems), and instructor says same words... I have had lots of problems at $work with Solaris UFS quietly corrupting data following crashes. At least with ZFS, you have a detter chance of knowing when your data has been corrupted. -- Peter Jeremy pgpk4t2qrNnV7.pgp Description: PGP signature
Re: UFS+J panics on HEAD
On Wed, May 23, 2012 at 12:40:34AM +, Bjoern A. Zeeb wrote: Hi, I have a machine that since updated to r235609 started to constantly panic in the FS code while building universe, first with ufs_dirbad: /scratch: bad dir ino 1137225 at offset 17920: mangled entry which a clri and a fully forced fsck -y -f seems to have cleared (thanks to kib) and now it is giving me: mode = 040700, inum = 14560, fs = /scratch panic: ffs_valloc: dup alloc cpuid = 0 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a kdb_backtrace() at kdb_backtrace+0x37 panic() at panic+0x1ce ffs_valloc() at ffs_valloc+0x70c ufs_makeinode() at ufs_makeinode+0x86 VOP_CREATE_APV() at VOP_CREATE_APV+0x44 vn_open_cred() at vn_open_cred+0x4c8 kern_openat() at kern_openat+0x1f9 amd64_syscall() at amd64_syscall+0x61e Xfast_syscall() at Xfast_syscall+0xf7 --- syscall (5, FreeBSD ELF64, sys_open), rip = 0x4b94bc, rsp = 0x7fffc998, rbp = 0x10 --- /scratch has USF+J enabled as another hint. The machine is also reporting ECC memory corrections once in a while (replacement is on its way) but had done that the months before the update to the latest HEAD as well w/o the FS trouble. Anyone an idea on what's going on there or what had changed since Feb/March that could cause this? I am willing to try things if I manage to get a kernel compile for testing;-) otherwise I might dump/dd/newfs/restore and see if I can still reproduce it afterwards or whether it just got into a state that fsck is failing to correct... This panic is another protective panic caused by on-disk inconsistent structures. The bitmap indicated that an inode was free, but actual inode context suggested that the inode is in use. I would not worry much about ffs code until known hardware problem on the machine are fixed. You could try another pass of the full fsck on the volume, but my expectations are that bad hardware causes continuous damage to the data and metainformation. pgp04Bg1jzI8R.pgp Description: PGP signature
Re: UFS+J panics on HEAD
Hello, Konstantin. You wrote 23 мая 2012 г., 17:10:46: KB This panic is another protective panic caused by on-disk inconsistent KB structures. The bitmap indicated that an inode was free, but actual inode KB context suggested that the inode is in use. KB I would not worry much about ffs code until known hardware problem on KB the machine are fixed. Konstantin, it is very sad, that official position of one of FFS maintainers (according to mailing list activity), is to blame hardware on every FFS/SU/SUJ inconsistency and do nothing with code. According to my experience, we all live in real world, where HDDs could lost write cache on power failure or kernel panic unrelated to FFS, UPSes and PSUs fails (which leads to power failures) and HDDs with turned off write cache are unusable -- because they become slow like hell without writing cache. You could name it broken hardware, but let face it -- all non-top-server hardware, everything, but HBAs with battery installed in double-PSU-equipped cases, are broken now in this sense. My home server with almost-read-only load crashes due to burned out PSU twice in last 2 years (and I buy good desktop PSUs, in $150-$200 price range, not china boxes for $30) and I've got one memory problem in this time period (DIMM was detected and replaced, but I've got two or three panics before I become sure, that it is memory problem, because memtest86+ doesn;t find any problems in 12+ hours run). It is good desktop hardware, with good cooling system, not something low-end, but not server-grade one, of course. And after EVERY of such crashes my main storage area (95% read, 5% write) had dozens of unexpected SU inconsistences, background fsck fails to create snapshot and I was need to run foreground fsck for many hours. It seems, that async mount without SU will not be worse that SU solution! And, if you read through mailing lists, you cold find dozens such reports. And answer almost always is broken hardware. Yes, Ok, it is broken hardware, all right. But we haven't other one! We need to live with what we have! What I want to say: FFS/SU become almost unusable on this hardware. Protective panic, my ass! Every solution (link this inode to lost+found and mark as used, mark it as free, etc) is better than protective panic. One mismatched inode is not the end of the world, it is even not end of cylinder group, not to say about whole FS, system could (and must) complain about it, but not panic! Did you hear term ``self-healing''? It seems, that modern hardware needed better solution, that just panic and blame hardware. Or should we call FFS officially dead and promote ZFS as only usable FS on modern FreeBSD now? -- // Black Lion AKA Lev Serebryakov l...@freebsd.org ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: UFS+J panics on HEAD
- Original Message - From: Lev Serebryakov l...@freebsd.org Or should we call FFS officially dead and promote ZFS as only usable FS on modern FreeBSD now? Slightly OT but I must say I'm a big fan of ZFS since I first tried it under FreeBSD a few years back. Not only is it a dream to use as an admin, making all traditionally hard tasks easy; its also helped us identify failing hardware which nothing else has spotted. While it might be a shame to see FFS go by the wayside are there any big reasons why you would rather stick with FFS instead of moving to ZFS with all the benefits that brings? With regards protective panics I must agree, in this case it does seem there are much better paths which could be chosen. Regards Steve This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmas...@multiplay.co.uk. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: UFS+J panics on HEAD
On Wed, May 23, 2012 at 10:58:48PM +0100, Steven Hartland wrote: While it might be a shame to see FFS go by the wayside are there any big reasons why you would rather stick with FFS instead of moving to ZFS with all the benefits that brings? - ZFS eats bytes for breakfast. It is completely inappropriate for anything with less than 4GB RAM. - ZFS performs poorly under disk-nearly-full conditions. mcl ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: UFS+J panics on HEAD
- Original Message - From: Mark Linimon lini...@lonesome.com On Wed, May 23, 2012 at 10:58:48PM +0100, Steven Hartland wrote: While it might be a shame to see FFS go by the wayside are there any big reasons why you would rather stick with FFS instead of moving to ZFS with all the benefits that brings? - ZFS eats bytes for breakfast. It is completely inappropriate for anything with less than 4GB RAM. It does like more RAM than FFS, but can be tuned to work under smaller amounts however given current pricing does anyone use HW for servers with that little RAM? Sure old i386 machines still running legacy ABC that just works so no need to upgrade it may still have less than 4GB RAM, but modern machines running large disk sets are a totally different matter. - ZFS performs poorly under disk-nearly-full conditions. Not experienced that here, care to clarify / quantify? While if this is the case I can see that may cause problems, but the solution to that is simple don't let your system get that full. Disks are still relatively cheap even after the disasters, so keep on top of your space and you wont see that problem; particularly as ZFS makes it so easy to expand existing pools :) Regards steve This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmas...@multiplay.co.uk. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
UFS+J panics on HEAD
Hi, I have a machine that since updated to r235609 started to constantly panic in the FS code while building universe, first with ufs_dirbad: /scratch: bad dir ino 1137225 at offset 17920: mangled entry which a clri and a fully forced fsck -y -f seems to have cleared (thanks to kib) and now it is giving me: mode = 040700, inum = 14560, fs = /scratch panic: ffs_valloc: dup alloc cpuid = 0 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a kdb_backtrace() at kdb_backtrace+0x37 panic() at panic+0x1ce ffs_valloc() at ffs_valloc+0x70c ufs_makeinode() at ufs_makeinode+0x86 VOP_CREATE_APV() at VOP_CREATE_APV+0x44 vn_open_cred() at vn_open_cred+0x4c8 kern_openat() at kern_openat+0x1f9 amd64_syscall() at amd64_syscall+0x61e Xfast_syscall() at Xfast_syscall+0xf7 --- syscall (5, FreeBSD ELF64, sys_open), rip = 0x4b94bc, rsp = 0x7fffc998, rbp = 0x10 --- /scratch has USF+J enabled as another hint. The machine is also reporting ECC memory corrections once in a while (replacement is on its way) but had done that the months before the update to the latest HEAD as well w/o the FS trouble. Anyone an idea on what's going on there or what had changed since Feb/March that could cause this? I am willing to try things if I manage to get a kernel compile for testing;-) otherwise I might dump/dd/newfs/restore and see if I can still reproduce it afterwards or whether it just got into a state that fsck is failing to correct... /bz -- Bjoern A. Zeeb You have to have visions! It does not matter how good you are. It matters what good you do! ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
