Re: [zfs-discuss] resilver that never finishes
On 18/09/10 15:25, George Wilson wrote: Tom Bird wrote: In my case, other than an hourly snapshot, the data is not significantly changing. It'd be nice to see a response other than you're doing it wrong, rebuilding 5x the data on a drive relative to its capacity is clearly erratic behaviour, I am curious as to what is actually happening. It sounds like you're hitting '6891824 7410 NAS head continually resilvering following HDD replacement'. If you stop taking and destroying snapshots you should see the resilver finish. George, I think you've won the prize. I suspended the snapshots last night and this morning one pool had completed, one left to go. Thanks, Tom ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] resilver that never finishes
On 18/09/10 09:02, Ian Collins wrote: On 09/18/10 06:47 PM, Carsten Aulbert wrote: Has someone an idea how it is possible to resilver 678G of data on a 500G drive? I see this all the time on a troublesome Thumper. I believe this happens because the data in the pool is continuously changing. In my case, other than an hourly snapshot, the data is not significantly changing. It'd be nice to see a response other than you're doing it wrong, rebuilding 5x the data on a drive relative to its capacity is clearly erratic behaviour, I am curious as to what is actually happening. All said and done though, we will have to live with snv_134's bugs from now on, or perhaps I could try Sol 10. Tom ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] resilver that never finishes
On 18/09/10 13:06, Edho P Arief wrote: On Sat, Sep 18, 2010 at 7:01 PM, Tom Birdt...@marmot.org.uk wrote: All said and done though, we will have to live with snv_134's bugs from now on, or perhaps I could try Sol 10. or OpenIllumos. Or Nexenta. Or FreeBSD. Orinsert osol distro name. ... none of which will receive ZFS code updates unless Oracle deigns to bestow them upon the community, this or ZFS dev is taken over by said community, in which case we end up with diverging code bases that would be a sisyphean task to try and merge. Tom ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] resilver that never finishes
Morning, c7t5000CCA221F4EC54d0 is a 2T disk, how can it resilver 5.63T of it? This is actually an old capture of the status output, it got to nearly 10T before deciding that there was an error and not completing, reseat disk and it's doing it all again. It's happened on another pool as well, looking at a load av of around 40 on the box currently, just sitting there churning disk IO. OS is snv_134 on x86. # zpool status -x pool: content4 state: DEGRADED status: One or more devices could not be opened. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Attach the missing device and online it using 'zpool online'. see: http://www.sun.com/msg/ZFS-8000-2Q scrub: resilver in progress for 147h39m, 100.00% done, 0h0m to go config: NAME STATE READ WRITE CKSUM content4 DEGRADED 0 0 0 raidz2-0 DEGRADED 0 0 0 c7t5000CCA221DE1E1Dd0ONLINE 0 0 0 c7t5000CCA221DE17BFd0ONLINE 0 0 0 c7t5000CCA221DE2229d0ONLINE 0 0 0 replacing-3 DEGRADED 0 0 0 c7t5000CCA221DE0CC7d0 UNAVAIL 0 0 0 cannot open c7t5000CCA221F4EC54d0 ONLINE 0 0 0 5.63T resilvered c7t5000CCA221DE200Ad0ONLINE 0 0 0 c7t5000CCA221DDFE6Ed0ONLINE 0 0 0 c7t5000CCA221DE0103d0ONLINE 0 0 0 c7t5000CCA221DE00C9d0ONLINE 0 0 0 c7t5000CCA221DE0D2Dd0ONLINE 0 0 0 c7t5000CCA221DE189Cd0ONLINE 0 0 0 c7t5000CCA221DE18A7d0ONLINE 0 0 0 c7t5000CCA221DE2A47d0ONLINE 0 0 0 c7t5000CCA221DE1E48d0ONLINE 0 0 0 c7t5000CCA221DE18A1d0ONLINE 0 0 0 c7t5000CCA221DE18A2d0ONLINE 0 0 0 c7t5000CCA221DE2A3Ed0ONLINE 0 0 0 c7t5000CCA221DE2A42d0ONLINE 0 0 0 c7t5000CCA221DE2225d0UNAVAIL 0 0 0 cannot open c7t5000CCA221DE28A3d0ONLINE 0 0 0 c7t5000CCA221DE2A46d0ONLINE 0 0 0 c7t5000CCA221DE0789d0ONLINE 0 0 0 c7t5000CCA221DE221Dd0ONLINE 0 0 0 c7t5000CCA221DE054Fd0ONLINE 0 0 0 c7t5000CCA221DE2EBEd0ONLINE 0 0 0 errors: No known data errors -- Tom // www.portfast.co.uk // hosted services, domains, virtual machines, consultancy ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] pool won't import
r...@cs6:~# zpool import pool: content3 id: 14184872052409584084 state: FAULTED status: The pool metadata is corrupted. action: The pool cannot be imported due to damaged devices or data. The pool may be active on another system, but can be imported using the '-f' flag. see: http://www.sun.com/msg/ZFS-8000-72 config: content3FAULTED corrupted data c6t8d0ONLINE r...@cs6:~# uname -a SunOS cs6.kw.bbc.co.uk 5.11 snv_116 sun4v sparc SUNW,SPARC-Enterprise-T5220 r...@cs6:~# zpool import -f content3 cannot import 'content3': I/O error Ideas, anyone? -- Tom // www.portfast.co.uk -- internet services and consultancy // hosting from 1.65 per domain ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] demise of community edition
Afternoon, I note to my dismay that I can't get the community edition any more past snv_129, this version was closest to the normal way of doing things that I am used to with Solaris = 10, the standard OpenSolaris releases seem only to have this horrible Gnome based installer that gives you only one option - install everything. Am I just doing it wrong or is there another way to get OpenSolaris installed in a sane manner other than just sticking with community edition at snv_129? -- Tom // www.portfast.co.uk -- internet services and consultancy // hosting from 1.65 per domain ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] demise of community edition
Richard Elling wrote: It is not true that there is only a horrible Gnome based installer. Try the Automated Installation (AI) version instead of the LiveCD if you've used JumpStart previously. But if you just want a text-based installer and AI is overkill, then b131 is available with the Text Installer Project. Downloads available on: http://www.genunix.org http://hub.opensolaris.org/bin/view/Project+caiman/TextInstallerProject Thanks, this looks useful. Nothing is sane about Solaris 10 installer, good riddance :-) It wasn't that bad! :) PS sorry for this being a non specifically ZFS question, but ZFS is the reason I use opensolaris so there's a link in there somewhere. Tom ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS tale of woe and fail
Ross wrote: Yup, that one was down to a known (and fixed) bug though, so it isn't the normal story of ZFS problems. Got a bug ID or anything for that, just out of interest? As an update on my storage situation, I've got some JBODs now, see how that goes. -- Tom // www.portfast.co.uk -- internet services and consultancy // hosting from 1.65 per domain ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS tale of woe and fail
Victor Latushkin wrote: This issue (and previous one reported by Tom) has got some publicity recently - see here http://www.uknof.org.uk/uknof13/Bird-Redux.pdf So i feel like i need to provide a little bit more information about the outcome (sorry that it is delayed and not as full as previous one). Morning, Right, the PDF on there doesn't really give the full story of the presentation, unfortunate as I see it seems to have got around a bit. In the actual presentation I wasn't perhaps as harsh as it seems on the slides! First permanent error means that root block of the filesystem named 'content' was corrupted (all copies), so it was not possible to open it and access any content of that filesystem. Fortunately enough, there were not too much activity on the pool, so we decided to try previous states of the pool. I do not remember exact txg number we tried, but it was something like hundred txg back or so. We checked it with zdb and discovered that that state was more or less good - at least filesystem content was openable and it was possible to access its content, so we decided to reactivate that previous state. Pool imported fine and contents of 'content' was there. Subsequent scrub did find some errors but I do not remember exactly how much. Tom may have exact number. I can't remember how many errors the check found, however all the data copied off successfully, as far as we know. -- Tom // www.portfast.co.uk -- internet services and consultancy // hosting from 1.65 per domain ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] poor performance / lots of disk activity with low throughput, seems prefetch related
Hi guys, I've been having trouble with my archival kit, in the performance department rather than data loss this time (phew!). At the point when I took these stats where was about 250 mbit of traffic outbound on an ixgb NIC on the thing, also about 100 mbit of new stuff incoming. As you can see, the amount of disk activity far exceeds the data being served. I've disabled prefetching: echo zfs_prefetch_disable/W0t1 | mdb -kw and now activity looks more reasonable, the machine seems able to cope with the load, so potentially the prefetch is a bit too aggressive. The machine itself has 8GB of RAM and is serving lots of different large files to users, so any cache will be fairly useless. Stats: (with prefetch ENABLED) With prefetch off most of the bandwidth figures are single / low double digits. # uname -a SunOS cs0 5.11 snv_116 sun4v sparc SUNW,Sun-Fire-T200 # zpool iostat 5 capacity operationsbandwidth pool used avail read write read write -- - - - - - - content037.2T 2.85T237 1 29.5M 9.54K content128.5T 11.5T300 53 37.4M 5.82M content219.0T 1.04T183 0 22.9M 4.27K content33.85T 11.2T312 65 39.0M 7.57M content43.85T 11.2T338 63 42.2M 7.47M content53.84T 11.2T312 65 39.0M 7.50M content619.0T 1.05T108 0 13.5M 4.12K content714.0T 991G115 0 14.4M 3.39K -- - - - - - - content037.2T 2.85T129 0 16.1M 0 content128.5T 11.5T564 57 70.5M 6.62M content219.0T 1.04T 99 0 12.5M 0 content33.85T 11.2T361 79 45.1M 8.08M content43.85T 11.2T995 0 124M 0 content53.84T 11.2T243 46 30.3M 2.64M content619.0T 1.05T 0 0 0 0 content714.0T 991G203 0 25.4M 0 -- - - - - - - content037.2T 2.85T135 0 16.9M 0 content128.5T 11.5T578 57 72.3M 5.79M content219.0T 1.04T 96 0 12.0M 0 content33.85T 11.2T377100 47.1M 9.01M content43.85T 11.2T980 0 122M 0 content53.84T 11.2T216106 26.9M 10.4M content619.0T 1.05T 0 0 0 0 content714.0T 991G200 0 25.1M 0 -- - - - - - - content037.2T 2.85T131 0 16.4M 0 content128.5T 11.5T578 50 72.3M 4.82M content219.0T 1.04T 99 0 12.4M 0 content33.85T 11.2T377 63 47.1M 6.40M content43.85T 11.2T944 0 118M 0 content53.84T 11.2T243 61 30.4M 5.97M content619.0T 1.05T 0 0 0 0 content714.0T 991G209 0 26.1M 0 -- - - - - - - ^C -- Tom // www.portfast.co.uk -- internet services and consultancy // hosting from 1.65 per domain ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS tale of woe and fail
Toby Thain wrote: On 18-Jan-09, at 6:12 PM, Nathan Kroenert wrote: Hey, Tom - Correct me if I'm wrong here, but it seems you are not allowing ZFS any sort of redundancy to manage. Every other file system out there runs fine on a single LUN, when things go wrong you have a fsck utility that patches it up and the world keeps on turning. I can't find anywhere that will sell me a 48 drive SATA JBOD with all the drives presented on a single SAS channel, so running on a single giant LUN is a real world scenario that ZFS should be able to cope with, as this is how the hardware I am stuck with is arranged. Which is particularly catastrophic when one's 'content' is organized as a monolithic file, as it is here - unless, of course, you have some way of scavenging that file based on internal structure. No, it's not a monolithic file, the point I was making there is that no files are showing up. r...@cs4:~# find /content /content r...@cs4:~# (yes that really is it) thanks -- Tom // www.portfast.co.uk -- internet services and consultancy // hosting from 1.65 per domain ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] ZFS tale of woe and fail
Morning, For those of you who remember last time, this is a different Solaris, different disk box and different host, but the epic nature of the fail is similar. The RAID box that is the 63T LUN has a hardware fault and has been crashing, up to now the box and host got restarted and both came up fine. However, just now as I have got replacement hardware in position and was ready to start copying, it went bang and my data has all gone. Ideas? r...@cs4:~# zpool list NAME SIZE USED AVAILCAP HEALTH ALTROOT content 62.5T 59.9T 2.63T95% ONLINE - r...@cs4:~# zpool status -v pool: content state: ONLINE status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://www.sun.com/msg/ZFS-8000-8A scrub: none requested config: NAMESTATE READ WRITE CKSUM content ONLINE 0 032 c2t8d0ONLINE 0 032 errors: Permanent errors have been detected in the following files: content:0x0 content:0x2c898 r...@cs4:~# find /content /content r...@cs4:~# (yes that really is it) r...@cs4:~# uname -a SunOS cs4.kw 5.11 snv_99 sun4v sparc SUNW,Sun-Fire-T200 from format: 2. c2t8d0 IFT-S12S-G1033-363H-62.76TB /p...@7c0/p...@0/p...@8/LSILogic,s...@0/s...@8,0 Also, content does not show in df output. thanks -- Tom // www.portfast.co.uk -- internet services and consultancy // hosting from 1.65 per domain ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS tale of woe and fail
Tim wrote: On Sun, Jan 18, 2009 at 8:02 AM, Tom Bird t...@marmot.org.uk mailto:t...@marmot.org.uk wrote: errors: Permanent errors have been detected in the following files: content:0x0 content:0x2c898 r...@cs4:~# find /content /content r...@cs4:~# (yes that really is it) Those are supposedly the two inodes that are corrupt. The 0x0 is a bit scary... you should be able to find out what file(s) they're tied to (if any) with: find /content -inum 0 find /content -inum 182424 If you can live without those files, delete them, export the pool, re-import, and resilver, and you should be good to go. Hi, well one of the problems is that find doesn't find anything as it is not presenting any files, so I can't delete anything. I've exported the pool but on reimport, I get the same error as I was getting last time something popped: r...@cs4:~# zpool import content cannot open 'content': I/O error Last time, Victor Latushkin fixed it by modifying the file system to point to an older copy of the data. I've not really been following the list of late, any more sign of a fsck.zfs...? thanks -- Tom // www.portfast.co.uk -- internet services and consultancy // hosting from 1.65 per domain ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] SDXC and the future of ZFS
Bob Friesenhahn wrote: On Sun, 11 Jan 2009, Eric D. Mudama wrote: My impression is not that other OS's aren't interested in ZFS, they are, it's that the licensing restrictions limit native support to Solaris, BSD, and OS-X. Perhaps the philosophical issues of the other OS's (i.e. Linux) are more significant than the actual licensing issues. Many/most Linux users could legally use a native optimized kernel implementation of Sun ZFS if it was offered to them to do so. GPLv2 only adds restrictions when copying binaries. A pure source based distribution like Gentoo has hardly any issues at all. Nobody in their right mind is using Gentoo. If you want it in Linux then it has to be a proper GPL compliant effort. I for one would like this to happen. Tom ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] more ZFS recovery
Victor Latushkin wrote: Hi Tom and all, [EMAIL PROTECTED]:~# uname -a SunOS cs3.kw 5.10 Generic_127127-11 sun4v sparc SUNW,Sun-Fire-T200 Btw, have you considered opening support call for this issue? As a follow up to the whole story, with the fantastic help of Victor, the failed pool is now imported and functional thanks to the redundancy in the meta data. This does however highlight the need and practical application of a fsck-like tool. Fine to say that if ZFS can't guarantee my data then I should restore from backups so I know what I've got, but in the case of this 42T device that would take days. Something to think about, Tom ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] more ZFS recovery
Victor Latushkin wrote: Hi Tom and all, Tom Bird wrote: Hi, Have a problem with a ZFS on a single device, this device is 48 1T SATA drives presented as a 42T LUN via hardware RAID 6 on a SAS bus which had a ZFS on it as a single device. There was a problem with the SAS bus which caused various errors including the inevitable kernel panic, the thing came back up with 3 out of 4 zfs mounted. It would be nice to see a panic stack. I'm afraid I don't have that but now have an open connection to the terminal server logging everything in case it should happen again. [EMAIL PROTECTED]:~# zpool import -f content cannot import 'content': I/O error As long as it does not panic and just returns I/O error which is rather generic, you may try to dig a little bit deeper with DTrace to have a chance to see where this I/O error is generated first, e.g. something like this with the attached dtrace script: dtrace -s /path/to/script -c zpool import -f content dtrace output was 6MB, a bit rude to post to the list so I've uploaded it here: http://picard.portfast.net/~tom/import.txt It is also interesting what impact SAS bus problem had on the storage controller. Btw, what is storage controller in question here? The controller is an LSI Logic PCI express with 2 external SAS ports which runs to an eonstor 2u 12 disk RAID chassis with 3 JBOD packs daisy chained from that. It seems I can't run the JBODs directly to the SAS controller when using SATA drives (may be a different story with proper SAS) and the RAID box has no JBOD mode so the redundancy has to stay in the box and can't be transferred to ZFS. The entire faulted array reads cleanly at /dev/rdsk level into /dev/null. There are 4 such arrays connected to the server via two SAS cards with a ZFS on each one, the supplied internal SAS card and an ixgb NIC are the only other cards installed. System boots from the standard internal disks. [EMAIL PROTECTED]:~# uname -a SunOS cs3.kw 5.10 Generic_127127-11 sun4v sparc SUNW,Sun-Fire-T200 Btw, have you considered opening support call for this issue? Would have thought that unless they have a secret zfsck utility there's probably not much they can do. It's not a Sun disk array or Sun branded SAS card. thanks -- Tom // www.portfast.co.uk -- internet services and consultancy // hosting from 1.65 per domain ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] more ZFS recovery
Hi, Have a problem with a ZFS on a single device, this device is 48 1T SATA drives presented as a 42T LUN via hardware RAID 6 on a SAS bus which had a ZFS on it as a single device. There was a problem with the SAS bus which caused various errors including the inevitable kernel panic, the thing came back up with 3 out of 4 zfs mounted. I've tried reading the partition table with format, works fine, also can dd the first 100G from the device quite happily so the communication issue appears resolved however the device just won't mount. Googling around I see that ZFS does have features designed to reduce the impact of corruption at a particular point, multiple meta data copies and so on, however commands to help me tidy up a zfs will only run once the thing has been mounted. Would be grateful for any ideas, relevant output here: [EMAIL PROTECTED]:~# zpool import pool: content id: 14205780542041739352 state: FAULTED status: The pool metadata is corrupted. action: The pool cannot be imported due to damaged devices or data. The pool may be active on on another system, but can be imported using the '-f' flag. see: http://www.sun.com/msg/ZFS-8000-72 config: content FAULTED corrupted data c2t9d0ONLINE [EMAIL PROTECTED]:~# zpool import content cannot import 'content': pool may be in use from other system use '-f' to import anyway [EMAIL PROTECTED]:~# zpool import -f content cannot import 'content': I/O error [EMAIL PROTECTED]:~# uname -a SunOS cs3.kw 5.10 Generic_127127-11 sun4v sparc SUNW,Sun-Fire-T200 Thanks -- Tom // www.portfast.co.uk -- internet services and consultancy // hosting from 1.65 per domain ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
