Re: failed to create gmirror with the handbook instructions
Thanks very much. Please could I make a suggestion that this be included in the handbook page? On 8 Oct 2013 01:31, Warren Block wbl...@wonkity.com wrote: On Tue, 8 Oct 2013, Andy Zammy wrote: Hi, I used the second section of the handbook (20.4) to create a gmirror. In my particular setup I had a 1GB /, 6GB swap, 1GB /tmp and the rest of the 1TB drive was left for /usr I had to deviate from the handbook when it came to running the dump + restore commands, as the dump failed due to an issue with the journalling. To get around this problem, I dropped into single user mode, so I could remount root as read-only. The dump commands then worked. It specified in the handbook to restart the machine, and boot from ada1. It was at this point that I noticed something wasn't quite right. There was a spew of 'not found/no such file or directory' messages. These were all trying to reference libs and binaries that live in /usr. I boot into single user mode, and upon checking the other partitions, I notice that /tmp and /usr are empty, apart from a .snap file, and the restoresymtable file. Please could someone help me troubleshoot this problem? Let me know if you need any more info, and I'll post it up asap. dump does not work reliably on filesystems with SUJ enabled. Turn off SUJ on the filesystems to be dumped by booting in single-user mode and running tunefs -j disable /dev/ada0whatever Do each filesystem, then use dump. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: failed to create gmirror with the handbook instructions
This is actually trickier than it first looked. First I got into single user mode by supplying 'shutdown now', but the tunefs commands all failed with the following: #tunefs -j disable /dev/ada0s1a Clearing journal flags from inode 4 tunefs: Failed to write journal inode: Operation not permitted tunefs: soft updates journalling cleared but soft updates still set. tunefs: remove .sujournal to reclaim space tunefs: /dev/ada0s1a: failed to write superblock I tried the dump command on the off-chance, and it failed with the original errors. Is there anything you can recommend? I then noticed you specified to boot into single user more, so I restarted the machine, with only ada0 attached. Because the handbook wants me to use the mirror/gm0sX devices, I swapped my fstab file back to the original. The boot loader now only seems to recognise the mirror/gm0 nodes, the original ada0sX are gone (though ada0 still shows up). I'm not sure if it's acceptable to do the dump by booting the 1st hard drive using the mirror/gm0, and then dump to the 2nd hard drive by mounting what will be ada1sX. Is this okay to do? On 8 October 2013 01:31, Warren Block wbl...@wonkity.com wrote: On Tue, 8 Oct 2013, Andy Zammy wrote: Hi, I used the second section of the handbook (20.4) to create a gmirror. In my particular setup I had a 1GB /, 6GB swap, 1GB /tmp and the rest of the 1TB drive was left for /usr I had to deviate from the handbook when it came to running the dump + restore commands, as the dump failed due to an issue with the journalling. To get around this problem, I dropped into single user mode, so I could remount root as read-only. The dump commands then worked. It specified in the handbook to restart the machine, and boot from ada1. It was at this point that I noticed something wasn't quite right. There was a spew of 'not found/no such file or directory' messages. These were all trying to reference libs and binaries that live in /usr. I boot into single user mode, and upon checking the other partitions, I notice that /tmp and /usr are empty, apart from a .snap file, and the restoresymtable file. Please could someone help me troubleshoot this problem? Let me know if you need any more info, and I'll post it up asap. dump does not work reliably on filesystems with SUJ enabled. Turn off SUJ on the filesystems to be dumped by booting in single-user mode and running tunefs -j disable /dev/ada0whatever Do each filesystem, then use dump. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: failed to create gmirror with the handbook instructions
I On 8 October 2013 01:31, Warren Block wbl...@wonkity.com wrote: On Tue, 8 Oct 2013, Andy Zammy wrote: Hi, I used the second section of the handbook (20.4) to create a gmirror. In my particular setup I had a 1GB /, 6GB swap, 1GB /tmp and the rest of the 1TB drive was left for /usr I had to deviate from the handbook when it came to running the dump + restore commands, as the dump failed due to an issue with the journalling. To get around this problem, I dropped into single user mode, so I could remount root as read-only. The dump commands then worked. It specified in the handbook to restart the machine, and boot from ada1. It was at this point that I noticed something wasn't quite right. There was a spew of 'not found/no such file or directory' messages. These were all trying to reference libs and binaries that live in /usr. I boot into single user mode, and upon checking the other partitions, I notice that /tmp and /usr are empty, apart from a .snap file, and the restoresymtable file. Please could someone help me troubleshoot this problem? Let me know if you need any more info, and I'll post it up asap. dump does not work reliably on filesystems with SUJ enabled. Turn off SUJ on the filesystems to be dumped by booting in single-user mode and running tunefs -j disable /dev/ada0whatever Do each filesystem, then use dump. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: failed to create gmirror with the handbook instructions
On Tue, 8 Oct 2013, Andy Zammy wrote: This is actually trickier than it first looked. First I got into single user mode by supplying 'shutdown now', but the tunefs commands all failed with the following: #tunefs -j disable /dev/ada0s1a Clearing journal flags from inode 4 tunefs: Failed to write journal inode: Operation not permitted tunefs: soft updates journalling cleared but soft updates still set. tunefs: remove .sujournal to reclaim space tunefs: /dev/ada0s1a: failed to write superblock I tried the dump command on the off-chance, and it failed with the original errors. Is there anything you can recommend? I then noticed you specified to boot into single user more, so I restarted the machine, with only ada0 attached. Because the handbook wants me to use the mirror/gm0sX devices, I swapped my fstab file back to the original. The boot loader now only seems to recognise the mirror/gm0 nodes, the original ada0sX are gone (though ada0 still shows up). I don't know what would do that. The device nodes on the original drive should be untouched until it is added back to the mirror. What does gpart show ada0s1 show? Did you make a backup of the original drive first? Is there an entry for vfs.root.mountfrom in /boot/loader.conf? I'm not sure if it's acceptable to do the dump by booting the 1st hard drive using the mirror/gm0, and then dump to the 2nd hard drive by mounting what will be ada1sX. Is this okay to do? Sorry, I don't quite understand the question. The mirror will not be usable until a good copy of the original drive is made to it. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: failed to create gmirror with the handbook instructions
# gpart show ada0s1 gpart: No such geom: ada0s1 By the way, this is after a restart of the machine. There's nothing to back up, I'm installing a fresh os, so I just install on one drive, plug the other in, and start following the handbook instructions for this method. So the only thing in loader.conf is geom_mirror_load=YES. I'll rephrase the question: given that the handbook originally wanted me to dump from ada0s1 to the mounted mirror/gm0s1 (which was ada1 at the time), and I cannot do this, would it be enough to dump from mirror/gm0s1 (which is what ada0 is now mounted as), to ada1s1 (even though this *should* be the other way around, it's equivalent as far as i can see, isn't it?)? On 8 October 2013 22:59, Warren Block wbl...@wonkity.com wrote: On Tue, 8 Oct 2013, Andy Zammy wrote: This is actually trickier than it first looked. First I got into single user mode by supplying 'shutdown now', but the tunefs commands all failed with the following: #tunefs -j disable /dev/ada0s1a Clearing journal flags from inode 4 tunefs: Failed to write journal inode: Operation not permitted tunefs: soft updates journalling cleared but soft updates still set. tunefs: remove .sujournal to reclaim space tunefs: /dev/ada0s1a: failed to write superblock I tried the dump command on the off-chance, and it failed with the original errors. Is there anything you can recommend? I then noticed you specified to boot into single user more, so I restarted the machine, with only ada0 attached. Because the handbook wants me to use the mirror/gm0sX devices, I swapped my fstab file back to the original. The boot loader now only seems to recognise the mirror/gm0 nodes, the original ada0sX are gone (though ada0 still shows up). I don't know what would do that. The device nodes on the original drive should be untouched until it is added back to the mirror. What does gpart show ada0s1 show? Did you make a backup of the original drive first? Is there an entry for vfs.root.mountfrom in /boot/loader.conf? I'm not sure if it's acceptable to do the dump by booting the 1st hard drive using the mirror/gm0, and then dump to the 2nd hard drive by mounting what will be ada1sX. Is this okay to do? Sorry, I don't quite understand the question. The mirror will not be usable until a good copy of the original drive is made to it. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: failed to create gmirror with the handbook instructions
On Tue, 8 Oct 2013, Andy Zammy wrote: Thanks very much. Please could I make a suggestion that this be included in the handbook page? Please do not top-post, it makes replies more difficult. I have added a warning about SUJ to the top of the gmirror section in the Handbook. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: failed to create gmirror with the handbook instructions
On Tue, 8 Oct 2013, Andy Zammy wrote: # gpart show ada0s1 gpart: No such geom: ada0s1 By the way, this is after a restart of the machine. There's nothing to back up, I'm installing a fresh os, so I just install on one drive, plug the other in, and start following the handbook instructions for this method. So the only thing in loader.conf is geom_mirror_load=YES. I'll rephrase the question: given that the handbook originally wanted me to dump from ada0s1 to the mounted mirror/gm0s1 (which was ada1 at the time), and I cannot do this, would it be enough to dump from mirror/gm0s1 (which is what ada0 is now mounted as), to ada1s1 (even though this *should* be the other way around, it's equivalent as far as i can see, isn't it?)? There is not much point in dumping from the mirror to another drive. The dump/restore is how the single drive is copied to the mirror. On a fresh install, use the Shell mode of the installer to set up the mirror, then install directly to it. There are some instructions on mountpoints in the bsdinstall man page. This will avoid the lag of waiting for the second drive to sync. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: failed to create gmirror with the handbook instructions
Andy Zammy wrote: # gpart show ada0s1 gpart: No such geom: ada0s1 By the way, this is after a restart of the machine. There's nothing to back up, I'm installing a fresh os, so I just install on one drive, plug the other in, and start following the handbook instructions for this method. So the only thing in loader.conf is geom_mirror_load=YES. [snip] Since you are beginning to reinstall from scratch, please allow/forgive a small interjection from some of my recent experience with this. Warren is more knowledgeable on this than I am, and I have followed many of his instructions in the past. With the shift towards GPT and away from the old DOS mbr/partition table stuff of the past, the current Handbook pages reflect this. The central point of contention arises from the fact that GPT, GEOM (gmirror), and many hardware RAID controllers require to claim the very last sector of a drive to store their metadata. Obviously, the effect of this collision is a whoever wrote last wrote best - so you can't use combinations of things that all want this sector. The most simple gmirroring is to slice an entire drive, with partitions contained within. The very end of the drive must NOT have any file system on it, and this is usually the case by default as most of the time slicing/partitioning leaves a little free space at the end anyway. This will not work with GPT; only with the old DOS compatible mbr and disklabel scheme. In order to use GPT and gmirror together you gmirror individual partitions (as opposed to the slice) , e.g. gmirror will write its metadata at the end of each partition leaving the very last sector at the end of the drive for GPT. This is what the content on the relevant Handbook pages reflects. More complicated, but allows for the demise of the ancient DOS/mbr partitioning. Notice that if you combine GPT and a hardware RAID controller card the same collision problem noted previously can still happen. If you utilize the BIOS on the controller card for anything it will save its metadata on the last drive sector. When not faced with terabyte sized humongous volumes and the huge amount of time an fsck will consume, the old DOS way with disklabel is still an option that works. The main reason for the journaling is to sidestep waiting for a very long fsck on a huge volume to run to completion before finishing a boot into a cleaned up/repaired file system. If your drive volume is small this is not so much a problem. Indeed my old gateway/firewall/IDS router box I did the old DOS/mbr scheme with gmirror (the old single-slice entire drive and mirror the drive) as the pair of drives are ancient 74GB Raptors. On my web/database test box I did go the GPT and SUJ+journaling route but am not using any mirroring here (yet). I have not experienced any problems with dump - but I also do not use the -L switch. It will show an error/warning about not dumping a live file system this way but I go ahead and do it anyway. IIRC the dump problem you may be seeing may be related to drive snapshotting. The caveat is I can sort of 'get away' with it as my boxen are largely quiescent, but would hesitate to do this on something like a public web/database box that was continually being hammered with lots of traffic. Just tossing out some ideas for your perusal and consideration. The way I used the old DOS/mbr and disklabel scheme on my router machine is very simple, quick to do, and has survived a few power outages now with no data loss (other than the time it takes to rebuild which it does automagically on boot). On the 74GB Raptors this rebuild takes about twenty minutes. Your situation and needs may force you in a different direction. Hence, the proverbial YMMV applies. FWIW. Now for to finally get around to purchasing a new UPS to replace the old one that went up in smoke and died horribly... -Mike ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: failed to create gmirror with the handbook instructions
I tried creating the mirror before the install. As the drives are now mirrored, the installer picked up on the face that there are two gm0 nodes - one on each hard drive. I installed onto ada0's gm0 node. After it reboots, the bootloader stops at the manual prompt. From what I can see that's not dissapeared up the screen, it tried and failed to mount from mirror/gm0s1a with error 19. I had to mount from ada0s1a in order for the boot to get further, but as it's been installed to boot from gm0s1x, it stops after it mounts /. After having checked my partition setup many times at this point, I know for a fact there's a rather large 500MB section free at the end of my hard drives with this partition set up. Is there any reason I can't just install as normal, do a 'gmirror label gm0 ada0', and then do a 'gmirror insert gm0 ada1', before changing my fstab to use mirror/gm0? I can't see why dumping and restoring is necessary, it's just manually doing what gmirror is there for in the first place. Correct me if I'm wrong :) On 9 October 2013 00:11, Warren Block wbl...@wonkity.com wrote: On Tue, 8 Oct 2013, Andy Zammy wrote: # gpart show ada0s1 gpart: No such geom: ada0s1 By the way, this is after a restart of the machine. There's nothing to back up, I'm installing a fresh os, so I just install on one drive, plug the other in, and start following the handbook instructions for this method. So the only thing in loader.conf is geom_mirror_load=YES. I'll rephrase the question: given that the handbook originally wanted me to dump from ada0s1 to the mounted mirror/gm0s1 (which was ada1 at the time), and I cannot do this, would it be enough to dump from mirror/gm0s1 (which is what ada0 is now mounted as), to ada1s1 (even though this *should* be the other way around, it's equivalent as far as i can see, isn't it?)? There is not much point in dumping from the mirror to another drive. The dump/restore is how the single drive is copied to the mirror. On a fresh install, use the Shell mode of the installer to set up the mirror, then install directly to it. There are some instructions on mountpoints in the bsdinstall man page. This will avoid the lag of waiting for the second drive to sync. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
failed to create gmirror with the handbook instructions
Hi, I used the second section of the handbook (20.4) to create a gmirror. In my particular setup I had a 1GB /, 6GB swap, 1GB /tmp and the rest of the 1TB drive was left for /usr I had to deviate from the handbook when it came to running the dump + restore commands, as the dump failed due to an issue with the journalling. To get around this problem, I dropped into single user mode, so I could remount root as read-only. The dump commands then worked. It specified in the handbook to restart the machine, and boot from ada1. It was at this point that I noticed something wasn't quite right. There was a spew of 'not found/no such file or directory' messages. These were all trying to reference libs and binaries that live in /usr. I boot into single user mode, and upon checking the other partitions, I notice that /tmp and /usr are empty, apart from a .snap file, and the restoresymtable file. Please could someone help me troubleshoot this problem? Let me know if you need any more info, and I'll post it up asap. Kind Regards Andy ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: failed to create gmirror with the handbook instructions
On Tue, 8 Oct 2013, Andy Zammy wrote: Hi, I used the second section of the handbook (20.4) to create a gmirror. In my particular setup I had a 1GB /, 6GB swap, 1GB /tmp and the rest of the 1TB drive was left for /usr I had to deviate from the handbook when it came to running the dump + restore commands, as the dump failed due to an issue with the journalling. To get around this problem, I dropped into single user mode, so I could remount root as read-only. The dump commands then worked. It specified in the handbook to restart the machine, and boot from ada1. It was at this point that I noticed something wasn't quite right. There was a spew of 'not found/no such file or directory' messages. These were all trying to reference libs and binaries that live in /usr. I boot into single user mode, and upon checking the other partitions, I notice that /tmp and /usr are empty, apart from a .snap file, and the restoresymtable file. Please could someone help me troubleshoot this problem? Let me know if you need any more info, and I'll post it up asap. dump does not work reliably on filesystems with SUJ enabled. Turn off SUJ on the filesystems to be dumped by booting in single-user mode and running tunefs -j disable /dev/ada0whatever Do each filesystem, then use dump. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: to gmirror or to ZFS
But then zfs doesn't access every block on the disk does it, only the allocated ones On 20 July 2013 21:07, Daniel Feenberg feenb...@nber.org wrote: On Sat, 20 Jul 2013, Steve O'Hara-Smith wrote: On Sat, 20 Jul 2013 18:14:20 +0100 Frank Leonhardt fra...@fjl.co.uk wrote: It's worth noting, as a warning for anyone who hasn't been there, that the number of times a second drive in a RAID system fails during a rebuild is higher than would be expected. During a rebuild the remaining drives get thrashed, hot, and if they're on the edge, that's when they're going to go. And at the most inconvenient time. Okay - obvious when you think about it, but this tends to be too late. Having the cabinet stuffed full of nominally identical drives bought at the same time from the same supplier tends to add to the probability that more than one drive is on the edge when one goes. It's a pity there are now only two manufacturers of spinning rust. Often this is presummed to be the reason for double failures close in time, also common mode failures such as environment, a defective power supply or excess voltage can be blamed. I have to think that the most common cause for a second failure soon after the first is that a failed drive often isn't detected until a particular sector is read or written. Since the resilvering reads and writes every sector on multiple disks, including unused sectors, it can detect latent problems that may have existed since the drive was new but which haven't been used for data yet, or have gone bad since the last write, but haven't been read since. The ZFS scrub processes only sectors with data, so it provides only partial protection against double failures. Daniel Feenberg NBER -- Steve O'Hara-Smith st...@sohara.org __**_ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/**mailman/listinfo/freebsd-**questionshttp://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-** unsubscr...@freebsd.org freebsd-questions-unsubscr...@freebsd.org __**_ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/**mailman/listinfo/freebsd-**questionshttp://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-** unsubscr...@freebsd.org freebsd-questions-unsubscr...@freebsd.org ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: to gmirror or to ZFS
On 21/07/2013 17:31, Steve O'Hara-Smith wrote: On Sun, 21 Jul 2013 14:13:39 +0930 Shane Ambler free...@shaneware.biz wrote: On 21/07/2013 04:42, Steve O'Hara-Smith wrote: It's a pity there are now only two manufacturers of spinning rust. I thought there was three left - Seagate WD and Toshiba I assumed Toshiba were out of the game, I've never seen anything bigger than 500GB with a Toshiba label. I have a 2.5 1TB Toshiba USB drive here. I see Toshiba 2 and 3TB 3.5 listed online. As I recall the Hitachi selloff - WD got the 2.5 Toshiba got the 3.5 I think the split was the only way to get the takeover approved. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: to gmirror or to ZFS
On Sun, 21 Jul 2013 14:13:39 +0930 Shane Ambler free...@shaneware.biz wrote: On 21/07/2013 04:42, Steve O'Hara-Smith wrote: It's a pity there are now only two manufacturers of spinning rust. I thought there was three left - Seagate WD and Toshiba I assumed Toshiba were out of the game, I've never seen anything bigger than 500GB with a Toshiba label. -- Steve O'Hara-Smith | Directable Mirror Arrays C:WIN | A better way to focus the sun The computer obeys and wins.|licences available see You lose and Bill collects. |http://www.sohara.org/ ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: to gmirror or to ZFS
Steve O'Hara-Smith st...@sohara.org wrote: It's a pity there are now only two manufacturers of spinning rust. I didn't think there were _any_! Haven't oxide-coated platters gone the way of the dodo bird? ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: to gmirror or to ZFS
On Sun, 21 Jul 2013 00:27:01 -0700 per...@pluto.rain.com (Perry Hutchison) wrote: Steve O'Hara-Smith st...@sohara.org wrote: It's a pity there are now only two manufacturers of spinning rust. I didn't think there were _any_! Haven't oxide-coated platters gone the way of the dodo bird? Ah the technicalities, this is a software group :-) -- Steve O'Hara-Smith | Directable Mirror Arrays C:WIN | A better way to focus the sun The computer obeys and wins.|licences available see You lose and Bill collects. |http://www.sohara.org/ ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: to gmirror or to ZFS
On 16/07/2013 20:48, Charles Swiger wrote: Hi-- On Jul 16, 2013, at 11:27 AM, Johan Hendriks joh.hendr...@gmail.com wrote: Well, don't do that. :-) When the server reboots because of a powerfailure at night, then it boots. Then it starts to rebuild the mirror on its own, and later the fsck kicks in. Not much i can do about it. Maybe i should have done it without the automatic attachment for a new device. It's normally the case that getting a hot spare automatically attached should be fine, but not if you also have the box go down entirely and need to fsck. I'm more used to needing to explicitly physically swap out a failed mirror component, in which case one can make sure the system is OK before the replacement drive goes in. Agreed. Blaming gmirror for this kind of thing overlooks the overall design and operating procedures of the system, and assuming ZFS would have been any better may be wishful thinking. I've had plenty of gmirror crashes over the years, and they have all been recoverable. One thing I never allow it to do is to rebuild automatically. That's something for a human to initiate once the problem has been identified, and if it's flaky power in the data centre the job is postponed until I'm satisfied it's not going to drop during the rebuild. IME, one power failure is normally followed by several more. It's worth noting, as a warning for anyone who hasn't been there, that the number of times a second drive in a RAID system fails during a rebuild is higher than would be expected. During a rebuild the remaining drives get thrashed, hot, and if they're on the edge, that's when they're going to go. And at the most inconvenient time. Okay - obvious when you think about it, but this tends to be too late. Regards, Frank. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: to gmirror or to ZFS
On Sat, 20 Jul 2013 18:14:20 +0100 Frank Leonhardt fra...@fjl.co.uk wrote: It's worth noting, as a warning for anyone who hasn't been there, that the number of times a second drive in a RAID system fails during a rebuild is higher than would be expected. During a rebuild the remaining drives get thrashed, hot, and if they're on the edge, that's when they're going to go. And at the most inconvenient time. Okay - obvious when you think about it, but this tends to be too late. Having the cabinet stuffed full of nominally identical drives bought at the same time from the same supplier tends to add to the probability that more than one drive is on the edge when one goes. It's a pity there are now only two manufacturers of spinning rust. -- Steve O'Hara-Smith st...@sohara.org ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: to gmirror or to ZFS
On Sat, 20 Jul 2013, Steve O'Hara-Smith wrote: On Sat, 20 Jul 2013 18:14:20 +0100 Frank Leonhardt fra...@fjl.co.uk wrote: It's worth noting, as a warning for anyone who hasn't been there, that the number of times a second drive in a RAID system fails during a rebuild is higher than would be expected. During a rebuild the remaining drives get thrashed, hot, and if they're on the edge, that's when they're going to go. And at the most inconvenient time. Okay - obvious when you think about it, but this tends to be too late. Having the cabinet stuffed full of nominally identical drives bought at the same time from the same supplier tends to add to the probability that more than one drive is on the edge when one goes. It's a pity there are now only two manufacturers of spinning rust. Often this is presummed to be the reason for double failures close in time, also common mode failures such as environment, a defective power supply or excess voltage can be blamed. I have to think that the most common cause for a second failure soon after the first is that a failed drive often isn't detected until a particular sector is read or written. Since the resilvering reads and writes every sector on multiple disks, including unused sectors, it can detect latent problems that may have existed since the drive was new but which haven't been used for data yet, or have gone bad since the last write, but haven't been read since. The ZFS scrub processes only sectors with data, so it provides only partial protection against double failures. Daniel Feenberg NBER -- Steve O'Hara-Smith st...@sohara.org ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: to gmirror or to ZFS
On 21/07/2013 04:42, Steve O'Hara-Smith wrote: It's a pity there are now only two manufacturers of spinning rust. I thought there was three left - Seagate WD and Toshiba ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: to gmirror or to ZFS
On Jul 16, 2013, at 11:42 AM, Warren Block wrote: On Tue, 16 Jul 2013, aurfalien wrote: On Jul 16, 2013, at 2:41 AM, Shane Ambler wrote: I doubt that you would save any ram having the os on a non-zfs drive as you will already be using zfs chances are that non-zfs drives would only increase ram usage by adding a second cache. zfs uses it's own cache system and isn't going to share it's cache with other system managed drives. I'm not actually certain if the system cache still sits above zfs cache or not, I think I read it bypasses the traditional drive cache. For zfs cache you can set the max usage by adjusting vfs.zfs.arc_max that is a system wide setting and isn't going to increase if you have two zpools. Tip: set the arc_max value - by default zfs will use all physical ram for cache, set it to be sure you have enough ram left for any services you want running. Have you considered using one or both SSD drives with zfs? They can be added as cache or log devices to help performance. See man zpool under Intent Log and Cache Devices. This is a very interesting point. In terms if SSDs for cache, I was planning on using a pair of Samsung Pro 512GB SSDs for this purpose (which I haven't bought yet). But I tire of buying stuff, so I have a pair of 40GB Intel SSDs for use as sys disks and several Intel 160GB SSDs lying around that I can combine with the existing 256GB SSDs for a cache. Then use my 36x3TB for the beasty NAS. Agreed that 256G mirrored SSDs are kind of wasted as system drives. The 40G mirror sounds ideal. Update; I went with ZFS as I didn't want to confuse the toolset needed to support this server. Although gmirror is not hard to figure out, I wanted consistency in systems. So I've a booted 9.1 rel using a mirrored ZFS system disk. The drives do support TRIM but am unsure how this plays with ZFS. I did the standard partition scheme of; root@kronos:/root # gpart show = 34 78165293 da0 GPT (37G) 34 1281 freebsd-boot (64k) 162 6 - free - (3.0k) 168 83886082 freebsd-swap (4.0G) 8388776 697765443 freebsd-zfs (33G) 78165320 7 - free - (3.5k) = 34 78165293 da1 GPT (37G) 34 1281 freebsd-boot (64k) 162 6 - free - (3.0k) 168 83886082 freebsd-swap (4.0G) 8388776 697765443 freebsd-zfs (33G) 78165320 7 - free - (3.5k) At any rate, thank you for the replies, very much appreciate it. Especially since building a rather large production worthy NAS not knowing a lick of freeBSD. The reasons going with freeBSD are 2 fold; ZFS stability,seems a better marriage then ZOL. Correctly provides NFS pre attributes on write reply; mtime. Linux does not. While its a steep learning curve, the 2 points above require the use of freeBSD or alike. - aurf ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: to gmirror or to ZFS
You would in theory as from what i remember every zfs filesystem takes up 64 kb of ram, so the savings could be massive 8) On 16 July 2013 10:41, Shane Ambler free...@shaneware.biz wrote: On 16/07/2013 14:41, aurfalien wrote: On Jul 15, 2013, at 9:23 PM, Warren Block wrote: On Mon, 15 Jul 2013, aurfalien wrote: ... thats the question :) At any rate, I'm building a rather large 100+TB NAS using ZFS. However for my OS, should I also ZFS or simply gmirror as I've a dedicated pair of 256GB SSD drives for it. I didn't ask for SSD sys drives, this system just came with em. This is more of a best practices q. ZFS has data integrity checking, gmirror has low RAM overhead. gmirror is, at present, restricted to MBR partitioning due to metadata conflicts with GPT, so 2TB is the maximum size. Best practices... depends on your use. gmirror for the system leaves more RAM for ZFS. Perfect, thanks Warren. Just what I was looking for. I doubt that you would save any ram having the os on a non-zfs drive as you will already be using zfs chances are that non-zfs drives would only increase ram usage by adding a second cache. zfs uses it's own cache system and isn't going to share it's cache with other system managed drives. I'm not actually certain if the system cache still sits above zfs cache or not, I think I read it bypasses the traditional drive cache. For zfs cache you can set the max usage by adjusting vfs.zfs.arc_max that is a system wide setting and isn't going to increase if you have two zpools. Tip: set the arc_max value - by default zfs will use all physical ram for cache, set it to be sure you have enough ram left for any services you want running. Have you considered using one or both SSD drives with zfs? They can be added as cache or log devices to help performance. See man zpool under Intent Log and Cache Devices. __**_ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/**mailman/listinfo/freebsd-**questionshttp://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-** unsubscr...@freebsd.org freebsd-questions-unsubscr...@freebsd.org ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: to gmirror or to ZFS
not recommended anymore you should run SU+J if your version supports it On 17 July 2013 00:08, Nikos Vassiliadis nv...@gmx.com wrote: On 07/16/13 21:27, Johan Hendriks wrote: Op dinsdag 16 juli 2013 schreef Charles Swiger (cswi...@mac.com) het volgende: Hi-- On Jul 16, 2013, at 10:33 AM, Johan Hendriks joh.hendr...@gmail.com** javascript:; wrote: [ ... ] I would us a zfs for the os. I have a couple of servers that did not survive a power failure with gmirror. The problems i had was when the power failed one disk was in a rebuilding state and then when the background fsck started or was busy for some time it would crash the whole server. Well, don't do that. :-) When the server reboots because of a powerfailure at night, then it boots. Then it starts to rebuild the mirror on its own, and later the fsck kicks in. Not much i can do about it. You could add geom_journal which will minimize the time of fsck to a second or something like that. Then you don't have to use background fsck anymore. Actually geom_journal's manual page mentions an interesting side-effect of geom_journal over a geom_mirror: you can turn off component synchronization. Geom_journal will re-play last writes so whatever was changed just before the crash will be re-written to both disks. I haven't used this but it makes sense in theory. Maybe i should have done it without the automatic attachment for a new device. I always turn off automatic synchronization or stale components as well. It seems to me that people don't really use geom_journal or maybe they just don't talk about it like it's some sort of secret:) just my two cents, Nikos __**_ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/**mailman/listinfo/freebsd-**questionshttp://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-** unsubscr...@freebsd.org freebsd-questions-unsubscr...@freebsd.org ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: to gmirror or to ZFS
On 16/07/2013 14:41, aurfalien wrote: On Jul 15, 2013, at 9:23 PM, Warren Block wrote: On Mon, 15 Jul 2013, aurfalien wrote: ... thats the question :) At any rate, I'm building a rather large 100+TB NAS using ZFS. However for my OS, should I also ZFS or simply gmirror as I've a dedicated pair of 256GB SSD drives for it. I didn't ask for SSD sys drives, this system just came with em. This is more of a best practices q. ZFS has data integrity checking, gmirror has low RAM overhead. gmirror is, at present, restricted to MBR partitioning due to metadata conflicts with GPT, so 2TB is the maximum size. Best practices... depends on your use. gmirror for the system leaves more RAM for ZFS. Perfect, thanks Warren. Just what I was looking for. I doubt that you would save any ram having the os on a non-zfs drive as you will already be using zfs chances are that non-zfs drives would only increase ram usage by adding a second cache. zfs uses it's own cache system and isn't going to share it's cache with other system managed drives. I'm not actually certain if the system cache still sits above zfs cache or not, I think I read it bypasses the traditional drive cache. For zfs cache you can set the max usage by adjusting vfs.zfs.arc_max that is a system wide setting and isn't going to increase if you have two zpools. Tip: set the arc_max value - by default zfs will use all physical ram for cache, set it to be sure you have enough ram left for any services you want running. Have you considered using one or both SSD drives with zfs? They can be added as cache or log devices to help performance. See man zpool under Intent Log and Cache Devices. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: to gmirror or to ZFS
On 16/07/2013 10:41, Shane Ambler wrote: On 16/07/2013 14:41, aurfalien wrote: On Jul 15, 2013, at 9:23 PM, Warren Block wrote: On Mon, 15 Jul 2013, aurfalien wrote: ... thats the question :) At any rate, I'm building a rather large 100+TB NAS using ZFS. However for my OS, should I also ZFS or simply gmirror as I've a dedicated pair of 256GB SSD drives for it. I didn't ask for SSD sys drives, this system just came with em. This is more of a best practices q. ZFS has data integrity checking, gmirror has low RAM overhead. gmirror is, at present, restricted to MBR partitioning due to metadata conflicts with GPT, so 2TB is the maximum size. Best practices... depends on your use. gmirror for the system leaves more RAM for ZFS. Perfect, thanks Warren. Just what I was looking for. I doubt that you would save any ram having the os on a non-zfs drive as you will already be using zfs chances are that non-zfs drives would only increase ram usage by adding a second cache. zfs uses it's own cache system and isn't going to share it's cache with other system managed drives. I'm not actually certain if the system cache still sits above zfs cache or not, I think I read it bypasses the traditional drive cache. For zfs cache you can set the max usage by adjusting vfs.zfs.arc_max that is a system wide setting and isn't going to increase if you have two zpools. Tip: set the arc_max value - by default zfs will use all physical ram for cache, set it to be sure you have enough ram left for any services you want running. Have you considered using one or both SSD drives with zfs? They can be added as cache or log devices to help performance. See man zpool under Intent Log and Cache Devices. I agree with the sentiment of using the SSD as ZFS cache - it's possibly the only logical use for them. I guess that with 100Tb worth of Winchesters you're not on a very tight budget, and not too tight on RAM for the OS either. If I was going to do this I'd stick with the OS on UFS and a gmirror because I simply don't trust ZFS. This is based on pure prejudice and inexperience. I know how to arrange disks on a UNIX file system for performance - what to use for swap, where tmp files should go and so on. I also know where every file will be, physically, in the event of trouble. And here's the clincher: If the machine blows up I can simply take one of the mirrored drives, slap it in to some new hardware and I've got a very reasonable chance that it'll boot. Can I do this with ZFS? I get the feeling that the answer is an emphatic maybe. So all things considered, I'd need a good reason not to stick with what I know works reliably and can be recovered in the event of a disaster (UFS), but I'm happy to watch and learn from everyone else's experience! ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: to gmirror or to ZFS
Op dinsdag 16 juli 2013 schreef Frank Leonhardt (fra...@fjl.co.uk) het volgende: On 16/07/2013 10:41, Shane Ambler wrote: On 16/07/2013 14:41, aurfalien wrote: On Jul 15, 2013, at 9:23 PM, Warren Block wrote: On Mon, 15 Jul 2013, aurfalien wrote: ... thats the question :) At any rate, I'm building a rather large 100+TB NAS using ZFS. However for my OS, should I also ZFS or simply gmirror as I've a dedicated pair of 256GB SSD drives for it. I didn't ask for SSD sys drives, this system just came with em. This is more of a best practices q. ZFS has data integrity checking, gmirror has low RAM overhead. gmirror is, at present, restricted to MBR partitioning due to metadata conflicts with GPT, so 2TB is the maximum size. Best practices... depends on your use. gmirror for the system leaves more RAM for ZFS. Perfect, thanks Warren. Just what I was looking for. I doubt that you would save any ram having the os on a non-zfs drive as you will already be using zfs chances are that non-zfs drives would only increase ram usage by adding a second cache. zfs uses it's own cache system and isn't going to share it's cache with other system managed drives. I'm not actually certain if the system cache still sits above zfs cache or not, I think I read it bypasses the traditional drive cache. For zfs cache you can set the max usage by adjusting vfs.zfs.arc_max that is a system wide setting and isn't going to increase if you have two zpools. Tip: set the arc_max value - by default zfs will use all physical ram for cache, set it to be sure you have enough ram left for any services you want running. Have you considered using one or both SSD drives with zfs? They can be added as cache or log devices to help performance. See man zpool under Intent Log and Cache Devices. I agree with the sentiment of using the SSD as ZFS cache - it's possibly the only logical use for them. I guess that with 100Tb worth of Winchesters you're not on a very tight budget, and not too tight on RAM for the OS either. If I was going to do this I'd stick with the OS on UFS and a gmirror because I simply don't trust ZFS. This is based on pure prejudice and inexperience. I know how to arrange disks on a UNIX file system for performance - what to use for swap, where tmp files should go and so on. I also know where every file will be, physically, in the event of trouble. And here's the clincher: If the machine blows up I can simply take one of the mirrored drives, slap it in to some new hardware and I've got a very reasonable chance that it'll boot. Can I do this with ZFS? I get the feeling that the answer is an emphatic maybe. So all things considered, I'd need a good reason not to stick with what I know works reliably and can be recovered in the event of a disaster (UFS), but I'm happy to watch and learn from everyone else's experience! I would us a zfs for the os. I have a couple of servers that did not survive a power failure with gmirror. The problems i had was when the power failed one disk was in a rebuilding state and then when the background fsck started or was busy for some time it would crash the whole server. Removing the disk that was rebuilding resolved the issue. This happened to me more than once. Most of the times it worked as advertised but not always. Before people tell me to use an UPS, i used a UPS but the damn thing gave way itself. Then after it came back from the warranty repair it gave way again. Some times it came back right away, leaving some servers survive and some in the state they where. It was hard to find the cause in the beginning because of the fact some servers did survive the power failure. We did not suspect the UPS at first. Anyway, gmirror did not work for me in all cases. I am now running a few servers with a zfs root. I did not have any problems with them till now (knock on wood). Since reading that swap on zfs root can cause trouble i have a separate freebsd-swap partition for the swap. Gr Johan __**_ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/**mailman/listinfo/freebsd-**questionshttp://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: to gmirror or to ZFS
On Jul 16, 2013, at 2:41 AM, Shane Ambler wrote: On 16/07/2013 14:41, aurfalien wrote: On Jul 15, 2013, at 9:23 PM, Warren Block wrote: On Mon, 15 Jul 2013, aurfalien wrote: ... thats the question :) At any rate, I'm building a rather large 100+TB NAS using ZFS. However for my OS, should I also ZFS or simply gmirror as I've a dedicated pair of 256GB SSD drives for it. I didn't ask for SSD sys drives, this system just came with em. This is more of a best practices q. ZFS has data integrity checking, gmirror has low RAM overhead. gmirror is, at present, restricted to MBR partitioning due to metadata conflicts with GPT, so 2TB is the maximum size. Best practices... depends on your use. gmirror for the system leaves more RAM for ZFS. Perfect, thanks Warren. Just what I was looking for. I doubt that you would save any ram having the os on a non-zfs drive as you will already be using zfs chances are that non-zfs drives would only increase ram usage by adding a second cache. zfs uses it's own cache system and isn't going to share it's cache with other system managed drives. I'm not actually certain if the system cache still sits above zfs cache or not, I think I read it bypasses the traditional drive cache. For zfs cache you can set the max usage by adjusting vfs.zfs.arc_max that is a system wide setting and isn't going to increase if you have two zpools. Tip: set the arc_max value - by default zfs will use all physical ram for cache, set it to be sure you have enough ram left for any services you want running. Have you considered using one or both SSD drives with zfs? They can be added as cache or log devices to help performance. See man zpool under Intent Log and Cache Devices. This is a very interesting point. In terms if SSDs for cache, I was planning on using a pair of Samsung Pro 512GB SSDs for this purpose (which I haven't bought yet). But I tire of buying stuff, so I have a pair of 40GB Intel SSDs for use as sys disks and several Intel 160GB SSDs lying around that I can combine with the existing 256GB SSDs for a cache. Then use my 36x3TB for the beasty NAS. - aurf ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: to gmirror or to ZFS
Op dinsdag 16 juli 2013 schreef Charles Swiger (cswi...@mac.com) het volgende: Hi-- On Jul 16, 2013, at 10:33 AM, Johan Hendriks joh.hendr...@gmail.comjavascript:; wrote: [ ... ] I would us a zfs for the os. I have a couple of servers that did not survive a power failure with gmirror. The problems i had was when the power failed one disk was in a rebuilding state and then when the background fsck started or was busy for some time it would crash the whole server. Well, don't do that. :-) When the server reboots because of a powerfailure at night, then it boots. Then it starts to rebuild the mirror on its own, and later the fsck kicks in. Not much i can do about it. Maybe i should have done it without the automatic attachment for a new device. Seriously, bring up the box on one disk, force a foreground fsck if needed to get the filesystem to known clean state, and then rebuild the mirror. Mixing the mirror rebuild with something like an fsck will just thrash the disks. [ ... ] Before people tell me to use an UPS, i used a UPS but the damn thing gave way itself. Then after it came back from the warranty repair it gave way again. Grr. That's when you want find another UPS vendor. Is apc not the right choice? I think i got a monday morning model. Some times things fail! Regards, -- -Chuck ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: to gmirror or to ZFS
On Tue, 16 Jul 2013, aurfalien wrote: On Jul 16, 2013, at 2:41 AM, Shane Ambler wrote: I doubt that you would save any ram having the os on a non-zfs drive as you will already be using zfs chances are that non-zfs drives would only increase ram usage by adding a second cache. zfs uses it's own cache system and isn't going to share it's cache with other system managed drives. I'm not actually certain if the system cache still sits above zfs cache or not, I think I read it bypasses the traditional drive cache. For zfs cache you can set the max usage by adjusting vfs.zfs.arc_max that is a system wide setting and isn't going to increase if you have two zpools. Tip: set the arc_max value - by default zfs will use all physical ram for cache, set it to be sure you have enough ram left for any services you want running. Have you considered using one or both SSD drives with zfs? They can be added as cache or log devices to help performance. See man zpool under Intent Log and Cache Devices. This is a very interesting point. In terms if SSDs for cache, I was planning on using a pair of Samsung Pro 512GB SSDs for this purpose (which I haven't bought yet). But I tire of buying stuff, so I have a pair of 40GB Intel SSDs for use as sys disks and several Intel 160GB SSDs lying around that I can combine with the existing 256GB SSDs for a cache. Then use my 36x3TB for the beasty NAS. Agreed that 256G mirrored SSDs are kind of wasted as system drives. The 40G mirror sounds ideal. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: to gmirror or to ZFS
Hi-- On Jul 16, 2013, at 10:33 AM, Johan Hendriks joh.hendr...@gmail.com wrote: [ ... ] I would us a zfs for the os. I have a couple of servers that did not survive a power failure with gmirror. The problems i had was when the power failed one disk was in a rebuilding state and then when the background fsck started or was busy for some time it would crash the whole server. Well, don't do that. :-) Seriously, bring up the box on one disk, force a foreground fsck if needed to get the filesystem to known clean state, and then rebuild the mirror. Mixing the mirror rebuild with something like an fsck will just thrash the disks. [ ... ] Before people tell me to use an UPS, i used a UPS but the damn thing gave way itself. Then after it came back from the warranty repair it gave way again. Grr. That's when you want find another UPS vendor. Regards, -- -Chuck ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: to gmirror or to ZFS
Hi-- On Jul 16, 2013, at 11:27 AM, Johan Hendriks joh.hendr...@gmail.com wrote: Well, don't do that. :-) When the server reboots because of a powerfailure at night, then it boots. Then it starts to rebuild the mirror on its own, and later the fsck kicks in. Not much i can do about it. Maybe i should have done it without the automatic attachment for a new device. It's normally the case that getting a hot spare automatically attached should be fine, but not if you also have the box go down entirely and need to fsck. I'm more used to needing to explicitly physically swap out a failed mirror component, in which case one can make sure the system is OK before the replacement drive goes in. [ ... ] Before people tell me to use an UPS, i used a UPS but the damn thing gave way itself. Then after it came back from the warranty repair it gave way again. Grr. That's when you want find another UPS vendor. Is apc not the right choice? I think i got a monday morning model. Some times things fail! APC is decent for desktops, but I'm dubious about them when it comes to entire racks or a DC. I like Leviton's PDUs/MDUs and TVSS; for a medium-sized UPS (10-40 kVA) Liebert and PowerWare (now Eaton) were good. Liebert's PDUs are also pretty good. Regards, -- -Chuck PS: I ran a small DC in NYC with a 20kVA PowerWare 9330 behind a Leviton 57000 TVSS; the Cupertino locals have ~650kVA worth of Bloom boxes and a Cummins diesel genset as a backup just for this building. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: to gmirror or to ZFS
On 07/16/13 21:27, Johan Hendriks wrote: Op dinsdag 16 juli 2013 schreef Charles Swiger (cswi...@mac.com) het volgende: Hi-- On Jul 16, 2013, at 10:33 AM, Johan Hendriks joh.hendr...@gmail.comjavascript:; wrote: [ ... ] I would us a zfs for the os. I have a couple of servers that did not survive a power failure with gmirror. The problems i had was when the power failed one disk was in a rebuilding state and then when the background fsck started or was busy for some time it would crash the whole server. Well, don't do that. :-) When the server reboots because of a powerfailure at night, then it boots. Then it starts to rebuild the mirror on its own, and later the fsck kicks in. Not much i can do about it. You could add geom_journal which will minimize the time of fsck to a second or something like that. Then you don't have to use background fsck anymore. Actually geom_journal's manual page mentions an interesting side-effect of geom_journal over a geom_mirror: you can turn off component synchronization. Geom_journal will re-play last writes so whatever was changed just before the crash will be re-written to both disks. I haven't used this but it makes sense in theory. Maybe i should have done it without the automatic attachment for a new device. I always turn off automatic synchronization or stale components as well. It seems to me that people don't really use geom_journal or maybe they just don't talk about it like it's some sort of secret:) just my two cents, Nikos ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
to gmirror or to ZFS
... thats the question :) At any rate, I'm building a rather large 100+TB NAS using ZFS. However for my OS, should I also ZFS or simply gmirror as I've a dedicated pair of 256GB SSD drives for it. I didn't ask for SSD sys drives, this system just came with em. This is more of a best practices q. Thanks in advance, - aurf ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: to gmirror or to ZFS
On Mon, 15 Jul 2013, aurfalien wrote: ... thats the question :) At any rate, I'm building a rather large 100+TB NAS using ZFS. However for my OS, should I also ZFS or simply gmirror as I've a dedicated pair of 256GB SSD drives for it. I didn't ask for SSD sys drives, this system just came with em. This is more of a best practices q. ZFS has data integrity checking, gmirror has low RAM overhead. gmirror is, at present, restricted to MBR partitioning due to metadata conflicts with GPT, so 2TB is the maximum size. Best practices... depends on your use. gmirror for the system leaves more RAM for ZFS. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: to gmirror or to ZFS
On Jul 15, 2013, at 9:23 PM, Warren Block wrote: On Mon, 15 Jul 2013, aurfalien wrote: ... thats the question :) At any rate, I'm building a rather large 100+TB NAS using ZFS. However for my OS, should I also ZFS or simply gmirror as I've a dedicated pair of 256GB SSD drives for it. I didn't ask for SSD sys drives, this system just came with em. This is more of a best practices q. ZFS has data integrity checking, gmirror has low RAM overhead. gmirror is, at present, restricted to MBR partitioning due to metadata conflicts with GPT, so 2TB is the maximum size. Best practices... depends on your use. gmirror for the system leaves more RAM for ZFS. Perfect, thanks Warren. Just what I was looking for. - aurf ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Troubleshooting a gmirror disk marked broken
On Wed, Jun 26, 2013 at 10:09:33PM -0500, Adam Vande More wrote: On Wed, Jun 26, 2013 at 9:38 PM, Nikola Pavlović n...@riseup.net wrote: Hi, Last night during a massive (~1 year worth :| ) portsnap fetch the server went unresponsive and ssh eventually disconnected. I decided to leave it during the night, and, sure enough, the situation was the same in the morning, so I had to do a hard reset. It came back up, but one of the two gmirror components was marked as broken and deactivated. The hang happened during the 'fetching new files or ports' (~24000 of them, there are currently ~1 snapshots in /var/db/portsnap) phase of postsnap fetch. /var/log/messages was completely silent during the period between the hang and the reset. Googling around I found a mention that it's possible to sometimes get a 'blip'[*] during busy periods, so I decided to just bite the bullet and reinsert the component with # gmirror forget gm0 # gmirror clean ad4 # gmirror insert gm0 ad4 Currently it's syncing and things *seem* OK. My question is how much should I be worried and what could be the cause of this? Is it possible that ports snapshot fetching caused this, or that perhaps it was the other way around (a failing disk causing the machine to choke during the huge portsnap fetch)? How to proceed? :) The messages log definitely shows problems with your io. The smart log of the disks are also at least mildly concerning and indicates the drives are in a preliminary stage of death. Some HD deaths take years to complete. Expect random glitches and intermittent reduced performance as a continuous degradation. You might be able to alleviate some of this by switching to the AHCI driver and bumping up timeouts but at the end of the day 2 flaky disks in a mirror don't inspire confidence. About AHCI, it didn't attach after setting ahci_load=YES in loader.conf so I assumed it wasn't enabled in BIOS. As I don't have physical access to the machine I asked the support to enable it, and presumably they did (that's what they said, and the machine was rebooted when they said they did). But still no luck. It's a VIA 6420 controller and maybe it doesn't support AHCI (couldn't find anything definitive on the net about that). If that's the case, is it even possible that there exists an option to enable it in BIOS? I'm confused because they didn't say it doesn't support it, but explicitly that they enabled it. It's possible to request KVM-over-IP, so I can look for myself, but I don't want to waste time (and install Java just for this) if it's useless. -- To criticize the incompetent is easy; it is more difficult to criticize the competent. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Troubleshooting a gmirror disk marked broken
On Thu, Jun 27, 2013 at 10:06:45AM -0700, Charles Swiger wrote: Hi-- On Jun 27, 2013, at 9:58 AM, Adam Vande More amvandem...@gmail.com wrote: On Thu, Jun 27, 2013 at 10:16 AM, Charles Swiger cswi...@mac.com wrote: If you haven't rebuilt the mirror already, running a full disk read scan against both drives (ie, via dd if=/dev/ad4 of=/dev/null bs=1m or similar) might be prudent. That will help identify/migrate any sectors which are failing but still recoverable via ECC to the spare sectors. I was going to say something like that too but AFAIK sectors aren't remapped on failed reads, has to be written to(dd if=/dev/zero of=/dev/ad4 bs=1m). If it were me, I make sure I had fully tested complete backups before I broke the mirror and did that. If the drive reads a sector with ECC-correctable errors, it's supposed to try to re-write that sector in order to fix up the ECC data. If that write fails, it remaps. Of course, your suggestion of blanking the entire drive and restoring from the mirror or a backup would be best, or perhaps better short of replacing the drive. OK, thank you both for suggestions. It rebuilt fine, and it's working fine. If it starts giving me trouble again I'll try your suggestions, or, ultimately, ask to get the disk replaced (although I don't expect a much better replacement, it's one of those cheap rental servers, you get what you pay :)) I'm still a bit reluctant to run the ports tree update again, but I'll ask on -ports@ for further assistance with that. -- Writing free verse is like playing tennis with the net down. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Troubleshooting a gmirror disk marked broken
On Sat, Jun 29, 2013 at 12:36:59AM +0200, Nikola Pavlović wrote: I'm still a bit reluctant to run the ports tree update again, but I'll ask on -ports@ for further assistance with that. Actually, no need. I retried it and it worked without any problem. -- Fantasies are free. NO!! NO!! It's the thought police ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Troubleshooting a gmirror disk marked broken
On Fri, Jun 28, 2013 at 5:28 PM, Nikola Pavlović n...@riseup.net wrote: About AHCI, it didn't attach after setting ahci_load=YES in loader.conf so I assumed it wasn't enabled in BIOS. As I don't have physical access to the machine I asked the support to enable it, and presumably they did (that's what they said, and the machine was rebooted when they said they did). But still no luck. It's a VIA 6420 controller and maybe it doesn't support AHCI (couldn't find anything definitive on the net about that). This appears to be the case. There may be some sysctl which can alter ata settings that might help like stuff under kern.geom.mirror. It's already been a long time since I've used 8.x so I don't remember everything. Just have to dig around. -- Adam Vande More ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Troubleshooting a gmirror disk marked broken
Hi-- On Jun 26, 2013, at 7:38 PM, Nikola Pavlović n...@riseup.net wrote: [ ... ] At the moment I'm attaching smartctl -a results for both disks (ad4 was marked broken). As I'm completely useless in deciphering smartctl results (apart from being thought by experience to pretty much ignore(tm) the 'Pre-fail' statuses), I'd appreciate assessment from more knowledgeable people. As Adam said, your drives haven't failed, but they are on the way out. Pay close attention to Reallocated_Sector_Ct, especially if it starts jumping upwards. You might also want to check the thermals; the Seagate is running at 52C, which is significantly hotter than it ought to be, except for maybe a laptop. I'd much rather see a drive running below 40C If you haven't rebuilt the mirror already, running a full disk read scan against both drives (ie, via dd if=/dev/ad4 of=/dev/null bs=1m or similar) might be prudent. That will help identify/migrate any sectors which are failing but still recoverable via ECC to the spare sectors. Regards, -- -Chuck ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Troubleshooting a gmirror disk marked broken
On Thu, Jun 27, 2013 at 10:16 AM, Charles Swiger cswi...@mac.com wrote: If you haven't rebuilt the mirror already, running a full disk read scan against both drives (ie, via dd if=/dev/ad4 of=/dev/null bs=1m or similar) might be prudent. That will help identify/migrate any sectors which are failing but still recoverable via ECC to the spare sectors. I was going to say something like that too but AFAIK sectors aren't remapped on failed reads, has to be written to(dd if=/dev/zero of=/dev/ad4 bs=1m). If it were me, I make sure I had fully tested complete backups before I broke the mirror and did that. -- Adam Vande More ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Troubleshooting a gmirror disk marked broken
Hi-- On Jun 27, 2013, at 9:58 AM, Adam Vande More amvandem...@gmail.com wrote: On Thu, Jun 27, 2013 at 10:16 AM, Charles Swiger cswi...@mac.com wrote: If you haven't rebuilt the mirror already, running a full disk read scan against both drives (ie, via dd if=/dev/ad4 of=/dev/null bs=1m or similar) might be prudent. That will help identify/migrate any sectors which are failing but still recoverable via ECC to the spare sectors. I was going to say something like that too but AFAIK sectors aren't remapped on failed reads, has to be written to(dd if=/dev/zero of=/dev/ad4 bs=1m). If it were me, I make sure I had fully tested complete backups before I broke the mirror and did that. If the drive reads a sector with ECC-correctable errors, it's supposed to try to re-write that sector in order to fix up the ECC data. If that write fails, it remaps. Of course, your suggestion of blanking the entire drive and restoring from the mirror or a backup would be best, or perhaps better short of replacing the drive. Regards, -- -Chuck ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Troubleshooting a gmirror disk marked broken
Hi, Last night during a massive (~1 year worth :| ) portsnap fetch the server went unresponsive and ssh eventually disconnected. I decided to leave it during the night, and, sure enough, the situation was the same in the morning, so I had to do a hard reset. It came back up, but one of the two gmirror components was marked as broken and deactivated. The hang happened during the 'fetching new files or ports' (~24000 of them, there are currently ~1 snapshots in /var/db/portsnap) phase of postsnap fetch. /var/log/messages was completely silent during the period between the hang and the reset. Googling around I found a mention that it's possible to sometimes get a 'blip'[*] during busy periods, so I decided to just bite the bullet and reinsert the component with # gmirror forget gm0 # gmirror clean ad4 # gmirror insert gm0 ad4 Currently it's syncing and things *seem* OK. My question is how much should I be worried and what could be the cause of this? Is it possible that ports snapshot fetching caused this, or that perhaps it was the other way around (a failing disk causing the machine to choke during the huge portsnap fetch)? How to proceed? :) At the moment I'm attaching smartctl -a results for both disks (ad4 was marked broken). As I'm completely useless in deciphering smartctl results (apart from being thought by experience to pretty much ignore(tm) the 'Pre-fail' statuses), I'd appreciate assessment from more knowledgeable people. Also attached is a trimmed /var/log/messages from the moment of hard reset on (I can fill in the snipped parts, just didn't want to choke people with possibly irrelevant details). $ uname -a FreeBSD isus 8.3-RELEASE-p3 FreeBSD 8.3-RELEASE-p3 #0: Tue Jun 12 00:39:29 UTC 2012 r...@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64 --- [*] http://www.eztiger.org/2008/08/removing-and-re-adding-a-disk-in-gmirror/ -- If you analyse anything, you destroy it. -- Arthur Miller smartctl 5.42 2011-10-20 r3458 [FreeBSD 8.3-RELEASE-p3 amd64] (local build) Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net === START OF INFORMATION SECTION === Model Family: Seagate Barracuda 7200.10 Device Model: ST3320620AS Serial Number:5QF1FDAS Firmware Version: 3.AAE User Capacity:320,072,933,376 bytes [320 GB] Sector Size: 512 bytes logical/physical Device is:In smartctl database [for details use: -P show] ATA Version is: 7 ATA Standard is: Exact ATA specification draft version not indicated Local Time is:Thu Jun 27 03:55:37 2013 CEST SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x82) Offline data collection activity was completed without error. Auto Offline Data Collection: Enabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection:( 430) seconds. Offline data collection capabilities:(0x5b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. No Conveyance Self-test supported. Selective Self-test supported. SMART capabilities:(0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability:(0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time:( 1) minutes. Extended self-test routine recommended polling time:( 115) minutes. ATA Version is: 7 ATA Standard is: Exact ATA specification draft version not indicated Local Time is:Thu Jun 27 03:55:37 2013 CEST SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x82) Offline data collection activity was completed without error. Auto Offline Data Collection: Enabled. Self-test execution status
Re: Troubleshooting a gmirror disk marked broken
On Wed, Jun 26, 2013 at 9:38 PM, Nikola Pavlović n...@riseup.net wrote: Hi, Last night during a massive (~1 year worth :| ) portsnap fetch the server went unresponsive and ssh eventually disconnected. I decided to leave it during the night, and, sure enough, the situation was the same in the morning, so I had to do a hard reset. It came back up, but one of the two gmirror components was marked as broken and deactivated. The hang happened during the 'fetching new files or ports' (~24000 of them, there are currently ~1 snapshots in /var/db/portsnap) phase of postsnap fetch. /var/log/messages was completely silent during the period between the hang and the reset. Googling around I found a mention that it's possible to sometimes get a 'blip'[*] during busy periods, so I decided to just bite the bullet and reinsert the component with # gmirror forget gm0 # gmirror clean ad4 # gmirror insert gm0 ad4 Currently it's syncing and things *seem* OK. My question is how much should I be worried and what could be the cause of this? Is it possible that ports snapshot fetching caused this, or that perhaps it was the other way around (a failing disk causing the machine to choke during the huge portsnap fetch)? How to proceed? :) The messages log definitely shows problems with your io. The smart log of the disks are also at least mildly concerning and indicates the drives are in a preliminary stage of death. Some HD deaths take years to complete. Expect random glitches and intermittent reduced performance as a continuous degradation. You might be able to alleviate some of this by switching to the AHCI driver and bumping up timeouts but at the end of the day 2 flaky disks in a mirror don't inspire confidence. -- Adam Vande More ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: 9.1 and gmirror with GPT?
On 2012-10-29 03:58, Jeremy Chadwick wrote: If you're truly using 4096-byte sectors disks -- specifically MECHANICAL hard disks (MHDDs) -- use of 4KByte alignment is fine. But if you ever plan on using an SSD the future, you need to align things to 1MBytes or 2MBytes. This is why Windows Vista and Windows 7 aligns its partitions to 1MByte boundaries. ...and quite honestly FreeBSD should too. I am aware 9.1-RELEASE supposedly addresses this -- however I have not determined if the alignment size chosen by the committer was 4096 or 1MB/2MB. I have a gut feeling it's the former, and that's bad. With 1MByte or 2MByte alignment, performance on 512-byte MHDDs would be fine, performance on 4096-byte MHDDs would be fine, and performance on SSDs would be fine. Next: in case it's not made clear to readers from Warren's statements: the magical 8 divisor he's using comes from 4096/512 (how many 512 bytes are there in a 4096-byte sector). Thus, for 1MByte alignment the value would be 1048576/512 or 2048. For 2MByte alignment the value would be 2097152/512 or 4096. Thank You Jeremy! In an effort to bring concluding info from the original thread, on some MRB partitioned drives (spinning media in this case) gpart seems unable to align the containing -t freebsd slice to 4K boundaries. However subsequent creation of -t freebsd-ufs and -t freebsd-swap partitions within the slice align correctly. To make this alignment on 1M boundaries instead of 4K boundaries the -a 1M should be used instead of -a 4K. Example gpart commands for MBR partition table aligned to 1M sector size for SSD: gpart create -s MBR mirror/gm0 gpart add -t freebsd -a 1M mirror/gm0 # ignore possible warning mirror/gm0s1 added, but partition is not aligned # create the bsdlabel partitions in slice 1 (s1) gpart create -s BSD mirror/gm0s1 gpart add -t freebsd-swap -a 1M -s 8g mirror/gm0s1 gpart add -t freebsd-ufs -a 1M mirror/gm0s1 # put bootcode on the MBR and mark the first slice active gpart bootcode -b /boot/mbr mirror/gm0 gpart set -a active -i 1 mirror/gm0 # put bootcode on the bsdlabel gpart bootcode -b /boot/boot mirror/gm0s1 ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: 9.1 and gmirror with GPT?
On 2012-10-27 16:01, Warren Block wrote: On Sat, 27 Oct 2012, free...@johnea.net wrote: I ended up just ignoring the not aligned warning from the gpart add -t freebsd and went on to add the freebsd-swap and freebsd-ufs partitions with -a 4k option. Do you think I'm aligned? ... =63 3907029104 mirror/gm0 MBR (1.8T) 63 63 - free - (31k) 126 3907028979 1 freebsd (1.8T) 3907029105 62 - free - (31k) = 0 3907028979 mirror/gm0s1 BSD (1.8T) 0 2- free - (1.0k) 216777216 1 freebsd-swap (8.0G) 16777218 3890251760 2 freebsd-ufs (1.8T) 3907028978 1- free - (512B) The slice starts at block 126, and then the swap partition starts an additional two blocks into the slice, which is block 128, evenly divisible by 8 (4096 = 512 * 8). The freebsd-ufs partition starts at 126+16777218, which is also evenly divisible by 8. So yes, that looks aligned to me. Thanks again Warren! I think I finally have this 9.1 system up and running with MBR and gmirror aligned to 4K sector size. After getting the gm0 running, I did a dump/restore to transfer the live system from ada0 to gm0, before adding ada0 to the mirror. I ran into the journaled soft-updates issue, and again relied on one of your posts for the solution: http://forums.freebsd.org/showthread.php?t=31257 Thank You for your contributions to FreeBSD! johnea ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: 9.1 and gmirror with GPT?
On 2012-10-23 17:46, Warren Block wrote: On Tue, 23 Oct 2012, free...@johnea.net wrote: To create a swap and then a root that fills the rest of the disk, must the swap be created first, like this: gpart add -t freebsd-swap -a 4k -s 4g mirror/gm0s1 gpart add -t freebsd-ufs -a 4k mirror/gm0s1 Is there any other way to tell gpart to create the / partition using all space except 4G? I'm afraid it requires one to Use Math(tm). gpart show will at least show the real capacity of a drive, instead of the diagonally-measured inflated units used by drive vendors. Thanks for your guidance Warren! I've also been reading a number of threads on the forums on this subject, to which you contributed. Rather than face the scary prospect of using actual Math(tm) 8-) I was just going to create swap first at 8G, and let the freebsd partition fill the rest of the disk, however as I try to destroy previous non-aligned MBR and gmirror metadata, I'm running into issues: mirror/gm0s1 added, but partition is not aligned on 4096 bytes Below is a short screen shot of the commands used to destroy and then recreate gmirror. I'm currently running with non-geom ada0 as root, and am attempting to create the aligned partitions of gm0 on ada1. I'm not sure what to do to cause the MBR scheme to be aligned. Thanks for any feedback! johnea orsbackup# mount /dev/ada0s1a on / (ufs, local, journaled soft-updates) devfs on /dev (devfs, local, multilabel) orsbackup# gmirror status NameStatus Components mirror/gm0 COMPLETE ada1 (ACTIVE) orsbackup# gpart destroy -F mirror/gm0 mirror/gm0 destroyed orsbackup# gpart create -s MBR mirror/gm0 mirror/gm0 created orsbackup# ls /dev/mirror/ gm0 orsbackup# gpart add -t freebsd -a 4k mirror/gm0 mirror/gm0s1 added, but partition is not aligned on 4096 bytes orsbackup# gpart show =63 3907029105 ada0 MBR (1.8T) 63 63- free - (31k) 126 3906994077 1 freebsd [active] (1.8T) 3906994203 34965- free - (17M) = 0 3906994077 ada0s1 BSD (1.8T) 0 3890216960 1 freebsd-ufs (1.8T) 389021696016777116 2 freebsd-swap (8G) 3906994076 1 - free - (512B) =63 3907029104 mirror/gm0 MBR (1.8T) 63 63 - free - (31k) 126 3907028979 1 freebsd (1.8T) 3907029105 62 - free - (31k) = 0 3907028979 mirror/gm0s1 BSD (1.8T) 0 3890216960 1 freebsd-ufs (1.8T) 389021696016777116 2 freebsd-swap (8G) 3906994076 34903- free - (17M) ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: 9.1 and gmirror with GPT?
On Sat, 27 Oct 2012, free...@johnea.net wrote:
On 2012-10-23 17:46, Warren Block wrote:
On Tue, 23 Oct 2012, free...@johnea.net wrote:
To create a swap and then a root that fills the rest of the disk, must the swap
be created first, like this:
gpart add -t freebsd-swap -a 4k -s 4g mirror/gm0s1
gpart add -t freebsd-ufs -a 4k mirror/gm0s1
Is there any other way to tell gpart to create the / partition using all space
except 4G?
I'm afraid it requires one to Use Math(tm). gpart show will at least show the
real capacity of a drive, instead of the diagonally-measured inflated units
used by drive vendors.
Thanks for your guidance Warren! I've also been reading a number of threads on
the forums on this subject, to which you contributed.
Rather than face the scary prospect of using actual Math(tm) 8-) I was just
going to create swap first at 8G, and let the freebsd partition fill the rest
of the disk, however as I try to destroy previous non-aligned MBR and gmirror
metadata, I'm running into issues:
mirror/gm0s1 added, but partition is not aligned on 4096 bytes
Below is a short screen shot of the commands used to destroy and then recreate
gmirror. I'm currently running with non-geom ada0 as root, and am attempting to
create the aligned partitions of gm0 on ada1.
I'm not sure what to do to cause the MBR scheme to be aligned. Thanks for any
feedback!
johnea
orsbackup# mount
/dev/ada0s1a on / (ufs, local, journaled soft-updates)
devfs on /dev (devfs, local, multilabel)
orsbackup# gmirror status
NameStatus Components
mirror/gm0 COMPLETE ada1 (ACTIVE)
orsbackup# gpart destroy -F mirror/gm0
mirror/gm0 destroyed
orsbackup# gpart create -s MBR mirror/gm0
mirror/gm0 created
orsbackup# ls /dev/mirror/
gm0
orsbackup# gpart add -t freebsd -a 4k mirror/gm0
mirror/gm0s1 added, but partition is not aligned on 4096 bytes
This is new to me, I have not seen it before. I had the impression that
gpart put the bsdlabel partition table at a misaligned offset so that
the actual filesystems in those partitions would land on an aligned
block, but it never gave that message. That comes from
sys/geom/part/g_part.c, function g_part_ctl_add which starts at line
645:
743 /* Provide feedback if so requested. */
744 if (gpp-gpp_parms G_PART_PARM_OUTPUT) {
745 sb = sbuf_new_auto();
746 G_PART_FULLNAME(table, entry, sb, gp-name);
747 if (pp-stripesize 0 entry-gpe_pp-stripeoffset != 0)
748 sbuf_printf(sb, added, but partition is not
749 aligned on %u bytes\n, pp-stripesize);
750 else
751 sbuf_cat(sb, added\n);
752 sbuf_finish(sb);
753 gctl_set_param(req, output, sbuf_data(sb), sbuf_len(sb) +
1);
754 sbuf_delete(sb);
755 }
756 return (0);
orsbackup# gpart show
=63 3907029105 ada0 MBR (1.8T)
63 63- free - (31k)
126 3906994077 1 freebsd [active] (1.8T)
3906994203 34965- free - (17M)
= 0 3906994077 ada0s1 BSD (1.8T)
0 3890216960 1 freebsd-ufs (1.8T)
389021696016777116 2 freebsd-swap (8G)
3906994076 1 - free - (512B)
=63 3907029104 mirror/gm0 MBR (1.8T)
63 63 - free - (31k)
126 3907028979 1 freebsd (1.8T)
3907029105 62 - free - (31k)
= 0 3907028979 mirror/gm0s1 BSD (1.8T)
0 3890216960 1 freebsd-ufs (1.8T)
389021696016777116 2 freebsd-swap (8G)
3906994076 34903- free - (17M)
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: 9.1 and gmirror with GPT?
On 2012-10-27 14:41, Warren Block wrote:
On Sat, 27 Oct 2012, free...@johnea.net wrote:
On 2012-10-23 17:46, Warren Block wrote:
On Tue, 23 Oct 2012, free...@johnea.net wrote:
orsbackup# gpart add -t freebsd -a 4k mirror/gm0
mirror/gm0s1 added, but partition is not aligned on 4096 bytes
This is new to me, I have not seen it before. I had the impression that
gpart put the bsdlabel partition table at a misaligned offset so that the
actual filesystems in those partitions would land on an aligned block, but it
never gave that message. That comes from sys/geom/part/g_part.c, function
g_part_ctl_add which starts at line 645:
743 /* Provide feedback if so requested. */
744 if (gpp-gpp_parms G_PART_PARM_OUTPUT) {
745 sb = sbuf_new_auto();
746 G_PART_FULLNAME(table, entry, sb, gp-name);
747 if (pp-stripesize 0 entry-gpe_pp-stripeoffset !=
0)
748 sbuf_printf(sb, added, but partition is not
749 aligned on %u bytes\n, pp-stripesize);
750 else
751 sbuf_cat(sb, added\n);
752 sbuf_finish(sb);
753 gctl_set_param(req, output, sbuf_data(sb), sbuf_len(sb)
+ 1);
754 sbuf_delete(sb);
755 }
756 return (0);
Thanks Warren!
I ended up just ignoring the not aligned warning from the gpart add -t
freebsd and went on to add the freebsd-swap and freebsd-ufs partitions with
-a 4k option.
Do you think I'm aligned?
Thanks!
johnea
orsbackup# gpart create -s BSD mirror/gm0s1
mirror/gm0s1 created
orsbackup# gpart add -t freebsd-swap -a 4k -s 8g mirror/gm0s1
mirror/gm0s1a added
orsbackup# gpart add -t freebsd-ufs -a 4k mirror/gm0s1
mirror/gm0s1b added
orsbackup# gpart bootcode -b /boot/mbr mirror/gm0
bootcode written to mirror/gm0
orsbackup# gpart set -a active -i 1 mirror/gm0
active set on mirror/gm0s1
orsbackup# gpart bootcode -b /boot/boot mirror/gm0s1
bootcode written to mirror/gm0s1
orsbackup#
orsbackup# gpart show
=63 3907029105 ada0 MBR (1.8T)
63 63- free - (31k)
126 3906994077 1 freebsd [active] (1.8T)
3906994203 34965- free - (17M)
= 0 3906994077 ada0s1 BSD (1.8T)
0 3890216960 1 freebsd-ufs (1.8T)
389021696016777116 2 freebsd-swap (8G)
3906994076 1 - free - (512B)
=63 3907029104 mirror/gm0 MBR (1.8T)
63 63 - free - (31k)
126 3907028979 1 freebsd (1.8T)
3907029105 62 - free - (31k)
= 0 3907028979 mirror/gm0s1 BSD (1.8T)
0 2- free - (1.0k)
216777216 1 freebsd-swap (8.0G)
16777218 3890251760 2 freebsd-ufs (1.8T)
3907028978 1- free - (512B)
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: 9.1 and gmirror with GPT?
On Sat, 27 Oct 2012, free...@johnea.net wrote: I ended up just ignoring the not aligned warning from the gpart add -t freebsd and went on to add the freebsd-swap and freebsd-ufs partitions with -a 4k option. Do you think I'm aligned? ... =63 3907029104 mirror/gm0 MBR (1.8T) 63 63 - free - (31k) 126 3907028979 1 freebsd (1.8T) 3907029105 62 - free - (31k) = 0 3907028979 mirror/gm0s1 BSD (1.8T) 0 2- free - (1.0k) 216777216 1 freebsd-swap (8.0G) 16777218 3890251760 2 freebsd-ufs (1.8T) 3907028978 1- free - (512B) The slice starts at block 126, and then the swap partition starts an additional two blocks into the slice, which is block 128, evenly divisible by 8 (4096 = 512 * 8). The freebsd-ufs partition starts at 126+16777218, which is also evenly divisible by 8. So yes, that looks aligned to me. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: 9.1 and gmirror with GPT?
On 2012.10.21 17:44, free...@johnea.net wrote: On 10/21/2012 07:32 AM, Warren Block wrote: On Sun, 21 Oct 2012, Lucas B. Cohen wrote: On 2012.10.20 20:17, free...@johnea.net wrote: Just wondering if 9.1 will bring any improvement to the situation of creating a full disk geom mirror while also using GPT partition table? I'm curious about what this is about. Could you refer me to an article or a discussion where this issue is described ? The GPT backup partition tables goes at the end of a disk, the same place gmirror(8) and other GEOM modules keep metadata. If GPT partitions are created inside a mirror, the backup GPT table is no longer at the end of the disk. Hiroki Sato created a patch which fixed the gptboot complaints, but there was concern about the nonstandard location of the backup table. At present, MBR partitioning is recommended with gmirror(8). Thank you Warren. That sums it up. It also seems greedy of GPT to require both the first and last sectors of the disk. This seems to almost guarantee it will have issues with other low level disk formatting tools. Of course, given the history of the WinTel partnership, perhaps not interoperating with other tools was a design specification 8-) What surprises me is that GEOM mirror provides a logical device that doesn't abstract the parts that hold its own metadata. It so happens that GPT wants to use one of those parts, but doesn't creating an MBR partition that spans the whole provider up to the last logical block create a similar - but in this case latent - problem, once that last block is written to by a filesystem living inside that partition ? ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: 9.1 and gmirror with GPT?
On 2012.10.24 22:10, Lucas B. Cohen wrote: On 2012.10.21 17:44, free...@johnea.net wrote: On 10/21/2012 07:32 AM, Warren Block wrote: On Sun, 21 Oct 2012, Lucas B. Cohen wrote: On 2012.10.20 20:17, free...@johnea.net wrote: Just wondering if 9.1 will bring any improvement to the situation of creating a full disk geom mirror while also using GPT partition table? I'm curious about what this is about. Could you refer me to an article or a discussion where this issue is described ? The GPT backup partition tables goes at the end of a disk, the same place gmirror(8) and other GEOM modules keep metadata. If GPT partitions are created inside a mirror, the backup GPT table is no longer at the end of the disk. Hiroki Sato created a patch which fixed the gptboot complaints, but there was concern about the nonstandard location of the backup table. At present, MBR partitioning is recommended with gmirror(8). Thank you Warren. That sums it up. It also seems greedy of GPT to require both the first and last sectors of the disk. This seems to almost guarantee it will have issues with other low level disk formatting tools. Of course, given the history of the WinTel partnership, perhaps not interoperating with other tools was a design specification 8-) What surprises me is that GEOM mirror provides a logical device that doesn't abstract the parts that hold its own metadata. It so happens that GPT wants to use one of those parts, but doesn't creating an MBR partition that spans the whole provider up to the last logical block create a similar - but in this case latent - problem, once that last block is written to by a filesystem living inside that partition ? Nevermind, I just got this. It's code working at the physical device level that gets confused and complains about a missing GPT backup in the single disks it examines; not code that's working against the provided GEOM mirror once it's assembled. My first understanding felt so weird, I knew I was missing something ! I guess Hiroki Sato's patch Warren mentions doesn't answer the danger of overwriting gmirror metadata by an unfriendly UEFI-BIOS, though. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: 9.1 and gmirror with GPT?
On Tue, 23 Oct 2012, free...@johnea.net wrote: In recent years I've just been creating a swap partition and one big root partition. It just seems as soon as I make all the traditional partitions, one runs out of room. Do you feel there are any major disadvantages of this approach? Backup of split filesystems can be easier. The traditional split-filesystem approach kind of separates things by use, and some people create a separate /home also. The advantage of one big root is efficient use of free space on small drives. To create a swap and then a root that fills the rest of the disk, must the swap be created first, like this: gpart add -t freebsd-swap -a 4k -s 4g mirror/gm0s1 gpart add -t freebsd-ufs -a 4k mirror/gm0s1 Is there any other way to tell gpart to create the / partition using all space except 4G? I'm afraid it requires one to Use Math(tm). gpart show will at least show the real capacity of a drive, instead of the diagonally-measured inflated units used by drive vendors. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: 9.1 and gmirror with GPT?
On 10/21/2012 07:32 AM, Warren Block wrote: On Sun, 21 Oct 2012, Lucas B. Cohen wrote: On 2012.10.20 20:17, free...@johnea.net wrote: Just wondering if 9.1 will bring any improvement to the situation of creating a full disk geom mirror while also using GPT partition table? I'm curious about what this is about. Could you refer me to an article or a discussion where this issue is described ? The GPT backup partition tables goes at the end of a disk, the same place gmirror(8) and other GEOM modules keep metadata. If GPT partitions are created inside a mirror, the backup GPT table is no longer at the end of the disk. Hiroki Sato created a patch which fixed the gptboot complaints, but there was concern about the nonstandard location of the backup table. At present, MBR partitioning is recommended with gmirror(8). Warren, I've been reading your article on formatting disks in FreeBSD: http://www.wonkity.com/~wblock/docs/html/disksetup.html It's a great description of using gpart to create GPT partitions, and using fdisk and bsdlabel to create MBR partitions. Would you still recommend this method, using fdisk and bsdlabel, for MRB setup? Do you have any docs on setting up MBR using gpart, to allign for 4K sector size drives? Thank you for your advice! johnea ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: 9.1 and gmirror with GPT?
On Mon, 22 Oct 2012, free...@johnea.net wrote: On 10/21/2012 07:32 AM, Warren Block wrote: On Sun, 21 Oct 2012, Lucas B. Cohen wrote: On 2012.10.20 20:17, free...@johnea.net wrote: Just wondering if 9.1 will bring any improvement to the situation of creating a full disk geom mirror while also using GPT partition table? I'm curious about what this is about. Could you refer me to an article or a discussion where this issue is described ? The GPT backup partition tables goes at the end of a disk, the same place gmirror(8) and other GEOM modules keep metadata. If GPT partitions are created inside a mirror, the backup GPT table is no longer at the end of the disk. Hiroki Sato created a patch which fixed the gptboot complaints, but there was concern about the nonstandard location of the backup table. At present, MBR partitioning is recommended with gmirror(8). I've been reading your article on formatting disks in FreeBSD: http://www.wonkity.com/~wblock/docs/html/disksetup.html It's a great description of using gpart to create GPT partitions, and using fdisk and bsdlabel to create MBR partitions. Thanks! Would you still recommend this method, using fdisk and bsdlabel, for MRB setup? For drives with 512-byte blocks, they are equivalent. Only gpart can align the bsdlabel partitions to 4K. Do you have any docs on setting up MBR using gpart, to allign for 4K sector size drives? This is a copy from the update of the gmirror section I'm planning to commit to the Handbook. For a single drive, replace mirror/gm0 with just the drive name, like ada4. # create the MBR and add a FreeBSD slice gpart create -s MBR mirror/gm0 gpart add -t freebsd -a 4k mirror/gm0 # create the bsdlabel partitions in slice 1 (s1) gpart create -s BSD mirror/gm0s1 gpart add -t freebsd-ufs -a 4k -s 2g mirror/gm0s1 gpart add -t freebsd-swap -a 4k -s 4g mirror/gm0s1 gpart add -t freebsd-ufs -a 4k -s 2g mirror/gm0s1 gpart add -t freebsd-ufs -a 4k -s 1g mirror/gm0s1 gpart add -t freebsd-ufs -a 4k mirror/gm0s1 # put bootcode on the MBR and mark the first slice active gpart bootcode -b /boot/mbr mirror/gm0 gpart set -a active -i 1 mirror/gm0 # put bootcode on the bsdlabel gpart bootcode -b /boot/boot mirror/gm0s1 ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: 9.1 and gmirror with GPT?
On Sun, 21 Oct 2012, Lucas B. Cohen wrote: On 2012.10.20 20:17, free...@johnea.net wrote: Just wondering if 9.1 will bring any improvement to the situation of creating a full disk geom mirror while also using GPT partition table? I'm curious about what this is about. Could you refer me to an article or a discussion where this issue is described ? The GPT backup partition tables goes at the end of a disk, the same place gmirror(8) and other GEOM modules keep metadata. If GPT partitions are created inside a mirror, the backup GPT table is no longer at the end of the disk. Hiroki Sato created a patch which fixed the gptboot complaints, but there was concern about the nonstandard location of the backup table. At present, MBR partitioning is recommended with gmirror(8). ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: 9.1 and gmirror with GPT?
On 10/21/2012 07:32 AM, Warren Block wrote: On Sun, 21 Oct 2012, Lucas B. Cohen wrote: On 2012.10.20 20:17, free...@johnea.net wrote: Just wondering if 9.1 will bring any improvement to the situation of creating a full disk geom mirror while also using GPT partition table? I'm curious about what this is about. Could you refer me to an article or a discussion where this issue is described ? The GPT backup partition tables goes at the end of a disk, the same place gmirror(8) and other GEOM modules keep metadata. If GPT partitions are created inside a mirror, the backup GPT table is no longer at the end of the disk. Hiroki Sato created a patch which fixed the gptboot complaints, but there was concern about the nonstandard location of the backup table. At present, MBR partitioning is recommended with gmirror(8). Thank you Warren. That sums it up. Lucas, I found this blag post informative: https://koitsu.wordpress.com/2012/09/18/using-freebsd-graid-geom-raid/ There are also several interesting posts on Michael Lucas' blag, such as: http://blather.michaelwlucas.com/archives/1071 This is a good discussion thread that dives into a specific configuration and the implications: http://freebsd.1045724.n5.nabble.com/disk-partitioning-with-gmirror-gpt-gjournal-RFC-td4912676.html I've tried to determine which came first GEOM or GPT. It seems GEOM is actually older, dating from FreeBSD 5, around 2003. While GPT seems to have been integrated with what is now known as UEFI in the later half of that decade. It also seems greedy of GPT to require both the first and last sectors of the disk. This seems to almost guarantee it will have issues with other low level disk formatting tools. Of course, given the history of the WinTel partnership, perhaps not interoperating with other tools was a design specification 8-) https://en.wikipedia.org/wiki/GUID_Partition_Table In any case, the upcoming wide spread use of UEFI/GPT (i.e. windoze on commodity PCs) compared to the FreeBSD specific nature of GEOM, pretty much insures that it will have to be GEOM that changes to accommodate this conflict. Even given the denial of who is David and who is Goliath, in the fact that the GEOM developers don't seem to consider this their bug: http://www.freebsd.org/cgi/query-pr.cgi?pr=162147 It seems inevitable that the FreeBSD devs will have to capitulate and find another way to store the GEOM meta-data or we're going to loose the great benefits of whole disk mirrors under GEOM. [please proceed with tongue in cheek] This may not occur any time soon, as time progresses at a different rate for BSD than it does with the rest of the world. A great example is this sentence from the Architecture Handbook: The Universal Serial Bus (USB) is a new way of attaching devices to personal computers. Of course USB is roughly 20 years old now 8-) There are some other great quotes regarding the new computer input device, called the mouse. [safe to freely operate tongue again] In any case, it seems my new 9.1-RC2 installation will be MBR partitioned with whole disk GEOM mirror. This motherboard is BIOS based, not UEFI. It's become fairly de rigueur to accommodate the 4K sector size disks with fdisk and MBR partitioning. As we propel forward into SSDs this may not stay the case. Any other comments or caveats are very greatly appreciated... johnea ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
9.1 and gmirror with GPT?
Hello, Just wondering if 9.1 will bring any improvement to the situation of creating a full disk geom mirror while also using GPT partition table? Is the fix for this a near term thing, or something in the farther future? Thanks for any insight! johnea ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: 9.1 and gmirror with GPT?
Hi, On 2012.10.20 20:17, free...@johnea.net wrote: Just wondering if 9.1 will bring any improvement to the situation of creating a full disk geom mirror while also using GPT partition table? I'm curious about what this is about. Could you refer me to an article or a discussion where this issue is described ? ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
gmirror degraded
Just looking for some advice, I had a server lock up that uses gmirror for two RAID-1 arrays of the primary drive and a data drive. The data drive was reported as degraded after a reset of the server, but is rebuilding. It is comprised of two TB drives with one reporting ACTIVE with no flags. The other synchronizing and taking days... Geom name: d1 State: DEGRADED Components: 2 Balance: round-robin Slice: 4096 Flags: NONE GenID: 0 SyncID: 2 ID: 2434624761 Providers: 1. Name: mirror/d1 Mediasize: 999653637632 (931G) Sectorsize: 512 Mode: r2w1e1 Consumers: 1. Name: ada1p1 Mediasize: 999653638144 (931G) Sectorsize: 512 Stripesize: 0 Stripeoffset: 17408 Mode: r1w1e1 State: ACTIVE Priority: 0 Flags: NONE GenID: 0 SyncID: 2 ID: 175176036 2. Name: ada3p1 Mediasize: 999653638144 (931G) Sectorsize: 512 Stripesize: 0 Stripeoffset: 17408 Mode: r1w1e1 State: SYNCHRONIZING Priority: 0 Flags: DIRTY, SYNCHRONIZING GenID: 0 SyncID: 2 Synchronized: 91% ID: 4158324973 I ran the smartctl command below on the synchronizing drive and it seems there are no errors? Should I trust this drive? backup# /usr/local/sbin/smartctl -a /dev/ada3 smartctl 5.43 2012-06-30 r3573 [FreeBSD 9.0-RELEASE i386] (local build) Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net === START OF INFORMATION SECTION === Model Family: Seagate Barracuda 7200.12 Device Model: ST31000528AS Serial Number:9VP8EKVV LU WWN Device Id: 5 000c50 026d2b322 Firmware Version: CC3E User Capacity:1,000,204,886,016 bytes [1.00 TB] Sector Size: 512 bytes logical/physical Device is:In smartctl database [for details use: -P show] ATA Version is: 8 ATA Standard is: ATA-8-ACS revision 4 Local Time is:Thu Oct 11 10:36:54 2012 EDT SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x82) Offline data collection activity was completed without error. Auto Offline Data Collection: Enabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection:( 600) seconds. Offline data collection capabilities:(0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities:(0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability:(0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time:( 1) minutes. Extended self-test routine recommended polling time:( 175) minutes. Conveyance self-test routine recommended polling time:( 2) minutes. SCT capabilities: (0x103f) SCT Status supported. SCT Error Recovery Control supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 10 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 113 097 006Pre-fail Always - 108410580 3 Spin_Up_Time0x0003 095 095 000Pre-fail Always - 0 4 Start_Stop_Count0x0032 100 100 020Old_age Always - 34 5 Reallocated_Sector_Ct 0x0033 076 076 036Pre-fail Always - 1003 7 Seek_Error_Rate 0x000f 074 060 030Pre-fail Always - 25881764 9 Power_On_Hours 0x0032 079 079 000Old_age Always - 18567 10 Spin_Retry_Count0x0013 100 100 097Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 100 100 020Old_age Always - 17 183 Runtime_Bad_Block 0x0032 100 100 000Old_age Always - 0
Re: gmirror degraded
On Thu, 11 Oct 2012, Robert Fitzpatrick wrote: Just looking for some advice, I had a server lock up that uses gmirror for two RAID-1 arrays of the primary drive and a data drive. The data drive was reported as degraded after a reset of the server, but is rebuilding. It is comprised of two TB drives with one reporting ACTIVE with no flags. The other synchronizing and taking days... You mean it does this repeatedly? I ran the smartctl command below on the synchronizing drive and it seems there are no errors? Should I trust this drive? 1,003 reallocated sectors is a bad sign. I would replace that drive. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Question about gmirror priorities
Hi, the manpage says for ``gmirror label'': The order of components is important, because a component's priority is based on its position (starting from 0 to 255). so I would expect to have different priorities for the components, yet both are listed with a priority of 0: gmirror list Geom name: gm0 State: COMPLETE Components: 2 Balance: load Slice: 4096 Flags: NONE GenID: 0 SyncID: 1 ID: 1162650455 Providers: 1. Name: mirror/gm0 Mediasize: 320072932864 (298G) Sectorsize: 512 Mode: r2w2e5 Consumers: 1. Name: ad4 Mediasize: 320072933376 (298G) Sectorsize: 512 Mode: r1w1e1 State: ACTIVE Priority: 0 Flags: NONE GenID: 0 SyncID: 1 ID: 2769583838 2. Name: ad6 Mediasize: 320072933376 (298G) Sectorsize: 512 Mode: r1w1e1 State: ACTIVE Priority: 0 Flags: NONE GenID: 0 SyncID: 1 ID: 540951176 Where is my misunderstanding? Regards, Michael ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Question about gmirror priorities
On Fri, Jul 6, 2012 at 8:10 AM, Michael Ross g...@ross.cx wrote: Hi, the manpage says for ``gmirror label'': The order of components is important, because a component's priority is based on its position (starting from 0 to 255). so I would expect to have different priorities for the components, yet both are listed with a priority of 0: I would expect components to have a different priority if I assigned them one. Otherwise I would assume they have the default priority. I don't know if makes it makes any difference for the algo you are running anyways. -- Adam Vande More ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
GPT + gmirror
Hello, I wondered if there is a way to gmirroring the whole disk (not slices separately) when using GPT? GPT puts its metadata at the end of the disk, and when I start to use gmirror it overwrites the GPT metadata (... as gmirror puts also its metadata at the end of the disk ...). I noticed a new option in the newfs manpage: -r reserved The size, in sectors, of reserved space at the end of the parti‐ tion specified in special. This space will not be occupied by the file system; it can be used by other consumers such as geom(4). Defaults to 0. I wondered if it could help .. ? Why does it default to 0? Thanks, Julien -- No trees were killed in the creation of this message. However, many electrons were terribly inconvenienced. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: FreeBSD 9, GPT and gmirror
On Thu, 09 Feb 2012 07:02:29 -0800 per...@pluto.rain.com wrote: Janos Dohanics w...@3dresearch.com wrote: 1. The Guided partitioning doesn't suggest any more to create /var, /tmp, /usr, etc. file systems. Is it really the recommendation to go with just / ? Depends on who you ask :) and on your intended usage. 2. Is there a way to use the old sysinstall to install FreeBSD 9? Not using the standard distribution IIUC. You might want to look at http://druidbsd.sf.net/ [...] This may be just what I need - thank you. -- Janos Dohanics ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: FreeBSD 9, GPT and gmirror
On Thu, 09 Feb 2012 20:20:03 +0100 Michael Cardell Widerkrantz m...@hack.org wrote: Janos Dohanics w...@3dresearch.com, 2012-02-08 19:42 (+0100): 4. Also, with GPT, one has to be in single user mode to synchronize disks - correct? I think the guide you linked to: http://blather.michaelwlucas.com/archives/1071 meant that you have to be in single user mode until you have edited /etc/fstab to point to the mirror, otherwise you wouldn't boot with root on the mirror. The synchronization between the disks works fine in multi-user mode as well. I have two 2 TiB disks in gmirror set up just like that. Synchronization was done running in multi-user. You are right - just removed and then re-inserted a component in one of the mirrors and the mirror synchronized fine in multi-user mode. -- Janos Dohanics ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: FreeBSD 9, GPT and gmirror
Janos Dohanics w...@3dresearch.com wrote: 1. The Guided partitioning doesn't suggest any more to create /var, /tmp, /usr, etc. file systems. Is it really the recommendation to go with just / ? Depends on who you ask :) and on your intended usage. 2. Is there a way to use the old sysinstall to install FreeBSD 9? Not using the standard distribution IIUC. You might want to look at http://druidbsd.sf.net/ 3. It seems that setting up gmirror is more involved with GPT (http://blather.michaelwlucas.com/archives/1071); now I have a mirror for each of the filesystems /, /var, /tmp, etc. Is it OK to use gmirror in this way at all? Yes, indeed it is the only way to combine GPT and gmirror without getting into trouble of one sort or another. (The conflict between GPT and a full-disk gmirror is actually not new.) 4. Also, with GPT, one has to be in single user mode to synchronize disks - correct? Dunno about this one. 3. Assuming one has enough RAM, is zfs mirror or raidz recommended over gmirror? Same situation as with #1. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: FreeBSD 9, GPT and gmirror
Janos Dohanics w...@3dresearch.com, 2012-02-08 19:42 (+0100): 4. Also, with GPT, one has to be in single user mode to synchronize disks - correct? I think the guide you linked to: http://blather.michaelwlucas.com/archives/1071 meant that you have to be in single user mode until you have edited /etc/fstab to point to the mirror, otherwise you wouldn't boot with root on the mirror. The synchronization between the disks works fine in multi-user mode as well. I have two 2 TiB disks in gmirror set up just like that. Synchronization was done running in multi-user. -- http://hack.org/mc/ Warning! Plain text e-mail, please. HTML e-mail deleted unread. OpenPGP: 673B 563E 3C78 1BA0 6525 2344 B22E 2C10 E4C9 2FA5 ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
FreeBSD 9, GPT and gmirror
Hello Everyone, May be I should have searched more for answers, but after installing FreeBSD 9 with gmirror, I am wondering if the experts here have some recommendations for best practices. 1. The Guided partitioning doesn't suggest any more to create /var, /tmp, /usr, etc. file systems. Is it really the recommendation to go with just / ? 2. Is there a way to use the old sysinstall to install FreeBSD 9? 3. It seems that setting up gmirror is more involved with GPT (http://blather.michaelwlucas.com/archives/1071); now I have a mirror for each of the filesystems /, /var, /tmp, etc. Is it OK to use gmirror in this way at all? 4. Also, with GPT, one has to be in single user mode to synchronize disks - correct? 3. Assuming one has enough RAM, is zfs mirror or raidz recommended over gmirror? Prior to FreeBSD 9, I used to take the the sysinstall defaults with some overrides as I thought appropriate and proceeded to set up gmirror - it was simple and not a lot of work, and a good way to make use of older systems... -- Janos Dohanics ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: FreeBSD 9, GPT and gmirror
On Wed, 8 Feb 2012 13:42:59 -0500 Janos Dohanics w...@3dresearch.com wrote: Hello Everyone, May be I should have searched more for answers, but after installing FreeBSD 9 with gmirror, I am wondering if the experts here have some recommendations for best practices. 1. The Guided partitioning doesn't suggest any more to create /var, /tmp, /usr, etc. file systems. Is it really the recommendation to go with just / ? This is a bad recommendation I think, but you can accept guidance and the adjust to your needs. 2. Is there a way to use the old sysinstall to install FreeBSD 9? Yes, harder to use, or no the new installer should have some more sane defaults 3. It seems that setting up gmirror is more involved with GPT (http://blather.michaelwlucas.com/archives/1071); now I have a mirror for each of the filesystems /, /var, /tmp, etc. Is it OK to use gmirror in this way at all? 4. Also, with GPT, one has to be in single user mode to synchronize disks - correct? 3. Assuming one has enough RAM, is zfs mirror or raidz recommended over gmirror? gmirror, still I think Prior to FreeBSD 9, I used to take the the sysinstall defaults with some overrides as I thought appropriate and proceeded to set up gmirror - it was simple and not a lot of work, and a good way to make use of older systems... I think the new installer is quite good, but needs some shaving around the rough edges Cheers Disclaimer: http://www.ose.nl/email ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: FreeBSD 9, GPT and gmirror
3. Assuming one has enough RAM, is zfs mirror or raidz recommended over gmirror? zfs mirror but I would not recommend a raidz root on zfs. -- George Kontostanos Aicom telecoms ltd http://www.aisecure.net ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: FreeBSD 9, GPT and gmirror
I can't speak to the mirror issue, but I had difficulty trying to tweak the defaults in the install on a 128G SSD: When manually configuring the SSD, I tried to leave some extra space at the end of the SSD. Not sure that is necessary or not. In any case, I had a 128GB SSD, reported as 119GB. Auto config laid it out as ada1 119GB ada1p1 64KB freebsd-boot ada1p2 115GB freebsd-ufs / ada1p34GB freebsd-swap I then deleted the last 2 and re-created as 100GB and 4GB, at which point it showed ada1 119GB ada1p1 64KB freebsd-boot ada1p2 100GB freebsd-ufs / ada1p3 -15GB freebsd-swap (I may have the -15 wrong; main point is it was negative) After deleting and recreating in different order I managed to get it to ada1 119GB ada1p1 64KB freebsd-boot ada1p34GB freebsd-swap ada1p2 100GB freebsd-ufs / but when I tried to commit it, I got the error: Error mounting partition /mnt: mount: /dev/ada1p2: Operation not permitted The only way I could get it to actually write the distribution was to use auto and keep what it came up with. Is this problem specific to SSDs (seems unlikely)? Is there some magic sequence needed to tweak the Auto result to get it to work? Gary On 2/8/2012 12:00 PM, Bas Smeelen wrote: On Wed, 8 Feb 2012 13:42:59 -0500 Janos Dohanicsw...@3dresearch.com wrote: Hello Everyone, May be I should have searched more for answers, but after installing FreeBSD 9 with gmirror, I am wondering if the experts here have some recommendations for best practices. 1. The Guided partitioning doesn't suggest any more to create /var, /tmp, /usr, etc. file systems. Is it really the recommendation to go with just / ? This is a bad recommendation I think, but you can accept guidance and the adjust to your needs. 2. Is there a way to use the old sysinstall to install FreeBSD 9? Yes, harder to use, or no the new installer should have some more sane defaults 3. It seems that setting up gmirror is more involved with GPT (http://blather.michaelwlucas.com/archives/1071); now I have a mirror for each of the filesystems /, /var, /tmp, etc. Is it OK to use gmirror in this way at all? 4. Also, with GPT, one has to be in single user mode to synchronize disks - correct? 3. Assuming one has enough RAM, is zfs mirror or raidz recommended over gmirror? gmirror, still I think Prior to FreeBSD 9, I used to take the the sysinstall defaults with some overrides as I thought appropriate and proceeded to set up gmirror - it was simple and not a lot of work, and a good way to make use of older systems... I think the new installer is quite good, but needs some shaving around the rough edges Cheers Disclaimer: http://www.ose.nl/email ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
gmirror failure booting 9.0 kernel upgrading from 8.2
I'm trying to upgrade a brand new server from 8.2 to 9.0 via source. I've done this upgrade twice so far, once on a vmware test system, and once on a Sun X4100m2, both with success. On this system, which is a Supermicro motherboard, I have gmirror boot disk. The other two did not (hardware RAID on the Sun). The boot fails as follows. The gmirror is not degraded. The part that concerns me is this: GEOM_PART: integrity check failed (mirror/gm0, MBR) This is loading a custom kernel, which pulls in geom_mirror as a module. I think the issue is that I have to convince gmirror to use ada0 and ada1 instead of ad4 and ad6 as the component devices. How does one accomplish this if one cannot boot? The server is remote, so plugging in the memstick image will be tricky :) KDB: debugger backends: ddb KDB: current backend: ddb Copyright (c) 1992-2012 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 9.0-RELEASE #0: Tue Jan 17 14:49:58 EST 2012 vi...@lorax.kcilink.com:/u/lorax1/usr9/obj.amd64/u/lorax1/usr9/src/sys/KCI64 amd64 CPU: Intel(R) Xeon(R) CPU E31220 @ 3.10GHz (3093.04-MHz K8-class CPU) Origin = GenuineIntel Id = 0x206a7 Family = 6 Model = 2a Stepping = 7 Features=0xbfebfbffFPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE Features2=0x15bae3ffSSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,XSAVE,AVX AMD Features=0x28100800SYSCALL,NX,RDTSCP,LM AMD Features2=0x1LAHF TSC: P-state invariant, performance statistics real memory = 8589934592 (8192 MB) avail memory = 8229543936 (7848 MB) Event timer LAPIC quality 600 ACPI APIC Table: SUPERM SMCI--MB FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs FreeBSD/SMP: 1 package(s) x 4 core(s) cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 2 cpu2 (AP): APIC ID: 4 cpu3 (AP): APIC ID: 6 ioapic0 Version 2.0 irqs 0-23 on motherboard kbd1 at kbdmux0 acpi0: SUPERM SMCI--MB on motherboard acpi0: Power Button (fixed) Timecounter ACPI-fast frequency 3579545 Hz quality 900 acpi_timer0: 24-bit timer at 3.579545MHz port 0x408-0x40b on acpi0 cpu0: ACPI CPU on acpi0 cpu1: ACPI CPU on acpi0 cpu2: ACPI CPU on acpi0 cpu3: ACPI CPU on acpi0 pcib0: ACPI Host-PCI bridge port 0xcf8-0xcff on acpi0 pci0: ACPI PCI bus on pcib0 pci0: simple comms at device 22.0 (no driver attached) pci0: simple comms at device 22.1 (no driver attached) ehci0: EHCI (generic) USB 2.0 controller mem 0xfbd03000-0xfbd033ff irq 16 at device 26.0 on pci0 usbus0: EHCI version 1.0 usbus0: EHCI (generic) USB 2.0 controller on ehci0 pcib1: ACPI PCI-PCI bridge irq 16 at device 28.0 on pci0 pci1: ACPI PCI bus on pcib1 pcib2: ACPI PCI-PCI bridge irq 16 at device 28.4 on pci0 pci2: ACPI PCI bus on pcib2 em0: Intel(R) PRO/1000 Network Connection 7.2.3 port 0xe000-0xe01f mem 0xfbc0-0xfbc1,0xfbc2-0xfbc23fff irq 16 at device 0.0 on pci2 em0: Using MSIX interrupts with 3 vectors em0: Ethernet address: 00:25:90:51:a3:aa pcib3: ACPI PCI-PCI bridge irq 17 at device 28.5 on pci0 pci3: ACPI PCI bus on pcib3 em1: Intel(R) PRO/1000 Network Connection 7.2.3 port 0xd000-0xd01f mem 0xfbb0-0xfbb1,0xfbb2-0xfbb23fff irq 17 at device 0.0 on pci3 em1: Using MSIX interrupts with 3 vectors em1: Ethernet address: 00:25:90:51:a3:ab pcib4: ACPI PCI-PCI bridge irq 18 at device 28.6 on pci0 pci4: ACPI PCI bus on pcib4 em2: Intel(R) PRO/1000 Network Connection 7.2.3 port 0xc000-0xc01f mem 0xfba0-0xfba1,0xfba2-0xfba23fff irq 18 at device 0.0 on pci4 em2: Using MSIX interrupts with 3 vectors em2: Ethernet address: 00:25:90:51:a3:ac pcib5: ACPI PCI-PCI bridge irq 19 at device 28.7 on pci0 pci5: ACPI PCI bus on pcib5 em3: Intel(R) PRO/1000 Network Connection 7.2.3 port 0xb000-0xb01f mem 0xfb90-0xfb91,0xfb92-0xfb923fff irq 19 at device 0.0 on pci5 em3: Using MSIX interrupts with 3 vectors em3: Ethernet address: 00:25:90:51:a3:ad ehci1: EHCI (generic) USB 2.0 controller mem 0xfbd02000-0xfbd023ff irq 23 at device 29.0 on pci0 usbus1: EHCI version 1.0 usbus1: EHCI (generic) USB 2.0 controller on ehci1 pcib6: ACPI PCI-PCI bridge at device 30.0 on pci0 pci6: ACPI PCI bus on pcib6 vgapci0: VGA-compatible display mem 0xfa00-0xfaff,0xfb80-0xfb803fff,0xfb00-0xfb7f irq 23 at device 3.0 on pci6 isab0: PCI-ISA bridge at device 31.0 on pci0 isa0: ISA bus on isab0 ahci0: Intel Cougar Point AHCI SATA controller port 0xf050-0xf057,0xf040-0xf043,0xf030-0xf037,0xf020-0xf023,0xf000-0xf01f mem 0xfbd01000-0xfbd017ff irq 19 at device 31.2 on pci0 ahci0: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported ahcich0: AHCI channel at channel 0 on ahci0 ahcich1: AHCI channel at channel 1 on ahci0 ahcich2: AHCI channel at channel 2 on ahci0
Re: Cannot create 2nd gmirror
On Fri, 6 Jan 2012 11:55:50 +1030 William Brown william.e.br...@adelaide.edu.au wrote: I am trying to add a second gmirror, gm1: # sysctl kern.geom.debugflags=16 kern.geom.debugflags: 16 - 16 # gmirror label -v -b round-robin gm1 /dev/ad4 Metadata value stored on /dev/ad4. Done. # gmirror insert gm1 /dev/ad6 gmirror: No such device: gm1. Why does gm1 fail to be created? What is the output of gmirror list after you run the gmirror label? Alternately, according to http://www.freebsd.org/cgi/man.cgi?query=gmirrorapropos=0sektion=0manpath=FreeBSD+6.1-RELEASEformat=html you should be able to just run gmirror label -v -b round-robin gm1 /dev/ad4 /dev/ad6 gmirror rebuild gm1 /dev/ad6 Sincerely, William Brown Research Teaching, Technology Services The University of Adelaide, AUSTRALIA 5005 gmirror list only showed gm0; it did did not show gm1, nor were there any gm1* entries in /dev/mirror. However, as soon as I have unmounted /dev/ad4s1d, /dev/ad4s1e, and /dev/ad4s1f, gm1 was automagically created: Jan 4 20:21:27 0.2 isolde kernel: GEOM_MIRROR: Device mirror/gm1 launched (1/1). Jan 4 20:21:27 0.2 isolde kernel: GEOM: mirror/gm1s1: geometry does not match label (16h,63s != 255h,63s). Jan 4 21:22:32 0.2 isolde kernel: GEOM_MIRROR: Device gm1: rebuilding provider ad6. Don't quite understand why creating gm1 happened only after unmounting the filesystems on ad4. -- Janos Dohanics ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Cannot create 2nd gmirror
Hello Everyone, I have system with gmirror gm0: # gmirror list Geom name: gm0 State: COMPLETE Components: 2 Balance: round-robin Slice: 4096 Flags: NONE GenID: 0 SyncID: 1 ID: 3516398316 Providers: 1. Name: mirror/gm0 Mediasize: 320072932864 (298G) Sectorsize: 512 Mode: r5w5e14 Consumers: 1. Name: ad8 Mediasize: 320072933376 (298G) Sectorsize: 512 Mode: r1w1e1 State: ACTIVE Priority: 0 Flags: NONE GenID: 0 SyncID: 1 ID: 95660722 2. Name: ad10 Mediasize: 320072933376 (298G) Sectorsize: 512 Mode: r1w1e1 State: ACTIVE Priority: 0 Flags: NONE GenID: 0 SyncID: 1 ID: 632264112 I am trying to add a second gmirror, gm1: # sysctl kern.geom.debugflags=16 kern.geom.debugflags: 16 - 16 # gmirror label -v -b round-robin gm1 /dev/ad4 Metadata value stored on /dev/ad4. Done. # gmirror insert gm1 /dev/ad6 gmirror: No such device: gm1. Why does gm1 fail to be created? -- Janos Dohanics ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
gmirror resync seems stalled
I had a drive which had some timeout problems, and got kicked out of a gmirror based RAID1 on my FreeBSD machine (now 8.2-RELEASE-p3). Normally, if the devices get out of sync, they rebuild relatively quickly, and I can watch the progress. This time, after running gmirror forget and inserting the device, it just seems to sit at '0%', despite the fact that it's supposedly synchronizing. I've tried removing and re-adding it a couple times, but the behavior is the same. Am I just being too impatient? Autosynchronization of stale components should have already been enabled (and the gmirror list output says 'synchronizing'), but I tried a gmirror configure -a gm0 just in case. Also, any opinions on how safe the '-F' option (Do not synchronize after a power failure or system crash) is? Following: # gmirror remove gm0 ad6 # gmirror forget gm0 ad6 # gmirror insert gm0 ad6 I see: # gmirror list Geom name: gm0 State: DEGRADED Components: 2 Balance: round-robin Slice: 4096 Flags: NONE GenID: 4 SyncID: 1 ID: 506994055 Providers: 1. Name: mirror/gm0 Mediasize: 1000204885504 (932G) Sectorsize: 512 Mode: r6w5e14 Consumers: 1. Name: ad10 Mediasize: 1000204886016 (932G) Sectorsize: 512 Mode: r1w1e1 State: ACTIVE Priority: 0 Flags: DIRTY GenID: 4 SyncID: 1 ID: 3348119132 2. Name: ad6 Mediasize: 1000204886016 (932G) Sectorsize: 512 Mode: r1w1e1 State: SYNCHRONIZING Priority: 0 Flags: DIRTY, SYNCHRONIZING GenID: 4 SyncID: 1 Synchronized: 0% ID: 3439333064 and the status seems to just stay at 0% relevant atacontrol output: ATA channel 3: Master: ad6 WDC WD10EVDS-63U8B0/01.00A01 SATA revision 2.x Slave: no device present ATA channel 5: Master: ad10 WDC WD10EVDS-63U8B0/01.00A01 SATA revision 2.x Slave: no device present ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
RE: glabel, gmirror, and gpart
Notes on a gmirrored partitions setup written into a draft article: http://www.wonkity.com/~wblock/docs/html/gmirror.html ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
RE: glabel, gmirror, and gpart
On Fri, Aug 26, 2011 at 11:04 PM, Warren Block wbl...@wonkity.com wrote: So it's cosmetic, but not really the kind of message that instills confidence. gptboot needs to be able to tell if it's reading from a gmirror. Or gmirror should provide one block less than it does, so it doesn't overwrite the GPT backup. I use the whole disk to gmirror setup on all my servers. And one day i thought i could try the gpart stuff. I also saw the message at startup, and disgarded it. This was on 8.1 But when i did an upgrade to 8.2 later on, the system could not boot. So i had to find another way. So i would not do it like you do now, it could turn against you in the long run. The thing i did was to mirror each partition. Like so http://unix-heaven.org/node/24 that way it worked. Do not forget to make your second disk bootable also regards, Johan Hendriks___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
RE: glabel, gmirror, and gpart
On Sat, 27 Aug 2011, Johan Hendriks wrote: On Fri, Aug 26, 2011 at 11:04 PM, Warren Block wbl...@wonkity.com wrote: So it's cosmetic, but not really the kind of message that instills confidence. gptboot needs to be able to tell if it's reading from a gmirror. Or gmirror should provide one block less than it does, so it doesn't overwrite the GPT backup. I use the whole disk to gmirror setup on all my servers. And one day i thought i could try the gpart stuff. I also saw the message at startup, and disgarded it. This was on 8.1 But when i did an upgrade to 8.2 later on, the system could not boot. So i had to find another way. So i would not do it like you do now, it could turn against you in the long run. The thing i did was to mirror each partition. Like so http://unix-heaven.org/node/24 that way it worked. Do not forget to make your second disk bootable also That was the idea I had last night trying to go back to sleep. Use gpart to partition the disks, and use gmirror to mirror the partitions. A little more setup, because you have to create multiple mirrors instead of one with partitions inside it. It also addresses Adam's question. For two dissimilar disks, create partitions of the same size and mirror them instead of the whole disk. gmirror apparently doesn't keep a list of outstanding writes, so smaller mirrors should be faster to sync. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
glabel, gmirror, and gpart
Trying to use labeled devices and filesystems where possible, and adding gmirror into the mix. (This is with 8-STABLE, i386.) glabel two disks primary and secondary. gmirror the two: gmirror label -v -b round-robin data /dev/label/primary /dev/label/secondary Use gpart to create *and label* partitions on the mirror device: =34 1465149099 mirror/data GPT (698G) 34 641 gpboot (32k) 981950 - free - (975k) 2048 41943042 gprootfs (2.0G) 4196352 83886083 gpswap (4.0G) 12584960167772164 gpvarfs (8.0G) 29362176167772165 gptmpfs (8.0G) 46139392 2516582406 gpusrfs (120G) 297797632 4194304007 gpotherfs (200G) 717228032 747921101 - free - (356G) Then install/restore as usual, refer to /dev/gpt/labelname in /etc/fstab. So far, it works. My concerns are with correctness, doing it right the first time to avoid unpleasant surprises later. Question 1 is whether using slashes in labels is going to be a problem later? gpart can find /dev/mirror/data and mirror/data, but not just data. gmirror status shows the, uh, path-like labels in both the Name and the Components: NameStatus Components mirror/data DEGRADED label/primary (ACTIVE) label/secondary (SYNCHRONIZING, 23%) Question 2 is maybe simpler. On boot, it shows this: gptboot: invalid backup GPT header I can speculate on what causes that (whether the various g-things use the absolute last block of the device, or the last block of their area and provide their total minus one block; or it could be that gptboot is seeing the disk rather than the gmirror device). Can it be fixed? ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: glabel, gmirror, and gpart
On Fri, Aug 26, 2011 at 8:09 PM, Warren Block wbl...@wonkity.com wrote: Question 2 is maybe simpler. On boot, it shows this: gptboot: invalid backup GPT header I can speculate on what causes that (whether the various g-things use the absolute last block of the device, or the last block of their area and provide their total minus one block; or it could be that gptboot is seeing the disk rather than the gmirror device). Can it be fixed? This is a long standing issue that still isn't fixed to the best of my knowledge. The problem is with gmirror and GPT. http://translate.google.com/translate?js=nprev=_thl=enie=UTF-8layout=2eotf=1sl=autotl=enu=http%3A%2F%2Fbu7cher.blogspot.com%2F2011%2F03%2Ffreebsd-gmirror-gpt-ufs.htmlact=url is the best I can do. -- Adam Vande More ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: glabel, gmirror, and gpart
On Fri, 26 Aug 2011, Adam Vande More wrote: On Fri, Aug 26, 2011 at 8:09 PM, Warren Block wbl...@wonkity.com wrote: Question 2 is maybe simpler. On boot, it shows this: gptboot: invalid backup GPT header I can speculate on what causes that (whether the various g-things use the absolute last block of the device, or the last block of their area and provide their total minus one block; or it could be that gptboot is seeing the disk rather than the gmirror device). Can it be fixed? This is a long standing issue that still isn't fixed to the best of my knowledge. The problem is with gmirror and GPT. http://translate.google.com/translate?js=nprev=_thl=enie=UTF-8layout=2eotf=1sl=autotl=enu=http%3A%2F%2Fbu7cher.blogspot.com%2F2011%2F03%2Ffreebsd-gmirror-gpt-ufs.htmlact=url is the best I can do. Thanks! So it's cosmetic, but not really the kind of message that instills confidence. gptboot needs to be able to tell if it's reading from a gmirror. Or gmirror should provide one block less than it does, so it doesn't overwrite the GPT backup.___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: glabel, gmirror, and gpart
On Fri, Aug 26, 2011 at 11:04 PM, Warren Block wbl...@wonkity.com wrote: So it's cosmetic, but not really the kind of message that instills confidence. gptboot needs to be able to tell if it's reading from a gmirror. Or gmirror should provide one block less than it does, so it doesn't overwrite the GPT backup. I would sort of agree with the latter although I'm afraid of what I don't know. One example is that in a transparent vs hardcoded gmirror, the -1 would have to apply to both to allow gmirror configure to continue to work without accidentally destroying something. While we're on the topic of desired gmirror changes I'd like an easier method to right-size a mirror so that mirroring similar sized hard drives from different manufacturers doesn't create an Uff Da when the number of bytes isn't consistent. You can do this with gnop, but that requires extra steps. -- Adam Vande More ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Drive selection for gmirror
On 28/04/2011 23:13, Ireneusz Pluta wrote: when selecting SATA drives for gmirror, boot device, connected to an on-board controller, should I look for so-called enterprise grade, or raid edition drives (like for instance http://www.wdc.com/en/products/products.aspx?id=40), or I should rather focus on models closer to consumer grade? Will gmirror configuration significantly benefit from things like TLER, or it would rather be harmful, like when using these disks as single ones in desktop applications? gmirror doesn't do all of the really fancy stuff an expensive hardware RAID card will. So, if one disk in a mirror fails a write, the mirror breaks and has to be resynch'd. With gmirror that means manually entering a sequence of commands to remove and re-add the drive to the gmirror. Enterprise grade drives can be much more reliable over all as they are designed for 100% duty, whereas with consumer grade the expectation is that the drive will be turned off much of the time. This is typically one of the distinguishing features between SAS and SATA as well. Green drives can be particularly bad with FreeBSD, as they frequently end up spinning up and down continuously, which both wastes a lot of power and also shortens the life of the drive: neither of which adds to the actual (rather than claimed) green credentials of the device. Decide how much disk space you want, and what your budget is, then buy the best quality drives that match your criteria. I'd turn off TLER. Cheers Matthew -- Dr Matthew J Seaman MA, D.Phil. 7 Priory Courtyard Flat 3 PGP: http://www.infracaninophile.co.uk/pgpkey Ramsgate JID: matt...@infracaninophile.co.uk Kent, CT11 9PW signature.asc Description: OpenPGP digital signature
Re: Drive selection for gmirror
Ireneusz Pluta wrote: Hello, when selecting SATA drives for gmirror, boot device, connected to an on-board controller, should I look for so-called enterprise grade, or raid edition drives (like for instance http://www.wdc.com/en/products/products.aspx?id=40), or I should rather focus on models closer to consumer grade? Will gmirror configuration significantly benefit from things like TLER, or it would rather be harmful, like when using these disks as single ones in desktop applications? This is an overly broad generalization, but I would tend to match RE drives with expensive hardware-only RAID controllers. They are touchy and really designed with TLER in mind. On the other end of the spectrum is a pure software RAID such as gmirror. I don't know all that much about the innards but I believe a pure software RAID run on JBOD will work with consumer grade drives just fine most of the time. The in-between caveat is the so-called 'fakeraid' sets of controllers. These are what you're most likely to find as a cheap 'feature-add' on a consumer desktop motherboard. Things like Intel RAID Matrix and such are known not to work well with non-Windows environments. However, much of the time these will still work fine if RAID functionality is turned off in BIOS and the connected drives are JBOD, with a pure software RAID applied instead of trying to utilize the motherboard BIOS RAID. My web development server at home has a pair of Raptors on an ICH5 controller with gmirror and has been completely trouble free for about a year now. I'd stay away from the Green drives that spin down all the time. Even a pure software RAID solution is likely not going to be happy with them. -Mike ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Drive selection for gmirror
Hello, when selecting SATA drives for gmirror, boot device, connected to an on-board controller, should I look for so-called enterprise grade, or raid edition drives (like for instance http://www.wdc.com/en/products/products.aspx?id=40), or I should rather focus on models closer to consumer grade? Will gmirror configuration significantly benefit from things like TLER, or it would rather be harmful, like when using these disks as single ones in desktop applications? Thanks Irek. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Safe to use GPT within gmirror?
Hi, can I safely use GPTs within a GEOM_MIRROR? I created a new mirror and then used gpart to create additinal partitions. dmesg gives: the secondary GPT header is not in the last LBA As far as I read by now it seems safe to ignore that message but I want to get sure. Or are mirrored GPTs only safe when using ZFS? Thanks, Helmut ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
gmirror and normal users?
Should a normal user be able to successfully: $ gmirror remove /dev/mirror/gm0 /dev/ad6 Or is this something that's just unlocked because I haven't mounted the drive yet? $ uname -a FreeBSD deathstar.example.com 8.2-STABLE FreeBSD 8.2-STABLE #1: Wed Apr 6 13:09:37 EDT 2011 root@dagobah:/usr/obj/usr/src/sys/GENERIC i386 $ id uid=1001(chris) gid=1001(chris) groups=1001(chris),0(wheel),5(operator),1000(users) Chris Hilton tildeChris -- http://myblog.vindaloo.com e: -- chris /at/ vindaloo /dot/ com .~~.--.~~.--.~~.--.~~.--.~~.--.~~.--.~~.--.~~.--.~~.--.~~.--.~~.--.~~.--.~~. I'm on the outside looking inside, What do I see? Much confusion, disillusion, all around me. -- Ian McDonald / Peter Sinfield ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: gmirror and normal users?
On 08/04/2011 16:43, Christopher Hilton wrote: Should a normal user be able to successfully: $ gmirror remove /dev/mirror/gm0 /dev/ad6 Or is this something that's just unlocked because I haven't mounted the drive yet? $ uname -a FreeBSD deathstar.example.com 8.2-STABLE FreeBSD 8.2-STABLE #1: Wed Apr 6 13:09:37 EDT 2011 root@dagobah:/usr/obj/usr/src/sys/GENERIC i386 $ id uid=1001(chris) gid=1001(chris) groups=1001(chris),0(wheel),5(operator),1000(users) It is because of the operator group. Normal users which are not in this groups would not be able to do it. If a user can communicate with the device (i.e. has at least reads rights to it), he can send GEOM commands to it. The operator group has read permissions by default: lara:~ ll /dev/mirror/ total 0 crw-r- 1 root operator0, 150 8 Apr 16:55 bla ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Can't Boot 8.2 with Gmirror
Once upon a time, I partitioned two disks identically and then added them to a mirror. It was good. Then I upgraded to 8.2-RELEASE and now I can't boot. Well, I did a little recovery work and I am currently booting without the gmirror so I am satisfied that my data is safe. Having read a few messages it sounds like there are some steps I need to take to fix up my partitioning scheme to make things work right in 8.2-RELEASE. But since gotchas got me once, what are the gotchas? Should I partition before adding a disk to a mirror? Unfortunately, one of the messages that I read said, There's no fix as yet. so I am quite leary of proceeding without a little help from my friends. Thanks, Jason C. Wells ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: a gmirror disappears after adding gjournals to its partitions
CyberLeo Kitsana cyber...@cyberleo.net wrote: ... I hope it makes sense! No problem with the explanation making sense; what I don't follow is the behavior of bsdlabel. Given the way I set it up this drive _should_ contain _two_ labels, but for some unfathomable reason bsdlabel seems to be using the second (inner) one while ignoring the first (outer) one entirely. The device itself is ad0. Its MBR contains a slice table, defining ad0s1 and ad0s2. (ad0s1 is FAT32 and AFAIK need not be considered further at this point.) ad0s2 starts with a bsdlabel, which defines ad0s2a and ad0s2b. (ad0s2b is intended to be used as swap and, like ad0s1, need not be considered further at this point -- but it _should_ be instantiated along with ad0s2a.) ad0s2a is supposed to be the provider for gm0, and it starts with a bsdlabel that is intended to partition gm0 into gm0[ade], but since geom_mirror.ko hasn't been loaded yet gm0 doesn't exist and ad0s2a is just a partition that happens to start with a bsdlabel and end with gmirror metadata. I could understand if bsdlabel tasted ad0s2a, found the label, and (recursively) instantiated ad0s2aa, ad0s2ad, and ad0s2ae; but that doesn't seem to happen. Instead, bsdlabel seems to ignore (or forget) the first label it tasted -- the one on ad0s2 -- and treats the one on ad0s2a as applying to ad0s2. We end up with ad0s2a containing the disk blocks intended for gm0a, ad0s2d containing the disk blocks intended for gm0d, ad0s2e containing the disk blocks intended for gm0e; and the blocks intended for ad0s2b (swap) -- which were not supposed to have been involved with any mirror or journal -- seem to have disappeared entirely. This seems like a bug. Now gjournal gets into the act, consuming the phony ad0s2[ade] before gmirror gets a chance to taste the real ad0s2a and instantiate gm0. That explains why gm0 and /dev/mirror are missing, but not why ad0s2b is missing, nor why we have ad0s2[ade] (and the corresponding .journal's) rather than ad0s2a[ade] (and their .journal's). (This machine is likely too old to understand GPT.) The machine's bios does not need to understand GPT to use it on a pure data disk; only as a boot disk. This is, however, intended as a boot disk -- gm0a, gm0d, and gm0e are supposed to be root, /var, and /usr respectively -- and it does seem to boot OK until it tries find the root FS (because /etc/fstab is set up to use gm0[ade].journal instead of ad0s2[ade].journal). I suppose I could try partitioning ad0s2a with gpt instead of with bsdlabel, but would the loader still be able to find the kernel? ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: a gmirror disappears after adding gjournals to its partitions
CyberLeo Kitsana cyber...@cyberleo.net wrote: If the kldstat Id numbers are assigned sequentially, it looks as if geom_journal got loaded first and this may somehow be related (although I don't entirely see how -- absent geom_mirror to make gm0 and its partitions visible, I'd think that geom_journal should not be able to find its metadata at all). From what I've found, this is because there is no taste difference between a bsdlabel on a gmirror and a bsdlabel on a non-mirror. ... which seems like a bug, unless I misunderstand how geoms work -- see diagram below: * If gjournal stores its metadata at the end of its provider, it should not be finding anything recognizable at the end of ad0s2a, because that block contains gmirror's metadata. * OTOH, if gjournal is looking for something at the beginning of its provider, it should be finding a bsdlabel -- not gjournal metadata -- at the beginning of ad0s2a and that should keep it from recognizing anything in ad0s2a (which is already known to be a partition, thus finding another bsdlabel at its beginning cannot be legitimate). It looks to me as if gjournal is confused: it claims to have found data and journal on each of ad0s2a, ad0s2d, and ad0s2e but in fact only the first of those even exists! The actual partitioning of ad0s2 is into ad0s2a and ad0s2b (plus the conventional ad0s2c entry covering all of ad0s2). It is gm0 (whose provider is ad0s2a) that is partitioned into gm0a, gm0d, and gm0e; if gm0's bsdlabel were interpreted as being directly on ad0s2a, shouldn't those partitions be named ad0s2aa, ad0s2ad, and ad0s2ae? __ | ad0s2 bsdlabel (on ad0s2) | ad0s2c _ || ad0s2a | gm0 bsdlabel (on gm0) ||| gm0c _ ||| | gm0a | data ||| | | ||| | |_ ||| | | journal ||| | |_ ||| |___|_gjMeta ||| | gm0d | data ||| | | ||| | | ||| | | ||| | | ||| | |_ ||| | | journal ||| | |_ ||| |___|_gjMeta ||| | gm0e | data ||| | | ||| | | ||| | | ||| | | ||| | | ||| | | ||| | | ||| | | ||| | | ||| | | ||| | | ||| | | ||| | |_ ||| | | journal ||| | |_ |||__|___|_gjMeta ||_gmMeta || ad0s2b || || ||_ ... either make the two look different somehow (use a different geom that stores its metadata at the beginning of the provider, instead of the end, thus eliminating ambiguity in the bsdlabel taste), When I asked earlier how to subdivide gm0, bsdlabel was recommended. Is there something else that would work better? (This machine is likely too old to understand GPT.) or to make the inner geom avoid the outer devices (hardcode provider names in metadata). Since you have an outer geom that provides a static name, hardcoding the name of the gmirror into the gjournal metadata shouldn't cause anything to break if your disks change places, either. But I suspect this may not scale well. Suppose I later decide to mirror the swap instead of using ad0s2b and ad8s2b as separate swap partitions. Is there not a 50/50 chance of the swap mirror becoming gm0 and my current gm0 becoming gm1, thereby breaking any metadata that depends on hard-coded provider names? ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: a gmirror disappears after adding gjournals to its partitions
On 11/24/2010 04:52 AM, per...@pluto.rain.com wrote: It looks to me as if gjournal is confused: It is not gjournal that is confused; it's bsdlabel. The gjournals lie entirely within the partitions defined within the bsdlabel, and don't care about anything outside of that. The ambiguity here is that the bsdlabel is stored at the beginning of the disk, and is very loose about what it accepts as valid, since there is no direct harm in being eager. The metadata for gmirror is stored at the end. The metadata for the bsdlabel is stored at the beginning. When bsdlabel tastes before gmirror, it sees the same label on the component disks that would be on the gm0 mirror. Moreover, all the partitions it then creates are identically sized, and contain exactly the same data, as they would on the mirror. It will complain that partition 'c' doesn't cover the whole unit, but this is not a fatal error as it doesn't take exclusive access, and so you are always free to use that same bsdlabel through another geom path. The problem arises when bsdlabel tastes ad0 before gmirror, and creates all the partitions thereupon, which triggers a taste of all the newly created devices by gjournal, which opens the devices exclusively once it finds the metadata it needs within the partitions. Now that they're opened exclusively somewhere, all the other paths to that device through the geom graph are withered, and cannot be tasted or used by anything else, including gmirror. Hardcoding provider names into gjournal makes it reject these ambiguously created devices. Since gjournal doesn't take exclusive access, gmirror can now taste the still-available ad0, see that it's a mirror, and launch gm0, which triggers a taste by bsdlabel (and creates the partitions) which triggers a taste by gjournal, which matches the names its expecting. That was difficult to keep clear. I hope it makes sense! ... either make the two look different somehow (use a different geom that stores its metadata at the beginning of the provider, instead of the end, thus eliminating ambiguity in the bsdlabel taste), When I asked earlier how to subdivide gm0, bsdlabel was recommended. Is there something else that would work better? (This machine is likely too old to understand GPT.) The machine's bios does not need to understand GPT to use it on a pure data disk; only as a boot disk. There are a few bioses that throw fits when not all the disks include mbr/slice tables, but those (thankfully) tend to be the minority. Plus, since GPT expects metadata at both the beginning and end of the disk, seeing gmirror metadata instead may prevent it from creating these ambiguous device nodes as well (but test this assumption before relying on it). or to make the inner geom avoid the outer devices (hardcode provider names in metadata). Since you have an outer geom that provides a static name, hardcoding the name of the gmirror into the gjournal metadata shouldn't cause anything to break if your disks change places, either. But I suspect this may not scale well. Suppose I later decide to mirror the swap instead of using ad0s2b and ad8s2b as separate swap partitions. Is there not a 50/50 chance of the swap mirror becoming gm0 and my current gm0 becoming gm1, thereby breaking any metadata that depends on hard-coded provider names? When you create a mirror, you give it an explicit name, which will not change over the life of the mirror without your explicit action. This name does not have to be 'gm0' or some such. I have named mirrors after the hostname, or 'hostname-purpose', such as 'sc1425-root' and 'sc1425-swap' -- Fuzzy love, -CyberLeo Technical Administrator CyberLeo.Net Webhosting http://www.CyberLeo.Net cyber...@cyberleo.net Furry Peace! - http://.fur.com/peace/ ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: a gmirror disappears after adding gjournals to its partitions
On 11/22/2010 10:19 PM, per...@pluto.rain.com wrote: krad kra...@gmail.com wrote: On 21 November 2010 06:10, per...@pluto.rain.com wrote: ... manually-created config files, while still in chroot after install Fixit# cat /boot/loader.conf geom_mirror_load=YES geom_journal_load=YES vfs.root.mountfrom=ufs:/dev/mirror/gm0a.journal vfs.root.mountfrom.options=rw ... output from kldstat, after booting the newly-installed system -- and manually mounting the root FS -- showing that geom_mirror.ko did get loaded. Id Refs AddressSize Name 16 0xc040 bb5504 kernel 21 0xc0fb6000 14540geom_journal.ko 31 0xc0fcb000 16ed4geom_mirror.ko ... sounds silly but are you loading the gmirror kernel module via loader.conf Yes, I'm even setting geom_mirror_load to YES before setting geom_journal_load to YES (although I doubt the order of these settings in loader.conf makes any difference). If the kldstat Id numbers are assigned sequentially, it looks as if geom_journal got loaded first and this may somehow be related (although I don't entirely see how -- absent geom_mirror to make gm0 and its partitions visible, I'd think that geom_journal should not be able to find its metadata at all). From what I've found, this is because there is no taste difference between a bsdlabel on a gmirror and a bsdlabel on a non-mirror. Since both gmirror and gjournal are greedy (they take exclusive access of their parent providers upon successful taste, and not upon exclusive access to their own providers like glabel), the first one to successfully taste and start is the winner; the other will never get to taste those devices. The trick here is to either make the two look different somehow (use a different geom that stores its metadata at the beginning of the provider, instead of the end, thus eliminating ambiguity in the bsdlabel taste), or to make the inner geom avoid the outer devices (hardcode provider names in metadata). Since you have an outer geom that provides a static name, hardcoding the name of the gmirror into the gjournal metadata shouldn't cause anything to break if your disks change places, either. http://pb.cyberleo.net/?show=m7fcbcef7 -- Fuzzy love, -CyberLeo Technical Administrator CyberLeo.Net Webhosting http://www.CyberLeo.Net cyber...@cyberleo.net Furry Peace! - http://.fur.com/peace/ ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: a gmirror disappears after adding gjournals to its partitions
2010/11/23 CyberLeo Kitsana cyber...@cyberleo.net On 11/22/2010 10:19 PM, per...@pluto.rain.com wrote: krad kra...@gmail.com wrote: On 21 November 2010 06:10, per...@pluto.rain.com wrote: ... manually-created config files, while still in chroot after install Fixit# cat /boot/loader.conf geom_mirror_load=YES geom_journal_load=YES vfs.root.mountfrom=ufs:/dev/mirror/gm0a.journal vfs.root.mountfrom.options=rw ... output from kldstat, after booting the newly-installed system -- and manually mounting the root FS -- showing that geom_mirror.ko did get loaded. Id Refs AddressSize Name 16 0xc040 bb5504 kernel 21 0xc0fb6000 14540geom_journal.ko 31 0xc0fcb000 16ed4geom_mirror.ko ... sounds silly but are you loading the gmirror kernel module via loader.conf Yes, I'm even setting geom_mirror_load to YES before setting geom_journal_load to YES (although I doubt the order of these settings in loader.conf makes any difference). If the kldstat Id numbers are assigned sequentially, it looks as if geom_journal got loaded first and this may somehow be related (although I don't entirely see how -- absent geom_mirror to make gm0 and its partitions visible, I'd think that geom_journal should not be able to find its metadata at all). From what I've found, this is because there is no taste difference between a bsdlabel on a gmirror and a bsdlabel on a non-mirror. Since both gmirror and gjournal are greedy (they take exclusive access of their parent providers upon successful taste, and not upon exclusive access to their own providers like glabel), the first one to successfully taste and start is the winner; the other will never get to taste those devices. The trick here is to either make the two look different somehow (use a different geom that stores its metadata at the beginning of the provider, instead of the end, thus eliminating ambiguity in the bsdlabel taste), or to make the inner geom avoid the outer devices (hardcode provider names in metadata). Since you have an outer geom that provides a static name, hardcoding the name of the gmirror into the gjournal metadata shouldn't cause anything to break if your disks change places, either. http://pb.cyberleo.net/?show=m7fcbcef7 -- Fuzzy love, -CyberLeo Technical Administrator CyberLeo.Net Webhosting http://www.CyberLeo.Net cyber...@cyberleo.net Furry Peace! - http://.fur.com/peace/ I think what he is saying is slice it up 1st, then mirror the slices, and slap the journal on top of that ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: a gmirror disappears after adding gjournals to its partitions
On 21 November 2010 06:10, per...@pluto.rain.com wrote: Is there something wrong with this sequence, in Fixit: * create a mirror * partition it with disklabel * create journals on the partitions * install * reboot? After rebooting, the mirror had disappeared and the journals seemed to exist directly on partitions of the mirror's provider rather than on the mirror itself. Details: Using Fixit# from the 8.1-RELEASE memstick I defined a gmirror (initially containing only one provider; the other to be added later), partitioned it using disklabel, added a journal to each of the partitions, ran newfs -J on them, and installed FreeBSD using http://wiki.freebsd.org/RootOnZFS/GPTZFSBoot/Mirror as a guide (with a few differences due to this installation being UFS rather than ZFS). In Fixit the /dev tree contained entries for both the mirror and the journal devices, but when I rebooted the mirror did not show up -- even though geom_mirror.ko was loaded. What would cause this sort of mixup, and how do I fix it? Is any more info needed? contents of /dev/mirror before creating the journals Fixit# ls -la /dev/mirror total 1 dr-xr-xr-x 2 root 0 512 Nov 14 00:17 . dr-xr-xr-x 9 root 0 512 Nov 14 00:11 .. crw-r- 1 root operator0, 80 Nov 14 00:36 gm0 crw-r- 1 root operator0, 126 Nov 14 00:36 gm0a crw-r- 1 root operator0, 128 Nov 14 00:36 gm0d crw-r- 1 root operator0, 129 Nov 14 00:36 gm0e corresponding disklabel report Fixit# disklabel /dev/mirror/gm0 # /dev/mirror/gm0: 8 partitions: #size offsetfstype [fsize bsize bps/cpg] a: 8388608 164.2BSD 1024 819216 c: 6199075170unused0 0 d: 25165824 83886244.2BSD0 0 0 e: 586353069 335544484.2BSD0 0 0 journal creation, with resulting dmesg reports Fixit# gjournal label -s 2G /dev/mirror/gm0a GEOM_JOURNAL: Journal 1098378706: mirror/gm0a contains data. GEOM_JOURNAL: Journal 1098378706: mirror/gm0a contains journal. GEOM_JOURNAL: Journal mirror/gm0a clean. Fixit# gjournal label -s 2G /dev/mirror/gm0d GEOM_JOURNAL: Journal 3795372090: mirror/gm0d contains data. GEOM_JOURNAL: Journal 3795372090: mirror/gm0d contains journal. GEOM_JOURNAL: Journal mirror/gm0d clean. Fixit# gjournal label -s 2G /dev/mirror/gm0e GEOM_JOURNAL: Journal 2063379813: mirror/gm0e contains data. GEOM_JOURNAL: Journal 2063379813: mirror/gm0e contains journal. GEOM_JOURNAL: Journal mirror/gm0e clean. contents of /dev/mirror after creating the journals Fixit# ls -la /dev/mirror total 1 dr-xr-xr-x 2 root 0 512 Nov 14 00:17 ./ dr-xr-xr-x 9 root 0 512 Nov 14 00:11 ../ crw-r- 1 root operator0, 80 Nov 14 02:01 gm0 crw-r- 1 root operator0, 78 Nov 14 02:06 gm0a crw-r- 1 root operator0, 126 Nov 14 02:06 gm0a.journal crw-r- 1 root operator0, 125 Nov 14 02:07 gm0d crw-r- 1 root operator0, 128 Nov 14 02:07 gm0d.journal crw-r- 1 root operator0, 130 Nov 14 02:07 gm0e crw-r- 1 root operator0, 129 Nov 14 02:07 gm0e.journal newfs commands Fixit# newfs -J /dev/mirror/gm0a.journal Fixit# newfs -J /dev/mirror/gm0d.journal Fixit# newfs -J /dev/mirror/gm0e.journal mount the resulting filesystems (and one ordinary partition, neither mirrored nor journalled, to be used as /tmp -- I figure /tmp is expendable), resulting in this FS configuration Fixit# mount /dev/md0 on / (ufs, local) devfs on /dev (devfs, local, multilabel) /dev/da1a on /dist (ufs, local, read-only) /dev/mirror/gm0a.journal on /mnt (ufs, local, gjournal) /dev/ad8s2d on /mnt/tmp (ufs, local) /dev/mirror/gm0d.journal on /mnt/var (ufs, local, gjournal) /dev/mirror/gm0e.journal on /mnt/usr (ufs, local, gjournal) install per http://wiki.freebsd.org/RootOnZFS/GPTZFSBoot/Mirror, no detailed log kept manually-created config files, while still in chroot after install Fixit# cat /boot/loader.conf geom_mirror_load=YES geom_journal_load=YES vfs.root.mountfrom=ufs:/dev/mirror/gm0a.journal vfs.root.mountfrom.options=rw Fixit# cat /etc/fstab /dev/mirror/gm0a.journal / ufs rw 0 1 /dev/ad0s2b none swapsw 0 0 /dev/ad8s2b none swapsw 0 0 /dev/ad8s2d /tmp ufs rw 0 2 /dev/mirror/gm0d.journal /var ufs rw 0 3 /dev/mirror/gm0e.journal /usr ufs rw 0 4 /dev/da1a/dist ufs ro 0 0 devfs/dev devfs multilabel 0 0 output from kldstat, after booting the newly-installed system -- and manually mounting the root FS -- showing that geom_mirror.ko did get loaded. Id Refs
Re: a gmirror disappears after adding gjournals to its partitions
krad kra...@gmail.com wrote: On 21 November 2010 06:10, per...@pluto.rain.com wrote: ... manually-created config files, while still in chroot after install Fixit# cat /boot/loader.conf geom_mirror_load=YES geom_journal_load=YES vfs.root.mountfrom=ufs:/dev/mirror/gm0a.journal vfs.root.mountfrom.options=rw ... output from kldstat, after booting the newly-installed system -- and manually mounting the root FS -- showing that geom_mirror.ko did get loaded. Id Refs AddressSize Name 16 0xc040 bb5504 kernel 21 0xc0fb6000 14540geom_journal.ko 31 0xc0fcb000 16ed4geom_mirror.ko ... sounds silly but are you loading the gmirror kernel module via loader.conf Yes, I'm even setting geom_mirror_load to YES before setting geom_journal_load to YES (although I doubt the order of these settings in loader.conf makes any difference). If the kldstat Id numbers are assigned sequentially, it looks as if geom_journal got loaded first and this may somehow be related (although I don't entirely see how -- absent geom_mirror to make gm0 and its partitions visible, I'd think that geom_journal should not be able to find its metadata at all). ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
a gmirror disappears after adding gjournals to its partitions
Is there something wrong with this sequence, in Fixit: * create a mirror * partition it with disklabel * create journals on the partitions * install * reboot? After rebooting, the mirror had disappeared and the journals seemed to exist directly on partitions of the mirror's provider rather than on the mirror itself. Details: Using Fixit# from the 8.1-RELEASE memstick I defined a gmirror (initially containing only one provider; the other to be added later), partitioned it using disklabel, added a journal to each of the partitions, ran newfs -J on them, and installed FreeBSD using http://wiki.freebsd.org/RootOnZFS/GPTZFSBoot/Mirror as a guide (with a few differences due to this installation being UFS rather than ZFS). In Fixit the /dev tree contained entries for both the mirror and the journal devices, but when I rebooted the mirror did not show up -- even though geom_mirror.ko was loaded. What would cause this sort of mixup, and how do I fix it? Is any more info needed? contents of /dev/mirror before creating the journals Fixit# ls -la /dev/mirror total 1 dr-xr-xr-x 2 root 0 512 Nov 14 00:17 . dr-xr-xr-x 9 root 0 512 Nov 14 00:11 .. crw-r- 1 root operator0, 80 Nov 14 00:36 gm0 crw-r- 1 root operator0, 126 Nov 14 00:36 gm0a crw-r- 1 root operator0, 128 Nov 14 00:36 gm0d crw-r- 1 root operator0, 129 Nov 14 00:36 gm0e corresponding disklabel report Fixit# disklabel /dev/mirror/gm0 # /dev/mirror/gm0: 8 partitions: #size offsetfstype [fsize bsize bps/cpg] a: 8388608 164.2BSD 1024 819216 c: 6199075170unused0 0 d: 25165824 83886244.2BSD0 0 0 e: 586353069 335544484.2BSD0 0 0 journal creation, with resulting dmesg reports Fixit# gjournal label -s 2G /dev/mirror/gm0a GEOM_JOURNAL: Journal 1098378706: mirror/gm0a contains data. GEOM_JOURNAL: Journal 1098378706: mirror/gm0a contains journal. GEOM_JOURNAL: Journal mirror/gm0a clean. Fixit# gjournal label -s 2G /dev/mirror/gm0d GEOM_JOURNAL: Journal 3795372090: mirror/gm0d contains data. GEOM_JOURNAL: Journal 3795372090: mirror/gm0d contains journal. GEOM_JOURNAL: Journal mirror/gm0d clean. Fixit# gjournal label -s 2G /dev/mirror/gm0e GEOM_JOURNAL: Journal 2063379813: mirror/gm0e contains data. GEOM_JOURNAL: Journal 2063379813: mirror/gm0e contains journal. GEOM_JOURNAL: Journal mirror/gm0e clean. contents of /dev/mirror after creating the journals Fixit# ls -la /dev/mirror total 1 dr-xr-xr-x 2 root 0 512 Nov 14 00:17 ./ dr-xr-xr-x 9 root 0 512 Nov 14 00:11 ../ crw-r- 1 root operator0, 80 Nov 14 02:01 gm0 crw-r- 1 root operator0, 78 Nov 14 02:06 gm0a crw-r- 1 root operator0, 126 Nov 14 02:06 gm0a.journal crw-r- 1 root operator0, 125 Nov 14 02:07 gm0d crw-r- 1 root operator0, 128 Nov 14 02:07 gm0d.journal crw-r- 1 root operator0, 130 Nov 14 02:07 gm0e crw-r- 1 root operator0, 129 Nov 14 02:07 gm0e.journal newfs commands Fixit# newfs -J /dev/mirror/gm0a.journal Fixit# newfs -J /dev/mirror/gm0d.journal Fixit# newfs -J /dev/mirror/gm0e.journal mount the resulting filesystems (and one ordinary partition, neither mirrored nor journalled, to be used as /tmp -- I figure /tmp is expendable), resulting in this FS configuration Fixit# mount /dev/md0 on / (ufs, local) devfs on /dev (devfs, local, multilabel) /dev/da1a on /dist (ufs, local, read-only) /dev/mirror/gm0a.journal on /mnt (ufs, local, gjournal) /dev/ad8s2d on /mnt/tmp (ufs, local) /dev/mirror/gm0d.journal on /mnt/var (ufs, local, gjournal) /dev/mirror/gm0e.journal on /mnt/usr (ufs, local, gjournal) install per http://wiki.freebsd.org/RootOnZFS/GPTZFSBoot/Mirror, no detailed log kept manually-created config files, while still in chroot after install Fixit# cat /boot/loader.conf geom_mirror_load=YES geom_journal_load=YES vfs.root.mountfrom=ufs:/dev/mirror/gm0a.journal vfs.root.mountfrom.options=rw Fixit# cat /etc/fstab /dev/mirror/gm0a.journal / ufs rw 0 1 /dev/ad0s2b none swapsw 0 0 /dev/ad8s2b none swapsw 0 0 /dev/ad8s2d /tmp ufs rw 0 2 /dev/mirror/gm0d.journal /var ufs rw 0 3 /dev/mirror/gm0e.journal /usr ufs rw 0 4 /dev/da1a/dist ufs ro 0 0 devfs/dev devfs multilabel 0 0 output from kldstat, after booting the newly-installed system -- and manually mounting the root FS -- showing that geom_mirror.ko did get loaded. Id Refs AddressSize Name 16 0xc040 bb5504 kernel 21 0xc0fb6000 14540geom_journal.ko 31 0xc0fcb000 16ed4geom_mirror.ko dmesg
partitioning a gmirror (was Re: sysinstall vs gmirror)
binE6c8fkIE6U.bin Description: Binary data ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: sysinstall vs gmirror
Adam Vande More amvandem...@gmail.com wrote: On Fri, Sep 17, 2010 at 11:09 PM, per...@pluto.rain.com wrote: Next fdisk/gpart accordingly (don't forget to make it bootable). This is where I get stuck. I've partitioned the physical drives using sysinstall, but how do I go about partitioning gm0? Your problem is that you are still using sysinstall. No, I'm not. You can't for your purposes(this was pointed out earlier). Fixit only! The question is, how do I go about partitioning gm0 from Fixit? I've seen nothing so far that describes how to go about creating multiple partitions on a gmirror (or on anything else, for that matter) without either using sysinstall or having to understand gpart. Notice in the example it creates some basic filesystems/diretories using gpart and ZFS ... If your setup if GPT compatible, I recommend using it. How do I find out whether this setup is GPT compatible? Hardware(BIOS) dependent. OK, given the system's age I will presume that it is not, thus (I suppose) no reason to deal with gpart. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
