Re: [gentoo-user] Re: Strategy for using SAN/NAS for storage with Gentoo...
Am 17.03.2010 22:00, schrieb Neil Bothwick: On Wed, 17 Mar 2010 21:44:34 +0100, Florian Philipp wrote: Just for clarification: Is it really necessary to unplug the broken disk for this to work? If read access fails on sda and the BIOS tries sdb, would this also work? Isn't grub's hd0 always the disk on which grub resides (e.g. the disk from which grub managed to boot)? I suspect that may be dependent on the nature of the failure. For example, if /boot is corrupted, the BIOS will still boot from the broken disk's MBR before failing later. Most BIOSes now enable you to disable individual SATA ports, so you could disappear the disk without unplugging it, although I'm not sure why you'd want to leave a broken disk in the box. Just in case I ever face high demands on uptime. It's good to know whether I can still (remote) reboot a machine and it will come up although one of its drives is broken. signature.asc Description: OpenPGP digital signature
Re: [gentoo-user] Re: Strategy for using SAN/NAS for storage with Gentoo...
On Mon, 2010-03-15 at 09:37 -0500, Harry Putnam wrote: There was talk of opensolaris going by the wayside with the Oracle takeover of Sun... but Oracle has since announced its intention of puttin even more resources into `opensolaris' development than Sun was doing. that will kill it for sure! (ok, maybe not, but you know the mythical man month...) -- Iain Buchanan iaindb at netspace dot net dot au Life is like a sewer. What you get out of it depends on what you put into it. -- Tom Lehrer
Re: [gentoo-user] Re: Strategy for using SAN/NAS for storage with Gentoo...
Am 16.03.2010 22:26, schrieb Neil Bothwick: On Tue, 16 Mar 2010 20:13:29 +, Stroller wrote: How does your system boot if your RAID1 system volume fails? You put GRUB on both disks, then you can boot from either on its own. Is this reliable? I don't contest it, I'm just asking. It's just this was one of my considerations when choosing hardware RAID. Yes it is, if sda fails unplug it and sdb becomes sda (or hd1 becomes hd0 in GRUB terms) and the boot continues. Because RAID1 puts the RAID superblock in a different location from the ordinary one, you can use either disk from a RAID1 array as a single disk. Just for clarification: Is it really necessary to unplug the broken disk for this to work? If read access fails on sda and the BIOS tries sdb, would this also work? Isn't grub's hd0 always the disk on which grub resides (e.g. the disk from which grub managed to boot)? Thanks in advance! Florian Philipp signature.asc Description: OpenPGP digital signature
Re: [gentoo-user] Re: Strategy for using SAN/NAS for storage with Gentoo...
On Wed, 17 Mar 2010 21:44:34 +0100, Florian Philipp wrote: Just for clarification: Is it really necessary to unplug the broken disk for this to work? If read access fails on sda and the BIOS tries sdb, would this also work? Isn't grub's hd0 always the disk on which grub resides (e.g. the disk from which grub managed to boot)? I suspect that may be dependent on the nature of the failure. For example, if /boot is corrupted, the BIOS will still boot from the broken disk's MBR before failing later. Most BIOSes now enable you to disable individual SATA ports, so you could disappear the disk without unplugging it, although I'm not sure why you'd want to leave a broken disk in the box. -- Neil Bothwick This is the day for firm decisions! Or is it? signature.asc Description: PGP signature
Re: [gentoo-user] Re: Strategy for using SAN/NAS for storage with Gentoo...
On 15/03/2010 22:29, Andrea Conti wrote: This IMHO pretty much rules out any kind of server-class hardware, which tends to be both costly and power-hungry. If you're thinking about buying used stuff, be sure to factor in the cost and difficulty of finding spares in some years' time. I'm considering neither used equipment nor 'server-class' - the workload simply doesn't demand it. Given the point above I would also stick with software RAID. ... If reliability is your primary concern, I would go for a simple RAID1 setup; Absolutely. Software raid is cheaper and implies less hardware to fail. Similarly, RAID1 minimises the total number of disks required to survive a failure. It's the only way for me to go. If you do not need data sharing (i.e. if your volumes are only mounted by one client at a time), the simplest solution is to completely avoid having a FS on the storage server side -- just export the raw block device via iSCSI, and do everything on the client. This idea is on my wavelength. Has anyone on this tried this? My concerns are: 1. Are there reliability issues surrounding this technology in Gentoo? 2. Are there any howtos about putting as much of the file-system as possible onto an iSCSI device. 3. What's the best (most lightweight) way to expose the disk as a block device. I don't want to manage three fully-fledged Linux boxes. Can (cheap) NAS devices be used to export iSCSI to Gentoo? 4. What would be the strategy to 'secure' this iSCSI device... it would be a disaster if my WiFi were cracked and my data corrupted from a non-authorised host. In my experience this also works very well with Windows clients using the free MS iSCSI initiator. That's fantastic - I had no idea that such software existed. Now, I wonder, what's the most lightweight solution to get a couple of iSCSI devices? Does it help that MS supports attaching devices this way? File systems: avoid complexity. As technically superior as it might be, in this kind of setup ZFS is only going to be resource hog and a maintenance headache; your priority should be having a rock-solid implementation and a reliable set of diagnostic/repair tools in case disaster strikes. Yes. Separate arguments for snapshot support are compelling... but there are alternatives without tackling the additional complexity. That said, the iSCSI approach would work as well with ZFS as something mundane. Snap-shots, of course, are only really valuable for non-archive data... so, in future, I could add a ZFS volume using the same iSCSI strategy.
Re: [gentoo-user] Re: Strategy for using SAN/NAS for storage with Gentoo...
On 16 Mar 2010, at 16:32, Steve wrote: ... Given the point above I would also stick with software RAID. ... If reliability is your primary concern, I would go for a simple RAID1 setup; Absolutely. Software raid is cheaper and implies less hardware to fail. Similarly, RAID1 minimises the total number of disks required to survive a failure. It's the only way for me to go. How does your system boot if your RAID1 system volume fails? The one you have grub on? I think you mentioned a flash drive, which I've seen mentioned before. This seems sound, but just to point out that's another, different, single point of failure. If you do not need data sharing (i.e. if your volumes are only mounted by one client at a time), the simplest solution is to completely avoid having a FS on the storage server side -- just export the raw block device via iSCSI, and do everything on the client. ... Snap-shots, of course, are only really valuable for non-archive data... so, in future, I could add a ZFS volume using the same iSCSI strategy. I have wondered if it might be possible to create a large file (`dd if=/dev/zero of=/path/to/large/file` constrain at a size of 20gig or 100gig or whatever) and treat it as a loopback device for stuff like this. It's not true snapshotting (in the ZFS / BTFS sense), but you can unmount it and make a copy quite quickly. Stroller.
Re: [gentoo-user] Re: Strategy for using SAN/NAS for storage with Gentoo...
On Tue, 16 Mar 2010 19:57:49 +, Stroller wrote: How does your system boot if your RAID1 system volume fails? You put GRUB on both disks, then you can boot from either on its own. -- Neil Bothwick Growing old is mandatory; growing up is optional!! signature.asc Description: PGP signature
Re: [gentoo-user] Re: Strategy for using SAN/NAS for storage with Gentoo...
On 16 Mar 2010, at 20:04, Neil Bothwick wrote: On Tue, 16 Mar 2010 19:57:49 +, Stroller wrote: How does your system boot if your RAID1 system volume fails? You put GRUB on both disks, then you can boot from either on its own. Is this reliable? I don't contest it, I'm just asking. It's just this was one of my considerations when choosing hardware RAID. Stroller.
Re: [gentoo-user] Re: Strategy for using SAN/NAS for storage with Gentoo...
On Tuesday 16 March 2010 21:13:29 Stroller wrote: On 16 Mar 2010, at 20:04, Neil Bothwick wrote: On Tue, 16 Mar 2010 19:57:49 +, Stroller wrote: How does your system boot if your RAID1 system volume fails? You put GRUB on both disks, then you can boot from either on its own. Is this reliable? I don't contest it, I'm just asking. It's just this was one of my considerations when choosing hardware RAID. Stroller. This is the generally recommended method and I found this method in the Gentoo documentation. If this wouldn't be reliable, I would have expected this not to be in the docs for long. I have /boot mirrored, not actually tested with removing a disk yet, but I am confident it will work. If not, I have other methods of booting the system and getting to the data. -- Joost
Re: [gentoo-user] Re: Strategy for using SAN/NAS for storage with Gentoo...
On 16/03/2010 19:57, Stroller wrote: How does your system boot if your RAID1 system volume fails? The one you have grub on? I think you mentioned a flash drive, which I've seen mentioned before. This seems sound, but just to point out that's another, different, single point of failure. Well, at the moment, I don't have a RAID system... A flash drive (USB key) seems a reasonable strategy - I could even have two containing identical data - so, if the first were to fail then the second would kick in - if not automatically - then after the duff flash-drive is removed. A neat side effect of this would be to eliminate a moving part on the server - making it quieter... and the drives themselves can be located at two physically remote places on my LAN. by one client at a time), the simplest solution is to completely avoid having a FS on the storage server side -- just export the raw block device via iSCSI, and do everything on the client. ... Snap-shots, of course, are only really valuable for non-archive data... so, in future, I could add a ZFS volume using the same iSCSI strategy. If you do not need data sharing (i.e. if your volumes are only mounted Yes - I don't think I'd need sharing. It strikes me that it should be possible to have a 'live' backup server which just reads until fail-over... with a different /var/* - of course. I have wondered if it might be possible to create a large file (`dd if=/dev/zero of=/path/to/large/file` constrain at a size of 20gig or 100gig or whatever) and treat it as a loopback device for stuff like this. It's not true snapshotting (in the ZFS / BTFS sense), but you can unmount it and make a copy quite quickly. You could, but the advantage of ZFS is the efficiency of snap-shots. With your strategy I'd need to process all of the large file every time I want to make a snapshot... which, even for a mere 100gig, won't be quick.
Re: [gentoo-user] Re: Strategy for using SAN/NAS for storage with Gentoo...
On Tue, 16 Mar 2010 20:13:29 +, Stroller wrote: How does your system boot if your RAID1 system volume fails? You put GRUB on both disks, then you can boot from either on its own. Is this reliable? I don't contest it, I'm just asking. It's just this was one of my considerations when choosing hardware RAID. Yes it is, if sda fails unplug it and sdb becomes sda (or hd1 becomes hd0 in GRUB terms) and the boot continues. Because RAID1 puts the RAID superblock in a different location from the ordinary one, you can use either disk from a RAID1 array as a single disk. -- Neil Bothwick Minds are like parachutes; they only function when fully open. * Sir James Dewar signature.asc Description: PGP signature
[gentoo-user] Re: Strategy for using SAN/NAS for storage with Gentoo...
Steve gentoo_...@shic.co.uk writes: I have recently started looking at server resilience and availability in the context of a hardware failure or hardware upgrade. I've come to the conclusion that it would be very desirable if terrabyte-scale data did not need to be restored from backup. This isn't a commercial server - so I'm interested in minimum cost approaches. With this in mind, I'm interested to discover what represents state-of-the-art from the perspective of the OS and its configuration. Issues I envisage are: * With NAS, it would be desirable to have a Linux filesystem rather than access files over CIFS - this raises further questions about protocol... is NFS as hopelessly outdated as it seems? Are there any products that offer NFS access? Are any of them secure? * With a SAN, questions of filesystem features are diminished - but questions of access protocol remain. What is best supported by gentoo? * Do any gentooists have any inexpensive hardware configurations that work especially well? Any hints or tips? Someone here, a yr or two ago recommended to me when I asked that question to install opensolaris on a machine and set it up as NAS. Opensolaris offers the zfs file system, that is really advanced compared to others. I'll admit I've had some issues along the way. And have to do lots of boning up on opensolaris. I access the zfs server by cifs from windows machines, and by NFS from linux. Opensolaris doesn't use samba by default. Instead they have their own CIFS server which works fine. They do have samba pkgs but no one hardly use it, preferring their CIFS server. Mine is only a home lan setup, but even then the opensolaris server has over a terabyte of capacity. I have it setup in 3 mirrors of 2 disks each. 2 prs of 500gb sata hdd and one pr of 750 sata hdd There are many ways so setup `zraid' systems on `zfs' that are excellent for reliability. ( I have been told I haven't tried that route). For me the mirror setup seemed good for my small needs. Even more reliable I'm told, if a bit higher in disk usage. opensolaris zfs fs offers a `timeslider' interface to a system of snapshotting the filesystems in 15 min, 1hr 1day etc snapshots in a very small footprint way. You'd have to read up on it.. It would take too much off topic to cover here. The system takes an amazing small amount of disk space for the snapshots based on COW (Copy on write). There was talk of opensolaris going by the wayside with the Oracle takeover of Sun... but Oracle has since announced its intention of puttin even more resources into `opensolaris' development than Sun was doing. These may be a good intro to zfs: http://www.sun.com/bigadmin/features/articles/zfs_part1.scalable.jsp http://all-unix.blogspot.com/2007/03/zfs-cow-and-relate-features.html http://wikis.sun.com/display/OpenSolarisInfoPL/How+to+Manage+the+Automatic+ZFS+Snapshot+Service
Re: [gentoo-user] Re: Strategy for using SAN/NAS for storage with Gentoo...
+1 on zfs w/ solaris for storage, just don't go cheap and get desktop disks. -- Kyle
Re: [gentoo-user] Re: Strategy for using SAN/NAS for storage with Gentoo...
On 15/03/2010 15:49, Kyle Bader wrote: +1 on zfs w/ solaris for storage, just don't go cheap and get desktop disks. I have to admit, I do like the idea of ZFS, though not quite enough to justify maintaining Solaris in addition to my other infrastructure. I was thinking about something rather different entirely. I was thinking about bunging disk on my LAN and shifting as much data from local storage on my server as possible. This would mean that the server could be swapped out with minimum effort. If 'disk on the net' allowed mirroring etc. then storage could be expanded and contracted as necessary without any downtime... essentially, only my hub would then be a single-point-of-failure. I'd love to be able to run a VM on my desktop, for example, and use that as a 'stand-in' while I take-down my main server for maintenance. For this to work, I'd need to access the same file system and be able to switch responsibility for services between the two 'servers' quickly. From ages ago, I remember iSCSI being bandied about. Did that ever go anywhere (i.e. is this easy to do from Gentoo?)
Re: [gentoo-user] Re: Strategy for using SAN/NAS for storage with Gentoo...
On 15 Mar 2010, at 16:26, Steve wrote: ... From ages ago, I remember iSCSI being bandied about. Did that ever go anywhere (i.e. is this easy to do from Gentoo?) I believe it is quite widely used - it is mentioned often on the linux- poweredge list. I would imagine the Linux kernel allows mounting and sharing by iSCSI - check `make menuconfig` and type /iscsi. It's hard to be more specific without knowing your usage. For storage of a mere terabyte you can buy a networked storage enclosure which will accommodate two drives. These are cheap, do mirroring, will accommodate standard 1TB, 1.5TB, 2TB drives, but are probably not too fast. One reads a lot posted by people who have large movie collections stored on the network, whether they be MythTV users or the mutineer sailors of 17th century galleons. A PC-based solution gives you more room for this - you can fit perhaps 4 drives in a standard PC case you find at the tip, or you can get 12 or 16 drives in a dedicated rackmount server case. This allows capacity of upto 32TB with current drives, if you can afford that, or to use cheaper drives (1TB or 1.5TB are best gigabytes-per-dollar at present, I think; 500gb drives seem recently to have become disproportionately expensive) and have better RAID levels. The Norco one is popular amongst enthusiasts, because it's really cheap [1]; it uses 2 x standard ATX power supplies, one for the mainboard, one for the drives. You can get similar cases with the option of hot-swap PSUs - Chenbro used to be the main brand for this, I think, but in the last couple of years TST http://TSTcom.com have started producing nicer cases; I use a TST ESR-316, which is utterly lush, but which was expensive. I have one slight reservation about the TST, which I will not spend time detailing unless you ask. I use only half the TST's capacity at present, but it is a pleasure and a relief to have so much room available; expansion of network drive capacity is never a problem - just slap a drive in and you're ready to go. Even with as many as 6 or 8 drive bays there are corner cases which can make expansion a bit of a headache (at least if uptime is important). Since these cases accommodate standard ATX motherboards, you get to use an old Pentium 4 motherboard salvaged from an old PC or an Atom- based motherboard for £100 or so. The latter price is a bit shocking, IMO, compared to (say) the Asus EE-PC, but it reflects the demand for them; they're prolly only $100 in the US. These atom motherboards have minimal expansion slots, but if you only want to use it for storage then you're probably fine with just one. If you build your own server you can use software or hardware RAID. Fast hardware RAID, based on an PCIe controller card, is expensive. You can get PCI or PCI-X hardware RAID very cheaply on eBay these days, but it's slow. That is to say that PCI or PCI-X hardware RAID is fast enough to stream a couple of movies at the same time, fast enough to copy 5gb files only a couple of minutes, but production server systems (if you were buying a database server for work) would be expected to use a PCIe-based hard-drive controller. Hardware RAID is nice in its ability to hot-swap out a failed hard-drive without interruption. I have not found non-RAID SATA controllers that satisfy me with their ability to do hot-swap (although I would love to). Managing RAID on a PC-based server - rather than a dedicated NAS enclosure - very easily allows expansion. With RAID5 or 6 you can just add in another drive and expand on to it. I use an old PCI-X (fits in a PCI slot) 3ware 9500 card, and it *seems* like if you have a RAID1 (haven't tried RAID5) on two drives of capacity X, then remove each of those drives in turn, rebuilding onto drives of X+Y capacity, then upon completion the array appears to the o/s as the larger X+Y size. I think some LSI cards do this, also. I would not bet on the ability of low-end NAS boxes to do this. A company called Drobo makes some high-end NAS hardware with space for plenty of drives (on some models) and some fancy features. I find UK prices a bit shocking, but depending upon your application they might be justified; the US prices seem quite reasonable to me. I wouldn't get too het up about Samba / CIFS vs NFS. Samba / CIFS can be faster than NFS, even in an all-Linux environment. Other times it's not. This seems pretty much random, depending upon whom is doing the benchmarking. On an intellectual level, at least, I find neither wholly satisfying - it would be really nice to have a Linux-native network filesystem that does authentication / permissions properly. But both do work. I looked at ZFS, but decided that Solaris, from a look at the HCL, was too picky over hardware. I think ZFS is great, I no longer think it's the future. My selection of cheap hardware is far wider under Linux, I
Re: [gentoo-user] Re: Strategy for using SAN/NAS for storage with Gentoo...
On 15/03/2010 18:21, Stroller wrote: It's hard to be more specific without knowing your usage. Yes... I was deliberately vague to see what options came up... but I can be more specific. The budget is miniscule - and the performance demands (bandwidth and latency) are completely non-challenging. It's in this context that I'm looking for reliability and availability... and I'd like to have unix permissions working properly. Security is a moderate concern - the physical network is secured - but there is a broadband connection which exposes various services. For storage of a mere terabyte you can buy a networked storage enclosure which will accommodate two drives. These are cheap, do mirroring, will accommodate standard 1TB, 1.5TB, 2TB drives, but are probably not too fast. A cheap NAS enclosure is a definite possibility - there'd be no performance issue - though this leaves three key questions: 1) Will it support unix file-permissions and can I be (fairly sure) it will be secure if someone hacks my Wi-Fi? 2) Will I be able to put the (majority of the) gentoo filesystem on it - or will I need to have a fully booted system to connect? 3) Can I use two entirely separate devices and mirror to both? (I expect the failure of the enclosure to be at least as likely as the failure of a drive.) If you build your own server you can use software or hardware RAID. Hmmm... building my own server - I've done that in the past, but my plan is to minimize DIY with a view to minimizing the number of components that might fail. Ideally, I'd have four devices - one with a CPU and memory (the server)... booting from Flash or CD or whatever (+a replacement in the cupboard); two separate boxes with drives in them (mirrored storage); one (wired) Ethernet hub and broadband gateway. I'd connect to the network from a separate desktop/laptop to interact with it - either locally or remotely. I wouldn't get too het up about Samba / CIFS vs NFS. Samba / CIFS can be faster than NFS, even in an all-Linux environment. Other times it's not. This seems pretty much random, depending upon whom is doing the benchmarking. On an intellectual level, at least, I find neither wholly satisfying - it would be really nice to have a Linux-native network filesystem that does authentication / permissions properly. But both do work. Well the 'server' will be running Samba - and it's the back-end storage for that I'm trying to resolve. CIFS definitely looks problematic - since Unix permissions for server data are one valuable separation between publicly accessible services and my private data. NFS might be OK (it doesn't feel great) - though I *really* don't want to move from one server to two when I'm aiming for reliability. I looked at ZFS, but decided that Solaris, from a look at the HCL, was too picky over hardware. I think ZFS is great, I no longer think it's the future. My selection of cheap hardware is far wider under Linux, I can install Gentoo and just `emerge mediatomb` and stream movies to my PS3. I like ZFS, conceptually, though I don't like Solaris. I'm aware that Apple have toyed with adopting ZFS and that it is available for BSD... A *really* neat solution would be a (pair of) cheap NAS devices running an appliance distribution of BSD with ZFS - exporting a NFS mount... possibly over a VPN? Hmmm - I'm trying to avoid complexity, too. Hmmm.
Re: [gentoo-user] Re: Strategy for using SAN/NAS for storage with Gentoo...
Hi, The budget is miniscule - and the performance demands (bandwidth and latency) are completely non-challenging. This IMHO pretty much rules out any kind of server-class hardware, which tends to be both costly and power-hungry. If you're thinking about buying used stuff, be sure to factor in the cost and difficulty of finding spares in some years' time. Given the point above I would also stick with software RAID. True HW RAID controllers are quite expensive and generally come with a x8 PCIe interface which will require a server motherboard -- x16 PCIe video card slots in commodity boards are usually only certified for x16 and x1 operation, so don't expect them to work reliably with other bus widths. Linux software RAID also has the advantage that the kernel is not tied to any specific piece of hardware. In case of a failure, your volumes will be readable on any other Linux system -- provided the disks themselves are not toast. If reliability is your primary concern, I would go for a simple RAID1 setup; if your volumes need to be bigger than a physical disk you can build a spanned volume over multiple mirrored pairs. Network throughput will mostly likely be your primary bottleneck, so I'd avoid striping as it would offer little in the way of performace at the expense of making data recovery extremely difficult in case the worst should happen. As for availability, I think the best strategy with a limited budget is to focus on reducing downtime: make sure your data can survive the failure of any single component, and choose hardware that you can get easily and for a reasonable price. Sh*t happens, so make it painless to clean up. Network protocol: If you do not need data sharing (i.e. if your volumes are only mounted by one client at a time), the simplest solution is to completely avoid having a FS on the storage server side -- just export the raw block device via iSCSI, and do everything on the client. In my experience this also works very well with Windows clients using the free MS iSCSI initiator. Alternatively, you can consider good old NFS, which performs decently and tends to behave a bit better -- especially when used over UDP -- in case of network glitches, like accidentally powering off a switch, yanking cables, losing wireless connectivity... CIFS should be avoided at all costs if your clients are not Windows machines. File systems: avoid complexity. As technically superior as it might be, in this kind of setup ZFS is only going to be resource hog and a maintenance headache; your priority should be having a rock-solid implementation and a reliable set of diagnostic/repair tools in case disaster strikes. Tried-and-true ext3 fits the bill nicely if you ask me; just remember to tune it properly according to your planned use -- eg. if a volume is going to be used to host huge VM disk images, be sure to create its filesystem with -T largefile4. Just my 2 cents, Andrea
