Re: [CentOS] Upcoming OwnCloud changes
Just a FYI folks ... I am running OwnCloud 9.0.2 on CentOS 6.7 and php-7.0 with no issues. I installed the Webtatic repo which has several versions of PHP available for CentOS 6 and 7. I then used the official OwnCloud ce:stable repo to add the cloud software. In a leap of faith, and because this CentOS VM doesn't run anything other than OwnCloud, I used the 'php70w' PHP repo and have had no problems at all. The OwnCloud server is very lightly loaded so I have not had an opportunity to stress test it and have not tried to use MySQL (I configured it with the standard CentOS SQLite). YMMV of course, especially if you're running other PHP applications! Chuck ___ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
[CentOS] Native ZFS on Linux
I have a question that has been puzzling me for some time ... what is the reason RedHat chose to go with btrfs rather than working with the ZFS-on-Linux folks (now OpenZFS)? Is it a licensing issue, political, etc? Although btrfs is making progress, ZFS is far more mature, has a few more stable features (especially Raid-z3) and has worked flawlessly for me on CentOS-6 and Scientific Linux-6. I used ZFS for a couple of large file servers with no problems. The only annoying part was having to manually install the kernel module each time a kernel update was issued. Because FreeBSD has built-in drivers for LSI cards (I use JBOD mode) and strong ZFS support, I've switched to FreeNAS with it's default native ZFS and driver support for LSI. I would have preferred to stick with CentOS, but FreeNAS just made things easier to manage. It also seems about twice as fast as ZFS on Linux (same hardware), but that may simply be a tuning issue. Is there any chance that a native ZFS (rather than btrfs) will ever emerge in RHEL/CentOS? Just curious, because I'd love to return to CentOS/SL next time I build large file servers. As an aside, I have used only WD Black and WD RedPro drives for RAID, and not had any issues. Green drives are scary :-) Chuck ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] ZFS
On 2014-09-15 , kkel...@wombat.san-francisco.ca.us wrote: So the ZoL folks want one more feature before calling it 1.0; otherwise they believe it's production ready. Only your own testing can convince you that it's truly production ready. --keith That's encouraging news, something I've been looking forward to. FWIW, I've been running ZoL Raid-z3 on two large file servers for quite some time (0.6.2-1 currently, on CentOS-6 x64) with zero problems other than a couple of disk drive failures which ZFS handled nicely. I've had no real reason to upgrade and will wait for a 1.x release before doing anything major. Performance has been very good (although the application is not all that demanding) and stability has been excellent. Over the past year or so the routine twice-weekly scrubs have found no issues, and we've detected no file corruption. The scrubbing operation doesn't seem to affect concurrent use all that much. The servers run for months without reboots (only done for security updates). I don't miss those hours-long ext4 fsck's at all :-) Chuck ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] ZFS on Linux testing
On 12/19/2013, 04:00 , li...@benjamindsmith.com wrote: BackupPC is a great product, and if I knew of it and/or it was available when I started, I would likely have used it instead of cutting code. Now that we've got BackupBuddy working and integrated, we aren't going to be switching as it has worked wonderfully for a decade with very few issues and little oversight. I would differentiate BackupBuddy in that there is no incremental and full distinction. All backups are full in the truest sense of the word, and all backups are stored as native files on the backup server. This works using rsync's hard-link option to minimize wasted disk space. This means that the recovery process is just copying the files you need. Also, graceful recovery for downtime and optimistic disk space usage are both very nice. (it will try to keep as many backup savepoints as it can disk space depending) I'm evaluating ZFS and will likely include some features of ZFS into BBuddy as we integrate these capabilities into our backup processes. We're free to do this in part because we have redundant backup sets, so a single failure wouldn't be catastrophic in the short/medium term. -Ben I would agree that BackupPC is a great tool, stable and well respected, and I have to admit I didn't give it a fair chance before moving on to a simple, crude homebrew solution many years ago. One thing I neglected to mention about using inotify+lsync for a near-real-time secondary file server is that if the primary server croaks, I can quickly switch users to the secondary. When the primary server is repaired, a one-shot rsync to it brings it back to the current state of users' files. This conflicts a bit with non-deletion of old files on the secondary, but it keeps users going while repairs are made. I haven't had to use this trick ... yet. ZFS Raid-z3 and disk hot swap have done their jobs well so far. I'm toying with the idea of keeping a marker file on the secondary whose timestamp can be used to limit rsync back to the primary to only those files which got added while the secondary was being used. That would take care of the non-deleted files issue on the secondary. Fortunately there are no database files involved in any of this. Chuck ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] ZFS on Linux testing
On 12/18/2013, 04:00 , li...@benjamindsmith.com wrote: I may be being presumptuous, and if so, I apologize in advance... It sounds to me like you might consider a disk-to-disk backup solution. I could suggest dirvish, BackupPC, or our own home-rolled rsync-based solution that works rather well:http://www.effortlessis.com/backupbuddy/ Note that with these solutions you get multiple save points that are deduplicated with hardlinks so you can (usually) keep dozens of save points in perhaps 2x the disk space of a single copy. Also, because of this, you can go back a few days / weeks / whatever when somebody deletes a file. In our case, we make the backed up directories available via read-only ftp so that end users can recover their files. I don't know if dirvish offers this, but backupbuddy also allows you to run pre and post backup shell scripts, which we use (for example) for off-site archiving to permanent storage since backup save points expire. -Ben Not presumptuous at all! I have not heard of backupbuddy (or dirvish), so I should investigate. Your description makes it sound somewhat like OS-X Time Machine, which I like a lot. I did try backuppc but it got a bit complex to manage IMHO. Thanks for the tip! Chuck ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] ZFS on Linux testing
On 12/14/2013, 04:00 , li...@benjamindsmith.com wrote:
We checked lsyncd out and it's most certainly an very interesting tool.
I*will* be using it in the future!
However, we found that it has some issues scaling up to really big file
stores that we haven't seen (yet) with ZFS.
For example, the first thing it has to do when it comes online is a
fully rsync of the watched file area. This makes sense; you need to do
this to ensure integrity. But if you have a large file store, EG: many
millions of files and dozens of TB then this first step can take days,
even if the window of downtime is mere minutes due to a restart. Since
we're already at this stage now (and growing rapidly!) we've decided to
keep looking for something more elegant and ZFS appears to be almost an
exact match. We have not tested the stability of lsyncd managing the
many millions of inode write notifications in the meantime, but just
trying to satisfy the write needs for two smaller customers (out of
hundreds) with lsyncd led to crashes and the need to modify kernel
parameters.
As another example, lsyncd solves a (highly useful!) problem of
replication, which is a distinctly different problem than backups.
Replication is useful, for example as a read-only cache for remote
application access, or for disaster recovery with near-real-time
replication, but it's not a backup. If somebody deletes a file
accidentally, you can't go to the replicated host and expect it to be
there. And unless you are lsyncd'ing to a remote file system with it's
own snapshot capability, there isn't an easy way to version a backup
short of running rsync (again) on the target to create hard links or
something - itself a very slow, intensive process with very large
filesystems. (days)
I'll still be experimenting with lsyncd further to evaluate its real
usefulness and performance compared to ZFS and report results. As said
before, we'll know much more in another month or two once our next stage
of roll out is complete.
-Ben
Hi Ben,
Yes, the initial replication of a large filesystem is *very* time
consuming! But it makes sleeping at night much easier. I did have to
crank up the inotify kernel parameters by a significant amount.
I did the initial replication using rsync directly, rather than asking
lsyncd to do it. I notice that if I reboot the primary server, it takes
a while for the inotify tables to be rebuilt ... after that it's smooth
sailing.
If you want to prevent deletion of files from your replicated filesystem
(which I do), you can modify the rsync{} array in the lsyncd.lua file by
adding the line 'delete = false' to it. This has saved my butt a few
times when a user has accidentally deleted a file on the primary server.
I agree that filesystem replication isn't really a backup, but for now
it's all I have available, but at least the replicated fs is on a
separate machine.
As a side note for anyone using a file server for hosting OS-X Time
Machine backups, the 'delete' parameter in rsync{} must be set to 'true'
in order to prevent chaos should a user need to point their Mac at the
replicate filesystem (which should be a very rare event). I put all TM
backups in a separate ZFS sub-pool for this reason.
Chuck
___
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] ZFS on Linux testing effort
On 04.12.2013 14:05, n...@li.nux.ro wrote: On 04.12.2013 14:05, John Doe wrote: From: Listsli...@benjamindsmith.com Our next big test is to try out ZFS filesystem send/receive in lieu of our current backup processes based on rsync. Rsync is a fabulous tool, but is beginning to show performance/scalability issues dealing with the many millions of files being backed up, and we're hoping that ZFS filesystem replication solves this. Not sure if I already mentioned it but maybe have a look at: ?http://code.google.com/p/lsyncd/ I'm not so sure inotify works well with millions of files, not to mention it uses rsync. :D -- Sent from the Delta quadrant using Borg technology! Nux! I can attest to the usefulness of 'lsyncd' for large numbers of files (our file server has almost 2 million in active use, with a second backup server that's lsync'd to the first. Things to note: - Yes, lsyncd does use rsync, but it issues an 'exclude *' followed by the list of only the file(s) that need updating at that moment. - The inotify service can be jacked wy up (three kernel parameters) to handle millions of files if you wish. Just make sure you have lots of RAM. Be careful with it. Sadly I found out that inotify would consistently fail on InnoDB files (ibd); I had to use stupid while loops and check mtimes to perform some stuff that inotify-cron would've done much more elegantly ... -- Sent from the Delta quadrant using Borg technology! Nux! Interesting point, something I didn't know. Fortunately in my case there are no db files involved directly, just db dumps wrapped in a tarball along with other associated stuff, sent from other servers. I would expect that lsync'ing db files could be a nasty non-stop process if the database is constantly being updated, so using db tools for replication would be best, configuring inotify/lsyncd to ignore the db directories. I believe by default that lsyncd instructs rsync to do whole-file transfers, so a large db could be a real problem. Thanks for the important heads-up! ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] ZFS on Linux testing effort
On 04.12.2013 14:05, John Doe wrote: From: Listsli...@benjamindsmith.com Our next big test is to try out ZFS filesystem send/receive in lieu of our current backup processes based on rsync. Rsync is a fabulous tool, but is beginning to show performance/scalability issues dealing with the many millions of files being backed up, and we're hoping that ZFS filesystem replication solves this. Not sure if I already mentioned it but maybe have a look at: ?http://code.google.com/p/lsyncd/ I'm not so sure inotify works well with millions of files, not to mention it uses rsync. :D -- Sent from the Delta quadrant using Borg technology! Nux! I can attest to the usefulness of 'lsyncd' for large numbers of files (our file server has almost 2 million in active use, with a second backup server that's lsync'd to the first. Things to note: - Yes, lsyncd does use rsync, but it issues an 'exclude *' followed by the list of only the file(s) that need updating at that moment. - The inotify service can be jacked wy up (three kernel parameters) to handle millions of files if you wish. Just make sure you have lots of RAM. It's wise to tune ZFS to *not* use all available RAM. - Updating is very quick and has never failed. Regarding ZFS, our two ZFS-on-Linux servers have been in full production for several months, with zero problems so far. Updates to the latest version is quite painless. Today I had to replace a failed 4TB drive ... it took just a few minutes and required only two commands to do the replacement and start the resilvering process. This was done while the server was in active use, with only a small performance hit. Sweet! Chuck ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] ZFS on Linux in production?
On 10/25/2013, 05:00 , centos-requ...@centos.org wrote: We are a CentOS shop, and have the lucky, fortunate problem of having ever-increasing amounts of data to manage. EXT3/4 becomes tough to manage when you start climbing, especially when you have to upgrade, so we're contemplating switching to ZFS. As of last spring, it appears that ZFS On Linuxhttp://zfsonlinux.org/ calls itself production ready despite a version number of 0.6.2, and being acknowledged as unstable on 32 bit systems. However, given the need to do backups, zfs send sounds like a godsend over rsync which is running into scaling problems of its own. (EG: Nightly backups are being threatened by the possibility of taking over 24 hours per backup) Was wondering if anybody here could weigh in with real-life experience? Performance/scalability? -Ben FWIW, I manage a small IT shop with a redundant pair of ZFS file servers running the zfsonlinux.org package on 64-bit ScientificLinux-6 platforms. CentOS-6 would work just as well. Installing it with yum couldn't be simpler, but configuring it takes a bit of reading and experimentation. I reserved a bit more than 1GByte of RAM for each TByte of disk. One machine (20 useable TBytes in raid-z3) is the SMB server for all of the clients, and the other machine (identically configured) sits in the background acting as a hot spare. Users tell me that performance is quite good. After about 2 months of testing, there have been no problems whatsoever, although I'll admit the servers do not operate under much stress. There is a cron job on each machine that does a scrub every Sunday. The old ext4 primary file servers have been shut down and the ZFS boxes put into production, although one of the old ext4 servers will remain rsync'd to the new machines for a few more months (just in case). The new servers have the zfsonlinux repositories configured for manual updates, but the two machines tend to be left alone unless there are important security updates or new features I need. To keep the two servers in sync I use 'lsyncd' which is essentially a front-end for rsync that cuts down thrashing and overhead dramatically by excluding the full filesystem scan and using inotify to figure out what to sync. This allows almost-real-time syncing of the backup machine. (BTW, you need to crank the resources for inotify way up for large filesystems with a couple million files.) So far, so good. I still have a *lot* to learn about ZFS and its feature set, but for now it's doing the job very nicely. I don't miss the long ext4 periodic fsck's one bit :-) YMMV, of course, Chuck ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Failing Network card
Date: Wed, 20 Jun 2012 10:54:33 -0700 From: John R Piercepie...@hogranch.com Subject: Re: [CentOS] Failing Network card To:centos@centos.org Message-ID:4fe20e59.20...@hogranch.com Content-Type: text/plain; charset=ISO-8859-1; format=flowed On 06/20/12 8:44 AM, Gregory P. Ennis wrote: 01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller (rev ff) pure unmitigated junk. -- john r pierce N 37, W 122 santa cruz ca mid-left coast I agree with John's comment. Realtek chips are junk with unpredictable reliability, especially under heavy load. I have had several problems with various versions of the 81xx chips. When I tossed the cards in the garbage and switched to Intel-based NICs, all the problems went away. Every time I build systems with Realtek network chips on the motherboard, I disable them in the BIOS and add Intel NICs instead. YMMV, but please consider ditching Realtek altogether. Chuck ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Software RAID1 with CentOS-6.2
On 03/01/2012 09:00 AM, Mark Roth wrote: Miguel Medalha wrote: A few months ago I had an enormous amount of grief trying to understand why a RAID array in a new server kept getting corrupted and suddenly changing configuration. After a lot of despair and head scratching it turned out to be the SATA cables. This was a rack server from Asus with a SATA backplane. The cables, made by Foxconn, came pre-installed. After I replaced the SATA cables with new ones, all problems were gone and the array is now rock solid. Thanks for this info, Miguel. snip As an additional information, I quote from the Caviar Black range datasheet: Desktop / Consumer RAID Environments - WD Caviar Black Hard Drives are tested and recommended for use in consumer-type RAID applications (RAID-0 /RAID-1). - Business Critical RAID Environments ? WD Caviar Black Hard Drives are not recommended for and are not warranted for use in RAID environments utilizing Enterprise HBAs and/or expanders and in multi-bay chassis, as they are not designed for, nor tested in, these specific types of RAID applications. For all Business Critical RAID applications, please consider WD?s Enterprise Hard Drives that are specifically designed with RAID-specific, time-limited error recovery (TLER), are tested extensively in 24x7 RAID applications, and include features like enhanced RAFF technology and thermal extended burn-in testing. Wonderful... NOT. We've got a number of Caviar Green, so I looked up its datasheet... and it says the same. That rebuild of my system at home? I think I'll look at commercial grade drives mark Interesting thread ... I have had problems with SATA cables in the past, and prefer those with the little metal latches. The problem is that you can't easily tell by looking at the connectors whether or not they're flakey. I've had positive experience with Caviar Black and Scorpio Black drives. The WD Green and Blue drives are built more cheaply than the Blacks (which have close to enterprise-grade construction). The dealer I buy drives from has told me that the Blacks have far lower return/defect rates. Of the approximately 30 2TB Blacks I have in RAID-6 service, I've only experienced two failures, which were handled quickly by the WD warranty program. It's interesting to note that while all the drive manufacturers are going back to 1 or 2-year warranties, the WD Black series remains at 5 years. A friend of mine has had a couple of strange problems with the RE (RAID) series of Caviars, which utilize the same mechanics as the non-RE Blacks. For software RAID, I would recommend that you stick with the non-RE versions because of differences in the firmware. It has come down to me buying *only* WD Black-series drives and nothing else. If I could afford them, I'd consider enterprise-grade drives. Having said that, I have a pair of 1TB Green drives in RAID-1 for the TimeMachine backups on my Mac, and they've been spinning 24x7 non-stop for 3 years without failure. I'm almost afraid to switch them off. Now, if WD can just get their post-flood production back in gear so prices can drop. My 2c, FWIW :-) Chuck ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Connecting ethX devices directly to a KVM/QEMU guest OS (SOLVED)
On 01/20/2012 09:00 AM, Tait Clarridge wrote: But a new problem arises ... one of the m0n0wall instances needs to use DHCP to get a dynamic IP from my ISP (the others are static, so they're ok). How do I get the bridge to proxy the DHCP request and forward the response to the VM guest? I brought up a test instance of m0n0wall but when it asks for an IP on its WAN side, the result is 0.0.0.0/8 so it would appear the request never makes it to the ISP. As a sanity check, I configured the WAN-side bridge to use DHCP and that worked, but it's the m0n0wall VM guest that needs to make the request, not the CentOS host. Suggestions anyone? In the mean time, I'll keep looking for a solution. Thanks, Chuck Hmmm. Is the bridge up and you can see the interface attached to it from running brctl show and ifconfig? Could you attach that output? If yes, then I would do a packet capture on the CentOS host on the eth interface attached to the bridge to see if the request makes it out. I'm running XEN at home and I have a very similar setup where I pass a VLAN from my switch that a provider modem is connected to into a bridge and it gets DHCP no problem (as long as the bridge and dot1q subinterfaces show up in ifconfig). The bridge should automatically send all the traffic out regardless of protocol, I doubt ebtables would be actively blocking anything as well and same goes for the ISP locking down with MAC addresses. I'm embarrassed to say the problem was solved in the most simplistic possible way ... reboot the cable modem. I notice that when the host boots up, all NIC ports are placed in promiscuous mode, which means that DHCP broadcasts should be free to travel across the bridge, so it had to be something about a mismatch in the cable modem's arp cache. Thanks for your suggestions. It's full steam ahead now, and I'm pleased to say the performance of the four m0n0wall VM instances has been very good. Chuck ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Connecting ethX devices directly to a KVM/QEMU guest OS (no bridging)
On 01/19/2012 09:00 AM, Tait Clarridge wrote: I've Googled until I'm blue in the face, but haven't found a clear explanation of how I can assign each ethX device directly to the guest OS rather than going thru the usual bridge configuration. I need to allow the m0n0wall OS to assign the LAN and WAN IP addresses, if that's possible without using a bridge. I would think that it is a little easier to setup a bridge but not assign an IP address for it on the CentOS host. eg. /etc/sysconfig/network-scripts/ifcfg-br0 DEVICE=br0 TYPE=Bridge BOOTPROTO=none ONBOOT=yes NM_CONTROLLED=no /etc/sysconfig/network-scripts/ifcfg-ethX (where X is the interface number of one of your intel ports and the HWADDR is the MAC address) DEVICE=ethX ONBOOT=yes BRIDGE=br0 HWADDR=xx:xx:xx:xx:xx:xx NM_CONTROLLED=no Restart the network service, then run brctl show - the new bridges should show up. Create the 8 bridges that you need and go from there, you should be able to assign them in Virtual Machine Manager to the VMs. Then in your VM guests you should be able to assign the IPs that you want. Thanks for your reply, Tait. I came to much the same conclusion after I eventually found an article about tinkering with the host kernel device tables to remap each NIC instance. I was going to try it until I discovered that the new motherboard I'm using doesn't properly support IOMMU in the BIOS, which the mapping trick requires. So much for that idea. Your method of not configuring a bridge IP address gets around having the NIC appear to my ISP as having some sort of rogue IP (even if I assign something in the 172.16... range). I wasn't aware that I could bring up the bridge without an IP. But a new problem arises ... one of the m0n0wall instances needs to use DHCP to get a dynamic IP from my ISP (the others are static, so they're ok). How do I get the bridge to proxy the DHCP request and forward the response to the VM guest? I brought up a test instance of m0n0wall but when it asks for an IP on its WAN side, the result is 0.0.0.0/8 so it would appear the request never makes it to the ISP. As a sanity check, I configured the WAN-side bridge to use DHCP and that worked, but it's the m0n0wall VM guest that needs to make the request, not the CentOS host. Suggestions anyone? In the mean time, I'll keep looking for a solution. Thanks, Chuck ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
[CentOS] Connecting ethX devices directly to a KVM/QEMU guest OS (no bridging)
Hello CentOS gurus, and a belated Happy New Year (or an early Happy Chinese New Year). I'm in the process of building a CentOS-6 KVM host machine whose only function will be to run four independent 'm0n0wall' firewall guest instances. I have a couple of quad-port Intel NICs to provide four WAN-side and four LAN-side ports, which will not be shared with any other guests. Remote access to the host CentOS is via a separate on-board NIC, so the Intel ports are free to assign. I've Googled until I'm blue in the face, but haven't found a clear explanation of how I can assign each ethX device directly to the guest OS rather than going thru the usual bridge configuration. I need to allow the m0n0wall OS to assign the LAN and WAN IP addresses, if that's possible without using a bridge. Using the Virtual Machine Manager GUI to create the VM guests (which boot up and run nicely), the drop-down list shows all of the ethX devices greyed out. Do I need to somehow start the NICs without assigning IP addresses, so that they show up in 'ifconfig'? Can anyone provide a clear, easy-to-understand procedure for doing this? If necessary, I don't mind using the command-line to create the VMs but the docs aren’t totally clear (to me, at least) regarding NIC assignment. Thanking you in advance for your suggestions, Chuck ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] EL 6 rollout strategies? (Scientific Linux)
On 05/07/2011 09:00 AM, Benjamin Smith wrote: I was wondering what feedback might be offered by the CentOS community on their experiences using Scientific Linux? I'm a long-time Centos user, and am basically happy with CentOS. I understand there are delays getting EL 6 out. We have been long anxious to roll out EL 6 as soon as it's ready, but our time window for rollout is looming and we will need to act. (for business reasons, we need to rollout over summer, before Aug 1, we need to start regression testing now!) I was once a WhiteBox Enterprise Linux user and switched to CentOS 4 without issue, and am assuming that I might need to do something similar if we decide to go this route. Any feedback is appreciated! I had to complete a new VM Host machine before CentOS-6 was ready, so I used SL-6 as the underpinnings. It simply worked, once I ironed out a couple of unrelated hardware issues. It auto-updates and lets me know when that's been done, but it doesn't reboot unless I want to, so there's little risk in allowing the updates. Auto-update can be disabled if you prefer. So far it has been flawless. It's managing two very large software RAID-6 arrays and 7 Guest VMs on a dual-Xeon Supermicro motherboard. All of the VM Guest OS's are CentOS-5.6 which works well for my applications. I may or may not upgrade the Guests to C6, depending on need. CentOS has been so reliable that I'm reluctant to move to a different platform. I've used Ubuntu Server and it works well, but I'm more familiar with the RHEL way of doing things so I'll stick with CentOS. One thing I don't know much about is an in-place upgrade of C5 to C6. There are some under-the-hood differences that must be taken into account, which I haven't looked into. YMMV, Chuck ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
[CentOS] A round of applause!
Hello All, Just a short note to add my vote for a HUGE round of applause to the CentOS team for their untiring efforts in getting releases out the door. I've just upgraded several servers to 5.6 and it all just works. None of the team's work is easy to accomplish, especially when less-than-useful complaints keep popping up from thoughtless users who don't appreciate the effort, and who waste the team's time trying to respond. RedHat's move to defend their support business against the freeloading distro vendors (we all know who those sharks are!) wasn't aimed at CentOS, but it has significantly increased the workload the team faces. Let's be patient and let them get the job done. Kudos to the CentOS team! Chuck ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] CentOS Digest, Vol 75, Issue 5
On 04/05/2011 09:00 AM, John R Pierce wrote: AFAIK, no standard raid modes verify parity on reads, as this would require reading the whole slice for every random read. Only raid systems like ZFS that use block checksuming can verify data on reads. parity (or mirrors) are verified by doing 'scrubs' Further, even if a raid DID verify parity/mirroring on reads, this would at best create a nonrecoverable error (bad data on one of the N drives in the slice, no way of knowing which one is the bad one). Thanks John, that's good information, something I didn't know. So I should think of RAID-5/6 parity as a mechanism for recovering from a drive fault that is more space-efficient than simple mirroring. Maybe RAID-10 with hot spares is more than good enough in most applications, but I do like dual parity for its ability to recover even in the face of a disk error popping up during the rebuild. Am I being too paranoid? Too bad ZFS on Linux is still up at the fuse layer. I understand Btrfs is rolled into newer kernels and should be in CentOS-6, but I read somewhere it's not yet in stable release and has some potential issues, so I'm reluctant to try it. It won't have RAID-6-like parity for a while. The fact that Oracle has both ZFS and Btrfs under its wing is, um, interesting. I'm only asking for the world :-) Chuck ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] ZFS @ centOS
On 04/05/2011 09:00 AM, rai...@ultra-secure.de wrote: That is really a no-brainer. In the time it takes to re-build such a RAID, another disk might just fail and the R in RAID goes down the toilet. Your 19-disk RAID5 just got turned into 25kg of scrap-metal. As for ZFS - we're using it with FreeBSD with mixed results. The truth is, you've got to follow the development very closely and work with the developers (via mailinglists), potentially testing patches/backports from current - or tracking current from the start. It works much better with Solaris. Frankly, I don't know why people want to do this ZFS on Linux thing. It works perfectly well with Solaris, which runs most stuff that runs on Linux just as well. I wouldn't try to run Linux-binaries on Solaris with lxrun, either. During my current work building a RAID-6 VM Host system (currently testing with SL-6 but later CentOS-6) I had a question rolling around in the back of my mind whether or not I should consider building the Host with OpenSolaris (or the OpenIndiana fork) and ZFS RAID-Z2, which I had heard performs somewhat better on Solaris. I'd then run CentOS Guest OS instances with VirtualBox. But ... I've been reading about some of the issues with ZFS performance and have discovered that it needs a *lot* of RAM to support decent caching ... the recommendation is for a GByte of RAM per TByte of storage just for the metadata, which can add up. Maybe cache memory starvation is one reason why so many disappointing test results are showing up. Chuck ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] KVM Host Disk Performance
On 04/04/2011 09:00 AM, compdoc wrote: It's possible to set up guests to use a block device that will get you the same disk I/O as the underlying storage. Is that what you're seeing? What speed does the host see when benchmarking the RAID volumes, and what speeds do the guests see? Yes, I have been going on the assumption that I get close to native block device performance, but the test results tell me otherwise. I see array rebuild data rates which seem reasonable ... in the order of 60 to 80 MBytes/sec. I'm using 256k chunks, with the stride size set to match the number of data drives. Using bonnie++, I mounted one of the Guest RAID-6 filesystems on the Host, ran the default tests, unmounted, then booted the Guest and ran the same default tests. The amount of RAM assigned was the same for both, to level the playing field a bit. Direct comparisons between the two were difficult to judge, but the general result was that the Host was between 2:1 and 3:1 better than the Guest, which seems to be a rather large performance gap. Latency differences were all over the map, which I find puzzling. The Host is 64-bit and the Guest 32-bit, if that makes any difference. Perhaps caching between Host and Guest accounts for some of the differences. At the moment my questions tend to be a bit academic. I'm primarily wondering if RAID-10 is paranoid enough given the current quality of WD CaviarBlack drives (better than dirt-cheap consumer drives, but not enterprise grade). My second question relates to whether or not the added overhead of using something like qcow2 would be offset by the advantages of more space efficiency and the copy-on-write feature. I'd love to hear what other software RAID users think, especially regarding large-capacity drives. It's rare for a modern drive to hand out bad data without an accompanying error condition (which the md driver should handle), but I have read that uncaught bad data is possible and would not be flagged in RAID arrays which don't use parity calculations. Chuck ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
[CentOS] KVM Host Disk Performance
Hello all, I'm having quite an interesting time getting up to speed with KVM/QEMU and the various ways of creating virtual Guest VMs. But disk I/O performance remains a bit of a question mark for me. I'm looking for suggestions and opinions This new machine has tons of disk space, lots of CPU cores and loads of RAM, so those are not issues. I currently have several software RAID-6 md's, each of which is about 700 GBytes. The md's are created as partitionable arrays, and I assign the resulting raw disk devices to each CentOS Guest OS, which then partitions the space according to its needs. Unlike using the more common filesystem-in-a-file method, the use of partitionable MD devices usually results in wasted disk space, but my aim is to provide the best performance possible with robust RAID. Is there a better-performing way of assigning disk space to Guest VMs? What about 'qcow2'? I understand its copy-on-write offers several advantages, but does it perform better than using partitionable md arrays or does the underlying filesystem overhead get in the way? Perhaps the native overhead of RAID-6 dual parity makes the whole question moot. Would the use of RAID-10 with hot spares be paranoid enough? I'm using 2 TByte drives which inherently have much greater chances of data errors, so I must keep data reliability in mind. I don't want to start a religious war over RAID and Guest VM disk methodology, but I have the luxury of trying various disk configurations until I get a reasonable balance between performance and reliability with these large drives. So many questions ... so much to learn. Lots of Googling around is quite useful but can lead to information overload :-) Cheers, Chuck ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Virtualization platform choice
On 03/27/2011 09:00 AM, Jerry Franz wrote: On 03/27/2011 02:57 AM, Jussi Hirvi wrote: Some may be bored with the subject - sorry... Still not decided about virtualization platform for my webhotel v2 (ns, mail, web servers, etc.). KVM would be a natural way to go, I suppose, only it is too bad CentOS 6 will not be out in time for me - I guess KVM would be more mature in CentOS 6. Any experience with the free VMware vSphere Hypervisor?. (It was formerly known as VMware ESXi Single Server or free ESXi.) http://www.vmware.com/products/vsphere-hypervisor/overview.html I would need a tutorial about that... For example, does that run without a host OS? Can it be managed only via Win clients? Issues with CentOS 4/5 guests (all my systems are currently CentOS 4/5). I'm currently using Ubuntu Server 10.04-LTS as a host for KVM running CentOS5.5 guests I migrated from VMware Server 2. Works fine. A nice feature of current generation KVM is that you are supposed to be able to do live migration even without shared storage (although I haven't tested that yet). I wrote some custom scripts to allow me to take LVM snapshots for whole-image backups and I'm pretty happy with the who setup. The only corners I encountered were 1) A lack of documentation on how to configure bridging over bonded interfaces for the host server. It turned out to be fairly easy - just not clearly documented anyplace I could find. 2) The default configuration for rebooting/shutting dow the host server just 'shoots the guests in the head' rather than having them shutdown cleanly.:( You will want to write something to make sure they get shutdown properly instead. Once in a while I find it's useful to compromise just a little, so I use Scientific Linux 6 as the Host OS, and run a bunch of CentOS-5.5 Guest VMs. It all simply works. KVM has improved quite a bit, and the management tools work well. One thing that requires a bit of skill is getting bridging configured (which I simply did by hand from the RHEL-6 documentation). I'm happy with the result, and see no reason to replace the underlying SL-6 Host distro. SL-6 as the Host is rather slow to shut down gracefully and reboot, because it hibernates the Guest OSs, one at a time, rather than just killing them. Hibernation takes a while to write out to disk if you've assigned a lot of RAM to the Guests. Bootup has to restore the saved state, so that's a bit slow too. But it works very well. I use partitionable RAID arrays for the Guests, and assign a raw md device to each one rather than using the 'filesystem-in-a-file' method. It seems to be a bit faster, but there's a learning curve to understanding how it works. One thing I found a bit annoying is the very long time it takes for a Guest to format its filesystems on the RAID-6 md device assigned to it. That's mostly due to array checksum overhead. RAID-10 would be a *lot* faster but somewhat less robust ... you pick what's best for your own situation. Chuck ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Race condition with mdadm at boot
On 03/12/2011 09:00 AM, compdoc wrote: On the particular Supermicro motherboard I'm using, there is a very long delay (10 or 15 sec) between power-on and initiation of visible BIOS activity, so all disk drives have ample time to spin up and stabilize. Yeah, I have used Supermicro in the past and they had the same long pause when you turn them on. Good boards, except I had one die recently. I was wondering how many drives total, and how many watts the PSU is? Also, is the controller's firmware up to date? I agree, Supermicro does design good boards, with a lot of server-grade features that consumer boards lack. Not cheap, but worth the money. I assume the long power-on pause is due to the BIOS silently checking RAM, with the side effect of giving the disks ample time to spin up. I have a total of 15 drives. Three of them are fast 2-1/2 drives used for the host OS filesystems, running on the ICH-10 SATA controller. No problem at all with those. The other 12 drives are 2TB WD Caviar Black series. 6 are connected to the on-board 8-port LSI 1068E chip, and 6 divided equally between 2 Tempo 'Sonnet' 4-port PCIe cards (Marvell chips). All drives have the same 8 partitions each. All drives have the same 8-partition configuration. The LSI controller supports 8 RAID-6 arrays using 5 active drives plus a hot spare. The Tempo controllers support the same arrangement using the other 6 drives. The firmware on everything is up to date. Other than the annoying failure to bind 1 or 2 random partitions on the last drive in each group of 6, the performance of this configuration is quite good. I still have the 'scsi_mod.scan=sync' kernel parameter in place, so there may be some other cause of the failure to bind some of the partitions. FWIW, in more than 10 system boots, there has never been any consistent pattern to these failures. Chuck ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Race condition with mdadm at boot [still mystifying]
On 03/11/2011 09:00 AM, Les Mikesell wrote: On 3/10/11 9:25 PM, Chuck Munro wrote: However, on close examination of dmesg, I found something very interesting. There were missing 'bindsd??' statements for one or the other hot spare drive (or sometimes both). These drives are connected to the last PHYs in each SATA controller ... in other words they are the last devices probed by the driver for a particular controller. It would appear that the drivers are bailing out before managing to enumerate all of the partitions on the last drive in a group, and missing partitions occur quite randomly. So it may or may not be a timing issue between the WD Caviar Black drives and both the LSI and Marvell SAS/SATA controller chips. I've seen some weirdness in powering up 6 or more SATA drives but never completely pinned down whether it was the controller, drive cage, or particular drives causing the problem. But I think my symptom was completely failing to detect some drives when certain combinations of disks were installed although each would work individually. Do you have any options about whether they power up immediately or wait until accessed? That's a good question, one I have experimented with. I don't have any choice as to when the drives are spun up (only on bootup), but I did try a controller card which pre-spun and checked the identification of the drives before handing off to the BIOS for bootup. That didn't help. On the particular Supermicro motherboard I'm using, there is a very long delay (10 or 15 sec) between power-on and initiation of visible BIOS activity, so all disk drives have ample time to spin up and stabilize. The drives' SMART data shows that the average spin-up time is well within the BIOS startup delay. Each drive activity indicator shows that they are always probed by the kernel's scsi scan process. I have since tried a couple of other tricks I found by Googling around ... setting the kernel parameters 'rootdelay=xx' and 'scsi_mod.scan=sync'. These had no effect on the problem. For some unfathomable reason, the last drives in each group of drives have one or more random partitions missing, with no 'bind' statement in dmesg. Other partitions on those drives are bound normally. This has been tested with at least two known-good replacement drives, with the same random results. On two occasions today, everything worked perfectly, but that was unusual. A friend of mine suggested an ugly hack - connect two 'dummy' unused old SATA drives to the last port of each controller (I'm using only 6 of 8 on each). I wonder if one of those $15 IDE-to-SATA converters would do the job (without a drive attached)? Foolish thought :-/ Chuck ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
[CentOS] Race condition with mdadm at boot [still mystifying]
This is a bit long-winded, but I wanted to share some info Regarding my earlier message about a possible race condition with mdadm, I have been doing all sorts of poking around with the boot process. Thanks to a tip from Steven Yellin at Stanford, I found where to add a delay in the rc.sysinit script, which invokes mdadm to assemble the arrays. Unfortunately it didn't help, so it likely wasn't a race condition after all. However, on close examination of dmesg, I found something very interesting. There were missing 'bindsd??' statements for one or the other hot spare drive (or sometimes both). These drives are connected to the last PHYs in each SATA controller ... in other words they are the last devices probed by the driver for a particular controller. It would appear that the drivers are bailing out before managing to enumerate all of the partitions on the last drive in a group, and missing partitions occur quite randomly. So it may or may not be a timing issue between the WD Caviar Black drives and both the LSI and Marvell SAS/SATA controller chips. So, I replaced the two drives (SATA-300) with two faster drives (SATA-600) on the off chance they might respond fast enough before the drivers move on to other duties. That didn't help either. Each group of arrays uses completely drivers (mptsas and sata_mv) but both exhibit the same problem, so I'm mystified as to where the real issue lies. Anyone care to offer suggestions? Chuck ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
[CentOS] Race condition with mdadm at bootup?
Hello folks, I am experiencing a weird problem at bootup with large RAID-6 arrays. After Googling around (a lot) I find that others are having the same issues with CentOS/RHEL/Ubuntu/whatever. In my case it's Scientific Linux-6 which should behave the same way as CentOS-6. I had the same problem with the RHEL-6 evaluation version. I'm posting this question to the SL mailing list as well. For some reason, each time I boot the server a random number of RAID arrays will come up with the hot-spare missing. This occurs with hot-spare components only, never with the active components. Once in a while I'm lucky enough to have all components come up correctly when the system boots. Which hot spares fail to be configured is completely random. I have 12 2TB drives, each divided into 4 primary partitions, and configured as 8 partitionable MD arrays. All drives are partitioned exactly the same way. Each R6 array consists of 5 components (partitions) plus a hot-spare. The small RAID-1 host OS array never has a problem with its hot spare. The predominant theory via Google is that there's a race condition at boot time between full enumeration of all disk partitions and mdadm assembling the arrays. Does anyone know of a way to have mdadm delay its assembly until all partitions are enumerated? Even if it's simply to insert a several-second wait time, that would probably work. My knowledge of the internal workings of the boot process isn't good enough to know where to look. I tried to issue 'mdadm -A -s /dev/md/md_dXX' after booting, but all it does is complain about No suitable drives found for /dev. Here is the mdadm.conf file: - MAILADDR root PROGRAM /root/bin/record_md_events.sh DEVICE partitions ##DEVICE /dev/sd* this didn't help. AUTO +imsm +1.x -all ## Host OS root arrays: ARRAY /dev/md0 metadata=1.0 num-devices=2 spares=1 UUID=75941adb:33e8fa6a:095a70fd:6fe72c69 ARRAY /dev/md1 metadata=1.1 num-devices=2 spares=1 UUID=7a96d82d:bd6480a2:7433f1c2:947b84e9 ARRAY /dev/md2 metadata=1.1 num-devices=2 spares=1 UUID=ffc6070d:e57a675e:a1624e53:b88479d0 ## Partitionable arrays on LSI controller: ARRAY /dev/md/md_d10 metadata=1.2 num-devices=5 spares=1 UUID=135f0072:90551266:5d9a126a:011e3471 ARRAY /dev/md/md_d11 metadata=1.2 num-devices=5 spares=1 UUID=59e05755:5b3ec51e:e3002cfd:f0720c38 ARRAY /dev/md/md_d12 metadata=1.2 num-devices=5 spares=1 UUID=7916eb13:cd5063ba:a1404cd7:3b65a438 ARRAY /dev/md/md_d13 metadata=1.2 num-devices=5 spares=1 UUID=9a767e04:e4e56a9d:c369d25c:9d333760 ## Partitionable arrays on Tempo controllers: ARRAY /dev/md/md_d20 metadata=1.2 num-devices=5 spares=1 UUID=1d5a3c32:eb9374ac:eff41754:f8a176c1 ARRAY /dev/md/md_d21 metadata=1.2 num-devices=5 spares=1 UUID=38ffe8c9:f3922db9:60bb1522:80fea016 ARRAY /dev/md/md_d22 metadata=1.2 num-devices=5 spares=1 UUID=ebb4ea67:b31b2105:498d81af:9b4f45d3 ARRAY /dev/md/md_d23 metadata=1.2 num-devices=5 spares=1 UUID=da07407f:deeb8906:7a70ae82:6b1d8c4a - Your suggestions are most welcome ... thanks. Chuck ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] CentOS and Marvell SAS/SATA drivers
On 03/07/2011 09:00 AM, Nico Kadel-Garcia wrote: On Sun, Mar 6, 2011 at 10:07 PM, Charles Polishercpol...@surewest.net wrote: https://secure.wikimedia.org/wikipedia/en/wiki/Fakeraid#Firmware.2Fdriver-based_RAID covers fake RAID. Ouch. That was*precisely* why I used the 2410, not the 1420, SATA card, some years back. It was nominally more expensive but well worth the reliability and support, which was very good for RHEL and CentOS. I hadn't been thinking about that HostRaid messiness because I read the reviews and avoided it early. Here's the latest info which I'll share ... it's good news, thankfully. The problem with terrible performance on the LSI controller was traced to a flaky disk. It turns out that if you examine 'dmesg' carefully you'll find a mapping of the controller's PHY to the id X string (thanks to an IT friend for that tip). The LSI error messages have dropped from several thousand/day to maybe 4 or 5/day when stressed. Now the LSI controller is busy re-syncing the arrays with speed consistently over 100,000K/sec, which is excellent. My scepticism regarding SMART data continues ... the flaky drive showed no errors, and a full test and full zero-write using the WD diagnostics revealed no errors either. If the drive is bad, there's no evidence that would cause WD to issue an RMA. Regarding fake raid controllers, I use them in several small machines, but only as JBOD with software RAID. I haven't used Adaptec cards for many years, mostly because their SCSI controllers back in the early days were junk. Using RAID for protecting the root/boot drives requires one bit of extra work ... make sure you install grub in the boot sector of at least two drives so you can boot from an alternate if necessary. CentOS/SL/RHEL doesn't do that for you, it only puts grub in the boot sector of the first drive in an array. Chuck ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] CentOS and Marvell SAS/SATA drivers
On 03/06/2011 09:00 AM, compdoc wrote: Regarding the Marvell drivers, I had good luck with the 'sata_mv' driver in Scientific Linux 6 just yesterday, running a pair of 4-port PCIe-x4 Tempo 'Sonnet' controller cards. Are those the Mac/Windows Sonnet cards that go for less than $200? What kind of performance you seeing? Are you doing software raid on them? Yes, those are the cards which target Windows and OS-X, but they work fine on Linux as well. They use the Marvell 88SX series chips. They control 6 2TB WD Caviar Black drives, arranged as 5 drives in a RAID-6 array with one hot spare. 3 drives are connected to each of two cards. mdstat shows array re-sync speed is usually over 100 MBytes/sec although that tends to vary quite a bit over time. -- On 03/06/2011 09:00 AM, John R Pierce wrote: On 03/05/11 7:01 AM, Eero Volotinen wrote: areca works.. for SAS, I prefer LSI Logic. The Supermicro mobo I'm using (X8DAL-3) has an on-board LSI 1068E SAS/SATA controller chip, although I have the RAID functionality disabled so I can use it as a bunch of drives for software RAID-6. Like the Tempo cards, it has 6 2TB WD SATA drives attached which provides a second set of arrays. Performance really sucks, for some unknown reason, and I get lots of I/O error messages logged when the drives get busy. There appears to be no data corruption, just a lot of retries that slow things down significantly. The LSI web site has no info about the errors. The firmware is passing back I/O abort code 0403 and LSI Debug info related to channel 0 id 9. There are only 8 ports so I don't know which disk drive may or may not be causing problems. The SMART data on all disks shows no issues, although I tend to treat some SMART data with scepticism. I need to track this error down because my understanding is that the LSI controller chip has very good performance. Chuck ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] CentOS and Marvell SAS/SATA drivers
On 03/05/2011 09:00 AM, Nico Kadel-Garcia wrote: On Fri, Mar 4, 2011 at 4:16 PM, compdoccomp...@hotrodpc.com wrote: If the Marvell drivers don't pan out, it looks like I'll have to either spend money on a 3Ware|LSI|Promise controller The 3ware are excellent... And Promise, historically, is*not*. Yes, I've had problems with Promise cards in the past, but haven't bought any for a long time. They seem to be moving upscale these days. Regarding the Marvell drivers, I had good luck with the 'sata_mv' driver in Scientific Linux 6 just yesterday, running a pair of 4-port PCIe-x4 Tempo 'Sonnet' controller cards. So it appears someone has fixed that particular driver. I've decided to stick with those cards rather than re-install the Supermicro/Marvell SAS/SATA 8-port controllers, which use the 'mvsas' driver that I had problems with on the RHEL-6 evaluation distro. So far, SL-6 has performed very well, all RAID-6 arrays re-synced properly, and running concurrent forced fscks on eight arrays was very fast (because the ext4 filesystems were still empty :-) ). I think I'll stick with SL-6 as the VM host OS, but will use CentOS for the guest VMs. CentOS-5.x will do fine for now, and I'll have the luxury of upgrading guest OSs to CentOS-6 as the opportunity arises. Chuck ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] CentOS and Marvell SAS/SATA drivers
On 03/04/2011 09:00 AM, Les Mikesell wrote: On 3/3/11 6:52 PM, Chuck Munro wrote: I've been on a real roller coaster ride getting a large virtual host up and running. One troublesome thing I've discovered (the hard way) is that the drivers for Marvell SAS/SATA chips still have a few problems. After Googling around quite a bit, I see a significant number of others have had similar issues, especially evident in the Ubuntu forums but also for a few RHEL/CentOS users. I have found that under heavy load (in my case, simply doing the initial sync of large RAID-6 arrays) the current 0.8 driver can wander off into the weeds after a while, less so for the older 0.5 driver in CentOS-5. It would appear that some sort of bug has been introduced into the newer driver. I've had to replace the Marvell-based controllers with LSI, which seem rock solid. I'm rather disappointed that I've wasted good money on several Marvell-based controller cards (2 SAS/SATA and 2 SATA). I replaced separate SII and promise controllers with a single 8-port Marvell based card and thought it was a big improvement. No problems with centos5.x, mostly running RAID1 pairs, one of which is frequently hot-swapped and re-synced. I hope its not going to have problems when I upgrade. Since I have the luxury of time to evaluate options, I've just downloaded Scientific Linux 6 to see what happens with either the mvsas or sata-mv driver. This is my first experience with SL but I wanted native ext4 rather than the preview version in CentOS-Plus. Even if I stick with SL-6 as the KVM host, I'll continue using CentOS as guest machines. If the Marvell drivers don't pan out, it looks like I'll have to either spend money on a 3Ware|LSI|Promise controller or revert to CentOS-Plus 5.5 for ext4. SL-6 is installing as I write this. Chuck ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
[CentOS] CentOS and Marvell SAS/SATA drivers
Hello all, I've been on a real roller coaster ride getting a large virtual host up and running. One troublesome thing I've discovered (the hard way) is that the drivers for Marvell SAS/SATA chips still have a few problems. After Googling around quite a bit, I see a significant number of others have had similar issues, especially evident in the Ubuntu forums but also for a few RHEL/CentOS users. I have found that under heavy load (in my case, simply doing the initial sync of large RAID-6 arrays) the current 0.8 driver can wander off into the weeds after a while, less so for the older 0.5 driver in CentOS-5. It would appear that some sort of bug has been introduced into the newer driver. I've had to replace the Marvell-based controllers with LSI, which seem rock solid. I'm rather disappointed that I've wasted good money on several Marvell-based controller cards (2 SAS/SATA and 2 SATA). My question ... is anyone aware of the *real* status of these drivers? The Internet is full of somewhat conflicting reports. I'm referring to 'mvsas' and 'sata-mv', both of which seem to have issues under heavy load. It sure would be nice to return to using what appear to be well-made controller cards. I understand that even Alan Cox has expressed some frustration with the current driver status. FWIW, I had similar problems under the RHEL-6 evaluation OS too. Chuck ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] security cameras
On 02/24/2011 09:00 AM, centos-requ...@centos.org wrote: On 02/23/2011 01:36 PM John R Pierce wrote: On 02/23/11 10:16 AM, Keith Roberts wrote: I think you will get far better video quality using CCTV cameras than a webcam on a USB port. you may think that, but those solutions you mentioned are all NTSC composite video, while even a $30 USB webcam now days is 2 megapixels or higher. anyways, the OP wants cameras that connect to the network and get their power off the ethernet cable, not a USB or a CCTV camera. Yes. True. I'm not interested in either USB or CCTV. Ethernet cams are much better and smarter technology and, from what I hear, easier to install and set up. From experience I can attest to the fact that PAL/NTSC CCTV cameras are significantly inferior to modern digital security cameras. I have used devices from Axis, who appear to be the largest and most diverse manufacturer (www.axis.com) but they're not the cheapest. As an aside, Axis cameras run embedded Linux. The newer Ethernet-enabled cameras can use POE (power over Ethernet) but you'll need either a power supply that you insert somewhere along the cable run, or a POE-enabled switch which supplies power to its Ethernet ports. Several brands are available. Using POE makes a lot of sense and saves a lot of trouble, but make sure your Ethernet cable installation is of high quality. Open-source software such as ZoneMinder works with cameras from several manufacturers, and runs on CentOS. I personally haven't tried it, but I understand it works well. Chuck ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] CentOS Digest, Vol 73, Issue 12
On 02/12/2011 09:00 AM, Gerhard Schneider wrote: Sorry, I don't use CentOS 6 now so I cannot check, but: Could it be that RHEL6 changed the default superblock mdadm is creating? CentOS 5 is creating a 0.9 superblock. Some other OS are creating 1.2 per default. But you can change that on the command line. If that's the case it's not a bug but a feature.. GS -- Gerhard Schneider Institute of Lightweight Design and e-Mail: g...@ilsb.tuwien.ac.at Structural Biomechanics (E317) Tel.: +43 664 60 588 3171 Vienna University of Technology / Austria Fax: +43 1 58801 31799 A-1040 Wien, Gusshausstrasse 27-29 http://www.ilsb.tuwien.ac.at/~gs/ Bingo! That hadn't occurred to me. And 1.2 is specified in the mdadm.conf file, which went right over my head. :-( Thanks Gerhard; mystery solved. Now all I need to do is wait for CentOS-6 to see if it correctly assembles all members of the arrays at boot time. I suspect the evaluation copy of RHEL-6 is not really up to date, so there may be issues with mdadm. I did take a look around the RedHat bugzilla site and saw a couple of things that might be related to the problem. Chuck ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
[CentOS] Some RAID-6 observations ... RHEL-6 vs CentOS-5.5
Hello all, In the process of building a new VM box to replace several individual CentOS servers, I've had the interesting experience of running both CentOS-5.5 and RHEL-6 (eval copy) as I build out the hardware based on a Supermicro motherboard. A couple of observations regarding RAID-6: - RAID-6 arrays created on RHEL-6 don't seem backward compatible with CentOS-5.5. This may or may not be expected behaviour, but it makes me wonder if forward compatibility might be a problem for those of us migrating large RAID-6 arrays from CentOS-5 to -6 systems. The main observations are that the md superblock is declared bad, and I get those boot-time messages claiming the available blocks are less than the configured size. As a result, the RAID arrays are not made available on CentOS-5.5. - A total of 8 RAID-6 arrays in RHEL-6 are built with 2TB data disks plus a hot spare. However, every time I boot the system some, but not all, of the hot spare partitions are randomly missing. Using mdadm to manually add the spare partition back into an array always works. There are 4 array partitions per disk, and the largest single array is just over 2 TBytes of useable space. There are two groups of 5+1 drives spread across two SAS/SATA controller cards, and this random behaviour can occur on either one. Rebooting gets me a different set of missing hot spare partitions. The only error messages I get are the mdmonitor missing spares event emails ... nothing in dmesg. The host OS has its own RAID-1 (+ hot spare) arrays on a separate set of disks and controller, and those arrays always come up correctly. I've checked and double ckecked the mdadm.conf file to make sure the ARRAY, DEVICE and 'devices=' statements are correct. Using mdadm to shut down an array and then reassemble it always seems to work properly, but something random is happening at boot-time assembly. It's only the hot spare partitions that don't appear, never the data partitions. Weird. I'm hoping CentOS-6 doesn't present me with the same problem. Because I'm not a registered RHEL user, I don't have the ability to submit a bug report at RedHat. Chuck ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Fan speed control on Supermicro X8DAL board with CentOS
On 02/07/11 20:42, RedShift wrote: Actually it's possible a hardware monitoring module is interfering with the FAN speed control duties. Try preventing any hardware monitoring related modules get modprobed. See modprobe.conf how to blacklist modules. Glenn Interesting thought Glenn, thanks. I've since gone back to CentOS-5.5 on the box (since it's not fully built anyway) and added the SuperoDoctor monitoring utility after resolving a couple of dependencies like SNMP and the correct kernel driver. I sent an enquiry to the Supermicro support folks, and they jumped on it right away, which impressed me. They wanted a screenshot of SuperoDoctor to try to analyze the problem. I don't think it's a case of a kernel module getting in the way because this issue crops up even during the BIOS boot process, and happens randomly if I run memtest86 as well. Sometimes the fans jump to light speed, other times not. I'm beginning to think it's either a bug in the latest BIOS or a sensor chip problem. Even when the CPU temperatures are reported as Low, the fans run full speed most of the time. The exhaust air leaving the box is almost cool, so ambient temperature is ok. I'll give the Supermicro support folks some time to sort this out, since I'm still waiting for CentOS-6 anyway. Chuck ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Fan speed control on Supermicro X8DAL board with CentOS
On Sun, 06 Feb 2011 12:09:12 + Ned Slider wrote: On 06/02/11 07:00, Chuck Munro wrote: Hello folks, I'm having a difficult time trying to figure out why the CPU cooling fans run at full speed on my Supermicro X8DAL-3 motherboard. There doesn't seem to be any variable speed (the fans are PWM compatible) ... they either idle at almost nothing, or suddenly burst into a high-pitched scream that gets my ears bleeding after a few seconds. Once they jump to warp-10, they remain there. The Super-I/O chip on this board is a Winbond W83627DHG which does the temperature and voltage monitoring. Is anyone aware of which driver or kernel module I need for that chip in order to get control of the fans? The Supermicro web site and the board's manual aren't any help. Fresh installs of CentOS-5.5 and RHEL-6 don't exert any control by default. Installing the lm_sensors package and probing with the 'sensors' command didn't help either. Slowly going deaf ... Chuck Hi Chuck, The correct kernel module for your chipset is w83627ehf.ko. I'm not sure the driver actually controls fan speed, I thought it was more for monitoring (fan speeds, temps, voltages) but I could be wrong. My current system (not a Supermicro) controls variable fan speed from options within the BIOS. I can enable/disable fan speed control and select either voltage or PWM based control. The stock w83627ehf driver in RHEL5.5 is oldish (they were updated in 5.5 I think but are still over a year old now). ELRepo.org have an updated driver available (kmod-w83627ehf) based on a backport from kernel-2.6.34. I've just checked upstream and a few more patches have been committed since kernel-2.6.34 and the current kernel-2.6.37 so I'll look at updating the elrepo driver with those latest patches. Hope that helps. Thanks Ned! I did go through the board's BIOS menus several times and could find only one fan control option, which ranges from always-fast for maximum performance to almost-silent for workstation use. No matter what the setting, the fans may start out slow but eventually jump to high speed. Updating the BIOS to the latest version made no difference. I also noticed that at all times the BIOS reports the CPU temperatures as Low no matter what the fan speed. The coolers are always cold to the touch. I sure hope I don't have a defective board ... it's a royal pain to have to remove one from a large server. I booted Ubuntu but the live-CD version doesn't have a working 'fancontrol' utility. I'd be tempted to install Ubuntu Server but I much prefer staying with CentOS and KVM to match all of the guest virtual machines it'll be running. Time to go through the mobo manual with a fine-tooth comb. :-) Chuck ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
[CentOS] Fan speed control on Supermicro X8DAL board with CentOS
Hello folks, I'm having a difficult time trying to figure out why the CPU cooling fans run at full speed on my Supermicro X8DAL-3 motherboard. There doesn't seem to be any variable speed (the fans are PWM compatible) ... they either idle at almost nothing, or suddenly burst into a high-pitched scream that gets my ears bleeding after a few seconds. Once they jump to warp-10, they remain there. The Super-I/O chip on this board is a Winbond W83627DHG which does the temperature and voltage monitoring. Is anyone aware of which driver or kernel module I need for that chip in order to get control of the fans? The Supermicro web site and the board's manual aren't any help. Fresh installs of CentOS-5.5 and RHEL-6 don't exert any control by default. Installing the lm_sensors package and probing with the 'sensors' command didn't help either. Slowly going deaf ... Chuck ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] CentOS Digest, Vol 73, Issue 3
On 02/03/2011 09:00 AM, Lamar Owen wrote: -- On Wednesday, February 02, 2011 08:04:43 pm Les Mikesell wrote: I think there are ways that drives can fail that would make them not be detected at all - and for an autodetected raid member in a system that has been rebooted, not leave much evidence of where it was when it worked. If your slots are all full you may still be able to figure it out but it might be a good idea to save a copy of the listing when you know everything is working. I'll echo this advice. I guess I'm spoiled to my EMC arrays, which light a yellow LED on the DAE and on the individual drive, as well as telling you which backend bus, which enclosure, and which drive in that enclosure. And the EMC-custom firmware is paranoid about errors. But my personal box is a used SuperMicro dual Xeon I got at the depth of the recession in December 2009, and paid a song and half a dance for it. It had the six bay hotswap SCSI, and I replaced it with the six bay hotswap SATA, put in a used (and cheap) 3Ware 9500S controller, and have a RAID5 of four 250GB drives for the boot and root volumes, and an MD RAID1 pair of 750GB drives for /home. The Supermicro motherboard didn't have SATA ports, but I got a 64-bit PCI-X dual internal SATA/dual eSATA low-profile board with the low-profile bracket to fit the 2U case. Total cost$500. Less than $500 for a Supermicro box? Holy crap, Batman! I'm using one of their motherboards (X8DAL-3) in a home-brew configuration, which has turned out to be somewhat less expensive than a factory-built Supermicro, but still not cheap. I'll say one thing for them ... they do make really nice stuff, with little extras the commodity makers like Asus or Gigabit don't include. Well worth the extra dollars. This box just screams, so it should handle KVM very nicely if I organize the arrays properly ... anyone with RAID-60 advice please chime in. I took Les' advice and built a connection map of the 15 drives. The Disk Utility GUI turned out to be useful once I determined the SATA breakout cable order and labelled each with its PHY number. Knowing that lets me use the GUI to clearly identify a failed drive, which shows up as a PHY#. I'm normally a command-line kinda guy but this utility is something I really like on RHEL-6. I hope it's the same on CentOS-6. I'll organize the disk arrays with this trial version of RHEL, on the assumption that CentOS-6 simply picks them up as-built. Chuck ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] RHEL-6 vs. CentOS-5.5 (was: Static assignment of SCSI device names?)
On 02/02/2011 09:00 AM, Lamar Owen wrote: On Wednesday, February 02, 2011 02:06:15 am Chuck Munro wrote: The real key is to carefully label each SATA cable and its associated drive. Then the little mapping script can be used to identify the faulty drive which mdadm reports by its device name. It just occurred to me that whenever mdadm sends an email report, it can also run a script which groks out the path info and puts it in the email message. Problem solved:-) Ok, perhaps I'm dense, but, if this is not a hot-swap bay you're talking about, wouldn't it be easier to have the drive's serial number (or other identifier found on the label) pulled into the e-mail, and compare with the label physically found on the drive, since you're going to have to open the case anyway? Using something like: hdparm -I $DEVICE | grep Serial.Number works here (the regexp Serial.Number matches the string Serial Number without requiring the double quotes). Use whatever $DEVICE you need to use, as long as it's on a controller compatible with hdparm usage. I have seen cases with a different Linux distribution where the actual module load order was nondeterministic (modules loaded in parallel); while upstream and the CentOS rebuild try to make things more deterministic, wouldn't it be safer to get a really unique, externally visible identifier from the drive? If the drive has failed to the degree that it won't respond to the query, then query all the good drives in the array for their serial numbers, and use a process of elimination. This, IMO, is more robust than relying on the drive detect order to remain deterministic. If in a hotswap or coldswap bay, do some data access to the array, and see which LED's don't blink; that should correspond to the failed drive. If the bay has secondary LED's, you might be able to blink those, too. Well no, you're not being dense. It's a case of making the best of what the physical hardware can do for me. In my case, the drives are segregated into several 3-drive bays which are bolted into the case individually, so removing each one to compare serial numbers would be a major pain, since I'd have to unbolt a bay and remove each drive one at a time to read the label. The use of the new RHEL-6/CentOS-6 'udevadm' command nicely maps out the hardware path no matter the order the drives are detected/named, and since hardware paths are fixed, I just have to attach a little tag to each SATA cable with that path number on it. One thing I did was reboot the machine *many* times to make sure the controller cards were always enumerated by Linux in the same slot order. I also notice that the RHEL-6 DriveInfo GUI application shows which drive is giving trouble, but it only maps the controllers in a vague way with respect to the hardware path. (At least that's what I remember seeing a couple of days ago, I could be mistaken.) On this particular machine I don't have the luxury of per-drive LED activity indicators, so whacking each drive with a big read won't point the way (but I have used that technique on other machines). I didn't have the funds to buy the hot-swap bays I would have preferred. I may retrofit later. Your suggestions are well taken, but the hardware I have doesn't readily allow my use of them. Thanks for the ideas. Chuck ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] RHEL-6 vs. CentOS-5.5 (was: Static assignment of, SCSI device names?)
Les Mikesell wrote: On 1/30/11 1:37 PM, Chuck Munro wrote: Hello list members, My adventure into udev rules has taken an interesting turn. I did discover a stupid error in the way I was attempting to assign static disk device names on CentOS-5.5, so that's out of the way. But in the process of exploring, I installed a trial copy of RHEL-6 on the new machine to see if anything had changed (since I intend this box to run CentOS-6 anyway). Lots of differences, and it's obvious that RedHat does things a bit differently here and there. My focus has been on figuring out how best to solve my udev challenge, and I found that tools like 'scsi_id' and udev admin/test commands have changed. The udev rules themselves seem to be the same. Do any of the names under /dev/disk/* work for your static identifiers? You should be able to use them directly instead of using udev to map them to something else, making it more obvious what you are doing. And are these names the same under RHEL6? I was happy to see that device names (at least for SCSI disks) have not changed. The more I look into the whole problem the more I realize that I've overstated the difficulty, now that I know how to map out the hardware path for any given /dev/sdX I might need to replace. I've never dug as deeply into this before, mostly because I never could find the spare time. I'm happy with simply writing a little script which accepts a /dev/sdX device name argument and reformats the output of: 'udevadm info --query=path --name=/dev/sdX' to extract the hardware path. It's a bit cleaner than the current RHEL-5/CentOS-5 'udevinfo' command. Using the numeric path assumes knowledge of how the motherboard sockets are laid out and the order in which I/O controller channels are discovered, of course. It's then not difficult to trace a failed drive by attaching little tags to the SATA cables from the controller cards. The real key is to carefully label each SATA cable and its associated drive. Then the little mapping script can be used to identify the faulty drive which mdadm reports by its device name. It just occurred to me that whenever mdadm sends an email report, it can also run a script which groks out the path info and puts it in the email message. Problem solved :-) So even though I figured out how to add 'alias' symlink names to each disk drive, I'm not going to bother with it. It was a very useful learning experience, though. Chuck ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
[CentOS] RHEL-6 vs. CentOS-5.5 (was: Static assignment of SCSI device names?)
Hello list members, My adventure into udev rules has taken an interesting turn. I did discover a stupid error in the way I was attempting to assign static disk device names on CentOS-5.5, so that's out of the way. But in the process of exploring, I installed a trial copy of RHEL-6 on the new machine to see if anything had changed (since I intend this box to run CentOS-6 anyway). Lots of differences, and it's obvious that RedHat does things a bit differently here and there. My focus has been on figuring out how best to solve my udev challenge, and I found that tools like 'scsi_id' and udev admin/test commands have changed. The udev rules themselves seem to be the same. Regarding networking, all of my 'ifcfg-*' files from CentOS-5.5 worked well, including bridging for KVM. Of course, one of the first things I did was remove that atrocious NetworkManager package ... it needs to be rewritten to make it a *lot* more intuitive. RedHat uses it during installation to manually configure the NICs, which I think is a mistake. I much prefer the way CentOS anaconda has done it so far, as a separate install menu form. The performance of the machine seemed to be better with the newer kernel, which is encouraging. I suspect we can look forward to a number of improvements. I've just managed to scratch the surface. I do expect there may be a few challenges for those of us upgrading a system from 5.x to 6, where some user-written admin scripts could break depending on the system commands they use. I look forward to CentOS-6 and all the goodies we can expect, and I'm quite happy to wait until the CentOS crew does their thing before releasing it ... they deserve a lot of credit for doing a thorough job all these years. Chuck ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Static assignment of SCSI device names?
Les Mikesell kindly wrote: Identifying drives by their ID string (which includes the drive's serial number) and assigning names in the rules works ok. BUT, what happens when I have to swap out a failed drive? The serial number (and possibly model number) changes, and the udev assignment should fail, probably assigning an unexpected /dev/sd? name. RAID rebuild would choke until I change the MD device assignment. If you can figure things out for the initial md device creation, subsequent assembly uses the uuid to match up the members and doesn't care if the disks have been moved around either physically or by detection order. And if you are hot-swapping drives and rebuilding manually, you should be able to find the just-assigned name with 'dmesg'. That's how I've been doing it with the two RAID-6 servers I have now, but in my advanced age I'm getting lazy. I did figure out how to tag the drive by path using the '-i' option to scsi_id and filtering by a RESULT==path * expression. However, something has gone very much astray, because now the rules don't do anything ... the system assigns names in the order scanned despite having added the udev rules before the default '50-udev.rules' is executed. I'm a bit mystified by this. Before I started messing around, filtering by drive id string did work, now it doesn't. Is there a small database somewhere I need to trash? Mike's suggestion is intriguing and makes me wonder if I can manually label a replacement drive with the same UUID as the failed one (assuming I keep records, of course). Since all drives in the array are partitioned exactly the same way, I can have one or two spares on the shelf pre-partitioned, ready to be given a UUID. I don't plan to hot swap. Still, I'd love to be able to use udev rules to assign names, if only to make life easier ... inquiring minds need to know why I suddenly can't get any rules to work. Maybe I need to re-install. I have just downloaded a trial version of RHEL-6 which I should try, to see if it handles this stuff more gracefully. It'll do for experimentation until CentOS-6 arrives. Chuck ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
[CentOS] Static assignment of SCSI device names?
Hello list members, In CentOS-5.5 I'm trying to achieve static assignment of SCSI device names for a bunch of RAID-60 drives on a Supermicro motherboard. The scsi_id command identifies all drives ok. The board has one SATA controller and three SAS/SATA controllers ... standard on-board ICH-10 ATA channels, an on-board LSI SAS/SATA controller, and two add-on SAS/SATA contoller cards. There are 13 drives in all, spread across the four controllers, all configured for Linux software RAID. The problem is in management of the drive names and figuring out which drive to pull in case of failure. Unfortunately the BIOS scan detects only the three drives connected to the ICH-10 SATA controller. That's ok because that's where the RAID-1 boot drives are. However, when the kernel starts it assigns those drives last, not first. For this reason I want to use a set of udev rules to assign specific names to the drives plugged into specific ports (to maintain my sanity :-) ). Identifying drives by their ID string (which includes the drive's serial number) and assigning names in the rules works ok. BUT, what happens when I have to swap out a failed drive? The serial number (and possibly model number) changes, and the udev assignment should fail, probably assigning an unexpected /dev/sd? name. RAID rebuild would choke until I change the MD device assignment. Is it possible to assign SCSI drive names by hardware path instead? I especially want the three RAID1+spare boot drives to always be assigned sda/sdb/sdc, because that sorts out other issues I'm having in CentOS-5. In the udev rules file I tried piping the output of scsi_id -g -i -u -s /block/... through cut to extract the path, but I get no match string when I run udevtest against that block device. Does the PROGRAM==. clause not recognize the pipe symbol? I tried a little shellscript to provide the RESULT match string, but udevtest didn't like that. Is there a supported way to predictably assign a drive name according to the hardware port it's plugged into ... it would make swapping drives a lot easier, since it becomes 'drive-id-string' agnostic. Better yet, is there any way to tell the kernel the order in which to scan the controllers? I'm also hoping the problem doesn't radically change when I install CentOS-6 on this box. I'm using CentOS-5 just to get practice in using KVM and RAID-60. Thanks for any advice you can offer. Chuck ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
[CentOS] Disk array format for CentOS virtual host
Hello List Members (and Holiday greetings!) This rambles a bit ... my apologies in advance. I am in the process of building a large CentOS-based VM host machine which will replace several individual boxes. I've done the usual hardware research and ended up with a SuperMicro motherboard, dual Xeons, lots of ECC RAM, solid power supply, and tons of cooling. When it comes to the best disk array layout for 7 guest instances of CentOS (mostly 64-bit) I run into the classical argument of disk I/O performance versus array robustness. I'm using Linux software RAID so I don't get locked into a vendor's controller, and because I've got CPU cycles to burn. The intent is to use CentOS-6 with KVM after climbing the usual experimental learning curve. I've got years of CentOS experience, but none with KVM. So, underneath all of this I could use RAID-10/RAID-6/RAID-60. Which one is better for the following work mix: - 4 web servers, not heavily loaded (yet), mostly CGI. - 2 independent Zimbra email servers, which seem to spend most of their time rejecting spam and malware. - A primarily-inbound FTP server with bandwidth throttling. - A batch ImageMagick video process which kicks in a couple of times/day for a few minutes. - A secondary/backup SMB server, not heavily used. - A very lightly loaded DDNS server. Two spindles will be used in RAID-1 for the host OS filesystems. I have 10 large drives for the guest machines, but because they're 2 TByte I want to ensure reasonable fault tolerance, meaning the ability to handle at least two drive failures. This really means that an array rebuild to replace *one* drive stands a good chance of succeeding. At first glance this would seem to be a system which will be doing more reads than writes, so RAID-6 or 60 might be a reasonable fit. RAID-10 offers much better write performance according to what I've been reading, but is less fault tolerant. RAID-60 is a bit of an unknown to me with regard to performance. In case you're wondering, I don't consider RAID to be a backup, so all of this will be automatically backed up to duplicate separate machines, each having tons of RAID-6 storage. Each individual guest OS will do its own rsync thing. Please forgive my lack of VM experience, but an additional question ... I'm thinking of creating a separate RAID-xx MD partition for each guest. Should I hand each over as a 'device' to KVM or is it better to place the guest filesystems in a big file on each separate MD partition? Or, is it better to create one huge RAID-xx partition and just use big filesystems-in-a-file. Either way would probably take the same amount of time (long!) to rebuild an array, if I understand this correctly. The collective wisdom of this list is most welcome. Thanks muchly, Chuck ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Small proxy appliance hardware
Regarding a small proxy appliance, my thanks to those who responded either on this list or by direct email. We are going to get one of the little Acer AspireRevo 3610 boxes and give it a try. It's small, quiet, runs cool, and has more than enough resources to handle our CentOS proxy application. It's a shame that we have to trash the Windows-7 install (a waste of money) but that's life. The only unanswered question is with regard to an external USB wired network adapter, so we'll have to go by the data on the various manufacturers' web sites about Linux support. Has anyone tried the Belkin USB2/Gbit, the D-Link DUB-E100 or the Linksys USB300M devices? Which adapter chips to they use (in other words, which driver)? ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
[CentOS] Small proxy appliance hardware
I need to configure a small proxy appliance which doesn't require a lot of CPU power. I'm looking for any warnings this group may have regarding CentOS-5 on the following: - Acer AspireRevo 3610 (Atom CPU) - Dell Inspiron 537s (Celeron) - Any other small machines you might recommend (UL approved) I need to implement dual NICs, and for the Dell it should be no problem finding a low-profile 3Com or Intel adapter, but for the Acer I'll need to use one of those USB wired network dongle things. Does anyone have recommendations for a USB dongle that is fully automatic (regarding speed and duplex negotiation) supported by CentOS? Gigabit speed is not required. Any other compatibility issues with the above machines? Normally I'd simply build something from components and squeeze it into a small box, but in this case I need something with UL/CSA/CE approvals, which lets out homebrew hardware. Many thanks in advance! ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] iptables - Forwarding with port translation thru an OpenVPN tunnel
Thanks to those who responded. The use of Apache's reverse proxy was
something I would never have though of (it's the mind-numbing cold
medication I'm on, LOL)
However, I did manage to get things rolling thru the tunnel by configuring
strong-end routing at the remote server. Requests were indeed arriving at
the remote, but because the request's origin IP address was that of the
outside user's browser, the remote server was simply trying to return
responses via its default route, which is not the tunnel.
I *have* to ask ... why is strong-end routing not the default behavior in
Linux? Anyway ...
Adding 'ip route ' and 'ip rule ' commands when establishing the
tunnel did the trick.
On the remote server, here are the commands run in a script launched by
rc.local:
---
#!/bin/sh
# NOTE: To allow VPNs under OpenVPN, IPv4 Forwarding
# must be enabled in the /etc/sysctl.conf file!
# Enable NAT for the OpenVPN tunnel from the main server:
WAN=eth0# The primary public IP interface
iptables -t nat -A POSTROUTING -s 172.17.xxx.0/24 -o ${WAN} -j MASQUERADE
# Enable strong-end routing for traffic coming in thru the VPN tunnel:
## Table 200 - In/Out traffic via tun0:
ip route add table 200 172.17.xxx.0/24 via 172.17.xxx.yy dev tun0
ip route add table 200 default via 172.17.xxx.yy dev tun0
## Engage! ...
ip rule add from 172.17.xxx.0/24 lookup 200
service openvpn start
---
In the example above, xxx.yy is tun0's 'P-t-P' IP address (usually it's inet
IP address minus 1).
-- and --
On the main server, here are the commands run in a script launched by
rc.local:
---
#!/bin/sh
# NOTE: To allow VPNs under OpenVPN, IPv4 Forwarding
# must be enabled in the /etc/sysctl.conf file!
# Enable NAT for the OpenVPN tunnels:
WAN=eth0# the public IP interface
/sbin/iptables -t nat -A POSTROUTING -s 172.17.xxx.0/24 -o ${WAN} -j
MASQUERADE
TunnelRemoteIP=172.17.xxx.zz # The inet IP address of the remote server
thru the VPN.
# Force any HTTP/HTTPS requests on eth0:1's secondary IP address
(64.aaa.bbb.ccc)
# to be forwarded to the remote server, with port translation.
# HTTP:
/sbin/iptables -t nat -A PREROUTING -i eth0 -d 64.aaa.bbb.ccc -p tcp --dport
80 -j DNAT --to ${TunnelRemoteIP}:29080
/sbin/iptables -A FORWARD -p tcp -m tcp -i eth0 -o tun0 -d ${TunnelRemoteIP}
--dport 29080 -j ACCEPT
#
# HTTPS:
/sbin/iptables -A PREROUTING -t nat -i eth0 -d 64.aaa.bbb.ccc -p tcp --dport
443 -j DNAT --to ${TunnelRemoteIP}:29443
/sbin/iptables -A FORWARD -p tcp -m tcp -i eth0 -o tun0 -d ${TunnelRemoteIP}
--dport 29443 -j ACCEPT
service openvpn start
sleep 2# Be polite.
/sbin/iptables -A FORWARD -p tcp -m tcp --dport 80 -j ACCEPT
/sbin/iptables -A FORWARD -p tcp -m tcp --dport 443 -j ACCEPT
---
Obviously the above iptables entries could simply be added to the recipe in
/etc/sysconfig/iptables, but I chose to put them in this script so that if I
don't want tunnels to be started I don't run the scripts. There may be
redundant commands in all of this, but at least it works flawlessly for me.
I didn't use any SNAT statements on the rash assumption POSTROUTING does the
same thing.
I hope this may be useful to anyone else out there who encounters this
issue.
Cheers,
Chuck
___
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
[CentOS] iptables - Forwarding with port translation thru an OpenVPN tunnel
Perhaps someone with a sharper brain than I can solve this little mystery. I've Googled until I'm blue in the face, read all TFM's I can find, and tried several iptables rule combinations but just can't get the following to work. Here's my challenge: I have a CentOS-5.3 main server with a static public IP address running Apache, OpenVPN, and a bunch of other services. The primary IP address for the only NIC in this box is used by Apache on standard ports 80 and 443. I have a secondary static public IP address assigned to this same NIC as eth0:1 I have a working OpenVPN tunnel going to another CentOS-5.3 server which has Apache listening on its eth0 NIC for requests on non-standard ports 29080 and 29443(don't ask!). This server is accessible on its static public IP address and thru OpenVPN tunnel from the main server. The main server is the OpenVPN server as well, with private IP address 172.16.xxx.1/32 auto-assigned to it. The second server is a VPN client with IP address 172.16.xxx.yyy/32 assigned. The narrow netmasks are the OpenVPN default, presumably used for isolation among multiple clients. OpenVPN is configured to run over the main server's NIC via its eth0:1 secondary IP address 64.aaa.bbb.cccThe main server has a static route set up between its eth0:1 IP and the remote server's public IP address (not the tunnel IP). This route is only there so that during initial VPN negotiations the data will move via eth0:1's IP address. Things like ping, SSH, scp and HTTP/HTTPS all work correctly thru the VPN in both directions, so the VPN itself is solid. Ok, so now I want external HTTP/HTTPS requests made to the main server's eth0:1 public IP address to be forwarded thru the VPN to the second remote server, with port translation along the way. Here is what I want to happen: Outside world HTTP/HTTPS requests to eth0:1 ports 80/443 | V CentOS eth0:1 (64.aaa.bbb.ccc) | V Translate to ports 29080/29443 | V Forward requests thru the tunnel to 172.16.xxx.yyy | V Remote Apache responds, packets return thru VPN to main | V Response goes back to the outside world via 64.aaa.bbb.ccc HTTP/HTTPS requests to the 64.aaa.bbb.ccc public IP just hang and time out. Direct requests to the remote server on the non-standard ports work fine. I've tried lots of iptables example entries found on several forums but so far none of them seem to work. Does anyone have a cookbook-like complete set of iptables rules that will accomplish what I need? Is it possible I need to set up strong-end routing on the remote server to send the response packets back thru the VPN? (such a pain to configure) Any assistance will be most gratefully received! Best Regards, and my sincere kudos to the CentOS team and helpful list members over the past few years. Chuck ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] [OT] Godaddy hell...
Jason Pyeron wrote: Can I get some recommendations: We are looking for a hosting provider (other than godaddy) with 1: SLA 2: SSH access 3: subversion/rsync or the ability to install binaries / compile source. Would like them to include http/https and email. Any suggestions? -Jason --- If you want serious control over the machine and are willing to pay a bit more for a dedicated server, take a look at ServerBeach. They offer several operating systems, including CentOS, and give you control over the box to the point that you can even re-image the disk remotely if you truly screw something up. If you do something nasty to it and cause it to lock up, you can still remotely reboot. Hardware RAID-1 is available. You get out-of-the-box CentOS and are allowed to install pretty well anything you like, including virtual machines on top. If you need reverse DNS lookups you can request whatever response you'd like for the IP address(es) you have. Just be aware that what you get is simply a complete machine fully exposed to the Internet, and you're responsible for your own iptables firewalling, etc. Their support staff is friendly and responsive, but they expect you to be your own geek. For DNS service I use Nettica because they are relatively inexpensive and give you control over all of the records. I use GoDaddy strictly as a registrar, and have been very happy with the service, but I don't consider them a serious hosting provider because there's not enough control. I have used several of the free hosting and DNS services out there, but when it comes down to the crunch, you get what you pay for (CentOS is a great exception!) BTW, I have no connection with these companies, I just happen to like their services. A lot. Chuck ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
