Re: vinum raid5 subdisks keep changing length?
Hi again, On Tue, Feb 17, 2004 at 11:01:36PM +1100, Tony Frank wrote: [ ... ] > Despite this logging oddity everything else appears to work just fine using > 4.9-RELEASE. > > I am currently building world based on RELENG_4 cvsup from this evening. > Will try it all again with the new kernel & world probably tomorrow. > > Main thing that is different is that I no longer have a vinum root environment. > I might try to rebuild that while I wait for world to build. > Vinum root needs a bit more planning from the start due to disk offsets and the > like... [ ... ] > > Will further report tomorrow on outcome of further testing. > Scenarios: > data raid5 with vinum root on 4.9-RELEASE > plain data raid5 with RELENG_4 > data raid5 with vinum root on RELENG_4 Well I have rebuilt my setup with vinum root + raid5 data volume. I originally rebuilt it on 4.9-RELEASE GENERIC kernel + world from CD #1. Without going so far as to do hardware failure testing, I did not observe any problems other than the lack of vinum entries from bootup in the logs. I then upgraded the system to RELENG_4 GENERIC kernel (cvsup yesterday) and world built from same sources. Still no problems after reboots or as a result of any other 'normal' operations. I then built & installed a custom kernel based on RELENG_4 GENERIC (remove lot of drivers and add IPFW etc) and still it all works fine and without any obvious issues. As such I suspect the previous scenario was likely a result of user error and some obscure combination of events. If I'm bored again tomorrow I might rebuild the lot again but starting with the RELENG_4 level instead of 4.9-RELEASE to see if that has any impacts. Thanks for your time, Tony ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: vinum raid5 subdisks keep changing length?
On Tue, Feb 17, 2004 at 05:53:06PM +1030, Greg 'groggy' Lehey wrote: > On Tuesday, 17 February 2004 at 11:39:26 +1100, Tony Frank wrote: > > On Tue, Feb 17, 2004 at 09:51:30AM +1030, Greg 'groggy' Lehey wrote: > >> On Monday, 16 February 2004 at 22:04:44 +1100, Tony Frank wrote: [... snip ...] > OK, I tried almost exactly the same thing. My disks are fractionally > smaller than yours, so I took a different integral number of stripes. > I didn't get any messages, and the config now looks like: > > volume data > plex name data.p0 org raid5 984s vol data > sd name data.p0.s0 drive drive1 plex data.p0 len 8374824s driveoffset 265s > plexoffset 0s > sd name data.p0.s1 drive drive2 plex data.p0 len 8374824s driveoffset 265s > plexoffset 984s > sd name data.p0.s2 drive drive3 plex data.p0 len 8374824s driveoffset 265s > plexoffset 1968s > sd name data.p0.s3 drive drive4 plex data.p0 len 8374824s driveoffset 265s > plexoffset 2952s > sd name data.p0.s4 drive drive5 plex data.p0 len 8374824s driveoffset 265s > plexoffset 3936s > > I've stopped and started vinum a couple of times, and all works well. > I then removed the objects and tried again with subdisks 4 sectors > longer. Vinum gives the message: > > vinum: removing 16 blocks of partial stripe at the end of data.p0 > > printconfig is then identical with the previous version. With no obvious known problems and this being a test system I have wiped the disks and started over. My near exact steps: Rebooted from 4.9-RELEASE CD1 Went to 'fixit' mode with live filesystem CD2 "Wiped" the disks: dd if=/dev/zero of=/dev/da0 bs=512 count=32 fdisk -BI /dev/da0 dd if=/dev/zero of=/dev/da0s1 bs=512 count=32 disklabel -w -B da0s1 auto First thing I notice is that the geometry for my SCSI disks differs by 1 cylinder between da0 and da0s1. I configured all the vinum slices as da0s1h etc. Anyway, then I went on to a 'standard' installation. I selected ad0 (space), q -no changes (already had a partition from above steps) Selected "BootMgr" Repeated for each disk (ad0, ad2, da0-da3) Deleted all the prelisted partitions & filesystems and performed an 'auto' based on ad2 (leaves ad0, da0-da3 unused) Selected "Minimal" installation type Sourced from CD Post install configured fxp0 for DHCP, Accepted NFS client, selected no for all other options Set timezone to Australia Victoria Added an extra user "tony" with additional group wheel. No ports, no packages, no other options. Rebooted System booted from ad2 with 4.9-RELEASE GENERIC kernel Ran disklabel -e da0s1 and added a 'h' slice of: # h: ** vinum Repeated for ad0s1, da0s1-da3s1. ad0 is size 16498692, da0-da3 is size 8803557 The scsi disk being smaller, I use it's size for stripe calculations. For stripe of 984s: (8803557 - 256) / 984 = 8946 (rounded to nearest whole number) 8946 * 984 = 8802864 8802864 + 256 = 8803120 which is less than total drive size so it should fit. I build a vinum config file test-config, just raid5 volume without vinum root: ### start test-config drive vinumdrive0 device /dev/ad0s1h drive vinumdrive1 device /dev/da0s1h drive vinumdrive2 device /dev/da1s1h drive vinumdrive3 device /dev/da2s1h drive vinumdrive4 device /dev/da3s1h volume data plex org raid5 984s sd drive vinumdrive0 len 8802864s driveoffset 265s sd drive vinumdrive1 len 8802864s driveoffset 265s sd drive vinumdrive2 len 8802864s driveoffset 265s sd drive vinumdrive3 len 8802864s driveoffset 265s sd drive vinumdrive4 len 8802864s driveoffset 265s ### end test-config Create the configuration: raider# vinum create test-config 5 drives: D vinumdrive0 State: up Device /dev/ad0s1h Avail: 3757/8056 MB (46%) D vinumdrive1 State: up Device /dev/da0s1h Avail: 0/4298 MB (0%) D vinumdrive2 State: up Device /dev/da1s1h Avail: 0/4298 MB (0%) D vinumdrive3 State: up Device /dev/da2s1h Avail: 0/4298 MB (0%) D vinumdrive4 State: up Device /dev/da3s1h Avail: 0/4298 MB (0%) 1 volumes: V data State: down Plexes: 1 Size: 16 GB 1 plexes: P data.p0R5 State: init Subdisks: 5 Size: 16 GB 5 subdisks: S data.p0.s0State: emptyPO:0 B Size: 4298 MB S data.p0.s1State: emptyPO: 492 kB Size: 4298 MB S data.p0.s2State: emptyPO: 984 kB Size: 4298 MB S data.p0.s3State: emptyPO: 1476 kB Size: 4298 MB S data.p0.s4State: emptyPO: 1968 kB Size: 4298 MB Initialise the plex: raider# vinum init data.p0 raider# vinum[215]: initializing subdisk /dev/vinum/sd/data.p0.s1 vinum[216]: initializing subdisk /dev/vinum/sd/data.p0.s2 vinum[217]: initializing subdisk /dev/vinum/sd/data.p0.s3 vinum[218]: initializing subdisk /dev/vinum/sd/data.p0.s4 vinum[214]: initializing subdisk /dev/vinum/sd/data.p0.s0 While waiting for my SCS
Re: vinum raid5 subdisks keep changing length?
On Tuesday, 17 February 2004 at 11:39:26 +1100, Tony Frank wrote: > On Tue, Feb 17, 2004 at 09:51:30AM +1030, Greg 'groggy' Lehey wrote: >> On Monday, 16 February 2004 at 22:04:44 +1100, Tony Frank wrote: >>> The original config looked as so: >>> >>> volume data >>> plex name data.p0 org raid5 984s vol data >>> sd name data.p0.s0 drive eightgig plex data.p0 len 8802864s driveoffset 6335201s >>> plexoffset 0s >>> sd name data.p0.s1 drive vinumdrive0 plex data.p0 len 8802864s driveoffset 265s >>> plexoffset 984s >>> sd name data.p0.s2 drive vinumdrive1 plex data.p0 len 8802864s driveoffset 265s >>> plexoffset 1968s >>> sd name data.p0.s3 drive vinumdrive2 plex data.p0 len 8802864s driveoffset 265s >>> plexoffset 2952s >>> sd name data.p0.s4 drive vinumdrive3 plex data.p0 len 8802864s driveoffset 265s >>> plexoffset 3936s >>> >>> Now after each system reboot I see messages on system console from vinum. >>> (they are not logged in vinum_history or messages) >> >> These messages should appear in /var/log/messages. Have you changed >> your syslogd.conf? > > No, default 4.9 config file, the vinum messages do not show up in 'dmesg' either. > I have changed to include the 'all.log' and 'console.log' in the syslogd.conf so > I will see if that changes this behaviour. Strange. It works fine for me: Feb 17 17:28:23 monorchid /kernel: vinum: loaded Feb 17 17:28:28 monorchid /kernel: vinum: reading configuration from /dev/da1s4h Feb 17 17:28:28 monorchid /kernel: vinum: updating configuration from /dev/da2s4h Feb 17 17:28:28 monorchid /kernel: vinum: updating configuration from /dev/da3s4h Feb 17 17:28:28 monorchid /kernel: vinum: updating configuration from /dev/da4s4h >>> vinum: updating configuration from /dev/ad0s1h >>> #(repeated for all 6 drives) >>> vinum: removing 2560 blocks of partial stripe at the end of data.p0 >> >> Yes, this is a feature, not a bug. > > While I accept this for a striped plex made up with subdisk of unequal length, > I had already taken that into account, hence the original subdisk length: > 8946 * 984 = 8802864 OK. I didn't go through the calculations. > As I understand it, this should have resulted in a whole number of > stripes. I have 5 subdisks so I'm not sure if I should have > factored that into my calculations? No, that makes no difference. >>> A "vinum printconfig" now shows difference values for the subdisks. >>> This value has changed after each reboot. >>> Current relevant parts of 'printconfig' are: >>> volume data >>> plex name data.p0 org raid5 984s vol data >>> sd name data.p0.s0 drive eightgig plex data.p0 len 8799978s driveoffset 6335201s >>> plexoffset 0s >>> sd name data.p0.s1 drive vinumdrive0 plex data.p0 len 8799486s driveoffset 265s >>> plexoffset 984s >>> sd name data.p0.s2 drive vinumdrive1 plex data.p0 len 8799158s driveoffset 265s >>> plexoffset 1968s >>> sd name data.p0.s3 drive vinumdrive2 plex data.p0 len 8798420s driveoffset 265s >>> plexoffset 2952s >>> sd name data.p0.s4 drive vinumdrive3 plex data.p0 len 8797600s driveoffset 265s >>> plexoffset 3936s OK, I tried almost exactly the same thing. My disks are fractionally smaller than yours, so I took a different integral number of stripes. I didn't get any messages, and the config now looks like: volume data plex name data.p0 org raid5 984s vol data sd name data.p0.s0 drive drive1 plex data.p0 len 8374824s driveoffset 265s plexoffset 0s sd name data.p0.s1 drive drive2 plex data.p0 len 8374824s driveoffset 265s plexoffset 984s sd name data.p0.s2 drive drive3 plex data.p0 len 8374824s driveoffset 265s plexoffset 1968s sd name data.p0.s3 drive drive4 plex data.p0 len 8374824s driveoffset 265s plexoffset 2952s sd name data.p0.s4 drive drive5 plex data.p0 len 8374824s driveoffset 265s plexoffset 3936s I've stopped and started vinum a couple of times, and all works well. I then removed the objects and tried again with subdisks 4 sectors longer. Vinum gives the message: vinum: removing 16 blocks of partial stripe at the end of data.p0 printconfig is then identical with the previous version. >> This is a bug, not a feature. I'll try to take a look at it today. > > Please advise if I can help - I have plenty of free time this week > and am willing to get my hands dirty. Hmm. Unfortunately, I don't have much time for the rest of the week. Does this mean you're not coming to the AUUG security symposium in Canberra? > Another item (which I think I read about somewhere but can no longer > find the reference) is that after a reboot the vinum drives all > reference da0h ad2h etc rather than the original value of ad2s1h > da0s1h. That in itself is not a worry. It's the way Vinum works (the two are equivalent). > Also after the reboot, each drive is shown as 100% free which is a > concern. That is indeed a concern. There's obviously something going on here which isn't immediately obvious. Take a look at http://www.vinumvm.org/vinum/how-to-debug.html and send me the information I as
Re: vinum raid5 subdisks keep changing length?
Hi, On Tue, Feb 17, 2004 at 09:51:30AM +1030, Greg 'groggy' Lehey wrote: > On Monday, 16 February 2004 at 22:04:44 +1100, Tony Frank wrote: > > Running vinum on RELENG_4. > > > > No changes to source, vinum loaded as a KLD as per instructions in handbook & > > man pages. > > > > As per the subject I am having what appears to be strange behaviour with vinum. > > > > After a number of teething problems I eventually setup a vinum root for > > my system which covers / swap /var and /usr. > > This works just fine, though I am yet to experiment with failure scenarios. > > > > However I also configured a raid5 volume and may be having issues with it. > > Currently just a test system and no obvious data loss but I have some > > concerns before I put it into production. > > > Specifically I originally configured five subdisks (on different > > drives) with identical lengths and then combined them into a raid5 > > plex. This worked ok and I successfully initialised the plex and > > used newfs on it. Successfully able to read&write to the volume > > (1G/14G used at the moment) > > > > The original config looked as so: > > > > volume data > > plex name data.p0 org raid5 984s vol data > > sd name data.p0.s0 drive eightgig plex data.p0 len 8802864s driveoffset 6335201s > > plexoffset 0s > > sd name data.p0.s1 drive vinumdrive0 plex data.p0 len 8802864s driveoffset 265s > > plexoffset 984s > > sd name data.p0.s2 drive vinumdrive1 plex data.p0 len 8802864s driveoffset 265s > > plexoffset 1968s > > sd name data.p0.s3 drive vinumdrive2 plex data.p0 len 8802864s driveoffset 265s > > plexoffset 2952s > > sd name data.p0.s4 drive vinumdrive3 plex data.p0 len 8802864s driveoffset 265s > > plexoffset 3936s > > > > Now after each system reboot I see messages on system console from vinum. > > (they are not logged in vinum_history or messages) > > These messages should appear in /var/log/messages. Have you changed > your syslogd.conf? No, default 4.9 config file, the vinum messages do not show up in 'dmesg' either. I have changed to include the 'all.log' and 'console.log' in the syslogd.conf so I will see if that changes this behaviour. > > vinum: updating configuration from /dev/ad0s1h > > #(repeated for all 6 drives) > > vinum: removing 2560 blocks of partial stripe at the end of data.p0 > > Yes, this is a feature, not a bug. While I accept this for a striped plex made up with subdisk of unequal length, I had already taken that into account, hence the original subdisk length: 8946 * 984 = 8802864 As I understand it, this should have resulted in a whole number of stripes. I have 5 subdisks so I'm not sure if I should have factored that into my calculations? > > A "vinum printconfig" now shows difference values for the subdisks. > > This value has changed after each reboot. > > Current relevant parts of 'printconfig' are: > > > > volume data > > plex name data.p0 org raid5 984s vol data > > sd name data.p0.s0 drive eightgig plex data.p0 len 8799978s driveoffset 6335201s > > plexoffset 0s > > sd name data.p0.s1 drive vinumdrive0 plex data.p0 len 8799486s driveoffset 265s > > plexoffset 984s > > sd name data.p0.s2 drive vinumdrive1 plex data.p0 len 8799158s driveoffset 265s > > plexoffset 1968s > > sd name data.p0.s3 drive vinumdrive2 plex data.p0 len 8798420s driveoffset 265s > > plexoffset 2952s > > sd name data.p0.s4 drive vinumdrive3 plex data.p0 len 8797600s driveoffset 265s > > plexoffset 3936s > > This is a bug, not a feature. > I'll try to take a look at it today. Please advise if I can help - I have plenty of free time this week and am willing to get my hands dirty. Another item (which I think I read about somewhere but can no longer find the reference) is that after a reboot the vinum drives all reference da0h ad2h etc rather than the original value of ad2s1h da0s1h. Also after the reboot, each drive is shown as 100% free which is a concern. Thanks, Tony ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: vinum raid5 subdisks keep changing length?
On Monday, 16 February 2004 at 22:04:44 +1100, Tony Frank wrote: > Hi all, > > Running vinum on RELENG_4. > > No changes to source, vinum loaded as a KLD as per instructions in handbook & > man pages. > > As per the subject I am having what appears to be strange behaviour with vinum. > > After a number of teething problems I eventually setup a vinum root for > my system which covers / swap /var and /usr. > This works just fine, though I am yet to experiment with failure scenarios. > > However I also configured a raid5 volume and may be having issues with it. > Currently just a test system and no obvious data loss but I have some > concerns before I put it into production. > Specifically I originally configured five subdisks (on different > drives) with identical lengths and then combined them into a raid5 > plex. This worked ok and I successfully initialised the plex and > used newfs on it. Successfully able to read&write to the volume > (1G/14G used at the moment) > > The original config looked as so: > > volume data > plex name data.p0 org raid5 984s vol data > sd name data.p0.s0 drive eightgig plex data.p0 len 8802864s driveoffset 6335201s > plexoffset 0s > sd name data.p0.s1 drive vinumdrive0 plex data.p0 len 8802864s driveoffset 265s > plexoffset 984s > sd name data.p0.s2 drive vinumdrive1 plex data.p0 len 8802864s driveoffset 265s > plexoffset 1968s > sd name data.p0.s3 drive vinumdrive2 plex data.p0 len 8802864s driveoffset 265s > plexoffset 2952s > sd name data.p0.s4 drive vinumdrive3 plex data.p0 len 8802864s driveoffset 265s > plexoffset 3936s > > Now after each system reboot I see messages on system console from vinum. > (they are not logged in vinum_history or messages) These messages should appear in /var/log/messages. Have you changed your syslogd.conf? > vinum: updating configuration from /dev/ad0s1h > #(repeated for all 6 drives) > vinum: removing 2560 blocks of partial stripe at the end of data.p0 Yes, this is a feature, not a bug. > A "vinum printconfig" now shows difference values for the subdisks. > > This value has changed after each reboot. > > Current relevant parts of 'printconfig' are: > > volume data > plex name data.p0 org raid5 984s vol data > sd name data.p0.s0 drive eightgig plex data.p0 len 8799978s driveoffset 6335201s > plexoffset 0s > sd name data.p0.s1 drive vinumdrive0 plex data.p0 len 8799486s driveoffset 265s > plexoffset 984s > sd name data.p0.s2 drive vinumdrive1 plex data.p0 len 8799158s driveoffset 265s > plexoffset 1968s > sd name data.p0.s3 drive vinumdrive2 plex data.p0 len 8798420s driveoffset 265s > plexoffset 2952s > sd name data.p0.s4 drive vinumdrive3 plex data.p0 len 8797600s driveoffset 265s > plexoffset 3936s This is a bug, not a feature. I'll try to take a look at it today. Greg -- When replying to this message, please copy the original recipients. If you don't, I may ignore the reply or reply to the original recipients. For more information, see http://www.lemis.com/questions.html See complete headers for address and phone numbers. pgp0.pgp Description: PGP signature
