Re: Rebuild after disk fail
On Sat, Jan 18, 2020 at 02:14:46PM +1100, Andrew McGlashan wrote: > Just some thoughts > > Way back, SSDs were expensive and less reliable than today. > > Given the cost of SSDs today, I would consider even RAIDING the SSDs. If it's physically possible to install a second SSD of the same storage capacity or larger then he absolutely should do so. I vaguely recall suggesting he should get a second SSD for the rootfs ages ago, but my understanding / assumption was that there was only physical space and connectors for one SSD in the machine. The 'btrfs snapshot' + 'btrfs send' suggestion was just a way of regularly backing up a single-drive btrfs filesystem onto his raid-1 btrfs array so that little or nothing was lost in case of another drive failure. It's less than ideal, but a LOT better than nothing. I personally would never use anything less than RAID-1 (or equivalent, such as a mirrored pair on zfs) for any storage. Which means, of course, that I'm used to paying double for my storage capacity - i can't just buy one, I have to buy a pair. Not as a substitute for regular backups, but for convenience when only one drive of a pair has died. Drives die, and the time & inconvenience of dealing with that (and the lost data) cost far more than the price of a second drive for raid-1/mirror. craig -- craig sanders ___ luv-main mailing list luv-main@luv.asn.au https://lists.luv.asn.au/cgi-bin/mailman/listinfo/luv-main
Re: Rebuild after disk fail
On Sat, Jan 18, 2020 at 01:41:05PM +1100, Andrew Greig wrote: > alg@andrewg:~$ sudo cat /etc/fstab > [sudo] password for alg: > # /etc/fstab: static file system information. > # > # Use 'blkid' to print the universally unique identifier for a > # device; this may be used with UUID= as a more robust way to name devices > # that works even if disks are added and removed. See fstab(5). > # > # > # / was on /dev/sda3 during installation > UUID=2dfcd965-625b-47d5-a267-b02276320922 / btrfs > defaults,subvol=@ 0 1 > # /home was on /dev/sda3 during installation > UUID=2dfcd965-625b-47d5-a267-b02276320922 /home btrfs > defaults,subvol=@home 0 2 > # swap was on /dev/sda2 during installation > UUID=b2c6d1c4-4b94-4171-954e-9f5d56704514 none swap sw > 0 0 > alg@andrewg:~$ > > Are these two following commands OK to apply to drives that were balanced > previously and hold data? > > sudo btrfs device add -f /dev/sdc1 /data > > sudo btrfs ballance start -dconvert=raid1 -mconvert=raid1 /data > No, don't run any of those commands, especially the 'btrfs add' command - you will destroy your existing data array if you run that. Run blkid to list all attached block devices. figure out which one of them is your data array and add an entry if you can't figure out which is the correct one, reply and include blkid's output. > and will issuing those commands write that into fstab? no. craig -- craig sanders BOFH excuse #376: Budget cuts forced us to sell all the power cords for the servers. ___ luv-main mailing list luv-main@luv.asn.au https://lists.luv.asn.au/cgi-bin/mailman/listinfo/luv-main
Re: Rebuild after disk fail
On Sat, Jan 18, 2020 at 01:20:39PM +1100, pushin.linux wrote: > I have elected to start with a Ubuntu 18.04 LTS desktop install.The Raid > drives were picked up, ie are available, but does the balance command need > to be issued again? You only need to run 'btrfs balance' when you're changing the number and/or size of drives (or partitions) in the btrfs array. The command re-balances all of the data on the array, roughly-equally across all the drives. So, if you're not adding drives to the array, you don't need to re-balance it. (btw, 'btrfs balance' is the one feature that btrfs has that I wish zfs had) > I had two lines to set up the raid and balance them at the start. IIRC, I think I advised you to do something like: 1. create a degraded btrfs array with just one of the drives; 2. copy your data to it; 3. add another drive to the btrfs array with 'btrfs add'; 4. re-balance the data so that it's on both drives with 'btrfs balance'. If so, that'll be why you have two commands written down. > I suspect that without those commands only one drive will be written to. nope. This time around, your btrfs array for /data ALREADY EXISTS, so you don't have to do any of that. And you certainly SHOULD NOT run mkfs.btrfs, that would erase your current btrfs array and re-format it. All you need to do this time is add an entry to /etc/fstab so that it mounts correctly on boot. Something like the following: UUID="c0483385-ca6f-abb3-aeeb-94793439a637" /databtrfs defaults,relatime 0 0 run 'blkid' to find the correct uuid for your /data fs and use it instead of the bogus one in the example above. craig -- craig sanders ___ luv-main mailing list luv-main@luv.asn.au https://lists.luv.asn.au/cgi-bin/mailman/listinfo/luv-main
Re: Rebuild after disk fail
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Hi, On 18/1/20 2:14 pm, Andrew McGlashan via luv-main wrote: > btrfs -- I never, ever considered that to be real production ready > and I believe that even dead hat has moved away from it somewhat > (not sure to what extent). Some links, none of which are new as this occurred some time ago now. https://www.theregister.co.uk/2017/08/16/red_hat_banishes_btrfs_from_rhe l https://www.marksei.com/red-hat-deprecates-btrfs-stratis/ https://news.ycombinator.com/item?id=14907771 Oh and one newer link, fwiw. https://access.redhat.com/discussions/3138231 Cheers A. -BEGIN PGP SIGNATURE- iHUEAREIAB0WIQTJAoMHtC6YydLfjUOoFmvLt+/i+wUCXiJ81QAKCRCoFmvLt+/i +yxIAQCwTtPSBOsBsJzf/yvs7j+PtVNSgoj2qELV0KbaM7NUUgEA1rBUQAsEdAeC lnxXo58Aw7lE7Qn6M5NzgZxXnbo2R4o= =gI1b -END PGP SIGNATURE- ___ luv-main mailing list luv-main@luv.asn.au https://lists.luv.asn.au/cgi-bin/mailman/listinfo/luv-main
Re: Rebuild after disk fail
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Hi, Just some thoughts Way back, SSDs were expensive and less reliable than today. Given the cost of SSDs today, I would consider even RAIDING the SSDs. btrfs -- I never, ever considered that to be real production ready and I believe that even dead hat has moved away from it somewhat (not sure to what extent). If you are storing photos and videos, then absolute data integrity might not be an issue, but what happens if you need recovery with btrfs failures of any kind? I would think you would be in trouble and will need plenty of help with this. I like the idea of btrfs, but prefer zfs, in any case I just use ext4 over encrypted RAID volumes. Cheers A -BEGIN PGP SIGNATURE- iHUEAREIAB0WIQTJAoMHtC6YydLfjUOoFmvLt+/i+wUCXiJ4IQAKCRCoFmvLt+/i +2tMAP9Yn9xMOJAB3Vvo2T1tEFEn2M2vOoLNh97oTl1DSJ2hNgD/QtMnQeKx0/79 5b2T9NXtjSnd6cTzwp18R6ulBE1Y8ss= =6lJW -END PGP SIGNATURE- ___ luv-main mailing list luv-main@luv.asn.au https://lists.luv.asn.au/cgi-bin/mailman/listinfo/luv-main
Re: Rebuild after disk fail
Hi All, Here is my fstab after the install, it seems that my two "RAID" drives are just "dwellers on the threshold" as they do not appear in fstab. alg@andrewg:~$ sudo cat /etc/fstab [sudo] password for alg: # /etc/fstab: static file system information. # # Use 'blkid' to print the universally unique identifier for a # device; this may be used with UUID= as a more robust way to name devices # that works even if disks are added and removed. See fstab(5). # # # / was on /dev/sda3 during installation UUID=2dfcd965-625b-47d5-a267-b02276320922 / btrfs defaults,subvol=@ 0 1 # /home was on /dev/sda3 during installation UUID=2dfcd965-625b-47d5-a267-b02276320922 /home btrfs defaults,subvol=@home 0 2 # swap was on /dev/sda2 during installation UUID=b2c6d1c4-4b94-4171-954e-9f5d56704514 none swap sw 0 0 alg@andrewg:~$ Are these two following commands OK to apply to drives that were balanced previously and hold data? sudo btrfs device add -f /dev/sdc1 /data sudo btrfs ballance start -dconvert=raid1 -mconvert=raid1 /data and will issuing those commands write that into fstab? Many thanks Andrew On 18/1/20 12:59 pm, Craig Sanders via luv-main wrote: On Fri, Jan 17, 2020 at 11:36:29AM +1100, Andrew Greig wrote: I recently experienced an SSD failure, and so I have purchased another to set up my system again. I received some substantial help from this list early in 2019 to build my machine with this SSD as / and /home under Ubuntu 18.04 with two x 2Tb conventional drives in RAID for storing my work, all are running btrfs. You lost your home dir and the data in it when your SSD failed Because your rootfs and /home on the SSD doesn't have any redundancy (i.e. it was a single partition, with no RAID). I strongly recommend setting up a cron job to regularly snapshot it (at least once/day) and do a 'btrfs send' of that snapshot to a sub-volume of your /data filesystem. That way you won't lose much data from that partition if your SSD dies again - you can retrieve it from the last snapshot backup, and will only lose any changes since then. If your / and /home are on separate partitions (or btrfs sub-volumes) you will need to do this for both of them. (if you weren't running btrfs on /, you could do this with rsync instead of 'btrfs send', but rsync would be a lot slower) IME, drives are fragile and prone to failure. It's always best to make plans and backup procedures so that WHEN (not IF) a drive fails, you don't lose anything important...or, at least, minimise your losses. Also, remember that RAID is not a substitute for backup so you should regularly backup your /data filesystem to tape or other drives. Ideally, you should try to have an off-site backup in case of fire/flood/etc (e.g. backup to an external USB drive and store it at your office, lawyer's safe, a friend's house or somewhere. Have at least two of these so you can rotate the offsite backups). After the machine was running I was asked if I had set up the machine using Ubuntu Server, I hadn't, because at that time I didn't see those options. I am thinking, then, for this build, perhaps I should set it up using Ubuntu Server. I will need to get my system to recognise the RAID drives as well. If the installer doesn't automatically detect your /data btrfs filesystem and add it to /etc/fstab, it's easy enough to add it yourself. So before I jump in the deep end again, are there any "gotchas" of which I should be aware. Will the server version make life more reliable? the only significant difference between the server and desktop versions of ubuntu are the packages which are installed by default. e.g. the desktop version installs a whole bunch of desktop stuff (X, desktop environment and GUI apps, etc) that the server version doesn't. Otherwise, they're the same - same kernel, same libc and other standard system libraries, etc. craig -- craig sanders ___ luv-main mailing list luv-main@luv.asn.au https://lists.luv.asn.au/cgi-bin/mailman/listinfo/luv-main ___ luv-main mailing list luv-main@luv.asn.au https://lists.luv.asn.au/cgi-bin/mailman/listinfo/luv-main
Re: Rebuild after disk fail
Thanks Craig,I have elected to start with a Ubuntu 18.04 LTS desktop install.The Raid drives were picked up, ie are available, but does the balance command need to be issued again?I had two lines to set up the raid and balance them at the start. I suspect that without those commands only one drive will be written to.Thanks for your assistanceAndrewSent from Samsung tablet. Original message From: Craig Sanders via luv-main Date: 18/1/20 12:59 pm (GMT+10:00) To: luv-main@luv.asn.au Subject: Re: Rebuild after disk fail On Fri, Jan 17, 2020 at 11:36:29AM +1100, Andrew Greig wrote:> I recently experienced an SSD failure, and so I have purchased another to> set up my system again. I received some substantial help from this list> early in 2019 to build my machine with this SSD as / and /home under Ubuntu> 18.04 with two x 2Tb conventional drives in RAID for storing my work, all> are running btrfs.You lost your home dir and the data in it when your SSD failed Because yourrootfs and /home on the SSD doesn't have any redundancy (i.e. it was a singlepartition, with no RAID). I strongly recommend setting up a cron job toregularly snapshot it (at least once/day) and do a 'btrfs send' of thatsnapshot to a sub-volume of your /data filesystem.That way you won't lose much data from that partition if your SSD dies again- you can retrieve it from the last snapshot backup, and will only lose anychanges since then.If your / and /home are on separate partitions (or btrfs sub-volumes) you willneed to do this for both of them.(if you weren't running btrfs on /, you could do this with rsync instead of'btrfs send', but rsync would be a lot slower)IME, drives are fragile and prone to failure. It's always best to make plansand backup procedures so that WHEN (not IF) a drive fails, you don't loseanything important...or, at least, minimise your losses.Also, remember that RAID is not a substitute for backup so you shouldregularly backup your /data filesystem to tape or other drives. Ideally,you should try to have an off-site backup in case of fire/flood/etc (e.g.backup to an external USB drive and store it at your office, lawyer's safe, afriend's house or somewhere. Have at least two of these so you can rotate theoffsite backups).> After the machine was running I was asked if I had set up the machine using> Ubuntu Server, I hadn't, because at that time I didn't see those options.>> I am thinking, then, for this build, perhaps I should set it up using Ubuntu> Server. I will need to get my system to recognise the RAID drives as well.If the installer doesn't automatically detect your /data btrfs filesystem andadd it to /etc/fstab, it's easy enough to add it yourself.> So before I jump in the deep end again, are there any "gotchas" of which I> should be aware.>> Will the server version make life more reliable?the only significant difference between the server and desktop versions ofubuntu are the packages which are installed by default. e.g. the desktopversion installs a whole bunch of desktop stuff (X, desktop environment andGUI apps, etc) that the server version doesn't. Otherwise, they're the same -same kernel, same libc and other standard system libraries, etc.craig--craig sanders ___luv-main mailing listluv-m...@luv.asn.auhttps://lists.luv.asn.au/cgi-bin/mailman/listinfo/luv-main___ luv-main mailing list luv-main@luv.asn.au https://lists.luv.asn.au/cgi-bin/mailman/listinfo/luv-main
Re: Rebuild after disk fail
On Fri, Jan 17, 2020 at 11:36:29AM +1100, Andrew Greig wrote: > I recently experienced an SSD failure, and so I have purchased another to > set up my system again. I received some substantial help from this list > early in 2019 to build my machine with this SSD as / and /home under Ubuntu > 18.04 with two x 2Tb conventional drives in RAID for storing my work, all > are running btrfs. You lost your home dir and the data in it when your SSD failed Because your rootfs and /home on the SSD doesn't have any redundancy (i.e. it was a single partition, with no RAID). I strongly recommend setting up a cron job to regularly snapshot it (at least once/day) and do a 'btrfs send' of that snapshot to a sub-volume of your /data filesystem. That way you won't lose much data from that partition if your SSD dies again - you can retrieve it from the last snapshot backup, and will only lose any changes since then. If your / and /home are on separate partitions (or btrfs sub-volumes) you will need to do this for both of them. (if you weren't running btrfs on /, you could do this with rsync instead of 'btrfs send', but rsync would be a lot slower) IME, drives are fragile and prone to failure. It's always best to make plans and backup procedures so that WHEN (not IF) a drive fails, you don't lose anything important...or, at least, minimise your losses. Also, remember that RAID is not a substitute for backup so you should regularly backup your /data filesystem to tape or other drives. Ideally, you should try to have an off-site backup in case of fire/flood/etc (e.g. backup to an external USB drive and store it at your office, lawyer's safe, a friend's house or somewhere. Have at least two of these so you can rotate the offsite backups). > After the machine was running I was asked if I had set up the machine using > Ubuntu Server, I hadn't, because at that time I didn't see those options. > > I am thinking, then, for this build, perhaps I should set it up using Ubuntu > Server. I will need to get my system to recognise the RAID drives as well. If the installer doesn't automatically detect your /data btrfs filesystem and add it to /etc/fstab, it's easy enough to add it yourself. > So before I jump in the deep end again, are there any "gotchas" of which I > should be aware. > > Will the server version make life more reliable? the only significant difference between the server and desktop versions of ubuntu are the packages which are installed by default. e.g. the desktop version installs a whole bunch of desktop stuff (X, desktop environment and GUI apps, etc) that the server version doesn't. Otherwise, they're the same - same kernel, same libc and other standard system libraries, etc. craig -- craig sanders ___ luv-main mailing list luv-main@luv.asn.au https://lists.luv.asn.au/cgi-bin/mailman/listinfo/luv-main
