Re: how best to do

[Gene Heskett]

On Wednesday 28 July 2021 23:32:11 Keith Bainbridge wrote:

> On 29/7/21 02:30, Dan Ritter wrote:
> > and mount where you will.
>
> G'day Gene
>
> I make back-up disks at fstab options
> noauto,noexec
>
> and mount/unmount as part of the cron script.  makes them 'less
> visible' to an uninvited visitor.
>
> Also, I run the back-up as root

I considered that when I started using amanda 20+ years ago, but have 
thanks to dd-wrt in my router(s) I have never had the unwanted visitor. 
This does have some exposure because this is also my web server, but I 
do keep iptables up to date on blocking the bots that burn up my dsl 
limited upload bandwidth mirroring my site. IMO, amanda's security model 
is better than root. 

Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis
Genes Web page 

Re: how best to do

[Keith Bainbridge]

On 29/7/21 02:30, Dan Ritter wrote:

and mount where you will.


G'day Gene

I make back-up disks at fstab options
noauto,noexec

and mount/unmount as part of the cron script.  makes them 'less visible' 
to an uninvited visitor.


Also, I run the back-up as root

--
All the best

Keith Bainbridge

keith.bainbridge.3...@gmail.com
0447 667 468

Re: how best to do

[piorunz]

On 28/07/2021 17:29, Charles Curley wrote:

I
plan to add drives to upgrade the RAID to add checksums, and that will
involve buying a SATA controller.


Btrfs filesystem does that by default, no need to buy any additional
hardware. It also does fake-software Raids on it's own without need for
mdadm or HW controller.

--

With kindest regards, piorunz.

⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ Debian - The universal operating system
⢿⡄⠘⠷⠚⠋⠀ https://www.debian.org
⠈⠳⣄

Re: how best to do

[Gene Heskett]

On Wednesday 28 July 2021 19:26:15 Greg Wooledge wrote:

> On Wed, Jul 28, 2021 at 11:03:01PM +0200, deloptes wrote:
> > Dan Ritter wrote:
> > >> Raid6 sounds promising. Equivalent capacity is also one drive?
> > >
> > > No, you actually get about 2 drives of capacity out of 4 here.
> >
> > You get N-1, so from 4x2TB disks you get close to 6TB RAID6 (you
> > have also various FS layer overhead)
>
> RAID-5 is N-1.  With 4 disks in a RAID-5 configuration, you'd get 3
> disks worth of usable storage.
>
> RAID-6 is N-2.  With 4 disks in a RAID-6 configuration, you'd get 2
> disks worth of usable storage.

Thanks Greg. Raid-6 sounds the most dependable.  Stuff is ordered.

Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis
Genes Web page 

Re: how best to do

[deloptes]

Greg Wooledge wrote:

> RAID-5 is N-1.  With 4 disks in a RAID-5 configuration, you'd get 3
> disks worth of usable storage.
> 
> RAID-6 is N-2.  With 4 disks in a RAID-6 configuration, you'd get 2
> disks worth of usable storage.

Sorry I mixed both :/

Re: how best to do

[Greg Wooledge]

On Wed, Jul 28, 2021 at 11:03:01PM +0200, deloptes wrote:
> Dan Ritter wrote:
> >> Raid6 sounds promising. Equivalent capacity is also one drive?
> > 
> > No, you actually get about 2 drives of capacity out of 4 here.
> 
> You get N-1, so from 4x2TB disks you get close to 6TB RAID6 (you have also
> various FS layer overhead)

RAID-5 is N-1.  With 4 disks in a RAID-5 configuration, you'd get 3
disks worth of usable storage.

RAID-6 is N-2.  With 4 disks in a RAID-6 configuration, you'd get 2
disks worth of usable storage.

Re: how best to do

[David Christensen]

On 7/28/21 6:44 AM, Gene Heskett wrote:

Greetings all;

This, my main machine that backs up 5 others here, but does that with
drives not involved with daily stuffs, and of course I'm gradually
replacing spinning rust with SSD's.

I've been dragging my feet on updating from stretch to buster, waiting
for 1TB SSD's to get affordable. Now they almost have.

But I do have a pair of 500 GB SamSung EVO 860's and one empty sata
socket. Spares for the rest of my multi-machine farm, currently at 5
other machines.

Total storage actually used on this machine ATM is around 318G. Some of
which isn't precious.

With 2 drives and 2 sata connectors, how can I configure this for maximum
redundancy, but not necessarily maximum working capacity?

Does that recommendation change if I order 2 more 1T SSD's? In which
case, which of the current buzzwords tech is the most dependable?

Thank you.
  
Cheers, Gene Heskett



I suggest creating a ZFS mirrored storage pool using the 500 GB SSD's:

https://docs.oracle.com/cd/E19253-01/819-5461/gaynr/index.html


Later, you could add mirrored 1 TB SSD's to the pool:

https://docs.oracle.com/cd/E37838_01/html/E61017/gayrd.html


And remove the 500 GB SSD mirror from the pool:

https://docs.oracle.com/cd/E37838_01/html/E61017/remove-devices.html


David

Re: how best to do

[deloptes]

Dan Ritter wrote:

>> Raid6 sounds promising. Equivalent capacity is also one drive?
> 
> No, you actually get about 2 drives of capacity out of 4 here.


You get N-1, so from 4x2TB disks you get close to 6TB RAID6 (you have also
various FS layer overhead)

Gene, IMO this is best option but I do not remember what was faster read or
write - there was a catch, so depends on the usecase. I did on one I use
for backup

On storage server for HA I bought 2 LSI controllers that manage 2x6 3.5"
disk bays - for now I have 4x4 disks that are equally distributed and in
RAID1 - might be the 2 LSI was overkill and a limitation into some extent,
but ... if one of them dies, the other will continue serve until
replacement is there (and I still use 1Gbps copper which is the actual
limitation for NFS SSH Samba or whatever goes trough)

regards

PS: just think twice, cause later it's harder to manage the amounts of data

Re: how best to do

[tomas]

On Wed, Jul 28, 2021 at 10:16:19AM -0600, Charles Curley wrote:
> On Wed, 28 Jul 2021 11:04:59 -0400
> Dan Ritter  wrote:
> 
> > However, if you don't need second-by-second simultaneity, you
> > could just set up a cron job to rsync your first drive to your
> > second drive once an hour or twice a day or overnight. 
> 
> Or use rsnapshot to get the same effect with less effort.

RAID and rsync/rsnapshot have different profiles (although they share,
of course, some traits).

The first is about availability: if one of your disks die, you can
just go on as if nothing happened. Take it out (as far as your
hardware supports hot swap ;-) , pop a fresh one in and let the
RAID fairy restore redundancy.

With the second, if what dies is your backup, all fine. If it's
your primary disk, you'll find yourself scuttling to set up a
new system and get the data in.

OTOH, with the second, you can set up things to stash away several
snapshots of your system. If you fat-finger your valuable data away,
you just can page back in your snapshot history until you find good
data (attention Windows users, with all that ransomware which seems
to be fashionable with you ;-)

Rsync provides ways to store snapshots in a reasonably storage-friendly
way (option --link-dest). Rsnapshot (which, AFAIK is based on rsync)
makes this a bit easier.

In a nutshell: what old sysadmins always say: RAID and backup are
different things. You want the first if you need high availability,
you want the second if your data is valuable.

> Does anybody read signatures any more?

I /never/ do.

Cheers
 - t


signature.asc
Description: Digital signature

Re: how best to do

[Gene Heskett]

On Wednesday 28 July 2021 12:29:42 Charles Curley wrote:

> On Wed, 28 Jul 2021 09:44:24 -0400
>
> Gene Heskett  wrote:
> > I've been dragging my feet on updating from stretch to buster,
> > waiting for 1TB SSD's to get affordable. Now they almost have.
> >
> > But I do have a pair of 500 GB SamSung EVO 860's and one empty sata
> > socket. Spares for the rest of my multi-machine farm, currently at 5
> > other machines.
>
> I have a setup similar to yours. However, I use SSDs for stuff I want
> readily available (OS, programs, /home, etc.).
>
> On my main computer, I have two 4TB spinning rust drives, with RAID0
> on them. I keep stuff on them where speed isn't as big a deal:
> backups, etc. Backups include amanda backups, rsnapshot backups, and
> others. I plan to add drives to upgrade the RAID to add checksums, and
> that will involve buying a SATA controller.
>
> If money is a concern, something like that might work for you.

ATM, it is not. Thanks Charles.

Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis
Genes Web page 

Re: how best to do

[Charles Curley]

On Wed, 28 Jul 2021 11:04:59 -0400
Dan Ritter  wrote:

> However, if you don't need second-by-second simultaneity, you
> could just set up a cron job to rsync your first drive to your
> second drive once an hour or twice a day or overnight. 

Or use rsnapshot to get the same effect with less effort.

-- 
Does anybody read signatures any more?

https://charlescurley.com
https://charlescurley.com/blog/

Re: how best to do

[Dan Ritter]

Gene Heskett wrote: 
> On Wednesday 28 July 2021 11:04:59 Dan Ritter wrote:
> > The standard SATA interface is one port, one drive. (There are
> > exceptions which are not worth talking about here.) If you can
> > plug in a fairly cheap PCIe to SATA card, you can get 2 or 4 or
> > 8 more connectors.
> >
> I have spares of both the very short slot and the longer slot, whatever 
> it is, this is an Asus Prime X370-A II motherboard, so name the 
> preferred poison for a new sata card.
>  
> > Your choices for 4 drives are:
> >
> > RAID10: write 2 copies to four drives, read from all four;
> > capacity is 2 drives worth. Survive any one drive failing and
> > also survive two drives failing if they are on different
> > stripes.
> >
> > RAID6: write one copy plus two sets of parity information to
> > four drives, read from all four to reconstruct data. Speed is
> > equivalent to one drive. You can survive any two drives failing.
> 
> Raid6 sounds promising. Equivalent capacity is also one drive?

No, you actually get about 2 drives of capacity out of 4 here.


> > All of these can be handled in software by the kernel, managed
> > by mdadm. You can easily transfer the drives to some other Linux
> > box.
> 
> Name the poison card, and I'll get it and 4 more drives as this tower has 
> lots of empty drive space suitable for hiding SSD's.

$20 for 2 ports:
https://www.newegg.com/syba-sy-pex40039-sata-iii/p/N82E16816124045

$80 for 6 ports:
https://www.newegg.com/p/14G-000G-00089?Item=14G-000G-00089

No drivers needed, kernel should just recognize them.

Plug in the drives and read https://wiki.debian.org/SoftwareRAID

You'll end up with something like

--create /dev/md0 --level=6 --raid-devices=4 /dev/sdc1 /dev/sdd1
/dev/sde1 /dev/sdf1

Then 

mkfs.ext4 /dev/md0

and mount where you will.


-dsr-

Re: how best to do

[Charles Curley]

On Wed, 28 Jul 2021 09:44:24 -0400
Gene Heskett  wrote:

> I've been dragging my feet on updating from stretch to buster,
> waiting for 1TB SSD's to get affordable. Now they almost have.
> 
> But I do have a pair of 500 GB SamSung EVO 860's and one empty sata 
> socket. Spares for the rest of my multi-machine farm, currently at 5 
> other machines.

I have a setup similar to yours. However, I use SSDs for stuff I want
readily available (OS, programs, /home, etc.).

On my main computer, I have two 4TB spinning rust drives, with RAID0 on
them. I keep stuff on them where speed isn't as big a deal: backups,
etc. Backups include amanda backups, rsnapshot backups, and others. I
plan to add drives to upgrade the RAID to add checksums, and that will
involve buying a SATA controller.

If money is a concern, something like that might work for you.

-- 
Does anybody read signatures any more?

https://charlescurley.com
https://charlescurley.com/blog/

Re: how best to do

[Gene Heskett]

On Wednesday 28 July 2021 11:04:59 Dan Ritter wrote:
on list, I am subbed.

> Gene Heskett wrote:
> > Greetings all;
> >
> > This, my main machine that backs up 5 others here, but does that
> > with drives not involved with daily stuffs, and of course I'm
> > gradually replacing spinning rust with SSD's.
> >
> > I've been dragging my feet on updating from stretch to buster,
> > waiting for 1TB SSD's to get affordable. Now they almost have.
> >
> > But I do have a pair of 500 GB SamSung EVO 860's and one empty sata
> > socket. Spares for the rest of my multi-machine farm, currently at 5
> > other machines.
> >
> > Total storage actually used on this machine ATM is around 318G. Some
> > of which isn't precious.
> >
> > With 2 drives and 2 sata connectors, how can I configure this for
> > maximum redundancy, but not necessarily maximum working capacity?
>
> Best redundancy is a RAID-1 mirror: everything written to one is
> also written to the other; reads are handled by both
> simultaneously. You get about the same write performance, much
> better read performance, and one of them dying doesn't kill your
> data.
>
> However, if you don't need second-by-second simultaneity, you
> could just set up a cron job to rsync your first drive to your
> second drive once an hour or twice a day or overnight.
>
> > Does that recommendation change if I order 2 more 1T SSD's? In which
> > case, which of the current buzzwords tech is the most dependable?
>
> Do you have an available PCI or PCIe slot?
>
> The standard SATA interface is one port, one drive. (There are
> exceptions which are not worth talking about here.) If you can
> plug in a fairly cheap PCIe to SATA card, you can get 2 or 4 or
> 8 more connectors.
>
I have spares of both the very short slot and the longer slot, whatever 
it is, this is an Asus Prime X370-A II motherboard, so name the 
preferred poison for a new sata card.
 
> Your choices for 4 drives are:
>
> RAID10: write 2 copies to four drives, read from all four;
> capacity is 2 drives worth. Survive any one drive failing and
> also survive two drives failing if they are on different
> stripes.
>
> RAID6: write one copy plus two sets of parity information to
> four drives, read from all four to reconstruct data. Speed is
> equivalent to one drive. You can survive any two drives failing.

Raid6 sounds promising. Equivalent capacity is also one drive?
>
> All of these can be handled in software by the kernel, managed
> by mdadm. You can easily transfer the drives to some other Linux
> box.

Name the poison card, and I'll get it and 4 more drives as this tower has 
lots of empty drive space suitable for hiding SSD's.

> -dsr-

Thanks dsr.

Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis
Genes Web page 

Re: how best to do

[Dan Ritter]

Gene Heskett wrote: 
> Greetings all;
> 
> This, my main machine that backs up 5 others here, but does that with 
> drives not involved with daily stuffs, and of course I'm gradually 
> replacing spinning rust with SSD's.
> 
> I've been dragging my feet on updating from stretch to buster, waiting 
> for 1TB SSD's to get affordable. Now they almost have.
> 
> But I do have a pair of 500 GB SamSung EVO 860's and one empty sata 
> socket. Spares for the rest of my multi-machine farm, currently at 5 
> other machines.
> 
> Total storage actually used on this machine ATM is around 318G. Some of 
> which isn't precious.
> 
> With 2 drives and 2 sata connectors, how can I configure this for maximum 
> redundancy, but not necessarily maximum working capacity?

Best redundancy is a RAID-1 mirror: everything written to one is
also written to the other; reads are handled by both
simultaneously. You get about the same write performance, much
better read performance, and one of them dying doesn't kill your
data.

However, if you don't need second-by-second simultaneity, you
could just set up a cron job to rsync your first drive to your
second drive once an hour or twice a day or overnight. 


> Does that recommendation change if I order 2 more 1T SSD's? In which 
> case, which of the current buzzwords tech is the most dependable?

Do you have an available PCI or PCIe slot?

The standard SATA interface is one port, one drive. (There are
exceptions which are not worth talking about here.) If you can
plug in a fairly cheap PCIe to SATA card, you can get 2 or 4 or
8 more connectors.

Your choices for 4 drives are:

RAID10: write 2 copies to four drives, read from all four;
capacity is 2 drives worth. Survive any one drive failing and
also survive two drives failing if they are on different
stripes.

RAID6: write one copy plus two sets of parity information to
four drives, read from all four to reconstruct data. Speed is
equivalent to one drive. You can survive any two drives failing.

All of these can be handled in software by the kernel, managed
by mdadm. You can easily transfer the drives to some other Linux
box. 

-dsr-

how best to do

[Gene Heskett]

Greetings all;

This, my main machine that backs up 5 others here, but does that with 
drives not involved with daily stuffs, and of course I'm gradually 
replacing spinning rust with SSD's.

I've been dragging my feet on updating from stretch to buster, waiting 
for 1TB SSD's to get affordable. Now they almost have.

But I do have a pair of 500 GB SamSung EVO 860's and one empty sata 
socket. Spares for the rest of my multi-machine farm, currently at 5 
other machines.

Total storage actually used on this machine ATM is around 318G. Some of 
which isn't precious.

With 2 drives and 2 sata connectors, how can I configure this for maximum 
redundancy, but not necessarily maximum working capacity?

Does that recommendation change if I order 2 more 1T SSD's? In which 
case, which of the current buzzwords tech is the most dependable?

Thank you.
 
Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis
Genes Web page