Re: OpenBSD Home Server: Hints and Advices

[Benny Lofgren]

On 2015-09-29 04:00, dominik...@openmailbox.org wrote:
> On 2015-09-28 20:15, Stuart Henderson wrote:
>> On 2015-09-28, dominik...@openmailbox.org 
>> wrote:
>>> I'll use a 500GB SATA drive for the OS installation and will setup two
>>> WD Red drives in mirror using softraid(4).
>> Any particular reason not to just put the OS on the mirrored drives?
> Good question. Thought it was preferable to isolate one from another.

There is no point nowadays, unless you intend to be able to break loose
the mirrored drives from the system at some point and use them
stand-alone or in another system, while still operating the first
server. In the past, there were for example limitations on booting from
raided volums, which I suspect is where this misconception stems from.
Now there is no problem.

>>> A bunch of people (on a certain FreeBSD based NAS forum) chastise users
>>> who lost data for not having backed up their NAS. Well, isn't your NAS
>>> already a backup? Of course, I'm talking about a home NAS here where the
>>> content is only occasionally accessed. Is it me that's mixing up the two
>>> responsibilities, file server and backup server when I shouldn't?
>> Take a different view: Mirrored drives and RAID are not really for data
>> protection, they're so you can keep operating in face of (some types of)
>> hardware failure.
> 
> Indeed, but in reality doesn't it do both?

Absolutely not. :-) As Stuart and others points out, you should look at
this from another perspective; from what I can deduce from your
description you are building a home server for the family to share data
on. That is absolutely excellent! There is no need for each of you to
carry your favourite movies and music around on your individual
workstations/laptops/ipads/phones/toasters when you can store them once,
centrally.

However, even with mirrored drives, IT IS NOT A BACKUP. What if there is
a fire? What if someone burglars your house and steals the server? What
if someone accidentally knocks it over and all disks in it are damaged
by G-force overload? As Stuart says, mirroring is redundancy for
OPERATION, not for backup. In other words, if your system is mirrored
your server won't go offline if one disk dies on you, but will give you
time to replace the drive and re-mirror it before the other one goes too.

If you want to set up a combined home server/backup solution, it would
in fact be better *not* to mirror your two drives, but to use one for
your server needs and the other for backing up what's on the first. Of
course, that still only takes care of two of the scenarios - drive
failure and operator/software failure, i e accidentally removing or
overwriting files. In a fire, all of your movies, tv series, music,
documents, digital photos and home videos will be just as gone forever
as with mirroring.

Look at this as your new home server for family file sharing.

THEN look at a backup solution, too. One with geographical redundancy,
which is absolutely crucial. That is, somewhere else but in your home.

But don't just do one without the other, because it WILL make you sorry
in the end.


>> Now, if you're intending to use this server to put a *second* copy of
>> your files onto from some other machine, always keeping another copy,
>> then mirrored drives might be good enough for you. But it involves a
>> lot more discipline than setting up some automated backup schedule,
>> and it means treating it differently than "the machine that you store
>> files on".
> 
> The files are currently strewn over a couple of machines all over the
> house.
> I intended on deleting them once pushed to the server but maybe I should
> keep them
> at least on my desktop once they're tidied up.

Here's where your plan becomes crucially flawed. The moment you have
only one copy of something, it IS NOT BACKED UP. And data on mirrored
disks always counts as *ONE* copy only. Always. And, I would also submit
that as long as you only have something in only one PHYSICAL location,
it is not backed up either.

In my business (and personally) I always have at least three copies of
every single piece of data on all of my systems. For example, I have
systems in three different data centers, and every single system except
the backup servers themselves is backed up every single night to all
three places. And in addition to that, I keep a fourth copy of
everything at home, also nightly backed up.

All of those 20+ servers are also either mirrored or use RAID5. Most of
them also have redundant power supplies and are connected with two
network interraces via trunk aggregation/link failover to two separate
switches, but those measures are ONLY to protect from common hardware
failures, to keep everything online.

Never ever make the mistake of putting an equal sign between raid and
backup.


>>> Would you suggest going more for a Supermicro X10 motherboard with a
>>> separate CPU that could be upgraded down the line (if need be)? I'm also
>>> 

Re: OpenBSD Home Server: Hints and Advices

[dominik . db]

Hummm I see all your points and that is good food for thoughts.

I now see that it is indeed a bad setup for a backup solution. I thought 
that for a home user it is not necessarily worse than someone using an 
attached drive to its router (Apple Time Capsule for example). Note that 
I said "not worse" not that a Time Capsule device is any good by the 
standards defined in this discussion.
I guess that's what I had in mind originally, but with OpenBSD, and 
thought that the mirrored drives would have the added benefit that if 
one drive died, data still survived for a while.
As always OpenBSD users go for the technically correct solution and I 
appreciate you guys telling me about it.


That said, my data is not business critical. If I can get Time Capsule 
degree of reliability (close to none you'll tell me), it would be a 
start. Right now I have nothing. If my desktop hard drive dies, 
everything is gone. Only advantage of the desktop machine as opposed to 
the server is that it is NOT always on.


So maybe I could start with that and see what's my minimum critical data 
set (let's say family pictures) and push that to Tarsnap for offsite 
backups.


Thanks for all the constructive and enlightening responses.

Dominik

On 2015-09-29 14:59, Benny Lofgren wrote:

On 2015-09-29 19:51, Devin Reade wrote:

--On Tuesday, September 29, 2015 11:38:00 AM -0600 Devin Reade
 wrote:


To the OP, while most of the advice on this thread has been good, I'd
be careful of that one.  *Keep* your drives in a mirrored 
configuration

and have *additional* disks for backup purposes.


Just to clarify, I was referring specifically to the comment about
not using mirroring; I didn't trim quite enough quoting in my original
response.


And I should perhaps also clarify myself. I was thinking about a
hypothetical scenario where the two disk drives were all we had to work
with, for budget constraint reasons or whatever, but that actual
construct never went from my brain to my fingers... so just to be
absolutely crystal clear: I do *not* recommend doing this! But *if* all
that got stranded on that remote uninhabited island was you, a server
with two disks and a two thousand mile long connected water proof
extension cord, then it would be wise to use one as a backup and the
other to work with. And make sure both are bootable. (And go look for
the keyboard, mouse and screen, they're on that beach SOMEWHERE!)


Regards,

/Benny

Re: OpenBSD Home Server: Hints and Advices

[Devin Reade]

--On Tuesday, September 29, 2015 01:14:39 PM +0200 Benny Lofgren 
 wrote:



However, even with mirrored drives, IT IS NOT A BACKUP. What if there is
a fire? What if someone burglars your house and steals the server? What
if someone accidentally knocks it over and all disks in it are damaged
by G-force overload? As Stuart says, mirroring is redundancy for
OPERATION, not for backup. In other words, if your system is mirrored
your server won't go offline if one disk dies on you, but will give you
time to replace the drive and re-mirror it before the other one goes too.

If you want to set up a combined home server/backup solution, it would
in fact be better *not* to mirror your two drives, but to use one for
your server needs and the other for backing up what's on the first.


To the OP, while most of the advice on this thread has been good, I'd
be careful of that one.  *Keep* your drives in a mirrored configuration
and have *additional* disks for backup purposes.

Over the years, even with buying what appeared to be the most robust
commodity drives available at any given time, I've had far instances
of failed drives than having to restore from backup due to accidental
deletion, etc.  But because those disks have always been part of a
redundant RAID configration, all a bad disk means is a few minutes
to replace the faulty disk and then the server is back in operation
without any loss. (Of course, it may take a large number of hours after
that for the resilvering to complete, depending on disk size and
RAID type, but the server is operational in the interim.)

Yes, the additional disks cost more but in the scheme of things disks
are cheap and this should be part of your initial budget.


THEN look at a backup solution, too. One with geographical redundancy,
which is absolutely crucial. That is, somewhere else but in your home.

But don't just do one without the other, because it WILL make you sorry
in the end.


Absolutely correct.

Deciding how to do backups is always a question of balancing things,
including but not limited to:
 - if the worst happens, how much data can you afford to lose?  A day?
   A week?  A month?  (You can take the answer to be less than a day,
   right down to "none", but you're talking about progressively more
   complex and expensive.  Even large corporations don't use "none"
   for most of their data, if any.)
 - cost of disks
 - availability / cost of network bandwidth
 - level of automation (the less automation, the more disciplined
   you need to be in keeping backups current)

Let's say you low-ball this.  Assume that if something bad happens, you're
willing to live with losing everything you did in the last month, and
if there was something you deleted by accident more than two months ago
you willing to say it's gone forever.  Let's also assume that the amount
of data that you're backing up is not more than the size of the largest
hard drive you can currently buy.  In that case, the ABSOLUTE MINIMUM
you're looking at for backups is four disks:

  - Disks 1 and 2 are in a mirrored RAID in your file server

  - Disks 3 and 4 are for backups

  - Each month, take a snapshot of what is on your fileserver.  The
first month it goes to disk 3 (bonus points for encrypting
disk 3).  As soon as your backup is complete, take disk 3 off-site
(such as to your office, to a safety deposit box, etc.  Note that
smaller safety deposit boxes may be too small for 3.5" drives).
Ideally your off-site is far enough away so that when the
tornado hits your house while everyone is away at work/school,
that your offsite isn't destroyed as well.  This becomes more
difficult if you're in an earthquake zone.

  - The second month you repeat the process with disk 4, *before*
moving disk 3.  When your backup to disk 4 is complete, take it
offsite and bring disk 3 back, ready to use for next month.  If
you find that you have to recover something from disk 4 during
the next month, return disk 3 to the offsite location before you
bring back disk 4.

  - Periodically do a test recovery of some of your files (into a
temporary directory) to ensure that the backups are actually
usable.  The first time you do this should be after your first-ever
backup.

This takes discipline; you need to remember to do this on a regular
basis, or at some point you'll find that your only available backup
is three years old and you've lost precious pictures of your kids'
early years.

Probably your best tools for doing this basic level are dump(8) and
restore(8), using a level zero dump.  Read the man pages.  Bonus points
for scripting them so that you get the correct invocation every time.

Remember, that's the MINIMUM strategy.  Doing a web search for "data backup
strategies" will give you more background information.  Some links
that (with a quick skim) seem to provide a reasonable background are:

Re: OpenBSD Home Server: Hints and Advices

[Devin Reade]

--On Tuesday, September 29, 2015 11:38:00 AM -0600 Devin Reade 
 wrote:



To the OP, while most of the advice on this thread has been good, I'd
be careful of that one.  *Keep* your drives in a mirrored configuration
and have *additional* disks for backup purposes.


Just to clarify, I was referring specifically to the comment about
not using mirroring; I didn't trim quite enough quoting in my original
response.

The other thing I forgot to mention is replacing failed drives from
a mirrored RAID works a lot better if you've got a daemon monitoring
the SMART attributes.

Devin

Re: OpenBSD Home Server: Hints and Advices

[Benny Lofgren]

On 2015-09-29 19:51, Devin Reade wrote:
> --On Tuesday, September 29, 2015 11:38:00 AM -0600 Devin Reade
>  wrote:
> 
>> To the OP, while most of the advice on this thread has been good, I'd
>> be careful of that one.  *Keep* your drives in a mirrored configuration
>> and have *additional* disks for backup purposes.
> 
> Just to clarify, I was referring specifically to the comment about
> not using mirroring; I didn't trim quite enough quoting in my original
> response.

And I should perhaps also clarify myself. I was thinking about a
hypothetical scenario where the two disk drives were all we had to work
with, for budget constraint reasons or whatever, but that actual
construct never went from my brain to my fingers... so just to be
absolutely crystal clear: I do *not* recommend doing this! But *if* all
that got stranded on that remote uninhabited island was you, a server
with two disks and a two thousand mile long connected water proof
extension cord, then it would be wise to use one as a backup and the
other to work with. And make sure both are bootable. (And go look for
the keyboard, mouse and screen, they're on that beach SOMEWHERE!)


Regards,

/Benny

Re: OpenBSD Home Server: Hints and Advices

[Quartz]

It's gonna be behind a 3020j surge protector


A $20 spikebar will NOT protect this machine from a lightning strike 
that hits the pole in front of your house.




Take a different view: Mirrored drives and RAID are not really for data
protection, they're so you can keep operating in face of (some types of)
hardware failure.


Indeed, but in reality doesn't it do both?


Not unless you have a very narrow definition of 'data protection'. RAID 
won't protect you against bad software corrupting your files, or 
accidental 'rm -rf'




The files are currently strewn over a couple of machines all over the
house. I intended on deleting them once pushed to the server


A single copy of your files is not a "backup" no matter what definition 
of the word you're using.

Re: OpenBSD Home Server: Hints and Advices

[Quartz]

Well, isn't your NAS
already a backup?


No. At least, not really. Any "online" backup (in other words, an 
actively running machine) is always subject to issues that could destroy 
your data. The power supply could go bad and fry your drives, software 
issues could cause silent corruption, and you could always accidentally 
just delete files. The only really good backup is an offline one, 
preferably stored in a fireproof safe.

Re: OpenBSD Home Server: Hints and Advices

[Romain FABBRI]

Nor a safe in case of a major fire.

-Message d'origine-
De : owner-m...@openbsd.org [mailto:owner-m...@openbsd.org] De la part de Quartz
Envoyé : mardi 29 septembre 2015 06:35
À : misc@openbsd.org
Objet : Re: OpenBSD Home Server: Hints and Advices

>It's gonna be behind a 3020j surge protector

A $20 spikebar will NOT protect this machine from a lightning strike that hits 
the pole in front of your house.


>> Take a different view: Mirrored drives and RAID are not really for data
>> protection, they're so you can keep operating in face of (some types of)
>> hardware failure.
>
> Indeed, but in reality doesn't it do both?

Not unless you have a very narrow definition of 'data protection'. RAID 
won't protect you against bad software corrupting your files, or 
accidental 'rm -rf'


> The files are currently strewn over a couple of machines all over the
> house. I intended on deleting them once pushed to the server

A single copy of your files is not a "backup" no matter what definition 
of the word you're using.

Re: OpenBSD Home Server: Hints and Advices

[dominik . db]

On 2015-09-28 20:15, Stuart Henderson wrote:
On 2015-09-28, dominik...@openmailbox.org  
wrote:

I'll use a 500GB SATA drive for the OS installation and will setup two
WD Red drives in mirror using softraid(4).


Any particular reason not to just put the OS on the mirrored drives?



Good question. Thought it was preferable to isolate one from another.

If I setup a script that'll shutdown the machine if it detects that 
one
of the drive went bad, how secure (from loss) do you think my data 
would

be in softraid mirror with FFS2?


There are lots of ways to lose data, a hard drive going bad is only one 
of
them. I'm sure you can think of many possibilities. I'd *really* 
recommend
some physical separation between backups and the server. At the very 
least,
removable disks so they can be unplugged so that something like an 
electrical
surge (perhaps from a nearby lightning strike) won't affect them - as 
long

as they're not connected at the time. If you have internet with decent
upload speeds, maybe an online service or a trade with a friend (you 
store

backups on their machine and vice-versa) would work out well for this.



You're right. It's gonna be behind a 3020j surge protector that filters 
cable and telephone lines
also but when lightning strikes... That said, it's replacing a hard 
drive in a usb2 enclosure. So basically right now while the drive isn't 
always plugged, its low availability means that backups are done 
infrequently. With the server being always on, it'll be easy for all the 
family to push their data to the NFS share. In any case you're right and 
highly critical files might be pushed to Tarsnap also.


A bunch of people (on a certain FreeBSD based NAS forum) chastise 
users

who lost data for not having backed up their NAS. Well, isn't your NAS
already a backup? Of course, I'm talking about a home NAS here where 
the
content is only occasionally accessed. Is it me that's mixing up the 
two

responsibilities, file server and backup server when I shouldn't?


Take a different view: Mirrored drives and RAID are not really for data
protection, they're so you can keep operating in face of (some types 
of)

hardware failure.


Indeed, but in reality doesn't it do both?



Now, if you're intending to use this server to put a *second* copy of
your files onto from some other machine, always keeping another copy,
then mirrored drives might be good enough for you. But it involves a
lot more discipline than setting up some automated backup schedule,
and it means treating it differently than "the machine that you store
files on".


The files are currently strewn over a couple of machines all over the 
house.
I intended on deleting them once pushed to the server but maybe I should 
keep them

at least on my desktop once they're tidied up.


Also, the server will probably also run OpenSSH, OpenNTPD (for the
internal network only), httpd and maybe some mail services and will
stand behind a separate dedicated OpenBSD pf gateway. I think httpd is
chroot jailed by default and will probably just serve static HTML 
files.
Do you guys see any unreasonable risks for my data in that setup? I 
know
that ideally those services should be on a separate machine than my 
file

server/backup but that won't be an option in my little home setup, at
least for now.

Lastly, hardware wise, anybody have impressions on the SuperMicro
A1SAM-2550F with OpenBSD or in general?
It's seems plenty powerful for the job (and more) while being fairly 
low

power.


I've been pretty happy with similar (A1SAi) for low-powered systems,
but the boards aren't all that cheap, and neither is RAM for them.

I have seen fairly good prices on Dell T20 systems recently (GBP 
200+vat
for a xeon e3), which run OpenBSD nicely (note there are cashback 
offers

in various european countries for systems bought this month i.e.
today/tomorrow, info at
https://plus.delltradetosave.com/gb/en/pages/promotions/qualifying
- s/gb//).


Would you suggest going more for a Supermicro X10 motherboard with a
separate CPU that could be upgraded down the line (if need be)? I'm 
also

hesitating between this and a much cheaper Biostar 1037U Celeron based
embedded motherboard, but weren't sure of the quality for something 
that

would stay on 24/7.


I wouldn't be too concerned about ability to do CPU upgrades later.
If you run out of performance, rather than getting rid of the old CPU
and adding a slightly faster one at quite a lot of expense, at that
point you could get another little machine and split off some of
those public/semipublic services.


Thanks for the friendly tips and the nice discussion Stuart.

Dominik

Re: OpenBSD Home Server: Hints and Advices

[Stuart Henderson]

On 2015-09-28, dominik...@openmailbox.org  wrote:
> I'll use a 500GB SATA drive for the OS installation and will setup two 
> WD Red drives in mirror using softraid(4).

Any particular reason not to just put the OS on the mirrored drives?

> If I setup a script that'll shutdown the machine if it detects that one 
> of the drive went bad, how secure (from loss) do you think my data would 
> be in softraid mirror with FFS2?

There are lots of ways to lose data, a hard drive going bad is only one of
them. I'm sure you can think of many possibilities. I'd *really* recommend
some physical separation between backups and the server. At the very least,
removable disks so they can be unplugged so that something like an electrical
surge (perhaps from a nearby lightning strike) won't affect them - as long
as they're not connected at the time. If you have internet with decent
upload speeds, maybe an online service or a trade with a friend (you store
backups on their machine and vice-versa) would work out well for this.

> A bunch of people (on a certain FreeBSD based NAS forum) chastise users 
> who lost data for not having backed up their NAS. Well, isn't your NAS 
> already a backup? Of course, I'm talking about a home NAS here where the 
> content is only occasionally accessed. Is it me that's mixing up the two 
> responsibilities, file server and backup server when I shouldn't?

Take a different view: Mirrored drives and RAID are not really for data
protection, they're so you can keep operating in face of (some types of)
hardware failure.

Now, if you're intending to use this server to put a *second* copy of
your files onto from some other machine, always keeping another copy,
then mirrored drives might be good enough for you. But it involves a
lot more discipline than setting up some automated backup schedule,
and it means treating it differently than "the machine that you store
files on".

> Also, the server will probably also run OpenSSH, OpenNTPD (for the 
> internal network only), httpd and maybe some mail services and will 
> stand behind a separate dedicated OpenBSD pf gateway. I think httpd is 
> chroot jailed by default and will probably just serve static HTML files. 
> Do you guys see any unreasonable risks for my data in that setup? I know 
> that ideally those services should be on a separate machine than my file 
> server/backup but that won't be an option in my little home setup, at 
> least for now.
>
> Lastly, hardware wise, anybody have impressions on the SuperMicro 
> A1SAM-2550F with OpenBSD or in general?
> It's seems plenty powerful for the job (and more) while being fairly low 
> power.

I've been pretty happy with similar (A1SAi) for low-powered systems,
but the boards aren't all that cheap, and neither is RAM for them.

I have seen fairly good prices on Dell T20 systems recently (GBP 200+vat
for a xeon e3), which run OpenBSD nicely (note there are cashback offers
in various european countries for systems bought this month i.e.
today/tomorrow, info at
https://plus.delltradetosave.com/gb/en/pages/promotions/qualifying
- s/gb//).

> Would you suggest going more for a Supermicro X10 motherboard with a 
> separate CPU that could be upgraded down the line (if need be)? I'm also 
> hesitating between this and a much cheaper Biostar 1037U Celeron based 
> embedded motherboard, but weren't sure of the quality for something that 
> would stay on 24/7.

I wouldn't be too concerned about ability to do CPU upgrades later.
If you run out of performance, rather than getting rid of the old CPU
and adding a slightly faster one at quite a lot of expense, at that
point you could get another little machine and split off some of
those public/semipublic services.