Re: Backups with 9-STABLE -- Options?

[Warren Block]

On Sun, 10 Jun 2012, Karl Denninger wrote:


1. Is it REALLY safer to have the root filesystem run WITHOUT
softupdates?  (As was previous default practice)


The FAQ has an entry which has been there for a while.
http://www.freebsd.org/doc/en_US.ISO8859-1/books/faq/disks.html#SAFE-SOFTUPDATES


2. In order of risk of data loss what are the risks and options for SU,
SU+J and neither?  Neither exposes you to huge time delays on a
post-crash boot due to the fsck requirement, but SU can expose you to a
failed background fsck and thus get you the huge time delay too.  Since
SU+J eliminates this the only argument for NOT using it is that it's
more dangerous to your data than running without either or with SU
alone.  Is this true?


AFAIK, they should be the same as far as filesystem integrity, it's just 
that SU+J cuts down the time spent waiting for fsck.



3. Is there intent to fix dump -L with SU+J?


Yes, and there have been commits in the last few months.
http://svnweb.freebsd.org/base/head/sys/ufs/ufs/inode.h?sortby=date&view=log


If so, is there a projection on when?


Sorry, no idea.
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: Backups with 9-STABLE -- Options?

[Karl Denninger]

On 6/10/2012 10:35 AM, Adam Strohl wrote:
> On 6/10/2012 22:26, Karl Denninger wrote:
>> Well, backup with snapshots don't do well EITHER on a database unless
>> you can snapshot BOTH the dbms data store(s) and the transaction log
>> store(s) /*at the exact same instant*/.  If you cannot then you're
>> asking for trouble and are likely to get it.  But I've dealt with that
>> particular "gotcha" problem in a different way for the DBMS I use
>> (Postgresql)
>
> You asked what would happen, not what was the best way to back up a
> SQL DB, but your point is valid.
>
> Snapshots don't fix this issue entirely but drastically reduce the
> chance of a 100% broken backup.
>
> SQL servers should be dumped out to disk (ie; mysql_dump) to avoid
> this or have a dedicated backup client (which means you're probably
> not using dump anyway).
Well, yes and no.

I have this "problem" here and solve it in a number of ways, as the
risks are multiple.  It's not just hardware failure, it's also the risk
of a malicious or broken code (or a stupid DBA, such as myself who
fat-fingers something) doing something utterly insane like "drop ."
and realizing after hitting the button that wasn't what I wanted -- but
having it dutifully mirrored almost-instantly to the slaves.  Oops. 
It's the same issue you have with ordinary files (the "fool" who does
"rm -rf /" by accident, as root) and then watches the color drain out of
his face.

My solution for Postgres is to take base backups as a "special case",
keep the WAL log files post the base and back those up and run a
mirrored server as a hot standby.  That makes the hardware fault
scenario a 5 minute deal to recover from while the stupid dba flaw
remains recoverable.

>> So basically what you're saying is that SU+J leaves you exposed to
>> having no real backup option that provides a rational guarantee of the
>> ability to restore the backup taken.
>
> That's a bit of a gloss over on what I said.  My point was that you
> might end up missing something if its changing at the time the backup
> was taken.  It really depends on what specifically that server is doing.
Well there's a difference there then. 

I'm trying to get my arms around the risks here.  For a very long time
before we had snapshots I took live Level 0s (because I had to; I
couldn't take the machines involved offline for the multiple hours
required to take the dumps) and never got bit by them (I've been running
FreeBSD since the 1990s!) and HAVE had to restore "in anger" before. 
There's a big difference between a user missing a file he had open in
emacs at the time the system crashed and having a dump that blows up
during restore.  One is a risk that one has to take, even with
snapshots, to some degree as the snapshot can always happen while a file
is open for write and has half its contents in RAM rather than on disk
at the instant of the snapshot.  I accept that risk and have multiple
versions of backups for that reason (the usual daily/weekly/monthly/full
hierarchy), which mitigates but does not entirely eliminate it.  In my
environment the largest risk there are in things like mailbox stores
(accessed via IMAP) which are open for write (not just append) at any
time the user is doing a move or compact operation.

But a dump that blows up restore at the 50% point (for example) and
refuses to proceed is a different animal entirely.  That leaves you with
nothing (or effectively so.)  The question is whether or not that latter
"oh crap!" scenario becomes exposed if you don't use "L" and do use
journaling.

> There is also a consistency issue too, using snapshots makes it so
> that all the files make sense together, instead of the files getting
> more and more recent as the end of the backup block approaches.

True.

I've shut off journaling for now on my 9-STABLE machines (I'm just
starting to roll these into production, now being satisfied with the
stability that I'm willing to trust them with data I can't afford to
lose); in general FreeBSD has gotten to the point that it doesn't crash
on me at all (I have a server at a colo in heavy production use with
roughly 18 months of uptime on it, with patches and code changes all
being made while in-operation for security patches and such) and as such
the crash reboot time is less material than knowing that if the
worst-case happens I can get the data back.

I'd really like to see a discussion as to the various risk modes here. 
That is:

1. Is it REALLY safer to have the root filesystem run WITHOUT
softupdates?  (As was previous default practice)

2. In order of risk of data loss what are the risks and options for SU,
SU+J and neither?  Neither exposes you to huge time delays on a
post-crash boot due to the fsck requirement, but SU can expose you to a
failed background fsck and thus get you the huge time delay too.  Since
SU+J eliminates this the only argument for NOT using it is that it's
more dangerous to your data than running without either or with SU
alone.  Is this true

Re: Backups with 9-STABLE -- Options?

[Adam Strohl]

On 6/10/2012 22:26, Karl Denninger wrote:

Well, backup with snapshots don't do well EITHER on a database unless
you can snapshot BOTH the dbms data store(s) and the transaction log
store(s) /*at the exact same instant*/.  If you cannot then you're
asking for trouble and are likely to get it.  But I've dealt with that
particular "gotcha" problem in a different way for the DBMS I use
(Postgresql)


You asked what would happen, not what was the best way to back up a SQL 
DB, but your point is valid.


Snapshots don't fix this issue entirely but drastically reduce the 
chance of a 100% broken backup.


SQL servers should be dumped out to disk (ie; mysql_dump) to avoid this 
or have a dedicated backup client (which means you're probably not using 
dump anyway).



So basically what you're saying is that SU+J leaves you exposed to
having no real backup option that provides a rational guarantee of the
ability to restore the backup taken.


That's a bit of a gloss over on what I said.  My point was that you 
might end up missing something if its changing at the time the backup 
was taken.  It really depends on what specifically that server is doing.


There is also a consistency issue too, using snapshots makes it so that 
all the files make sense together, instead of the files getting more and 
more recent as the end of the backup block approaches.


--
Adam Strohl
http://www.ateamsystems.com/
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: Backups with 9-STABLE -- Options?

[Karl Denninger]

On 6/10/2012 10:13 AM, Adam Strohl wrote:
> On 6/10/2012 3:08, Karl Denninger wrote:
>> With SU+J as the default filesystem, what options actually WORK now?
>>
>> 1. Dump "L" will NOT -- it doesn't hang any more but now just bitches
>> and refuses to run.  I suppose that beats a hang
>
> Heh, yeah that is improved from what it did before ;D
>
>> 2. Dump without "L" and take your chances?  What risks am I running by
>> doing this on a running system?
>
> Depends on what is running and how it does file writes.  For example
> SQL DB storage engines are unlikely to do well (ie; the restore will
> be corrupted if there are changes during the process).  Something like
> CouchDB though which is "always consistent on disk" probably wouldn't
> care.
Well, backup with snapshots don't do well EITHER on a database unless
you can snapshot BOTH the dbms data store(s) and the transaction log
store(s) /*at the exact same instant*/.  If you cannot then you're
asking for trouble and are likely to get it.  But I've dealt with that
particular "gotcha" problem in a different way for the DBMS I use
(Postgresql)
>
> Past specific applications (or user activity) the inherent risk is
> unpredictable usefulness of your backups.  Since you're doing backups
> as a safeguard (and are very likely your last hope if things really go
> wrong) you don't want to find out that a key piece corrupted or
> missing entirely due to files moving around during the dump when you
> end up needing it.
Yeah, that's the problem.
>> 3.  Other?
>>
>> Dump has been the canonical means of backing up... forever.  And it
>> still is claimed to be the canonical means in the documentation.
>>
>> So what options do we have now that actually work -- is there now a new
>> "canonical" backup method that is recommended?
>
> My solution is to turn off journals for any build.   Dump is a great
> tool (especially when scripted) and is very efficient.
>
> And as neat as journals are, backups using dump with snapshots is way
> more valuable and important in my book.
>
> My .02.

So basically what you're saying is that SU+J leaves you exposed to
having no real backup option that provides a rational guarantee of the
ability to restore the backup taken.

Yet this was made the default why?

-- 
-- Karl Denninger
/The Market Ticker ®/ 
Cuda Systems LLC
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: Backups with 9-STABLE -- Options?

[Adam Strohl]

On 6/10/2012 3:08, Karl Denninger wrote:

With SU+J as the default filesystem, what options actually WORK now?

1. Dump "L" will NOT -- it doesn't hang any more but now just bitches
and refuses to run.  I suppose that beats a hang


Heh, yeah that is improved from what it did before ;D


2. Dump without "L" and take your chances?  What risks am I running by
doing this on a running system?


Depends on what is running and how it does file writes.  For example SQL 
DB storage engines are unlikely to do well (ie; the restore will be 
corrupted if there are changes during the process).  Something like 
CouchDB though which is "always consistent on disk" probably wouldn't care.


Past specific applications (or user activity) the inherent risk is 
unpredictable usefulness of your backups.  Since you're doing backups as 
a safeguard (and are very likely your last hope if things really go 
wrong) you don't want to find out that a key piece corrupted or missing 
entirely due to files moving around during the dump when you end up 
needing it.



3.  Other?

Dump has been the canonical means of backing up... forever.  And it
still is claimed to be the canonical means in the documentation.

So what options do we have now that actually work -- is there now a new
"canonical" backup method that is recommended?


My solution is to turn off journals for any build.   Dump is a great 
tool (especially when scripted) and is very efficient.


And as neat as journals are, backups using dump with snapshots is way 
more valuable and important in my book.


My .02.

--
Adam Strohl
http://www.ateamsystems.com/
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Backups with 9-STABLE -- Options?

[Karl Denninger]

With SU+J as the default filesystem, what options actually WORK now?

1. Dump "L" will NOT -- it doesn't hang any more but now just bitches
and refuses to run.  I suppose that beats a hang

2. Dump without "L" and take your chances?  What risks am I running by
doing this on a running system?

3.  Other?

Dump has been the canonical means of backing up... forever.  And it
still is claimed to be the canonical means in the documentation. 

So what options do we have now that actually work -- is there now a new
"canonical" backup method that is recommended?

-- 
-- Karl Denninger
/The Market Ticker ®/ 
Cuda Systems LLC
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"