Re: backups & cloning

[Polytropon]

About the dd method:

On Wed, 30 Sep 2009 11:30:58 -0400, Jerry McAllister  wrote:
> It can be used, but it is not a good way to do it.

For regular backups or even for cloning, it's not very
performant, I agree. I'm mostly using this method for
forensic purposes, when I need a copy of a media (a
whole disk, one slice or a particular partition) to toy
around with, so I don't mess up the original data.



> That is because it copies sector by sector and the new 
> disk/filesystem may not match the old exactly. 

That's a known problem. Another problem is time complexity.
The dd program does copy everything - even the unused disk
blocks (which don't need to be copied). This makes this
process often last very long.



> Besides
> when it is newly written on a file by file basis, it can
> be more efficiently laid out and accomodate any changes in
> size and sector addressing.  dd cannot do that.

That's true. This is the point where tools like cpdup and
rsync come into mind (according to creating backups or
clones).



-- 
Polytropon
Magdeburg, Germany
Happy FreeBSD user since 4.0
Andra moi ennepe, Mousa, ...
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: backups & cloning

[Jerry McAllister]

On Wed, Sep 30, 2009 at 05:08:05AM +0200, Polytropon wrote:

> Forgot to mention this:
> 
> 
> On Tue, 29 Sep 2009 22:23:00 -0400, PJ  wrote:
> > 1. will the s1a slice dump the entire system, that is, the a, d, e, f
> > and g slices or is it partitions?
> 
> The ad0s1 slice (containing the a, d, e, f and g partitions) can
> be copied 1:1 with dd. By using dump + restore, the partitions
> need to be copied after another. In each case, the entire system
> will be copied. For this purpose, even the long lasting
> 
>   # dd if=/dev/ad0 of=/dev/da0 bs=1m
>   # dd if=/dev/ad0 of=/dev/da0 bs=512 count=1
> 
> method can be used.
> 

It can be used, but it is not a good way to do it.
That is because it copies sector by sector and the new 
disk/filesystem may not match the old exactly.  Besides
when it is newly written on a file by file basis, it can
be more efficiently laid out and accomodate any changes in
size and sector addressing.  dd cannot do that.

jerry


> -- 
> Polytropon
> Magdeburg, Germany
> Happy FreeBSD user since 4.0
> Andra moi ennepe, Mousa, ...
> ___
> freebsd-questions@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-questions
> To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: backups & cloning

[Jerry McAllister]

On Tue, Sep 29, 2009 at 10:48:30PM -0400, PJ wrote:

> Polytropon wrote:
> > On Tue, 29 Sep 2009 21:26:19 -0400, PJ  wrote:
> >   
> >> But what does that mean? But ad2s1a has just been newfs'd - so how can
> >> it be dumped if its been formatted?
> >> 
> > When you're working on this low level, triple-check all your
> > commands. Failure to do so can cause data loss. In the example
> > you presented, ad1 was the source disk, ad2 the target disk.
> > You DON'T want to newfs your source disk.
> >   
> >> And what exactly does stdout mean?
> >> 
> >
> > This refers to the standard output. In most cases, this is the
> > terminal, the screen, such as
> >
> > # cat /etc/fstab
> >
> > will write the /etc/fstab to stdout. If you redirect it, for
> > example by using > or |, you can make stdout a file, or the
> > input - stdin - for another program.
> >
> > This is how the dump | restore process works: It leaves out
> > the "use the tape" or "use the file", but instead directs the
> > output of dump - the dump itself - to the restore program as
> > input to be restored.
> >   
> >> What is dump doing? outputting what to where exactly?
> >> 
> > The dump program is outputting a dump of the specified partition
> > to the standard output, which in this case is directly trans-
> > mitted to the restore program, which "picks it up" and processes
> > it = restores it.
> >
> >> I don't see it or
> >> should I say, understand this at all.
> >
> > Have a look at the command line again, simplified:
> >
> > # dump -0 -f - /dev/ad0s1a | restore -r -f -
> >
> > Run the dump program, do a full backup of the 1st partition of
> > the 1st slice of the 1st disk, write this dump to the standard
> > output, pipe this output to the restore program, do a full
> > restore, read the dump to be restored from standard input.
> >   
> >> and then the restore is from what
> >> to where?
> >
> > The restore program gets the dump to be restored from the standard
> > input - remember, that's the output of the dump program - and
> > writes it to the current working directory. That's the reason
> > why you should always check with
> >
> > # pwd
> >
> > in which directory you're currently located, because that will
> > be the place where the restored data will appear.
> >   
> >> "write error 10 blocks into volume 1
> >> do you want to restart:"
> >> 
> >
> > Could you present the command you're actually using, especially
> > with where you issued it from?
> >   
> Duh I think I see where this is leading... I'm pretty sure it was
> issued from / which makes it redundant, right? I should have issued it
> from somewhere else, like from home, usr or whatever but not from / as
> that is what I was trying to dump :-[

No, that is not a problem.   You can be in any directory and do the dump
command, except if you want that restore to work you have to be in
the receiving filesystem/directory.

I just noticed that I missed that you were newfs-ing the wrong partition.
That was the one you wanted to read from and your newfs would wipe out
everything on it.If you do the newfs - a good idea - it has to be
on the new filesystem you will be writing to.

jerry



> >   
> >> The first time I tried with -L the error was 20 blocks...
> >> Both the slices for dump from and to are same size (2gb) and certainly
> >> not full by a long shot ( if I reccall correctly, only about 14% is used)
> >> 
> >
> > I'm not sure where you put the dump file. "Write error" seems
> > to indicate one of the following problems:
> > a) The snapshot cannot be created.
> > b) The dump file cannot be created.
> >
> >
> >
> >   
> >> And what's this about a snapshot? AFAIK, I'm not making a snapshot;
> >> anyway, there is no long pause except for the dumb look on my face upon
> >> seeing these messages.
> >> 
> >
> > Check "man dump" and search for the -L option. The dump program,
> > in order to obtain a dump from a file system that's currently in
> > use, will need to make a snapshot because it cannot handle data
> > that is changing. So it will dump the data with the state of the
> > snapshot, allowing the file system to be altered afterwards.
> >
> >
> >
> >   
> >> As it is, I am currently erasing the brand new 500gb disk on which I
> >> want to restore.
> >> 
> >
> > Excellent.
> >
> >
> >
> >   
> >> Things started out really bad... don't u;nderstand what is going on.
> >> 
> >
> > Polite question: Have you read the manpages and the section in the
> > Handbook?
> >   
> Yes... but my brain can't handle it all so quickly... and being as
> impatient as I am, I tend to miss things on the run... it usually comes
> to me sooner or later... unfortunately, it's more often later than
> sooner... I've been reading the stuff in the man pages, and getting more
> confused by googling... Actually, I've been trying to get things
> straightened ot for at least 3 days already.
> >
> >
> >   
> >> I
> >> installed a minimal 7.2, boot

Re: backups & cloning

[Jerry McAllister]

On Tue, Sep 29, 2009 at 07:44:38PM -0400, PJ wrote:

> I am getting more and more confused with all the info regarding backing
> up and cloning or moving systems from disk to disk or computer to computer.
> I would like to do 2 things:
> 1. clone several instances of 7.2 from and existing installation
> 2. set up a backup script to back up changes either every night or once
> a week
> 
> There are numerous solutions out there; but they are mostly confusing,
> erroneous or non functional.
> To start, could someone please explail to the the following, which I
> found here:http://forums.freebsd.org/showthread.php?t=185

This page is essentially correct.   But, it covers several situations.
You need to decide which situation you are working on.

Are  you trying to make a backup of your system in case something
fails or are you trying to make a clone to boot in another system?

As for the restore, are you trying to use it to create a disk to
move to another machine to boot with or to recover a failed disk
on the same machine or just have a bootable disk handy if your
current one fails?Each is different.

If you are just making a dump in case of a disk failure, then
just dump to a file on some removable media (USB drive, Tape, 
across the net, etc) and forget about doing the restore for now.
You do that if the disk fails and you have acquired a new disk
and prepared it for service including slicing and partitioning 
and putting an MBR on it and a boot sector.  Then you use the
fixit boot to restore those backups.

If you are making a clone drive to move to another system
then you have to slice and partition the new drive and then
do the piped dump-restores you indicate below.

If you are making a disk to switch to in case of a failure, you
start by making a slice and partitioned drive and do the dump-restores.
But, then you keep it current using rsync.   Note that in this case, you
only do the dump-restore once.  The rsync does all the updating.  
Alternatively you might use some of the mirroring software to make a 
mirror drive that is [almost] always an exact copy.  That is a completely 
different process.

If you are making a disk to move to another machine then you probably
do not want the -u switch on the dump command.That is meant for
making a series of full and change dumps as backups. 

> 
> You can move system from disk to disk on fly with
> Code:
> 
> $ newfs -U /dev/ad2s1a
> $ mount /dev/ad2s1a /target
> $ cd /target
> $ dump -0Lauf - /dev/ad1s1a  | restore -rf -
> 
> you can do the same using sudo
> Code:
> 
> $ sudo echo
> $ sudo dump -0Lauf - /dev/ad1s1a  | sudo restore -rf -
> 
> This may be clear to someone; it certainly is not to me.
> As I understand it, newfs will (re)format the slice.
> Ok,  But what is standard out in the above example.  The dump is from
> where to where?
> Could someone clarify all this for me?

The only thing the sudo does is make you root.
If you are already root, you don't need it - as in the first example
you give.   In this particular case, it is probably better to just
be root and run this as root.   That wouldn't always be the case in
every sudo situation.

The dump command as you give it reads the /dev/ad1s1a file system
and sends it to standard out.   That is what the  '-f -'  part of
the command tells it.   The restore command reads from standard in
and restores the data sent to it from the dump via the pipe which
is the '|'.  The pipe takes whatever is in the standard out from
where it is coming and puts it in the standard in where it is going.

> So far, I have been unable to dump the / slice, not even with the -L
> option. I am trying to dump the whole system (all the slices)except swap
> to a usb (sata2 500gb disk) and then restore to another computer with
> 7.2 minimal installation.

> Slices ad2s1d,e,f and g dump ok to usb. a does not - errors ("should use
> -L when dumping live filesystems)

So, what are the errors.

> Do you have to newfs each slice before restoring?  But if you are
> restoring on a running 7.2 system, don't you have to restore to another
> disk than the one the system is on?

You have asked two unrelated questions.   First:
No, but it is a convenient way to make sure there is a clean
receiving place.Actually, I don't bother doing the restore.
I just write a dump file and leave it there in case I need to
restore from it later.   So my dump command would look something like:

   dump -0Laf /target/ad1s1adump /dev/ad1s1a

So the file ad1s1adump would contain the dump.  You might add
in some characters that identify the date of the dump in the name
of the file.

Also, you can mount the source filesystem and dump using the
mount name.   So, if /dev/ad1s1a is normally mounted as /work,
then it would work - and be mnemonic to do:

  dump -0Laf /target/workdump.20090930 /work

Second: You cannot unmount a filesystem that is in use.  But if you have 
permissions you can write to it.   But, if you try to restore to the root 
filesys

Re: backups & cloning

[Warren Block]

On Wed, 30 Sep 2009, Polytropon wrote:


On Tue, 29 Sep 2009 21:49:01 -0600 (MDT), Warren Block  
wrote:

So usually I back up /, /var, and /usr to files
on a USB disk or sshfs.  Then I switch to the new target system, booting
it with a FreeBSD disk and doing a minimal install.  That makes sure the
MBR is installed, gives me a chance to set all the filesystem sizes, and
newfses them.


Similar here. In most cases, the FreeBSD live system is completely
sufficient: run sysinstall, slice, boot loader, partitions, drop
to shell; mount USB stick, restore from files located there.


Then I restore from the dump files created earlier, over the running
system.  First /usr, then /var, then /.  On reboot, it's a clone.


This means you bring up the minimal (installed) system first, then
do the restore? Why not do it right after the basic steps of
preparation right from the install CD?


Probably mostly inertia, but I also like that it makes certain 
everything has been done to make a complete bootable system.  Seems like 
when I do it manually, CRS syndrome kicks in and I forget a step which 
ends up taking more time.


-Warren Block * Rapid City, South Dakota USA
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: backups & cloning

[Giorgos Keramidas]

On Tue, 29 Sep 2009 22:23:00 -0400, PJ  wrote:
> Polytropon wrote:
>> Assuming nobody uses tape drives anymore, you need to specify
>> another file, which is the standard output in this case, which
>> may not be obvious, but it is if we reorder the command line:
>>
>> # dump -0 -L - a -u -f - /dev/ad1s1a | restore -r -f -

> 1. will the s1a slice dump the entire system, that is, the a, d, e, f
> and g slices or is it partitions?

No, dump will backup a single partition (or filesystem) specified by the
options you pass, i.e.:

*   Dump only the ad0s1a partition to standard output:

dump -0 -L -a -u -f - /dev/ad0s1a

*   Dump only the ad0s1d partition to standard output:

dump -0 -L -a -u -f - /dev/ad0s1d

You will have to run multiple `dump' instances to backup more than one
partition.  For example, a short script that I run daily to save dumps
at level 2 for all my laptop's UFS filesystems includes code that is
equivalent to the following set of commands:

TODAY=$( date -u '+%Y-%m-%d' )

dump -2 -L -a -u -f - /dev/ad0s1a > kobe.2.${TODAY}.ad0s1a
dump -2 -L -a -u -f - /dev/ad0s1d > kobe.2.${TODAY}.ad0s1d
dump -2 -L -a -u -f - /dev/ad0s1e > kobe.2.${TODAY}.ad0s1e
dump -2 -L -a -u -f - /dev/ad0s3d > kobe.2.${TODAY}.ad0s3d

Each partition is dumped to a separate output file, so I can restore
them separately.

>>> I am trying to dump the whole system (all the slices)except swap
>>> to a usb (sata2 500gb disk) and then restore to another computer with
>>> 7.2 minimal installation.
>>
>> I think that's not possible because dump operates on file system
>> level, which means on partitions, not on slices.
>
> I've been very confused with the slices/partitions.  I meant above, to
> dump the whole slice - but I guess that it has to be done with the
> partitions.

You cannot dump a full slice (what other operating systems call a "BIOS
partition") in a single dump run.  Use multiple dump commands like the
ones shown above.

>>> Slices ad2s1d,e,f and g dump ok to usb. a does not - errors ("should use
>>> -L when dumping live filesystems)
>
> and when I do dump -0Laf  /dev /ad1s1a  /dev/da0s1a
> the errors are
> "write error 10 blocks into volume 1
> do you want to restart:"

This is not a correct invocation of dump.  You can't pass multiple
partitions in one instance of dump, like /dev, /ad1s1a and /dev/da0s1a.

If the order of dump options confuses you, reorder them the same way
Polytropon did, so that the `-f OUTPUT' option stands out a bit more:

dump -0 -a -L -f - /ad1s1a

A break-down of these options is now easier to understand:

* This will save a level 0 dump (the -0 option) of partition
  /ad1s1a.

* The dump will not be split into multiple tape `archives' (the -a
  option).

* Before trying to save the current state of the input partition,
  dump will create a `snapshot' so that a consistent state of all
  files in the partition will be saved in the dumped archive (the -L
  option).

* The backup archive will be sent to standard output (the '-'
  argument of the -f option).

The special '-' value for the output file of the -f option may be a bit
confusing, but it is useful if you are going to immediately pipe dump's
output to the restore(8) program.  Otherwise, if you are just going to
save the dump archive to another disk, you could have used:

dump -0 -a -L -f dumpfile /ad1s1a

The dump utility would then save a dump archive to `dumpfile' instead of
writing everything to its standard output.

>> To illustrate a dump and restore process that involves several
>> partitions, just let me add this example:
>>
>> Stage 1: Initialize slice and partitions
>> # fdisk -I -B ad1
>> # bsdlabel -w -B ad1s1
>> # bsdlabel -e ad1s1
>> a: 512M * 4.2BSD 0 0 0
>> b: 1024M * swap
>> c: * * unused <--- don't change
>> e: 2G * 4.2BSD 0 0 0
>> f: 2G * 4.2BSD 0 0 0
>> g: 10G * 4.2BSD 0 0 0
>> h: * * 4.2BSD 0 0 0
>> ^KX (means: save & exit)
>> # newfs /dev/ad1s1a
>> # newfs -U /dev/ad1s1e
>> # newfs -U /dev/ad1s1f
>> # newfs -U /dev/ad1s1g
>> # newfs -U /dev/ad1s1h
>>
>> Stage 2: Go into SUM and prepare the source partitions
>
> why into SUM? I'm really the only user and I usually stay as root
> if SUM, shouldn't the # below be $?

Because when you are running in multi-user mode there may be services
and other background processes changing the files of the source
partitions while you are dumping them.

By going into single-user mode, you ensure that there is only one active
process in your system: the root shell you are using to back & restore
the files.

___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: backups & cloning

[Polytropon]

On Tue, 29 Sep 2009 21:49:01 -0600 (MDT), Warren Block  
wrote:
> So usually I back up /, /var, and /usr to files 
> on a USB disk or sshfs.  Then I switch to the new target system, booting 
> it with a FreeBSD disk and doing a minimal install.  That makes sure the 
> MBR is installed, gives me a chance to set all the filesystem sizes, and 
> newfses them.

Similar here. In most cases, the FreeBSD live system is completely
sufficient: run sysinstall, slice, boot loader, partitions, drop
to shell; mount USB stick, restore from files located there.

For automated cloning, there are good examples around that let
you boot from DVD or USB stick / USB hard disk and automatically
prepare the source disk, then restoring from files. This is a
common method especially via SSH, so a local media is needed only
for booting and maybe for preparing.



> Then I restore from the dump files created earlier, over the running 
> system.  First /usr, then /var, then /.  On reboot, it's a clone.

This means you bring up the minimal (installed) system first, then
do the restore? Why not do it right after the basic steps of
preparation right from the install CD?


-- 
Polytropon
Magdeburg, Germany
Happy FreeBSD user since 4.0
Andra moi ennepe, Mousa, ...
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: backups & cloning

[Warren Block]

On Wed, 30 Sep 2009, Polytropon wrote:


On Tue, 29 Sep 2009 21:37:50 -0600 (MDT), Warren Block  
wrote:

Why make it harder than it needs to be?  Call it / or /var or /usr
instead of /dev/ad0s1whatever.  dump will handle it.


This works without problems as long as it is running from the
system to be copied. In case you use a live system, it doesn't
know anything about the associations between devices and the
mountpoints; this information is, as far as I know, obtained
via /etc/fstab. This is important to know especially if the
source and target disk have different layouts and concepts,
e. g. /dev/ad0s1d = /var -> /dev/da0s1e = /var (different
partition names for same subtree).


Yes, you're right.  I only realized that after sending... so I just sent 
an additional message.


-Warren Block * Rapid City, South Dakota USA
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: backups & cloning

[Warren Block]

On Tue, 29 Sep 2009, Warren Block wrote:


On Wed, 30 Sep 2009, Polytropon wrote:


On Tue, 29 Sep 2009 22:48:30 -0400, PJ  wrote:

Duh I think I see where this is leading... I'm pretty sure it was
issued from / which makes it redundant, right? I should have issued it
from somewhere else, like from home, usr or whatever but not from / as
that is what I was trying to dump :-[


The working directory does only matter to the restore command.
The dump command just cares for the partition name. In order
to find out what partition corresponds with which subtree,
check /etc/fstab or run the

# mount
/dev/ad0s1a on / (ufs, local)
/dev/ad0s1d on /tmp (ufs, local, soft-updates)
/dev/ad0s1e on /var (ufs, local, soft-updates)
/dev/ad0s1f on /usr (ufs, local, soft-updates)
/dev/ad0s1g on /export/home (ufs, local, soft-updates)

command, as in the example above.


Why make it harder than it needs to be?  Call it / or /var or /usr instead of 
/dev/ad0s1whatever.  dump will handle it.  It's built for that.  If it's a 
live filesystem, add -L.


http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/backup-basics.html#AEN25814


Just to add a possibly more relevant example from the FAQ:

http://www.freebsd.org/doc/en_US.ISO8859-1/books/faq/disks.html#NEW-HUGE-DISK

That example has the user connect the new disk to the old system.  That 
works, but I've always felt it's too easy to get the disks mixed up and 
write to the wrong one.  So usually I back up /, /var, and /usr to files 
on a USB disk or sshfs.  Then I switch to the new target system, booting 
it with a FreeBSD disk and doing a minimal install.  That makes sure the 
MBR is installed, gives me a chance to set all the filesystem sizes, and 
newfses them.


Then I restore from the dump files created earlier, over the running 
system.  First /usr, then /var, then /.  On reboot, it's a clone.


-Warren Block * Rapid City, South Dakota USA
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: backups & cloning

[Polytropon]

On Tue, 29 Sep 2009 21:37:50 -0600 (MDT), Warren Block  
wrote:
> Why make it harder than it needs to be?  Call it / or /var or /usr 
> instead of /dev/ad0s1whatever.  dump will handle it. 

This works without problems as long as it is running from the
system to be copied. In case you use a live system, it doesn't
know anything about the associations between devices and the
mountpoints; this information is, as far as I know, obtained
via /etc/fstab. This is important to know especially if the
source and target disk have different layouts and concepts,
e. g. /dev/ad0s1d = /var -> /dev/da0s1e = /var (different
partition names for same subtree).




-- 
Polytropon
Magdeburg, Germany
Happy FreeBSD user since 4.0
Andra moi ennepe, Mousa, ...
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: backups & cloning

[Warren Block]

On Wed, 30 Sep 2009, Polytropon wrote:


On Tue, 29 Sep 2009 22:48:30 -0400, PJ  wrote:

Duh I think I see where this is leading... I'm pretty sure it was
issued from / which makes it redundant, right? I should have issued it
from somewhere else, like from home, usr or whatever but not from / as
that is what I was trying to dump :-[


The working directory does only matter to the restore command.
The dump command just cares for the partition name. In order
to find out what partition corresponds with which subtree,
check /etc/fstab or run the

# mount
/dev/ad0s1a on / (ufs, local)
/dev/ad0s1d on /tmp (ufs, local, soft-updates)
/dev/ad0s1e on /var (ufs, local, soft-updates)
/dev/ad0s1f on /usr (ufs, local, soft-updates)
/dev/ad0s1g on /export/home (ufs, local, soft-updates)

command, as in the example above.


Why make it harder than it needs to be?  Call it / or /var or /usr 
instead of /dev/ad0s1whatever.  dump will handle it.  It's built for 
that.  If it's a live filesystem, add -L.


http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/backup-basics.html#AEN25814

-Warren Block * Rapid City, South Dakota USA
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: backups & cloning

[Warren Block]

On Wed, 30 Sep 2009, Polytropon wrote:


Forgot to mention this:


On Tue, 29 Sep 2009 22:23:00 -0400, PJ  wrote:

1. will the s1a slice dump the entire system, that is, the a, d, e, f
and g slices or is it partitions?


The ad0s1 slice (containing the a, d, e, f and g partitions) can
be copied 1:1 with dd. By using dump + restore, the partitions
need to be copied after another. In each case, the entire system
will be copied. For this purpose, even the long lasting

# dd if=/dev/ad0 of=/dev/da0 bs=1m


This copies everything on the disk, including sectors not used by a 
filesystem.  So it usually takes a while.



# dd if=/dev/ad0 of=/dev/da0 bs=512 count=1


Not necessary, the first block was already copied, well, first.

-Warren Block * Rapid City, South Dakota USA
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: backups & cloning

[Warren Block]

On Tue, 29 Sep 2009, PJ wrote:


$ newfs -U /dev/ad2s1a
$ mount /dev/ad2s1a /target
$ cd /target
$ dump -0Lauf - /dev/ad1s1a  | restore -rf -


dump is reading /dev/ad1s1a and using stdout for output.
restore is writing to the current directory (/target) and is reading
from stdin.



But what does that mean? But ad2s1a has just been newfs'd


No.  Exact details are extremely important here.  ad2 is the target, 
dump is reading ad1.


And what exactly does stdout mean?  What is dump doing? outputting 
what to where exactly? I don't see it or should I say, understand this 
at all.and then the restore is from what to where?


The man page system is there to help you with this.  man dump and man 
restore show examples.  man stdout will help explain that.  Trying to do 
advanced operations without understanding these basics is going to be 
difficult, frustrating, and ultimately dangerous to your data.



A long pause while the system makes a snapshot is normal.

And what's this about a snapshot? AFAIK, I'm not making a snapshot;


But you are.  That's what the -L option to dump means, as described in 
the man page.


-Warren Block * Rapid City, South Dakota USA
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: backups & cloning

[Polytropon]

On Tue, 29 Sep 2009 22:48:30 -0400, PJ  wrote:
> Duh I think I see where this is leading... I'm pretty sure it was
> issued from / which makes it redundant, right? I should have issued it
> from somewhere else, like from home, usr or whatever but not from / as
> that is what I was trying to dump :-[

The working directory does only matter to the restore command.
The dump command just cares for the partition name. In order
to find out what partition corresponds with which subtree,
check /etc/fstab or run the

# mount
/dev/ad0s1a on / (ufs, local)
/dev/ad0s1d on /tmp (ufs, local, soft-updates)
/dev/ad0s1e on /var (ufs, local, soft-updates)
/dev/ad0s1f on /usr (ufs, local, soft-updates)
/dev/ad0s1g on /export/home (ufs, local, soft-updates)

command, as in the example above.



> Yes... but my brain can't handle it all so quickly... and being as
> impatient as I am, I tend to miss things on the run... it usually comes
> to me sooner or later... unfortunately, it's more often later than
> sooner...

As long as it doesn't damage your data, it's no real problem.



> I've been reading the stuff in the man pages, and getting more
> confused by googling...

FreeBSD has far the best documentation among operating systems
I've come around. The manpages give a good overview, and the
handbook illustrates many daily procedures with examples.



> Actually, I've been trying to get things
> straightened ot for at least 3 days already.

Maybe this "pattern" can help you understanding the "strange
piping dump into restore" command:

# cd targetdir
# dump -0 -L -a -u -f - sourcepartition | restore -r -f -

It's not that complicated, but you have to be SURE about certain
things.



> Well, that's why I'm really checking my new disk... but it could be the
> motherboard... I've always suspected it had something of a glitch in it
> ever since I got it... I don't think just a slower cpu should give it so
> many problems... a twin computer has the same hardware except for the
> cpu and it gives far less problems - only MS related.

You should consider checking some basic stuff, such as running
a memtest CD or building world + kernel (just for testing
purposes, load generating, and CPU utilization; GENERIC kernel
will be fine).



> Something about a boot sector - this is not the first time I have seen
> this identical error but on much older hdd's, though still satas.
> This does make me think that these problems are of hardware origin -
> motherboard or sata connectors - I find they are rather Disneyesque
> (Mickey Mouse) or just plain flimsy.

In this case, you should install the smartctl program by running

# pkg_add -r smartmontools

or installing them via ports by running

# cd /usr/ports/sysutils/smartmontools
# make install clean

Then run the

# smartctl -a da0

command to check the disk. Refer to

# man smartctl

for other options that can help to identify possible hardware
errors.



> Time to hit the sack... another day of computer frustration coming up...

Doesn't have to be.



> I'm under pressure to lear Flash and have to set up a reliable server to
> test a site I am designing and setting up. Have to do it myself... can't
> afford about anything today. :-(

You're learning things this way, and that's what makes "our" service
so expensive - because "we" know so much. :-)



-- 
Polytropon
Magdeburg, Germany
Happy FreeBSD user since 4.0
Andra moi ennepe, Mousa, ...
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: backups & cloning

[Polytropon]

Forgot to mention this:


On Tue, 29 Sep 2009 22:23:00 -0400, PJ  wrote:
> 1. will the s1a slice dump the entire system, that is, the a, d, e, f
> and g slices or is it partitions?

The ad0s1 slice (containing the a, d, e, f and g partitions) can
be copied 1:1 with dd. By using dump + restore, the partitions
need to be copied after another. In each case, the entire system
will be copied. For this purpose, even the long lasting

# dd if=/dev/ad0 of=/dev/da0 bs=1m
# dd if=/dev/ad0 of=/dev/da0 bs=512 count=1

method can be used.



-- 
Polytropon
Magdeburg, Germany
Happy FreeBSD user since 4.0
Andra moi ennepe, Mousa, ...
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: backups & cloning

[Polytropon]

On Tue, 29 Sep 2009 22:23:00 -0400, PJ  wrote:
> I feel a bit stupid, as usual, my carelessness led me to miss the
> difference between ad1 and ad2... dumb, dumb, dumb.

As long as you realize it BEFORE any writing operation, it's
no problem. Keep in mind that the numbering of ad*, as well
as of da* (which your USB disk will probably show up as)
depends on the position of the drive on the ATA controller,
e. g.   primary master   -> /dev/ad0
primary slave-> /dev/ad1
secondary master -> /dev/ad2
secondary slave  -> /dev/ad3
...
This numbering sceme even does take place if you're using
the master channels only, so in this example you would have
the ad0 and ad2 drives, no matter if ad1 is present or not.



> But... 2 questions:
> 1. will the s1a slice dump the entire system, that is, the a, d, e, f
> and g slices or is it partitions?

As far as I remember, you cannot dump slices. You can dump
partitions only, or, in other words, the dump program does
operate on file systems (and those are represented by one
partition per file system).

If you want to duplicate slice-wise, dd should be used, such
as the following example:

# dd if=/dev/ad0s1 of=/mnt/usb/slice1.dd bs=1m

If you want to duplicate partition-wise, you need to dump and
restore each partition, just as my verbose example showed.



> I've been very confused with the slices/partitions.

The term slice refers to what MICROS~1 calls "DOS primary
partitions" in the widest sense. Due to DOS limitations,
PCs do only support 4 slices per disk.

The term partition refers to a sub-area inside a FreeBSD
slice. Partitions can be used to separate functional
subtrees partition-wise. Of course, I often see settings
where there's only one partition on the slice, that's
a common thing.



> I meant above, to dump the whole slice - but I guess that it has to be
> done with the partitions.

Partitions: use dump + restore
Slices: use dd
Whole disks: use dd



> and when I do dump -0Laf  /dev /ad1s1a  /dev/da0s1a
> the errors are
> "write error 10 blocks into volume 1
> do you want to restart:"

Okay, everything is clear now. Just "interpret" the dump
command:

dump full snapshot autosize output=/dev /ad1s1a

First of all, /ad1s1a does not exist. Then, /dev/da0s1a is
ignored. The command doesn't make sense. The syntax of dump
can be simplified as follows:

dump (other options) -f outputfile inputdevice

Note that outputfile can be - (the standard output) which
can then be redirected somewhere else.

At this position, I need to ask you: What are you trying to do?
a) I want to dump ad1s1a as it is onto the disk that
   is da0.
b) I want to dump ad1s1a as a file on the disk that is
   da0.

Let's take a) first. I assume you have prepared the disk da0
as shown in my earlier example, or you simply have used sysinstall
to create a slice on the disk and partitions inside the slice that
are big enough to hold the data you want to transfer to them.

Check their existance:

# ll /dev/ad0* /dev/da0*

The listing should give you similar slices and partitions both
for the source and the target disk.

First you mount the target disk and change the working directory
to it:

# mount /dev/da0s1a /mnt
# cd /mnt

Now you dump from your ad0 disk to where you currently are:

# dump -0 -L -a -u -f - /dev/ad0s1a | restore -r -f -

Proceed with the other partitions. Mount them, change into
their mountpoints that are now relative to /mnt (e. g. /mnt/var,
/mnt/home) and repeat this command, substituting the source
/dev/ad0s1[defg]. Finally, change back to / and umount 
everything in a successive way, sync, done.

For case b) it's much easier. When you want to create data
files, you don't need to slice / partition your USB disk, just
newfs and mount it:

# newfs /dev/da0
# mount /dev/da0 /mnt

Now you can create all the data files for the different partitions:

# dump -0 -L -a -u -f /mnt/root.dump /dev/ad0s1a
# dump -0 -L -a -u -f /mnt/tmp.dump /dev/ad0s1d
# dump -0 -L -a -u -f /mnt/var.dump /dev/ad0s1e
# dump -0 -L -a -u -f /mnt/usr.dump /dev/ad0s1f
# dump -0 -L -a -u -f /mnt/home.dump /dev/ad0s1g



> The first time I tried with -L the error was 20 blocks...
> Both the slices for dump from and to are same size (2gb) and certainly
> not full by a long shot ( if I reccall correctly, only about 14% is used)

As far as I see, the command line just was wrong.


> why into SUM?

The idea behind doing dump / restore in SUM is - in addition with
unmounted partitions - to ensure that no write access disturbes
the reading process from the partitions. Of course, it's possible
to use -L and stay in MUM.



> I'm really the only user and I usually stay as root

It's valid to perform dump / restore as root.



> if SUM, shouldn't the # below be $?

No. The # indicates root permissions in any shell. The $ indicates
non-root acce

Re: backups & cloning

[PJ]

Polytropon wrote:
> On Tue, 29 Sep 2009 21:26:19 -0400, PJ  wrote:
>   
>> But what does that mean? But ad2s1a has just been newfs'd - so how can
>> it be dumped if its been formatted?
>> 
>
> When you're working on this low level, triple-check all your
> commands. Failure to do so can cause data loss. In the example
> you presented, ad1 was the source disk, ad2 the target disk.
> You DON'T want to newfs your source disk.
>
>   
>> And what exactly does stdout mean?
>> 
>
> This refers to the standard output. In most cases, this is the
> terminal, the screen, such as
>
>   # cat /etc/fstab
>
> will write the /etc/fstab to stdout. If you redirect it, for
> example by using > or |, you can make stdout a file, or the
> input - stdin - for another program.
>
> This is how the dump | restore process works: It leaves out
> the "use the tape" or "use the file", but instead directs the
> output of dump - the dump itself - to the restore program as
> input to be restored.
>
>
>
>   
>> What is dump doing? outputting what to where exactly?
>> 
>
> The dump program is outputting a dump of the specified partition
> to the standard output, which in this case is directly trans-
> mitted to the restore program, which "picks it up" and processes
> it = restores it.
>
>
>
>   
>> I don't see it or
>> should I say, understand this at all.
>> 
>
> Have a look at the command line again, simplified:
>
>   # dump -0 -f - /dev/ad0s1a | restore -r -f -
>
> Run the dump program, do a full backup of the 1st partition of
> the 1st slice of the 1st disk, write this dump to the standard
> output, pipe this output to the restore program, do a full
> restore, read the dump to be restored from standard input.
>
>
>
>   
>> and then the restore is from what
>> to where?
>> 
>
> The restore program gets the dump to be restored from the standard
> input - remember, that's the output of the dump program - and
> writes it to the current working directory. That's the reason
> why you should always check with
>
>   # pwd
>
> in which directory you're currently located, because that will
> be the place where the restored data will appear.
>
>
>
>   
>> "write error 10 blocks into volume 1
>> do you want to restart:"
>> 
>
> Could you present the command you're actually using, especially
> with where you issued it from?
>   
Duh I think I see where this is leading... I'm pretty sure it was
issued from / which makes it redundant, right? I should have issued it
from somewhere else, like from home, usr or whatever but not from / as
that is what I was trying to dump :-[
>
>
>   
>> The first time I tried with -L the error was 20 blocks...
>> Both the slices for dump from and to are same size (2gb) and certainly
>> not full by a long shot ( if I reccall correctly, only about 14% is used)
>> 
>
> I'm not sure where you put the dump file. "Write error" seems
> to indicate one of the following problems:
>   a) The snapshot cannot be created.
>   b) The dump file cannot be created.
>
>
>
>   
>> And what's this about a snapshot? AFAIK, I'm not making a snapshot;
>> anyway, there is no long pause except for the dumb look on my face upon
>> seeing these messages.
>> 
>
> Check "man dump" and search for the -L option. The dump program,
> in order to obtain a dump from a file system that's currently in
> use, will need to make a snapshot because it cannot handle data
> that is changing. So it will dump the data with the state of the
> snapshot, allowing the file system to be altered afterwards.
>
>
>
>   
>> As it is, I am currently erasing the brand new 500gb disk on which I
>> want to restore.
>> 
>
> Excellent.
>
>
>
>   
>> Things started out really bad... don't u;nderstand what is going on.
>> 
>
> Polite question: Have you read the manpages and the section in the
> Handbook?
>   
Yes... but my brain can't handle it all so quickly... and being as
impatient as I am, I tend to miss things on the run... it usually comes
to me sooner or later... unfortunately, it's more often later than
sooner... I've been reading the stuff in the man pages, and getting more
confused by googling... Actually, I've been trying to get things
straightened ot for at least 3 days already.
>
>
>   
>> I
>> installed a minimal 7.2, booted up and turned to another computer to do
>> some serious work. About 2 hours and 49 minutes later I notice messages
>> on the 7.2 about a page fault or something like that and then the system
>> reboots.
>> 
>
> This often indicates a hardware problem...
>   
Well, that's why I'm really checking my new disk... but it could be the
motherboard... I've always suspected it had something of a glitch in it
ever since I got it... I don't think just a slower cpu should give it so
many problems... a twin computer has the same hardware except for the
cpu and it gives far less problems - only MS related.
>
>
>   
>> Obviously with errors... but then I reboot again and it comes
>> up... I t

Re: backups & cloning

[PJ]

Olivier Nicole wrote:
 $ newfs -U /dev/ad2s1a
 $ mount /dev/ad2s1a /target
 $ cd /target
 $ dump -0Lauf - /dev/ad1s1a  | restore -rf -
 
>>> [...]
>>>   
>> But what does that mean? But ad2s1a has just been newfs'd - so how can
>> 
>
> Thats ad*1*s1a that has just been formatted, not ad2...
>
> Best,
>
> Olivier
>
>   
Thanks for that.  It took me a while to see that.
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: backups & cloning

[PJ]

You are a Master among masters... extraordianry understanding of the
genre and ver, very clear explanations...
I guess my filter between the brain and the computer is a bit foggy... :-(
I really appreciate your explanations.
But I still have a couple of small questions below...


Polytropon wrote:
> On Tue, 29 Sep 2009 19:44:38 -0400, PJ  wrote:
>> This may be clear to someone; it certainly is not to me.
>> As I understand it, newfs will (re)format the slice.
>
> No. The newfs program does create a new file system. In
> other terminology, this can be called a formatting process.
> Note that NOT a slice, but a PARTITION is subject to this
> process. So
>
> # newfs -U /dev/ad2s1a
>
> does format the first partition (a) of the first slice (s1)
> of the third disk (ad2).
>
>
>
>> Ok, But what is standard out in the above example. The dump is from
>> where to where?
>
> According to the command
>
> # dump -0Lauf - /dev/ad1s1a | restore -rf -
>
> you need to understand that the main purpose of dump is to
> dump unmounted (!) file systems to the system's tape drive.
> Assuming nobody uses tape drives anymore, you need to specify
> another file, which is the standard output in this case, which
> may not be obvious, but it is if we reorder the command line:
>
> # dump -0 -L - a -u -f - /dev/ad1s1a | restore -r -f -
>
> You can see that -f - specifies - to be the file to backup to.
> The backup comes from /dev/ad1s1a.
>
> The restore program, on the other side of the | pipe, does
> usually read from the system's tape drive. But in this case,
> it reads from standard input as the -f - command line option
> indicates. It restores the data to where the working directory
> at the moment is.
>
> Here's an example (ad1 is source disk, ad2 is target disk):
>
> # newfs -U /dev/ad2s1a
> # mount /dev/ad2s1a /mnt
> # cd /mnt
> # dump -0Lauf - /dev/ad1s1a | restore -rf -
>
>> Could someone clarify all this for me?
>
> Hopefully hereby done. :-)
I feel a bit stupid, as usual, my carelessness led me to miss the
difference between ad1 and ad2... dumb, dumb, dumb.
Ok, so I see that this works if you have two different drives on the
same machine...
But... 2 questions:
1. will the s1a slice dump the entire system, that is, the a, d, e, f
and g slices or is it partitions?
>
>> So far, I have been unable to dump the / slice, not even with the -L
>> option.
>
> Always keep in mind: Use dump only on unmounted partitions.
>
>> I am trying to dump the whole system (all the slices)except swap
>> to a usb (sata2 500gb disk) and then restore to another computer with
>> 7.2 minimal installation.
>
> I think that's not possible because dump operates on file system
> level, which means on partitions, not on slices.
I've been very confused with the slices/partitions.
I meant above, to dump the whole slice - but I guess that it has to be
done with the partitions.
>> Slices ad2s1d,e,f and g dump ok to usb. a does not - errors ("should use
>> -L when dumping live filesystems)
and when I do dump -0Laf  /dev /ad1s1a  /dev/da0s1a
the errors are
"write error 10 blocks into volume 1
do you want to restart:"
The first time I tried with -L the error was 20 blocks...
Both the slices for dump from and to are same size (2gb) and certainly
not full by a long shot ( if I reccall correctly, only about 14% is used)

>
> Keep an eye on terminology, you're swapping them here: The
> devices ad2s1[defg] are partitions, not slices. The corresponding
> slice that holds them is ad2s1.
Sorry; now it's getting clearer.
>
> Anyway, if you can, don't dump mounted file systems. Go into
> single user mode, mount / as ro, and run dump + restore. If you
> can, use a live system from CD, DVD or USB, which makes things
> easier.
>
>> Do you have to newfs each slice before restoring?
>
> Partitions. You don't have to newfs them once they are formatted.
> It's just the usual way to ensure they are free of any data.
>
>> But if you are
>> restoring on a running 7.2 system, don't you have to restore to another
>> disk than the one the system is on?
>
> I don't understand this question right... if you're using a running
> system for dump + restore - which is the system you want to be the
> source system, then do it in minimal condition. SUM is the most
> convenient way to do that, with all partitions unmounted, and
> only / in read-only mode so you can access the dump and restore
> binaries.
>
>> I am beginning to think that you have to have a system running and dumpt
>> to another disk on that system and then remove that disk and install in
>> another box and boot from that?
>> Am I getting close?
>
> Again, I'm not sure I understood you correctly. If you've done
> the dump + restore correctly, you always end up with a bootable
> system, so you can boot it in another box. Dumping and restoring
> just requires a running system, no matter if it is the source
> system itself or a live system from CD, DVD or USB. (I prefer
> tools like FreeSBIE for such tasks, but the FreeBSD live system

Re: backups & cloning

[Polytropon]

On Tue, 29 Sep 2009 21:26:19 -0400, PJ  wrote:
> But what does that mean? But ad2s1a has just been newfs'd - so how can
> it be dumped if its been formatted?

When you're working on this low level, triple-check all your
commands. Failure to do so can cause data loss. In the example
you presented, ad1 was the source disk, ad2 the target disk.
You DON'T want to newfs your source disk.

> And what exactly does stdout mean?

This refers to the standard output. In most cases, this is the
terminal, the screen, such as

# cat /etc/fstab

will write the /etc/fstab to stdout. If you redirect it, for
example by using > or |, you can make stdout a file, or the
input - stdin - for another program.

This is how the dump | restore process works: It leaves out
the "use the tape" or "use the file", but instead directs the
output of dump - the dump itself - to the restore program as
input to be restored.



> What is dump doing? outputting what to where exactly?

The dump program is outputting a dump of the specified partition
to the standard output, which in this case is directly trans-
mitted to the restore program, which "picks it up" and processes
it = restores it.



> I don't see it or
> should I say, understand this at all.

Have a look at the command line again, simplified:

# dump -0 -f - /dev/ad0s1a | restore -r -f -

Run the dump program, do a full backup of the 1st partition of
the 1st slice of the 1st disk, write this dump to the standard
output, pipe this output to the restore program, do a full
restore, read the dump to be restored from standard input.



> and then the restore is from what
> to where?

The restore program gets the dump to be restored from the standard
input - remember, that's the output of the dump program - and
writes it to the current working directory. That's the reason
why you should always check with

# pwd

in which directory you're currently located, because that will
be the place where the restored data will appear.



> "write error 10 blocks into volume 1
> do you want to restart:"

Could you present the command you're actually using, especially
with where you issued it from?



> The first time I tried with -L the error was 20 blocks...
> Both the slices for dump from and to are same size (2gb) and certainly
> not full by a long shot ( if I reccall correctly, only about 14% is used)

I'm not sure where you put the dump file. "Write error" seems
to indicate one of the following problems:
a) The snapshot cannot be created.
b) The dump file cannot be created.



> And what's this about a snapshot? AFAIK, I'm not making a snapshot;
> anyway, there is no long pause except for the dumb look on my face upon
> seeing these messages.

Check "man dump" and search for the -L option. The dump program,
in order to obtain a dump from a file system that's currently in
use, will need to make a snapshot because it cannot handle data
that is changing. So it will dump the data with the state of the
snapshot, allowing the file system to be altered afterwards.



> As it is, I am currently erasing the brand new 500gb disk on which I
> want to restore.

Excellent.



> Things started out really bad... don't u;nderstand what is going on.

Polite question: Have you read the manpages and the section in the
Handbook?



> I
> installed a minimal 7.2, booted up and turned to another computer to do
> some serious work. About 2 hours and 49 minutes later I notice messages
> on the 7.2 about a page fault or something like that and then the system
> reboots.

This often indicates a hardware problem...



> Obviously with errors... but then I reboot again and it comes
> up... I tried som copying from another disk and ended up with the disk
> all screwed up...

How that?



> yet the Seagate Seatools for Dos doesnt find any
> errors on it;

There's smartmontools (program: smartctl) for FreeBSD in the ports.
It can check various errors of modern hard disks.



> Partition magic found an error but couldn't fix it, so now
> Im wiping the whole thing and will try to reinstall tomorrow. Doesn't
> make sense.

What error was this?





-- 
Polytropon
Magdeburg, Germany
Happy FreeBSD user since 4.0
Andra moi ennepe, Mousa, ...
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: backups & cloning

[Polytropon]

On Tue, 29 Sep 2009 19:09:51 -0600 (MDT), Warren Block  
wrote:
> On Wed, 30 Sep 2009, Polytropon wrote:
> >> So far, I have been unable to dump the / slice, not even with the -L
> >> option.
> >
> > Always keep in mind: Use dump only on unmounted partitions.
> 
> That is unnecessary.  The -L option is there just for dumping mounted 
> filesystems.

You're right, but -L does require a certain time to create the
snapshot. Of course that's not problematic when you need to do
this only once. In other situations, especially when you're able
to boot from something else than the system you want to clone,
using the "pure" unmounted partitions is more convenient. This
is only my very individual opinion. :-)


-- 
Polytropon
Magdeburg, Germany
Happy FreeBSD user since 4.0
Andra moi ennepe, Mousa, ...
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: backups & cloning

[Olivier Nicole]

> >> $ newfs -U /dev/ad2s1a
> >> $ mount /dev/ad2s1a /target
> >> $ cd /target
> >> $ dump -0Lauf - /dev/ad1s1a  | restore -rf -
> >[...]
> But what does that mean? But ad2s1a has just been newfs'd - so how can

Thats ad*1*s1a that has just been formatted, not ad2...

Best,

Olivier
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: backups & cloning

[PJ]

Warren Block wrote:
> On Tue, 29 Sep 2009, PJ wrote:
>
>> I am getting more and more confused with all the info regarding backing
>> up and cloning or moving systems from disk to disk or computer to
>> computer.
>> I would like to do 2 things:
>> 1. clone several instances of 7.2 from and existing installation
>> 2. set up a backup script to back up changes either every night or once
>> a week
>>
>> There are numerous solutions out there; but they are mostly confusing,
>> erroneous or non functional.
>> To start, could someone please explail to the the following, which I
>> found here:http://forums.freebsd.org/showthread.php?t=185
>>
>> You can move system from disk to disk on fly with
>> Code:
>>
>> $ newfs -U /dev/ad2s1a
>> $ mount /dev/ad2s1a /target
>> $ cd /target
>> $ dump -0Lauf - /dev/ad1s1a  | restore -rf -
>
>> This may be clear to someone; it certainly is not to me.
>> As I understand it, newfs will (re)format the slice.
>> Ok,  But what is standard out in the above example.  The dump is from
>> where to where?
>
> dump is reading /dev/ad1s1a and using stdout for output.
> restore is writing to the current directory (/target) and is reading
> from stdin.
But what does that mean? But ad2s1a has just been newfs'd - so how can
it be dumped if its been formatted? And what exactly does stdout mean?
What is dump doing? outputting what to where exactly? I don't see it or
should I say, understand this at all.and then the restore is from what
to where?
>
>> Could someone clarify all this for me?
>> So far, I have been unable to dump the / slice, not even with the -L
>> option.
>
> It's hard to help without knowing the exact commands you are using and
> the errors they are producing.  Help us to help you by posting them.
>
>> I am trying to dump the whole system (all the slices)except swap
>> to a usb (sata2 500gb disk) and then restore to another computer with
>> 7.2 minimal installation.
>
> A minimal install makes it easier.  You don't need to copy /tmp, either.
>
>> Slices ad2s1d,e,f and g dump ok to usb. a does not - errors ("should use
>> -L when dumping live filesystems)
>
> Right.  So what happens when you use -L? 
"write error 10 blocks into volume 1
do you want to restart:"
The first time I tried with -L the error was 20 blocks...
Both the slices for dump from and to are same size (2gb) and certainly
not full by a long shot ( if I reccall correctly, only about 14% is used)

> A long pause while the system makes a snapshot is normal.
And what's this about a snapshot? AFAIK, I'm not making a snapshot;
anyway, there is no long pause except for the dumb look on my face upon
seeing these messages.
As it is, I am currently erasing the brand new 500gb disk on which I
want to restore.
Things started out really bad... don't u;nderstand what is going on. I
installed a minimal 7.2, booted up and turned to another computer to do
some serious work. About 2 hours and 49 minutes later I notice messages
on the 7.2 about a page fault or something like that and then the system
reboots. Obviously with errors... but then I reboot again and it comes
up... I tried som copying from another disk and ended up with the disk
all screwed up... yet the Seagate Seatools for Dos doesnt find any
errors on it; Partition magic found an error but couldn't fix it, so now
Im wiping the whole thing and will try to reinstall tomorrow. Doesn't
make sense.

>
>> Do you have to newfs each slice before restoring?
>
> The first time.  But your minimal install already did that for you.
>
>> But if you are restoring on a running 7.2 system, don't you have to
>> restore to another disk than the one the system is on?
>
> Nope.  You can overwrite the running system.  I restore in /usr, /var,
> and then / order.  Then reboot and you are running the new clone.
>
>> I am beginning to think that you have to have a system running and
>> dumpt to another disk on that system and then remove that disk and
>> install in another box and boot from that? Am I getting close? I know
>> it's a lot to ask, but then, I know you guys are capable...  :-)
>
> It's usually best to limit messages to a single question.
Sure, I agree... but when things are really complicated... I, at least,
don't know how to separate them when they are quite interdependent.
Thanks for responding.
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: backups & cloning

[Warren Block]

On Wed, 30 Sep 2009, Polytropon wrote:

So far, I have been unable to dump the / slice, not even with the -L
option.


Always keep in mind: Use dump only on unmounted partitions.


That is unnecessary.  The -L option is there just for dumping mounted 
filesystems.


-Warren Block * Rapid City, South Dakota USA
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: backups & cloning

[Polytropon]

On Tue, 29 Sep 2009 19:44:38 -0400, PJ  wrote:
> This may be clear to someone; it certainly is not to me.
> As I understand it, newfs will (re)format the slice.

No. The newfs program does create a new file system. In
other terminology, this can be called a formatting process.
Note that NOT a slice, but a PARTITION is subject to this
process. So

# newfs -U /dev/ad2s1a

does format the first partition (a) of the first slice (s1)
of the third disk (ad2).



> Ok,  But what is standard out in the above example.  The dump is from
> where to where?

According to the command

# dump -0Lauf - /dev/ad1s1a | restore -rf -

you need to understand that the main purpose of dump is to
dump unmounted (!) file systems to the system's tape drive.
Assuming nobody uses tape drives anymore, you need to specify
another file, which is the standard output in this case, which
may not be obvious, but it is if we reorder the command line:

# dump -0 -L - a -u -f - /dev/ad1s1a | restore -r -f -

You can see that -f - specifies - to be the file to backup to.
The backup comes from /dev/ad1s1a.

The restore program, on the other side of the | pipe, does
usually read from the system's tape drive. But in this case,
it reads from standard input as the -f - command line option
indicates. It restores the data to where the working directory
at the moment is.

Here's an example (ad1 is source disk, ad2 is target disk):

# newfs -U /dev/ad2s1a
# mount /dev/ad2s1a /mnt
# cd /mnt
# dump -0Lauf - /dev/ad1s1a | restore -rf -



> Could someone clarify all this for me?

Hopefully hereby done. :-)



> So far, I have been unable to dump the / slice, not even with the -L
> option.

Always keep in mind: Use dump only on unmounted partitions.



> I am trying to dump the whole system (all the slices)except swap
> to a usb (sata2 500gb disk) and then restore to another computer with
> 7.2 minimal installation.

I think that's not possible because dump operates on file system
level, which means on partitions, not on slices.



> Slices ad2s1d,e,f and g dump ok to usb. a does not - errors ("should use
> -L when dumping live filesystems)

Keep an eye on terminology, you're swapping them here: The
devices ad2s1[defg] are partitions, not slices. The corresponding
slice that holds them is ad2s1.

Anyway, if you can, don't dump mounted file systems. Go into
single user mode, mount / as ro, and run dump + restore. If you
can, use a live system from CD, DVD or USB, which makes things
easier.



> Do you have to newfs each slice before restoring? 

Partitions. You don't have to newfs them once they are formatted.
It's just the usual way to ensure they are free of any data.



> But if you are
> restoring on a running 7.2 system, don't you have to restore to another
> disk than the one the system is on?

I don't understand this question right... if you're using a running
system for dump + restore - which is the system you want to be the
source system, then do it in minimal condition. SUM is the most
convenient way to do that, with all partitions unmounted, and
only / in read-only mode so you can access the dump and restore
binaries.



> I am beginning to think that you have to have a system running and dumpt
> to another disk on that system and then remove that disk and install in
> another box and boot from that?
> Am I getting close?

Again, I'm not sure I understood you correctly. If you've done
the dump + restore correctly, you always end up with a bootable
system, so you can boot it in another box. Dumping and restoring
just requires a running system, no matter if it is the source
system itself or a live system from CD, DVD or USB. (I prefer
tools like FreeSBIE for such tasks, but the FreeBSD live system
CD is fine, too.)

As far as I now understood, you don't want to clone from source
disk to target disk, but use a USB "transfer disk"; in this case,
you first need to clone onto this disk, and then use it in the
other computers to fill their disks with the copy you made from
your "master system".

As a sidenote, it's worth mentioning that this can be achieved
in an easier way: You create a minimal bootable FreeBSD on this
USB disk, in case your computers can boot from it; if not, use
a live system to boot the computers. Then, format the USB disk
with only one file system (e. g. /dev/da0) and put dump files
onto it, so they are available then as /mnt/root.dump, tmp.dump,
var.dump, usr.dump and home.dump. Instead of using -f - for the
dump and restore program, use those file names.



> I know it's a lot to ask, but then, I know you guys are capable...  :-)

If you still have questions, try to ask them as precise as
possible.

I may add that this list is the most friendly and intelligent
community to ask, so you're definitely at the right place here.



To illustrate a dump and restore process that involves several
partitions, just let me add this example:

Stage 1: Initialize slice and partitions
 

Re: backups & cloning

[Warren Block]

On Tue, 29 Sep 2009, PJ wrote:


I am getting more and more confused with all the info regarding backing
up and cloning or moving systems from disk to disk or computer to computer.
I would like to do 2 things:
1. clone several instances of 7.2 from and existing installation
2. set up a backup script to back up changes either every night or once
a week

There are numerous solutions out there; but they are mostly confusing,
erroneous or non functional.
To start, could someone please explail to the the following, which I
found here:http://forums.freebsd.org/showthread.php?t=185

You can move system from disk to disk on fly with
Code:

$ newfs -U /dev/ad2s1a
$ mount /dev/ad2s1a /target
$ cd /target
$ dump -0Lauf - /dev/ad1s1a  | restore -rf -



This may be clear to someone; it certainly is not to me.
As I understand it, newfs will (re)format the slice.
Ok,  But what is standard out in the above example.  The dump is from
where to where?


dump is reading /dev/ad1s1a and using stdout for output.
restore is writing to the current directory (/target) and is reading 
from stdin.



Could someone clarify all this for me?
So far, I have been unable to dump the / slice, not even with the -L
option.


It's hard to help without knowing the exact commands you are using and 
the errors they are producing.  Help us to help you by posting them.



I am trying to dump the whole system (all the slices)except swap
to a usb (sata2 500gb disk) and then restore to another computer with
7.2 minimal installation.


A minimal install makes it easier.  You don't need to copy /tmp, either.


Slices ad2s1d,e,f and g dump ok to usb. a does not - errors ("should use
-L when dumping live filesystems)


Right.  So what happens when you use -L?  A long pause while the system 
makes a snapshot is normal.



Do you have to newfs each slice before restoring?


The first time.  But your minimal install already did that for you.

But if you are restoring on a running 7.2 system, don't you have to 
restore to another disk than the one the system is on?


Nope.  You can overwrite the running system.  I restore in /usr, /var, 
and then / order.  Then reboot and you are running the new clone.


I am beginning to think that you have to have a system running and 
dumpt to another disk on that system and then remove that disk and 
install in another box and boot from that? Am I getting close? I know 
it's a lot to ask, but then, I know you guys are capable...  :-)


It's usually best to limit messages to a single question.

-Warren Block * Rapid City, South Dakota USA
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

backups & cloning

[PJ]

I am getting more and more confused with all the info regarding backing
up and cloning or moving systems from disk to disk or computer to computer.
I would like to do 2 things:
1. clone several instances of 7.2 from and existing installation
2. set up a backup script to back up changes either every night or once
a week

There are numerous solutions out there; but they are mostly confusing,
erroneous or non functional.
To start, could someone please explail to the the following, which I
found here:http://forums.freebsd.org/showthread.php?t=185

You can move system from disk to disk on fly with
Code:

$ newfs -U /dev/ad2s1a
$ mount /dev/ad2s1a /target
$ cd /target
$ dump -0Lauf - /dev/ad1s1a  | restore -rf -

you can do the same using sudo
Code:

$ sudo echo
$ sudo dump -0Lauf - /dev/ad1s1a  | sudo restore -rf -

This may be clear to someone; it certainly is not to me.
As I understand it, newfs will (re)format the slice.
Ok,  But what is standard out in the above example.  The dump is from
where to where?
Could someone clarify all this for me?
So far, I have been unable to dump the / slice, not even with the -L
option. I am trying to dump the whole system (all the slices)except swap
to a usb (sata2 500gb disk) and then restore to another computer with
7.2 minimal installation.
Slices ad2s1d,e,f and g dump ok to usb. a does not - errors ("should use
-L when dumping live filesystems)
Do you have to newfs each slice before restoring?  But if you are
restoring on a running 7.2 system, don't you have to restore to another
disk than the one the system is on?
I am beginning to think that you have to have a system running and dumpt
to another disk on that system and then remove that disk and install in
another box and boot from that?
Am I getting close?
I know it's a lot to ask, but then, I know you guys are capable...  :-)
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"