Re: [BackupPC-users] Best way to copy backuppc data

[Carl W. Soderstrom]

On 02/06 08:51 , Adam Pribyl wrote:
> On Mon, 5 Feb 2018, Carl W. Soderstrom wrote:
> 
> > On 02/04 07:48 , Raoul Bhatia wrote:
> > > What's the issue with using LVM?  Unless you need to reinitialize
> > > the whole fs, i.e. increase EXT4 inode count or switch to another
> > > fs, believe this is a perfect example of where LVM shines.
> > 
> > I'd like to insert a word of caution based on experience. I don't know about
> > using LVM for copying data to a new disk; but I do know that using LVM
> > snapshots to get a quiesced pool for backup, was a poor idea in my case. It
> > took 12h to get a tar dump of a given BackupPC data pool when BPC was
> > fully stopped, but 2 days to try to backup the same data pool when it was
> > quiesced using an LVM snapshot.
> 
> This is the thing - I was thinking about snapshot too, luckily I did not
> used them. To make a 1:1 copy you may use the LVM mirror feature, which is a
> bit new. I used "pvmove" - that _moved_ the data from one disk to another
> fine, but of course is not a copy. I wanted to separated backuppc data from
> /var mount point to its own disk that would be mounted in /var/lib/backuppc
> - that was the reason to make a copy, but as it failed I changed my
> approach.

Thanks for the information. I haven't looked at LVM for many years. 

-- 
Carl Soderstrom
Systems Administrator
Real-Time Enterprises
www.real-time.com

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Best way to copy backuppc data

[Carl W. Soderstrom]

On 02/04 07:48 , Raoul Bhatia wrote:
> What's the issue with using LVM?  Unless you need to reinitialize the whole 
> fs, i.e. increase EXT4 inode count or switch to another fs, believe this is a 
> perfect example of where LVM shines.

I'd like to insert a word of caution based on experience. I don't know about
using LVM for copying data to a new disk; but I do know that using LVM
snapshots to get a quiesced pool for backup, was a poor idea in my case. It
took 12h to get a tar dump of a given BackupPC data pool when BPC was
fully stopped, but 2 days to try to backup the same data pool when it was
quiesced using an LVM snapshot. 

We decided it was better to have BPC offline for a short period (scheduled
on the weekend), but running full speed the rest of the time, rather than
degraded in performance for 4x as long.

Not fully relevant to the problem at hand, but still worth considering for
future reference.

-- 
Carl Soderstrom
Systems Administrator
Real-Time Enterprises
www.real-time.com

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Best way to copy backuppc data

[Iturriaga Woelfel, Markus]

tar -C /var/lib/backuppc --one-filesystem --acls --xattrs -cf - . | tar
-C /copy -xvf -

How does this deal with hardlinks?


Just to answer this question - this is how GNU tar handles hard links: 
https://www.gnu.org/software/tar/manual/html_node/hard-links.html



---
Markus A. Iturriaga Woelfel, IT Administrator
Department of Electrical Engineering and Computer Science
University of Tennessee
Min H. Kao Building, Suite 424 / 1520 Middle Drive
Knoxville, TN 37996-2250
mitur...@utk.edu / (865) 974-3837
http://twitter.com/UTKEECSIT




--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Best way to copy backuppc data

[Carl W. Soderstrom]

On 02/03 10:30 , Les Mikesell wrote:
> On Sat, Feb 3, 2018 at 5:50 AM, Adam Pribyl  wrote:
> > On Fri, 2 Feb 2018, Iturriaga Woelfel, Markus wrote:
> >
> >
> > I am running short on ideas how to copy this.
> 
> If you are copying to an identical disk you can use dd on the raw
> devices.The system may be confused by the identical UUID's if you
> have both drives connected when you reboot, though.

Even if it's not an identical disk, if the target disk is larger, you can
use dd to copy the data pool partition, then grow your filesystem after
this. I've used this successfully in the past on a few occasions. dd is
definitely the fastest way I found to copy a BackupPC data pool (tho I
haven't experimented with ZFS or btrfs).

-- 
Carl Soderstrom
Systems Administrator
Real-Time Enterprises
www.real-time.com

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Best way to copy backuppc data

[Raoul Bhatia]

On February 3, 2018 9:27:40 PM UTC, Adam Pribyl  wrote:
>On Sat, 3 Feb 2018, Les Mikesell wrote:
>
>> On Sat, Feb 3, 2018 at 5:50 AM, Adam Pribyl 
>wrote:
>>> On Fri, 2 Feb 2018, Iturriaga Woelfel, Markus wrote:
>>>
>>>
>>> I am running short on ideas how to copy this.
>>
>> If you are copying to an identical disk you can use dd on the raw
>> devices.The system may be confused by the identical UUID's if you
>> have both drives connected when you reboot, though.
>
>Yep, the problem is, it's live filesystem in /var mount and I'd need 
>another day offline. I ended up using LVM pvmove instead of copy of 
>/var/lib/backuppc only. This is not exacly what I want, but it was
>already 
>too long without backup to spend more time copying. Maybe LVM mirror
>could 
>be used too... anyway, this is a lesson, that copying filesystem via
>files 
>could become impossible, which is quite interesting.
>
>> --
>>   Les Mikesell
>
>
>Adam Pribyl
>
>--
>Check out the vibrant tech community on one of the world's most
>engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>___
>BackupPC-users mailing list
>BackupPC-users@lists.sourceforge.net
>List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
>Wiki:http://backuppc.wiki.sourceforge.net
>Project: http://backuppc.sourceforge.net/

What's the issue with using LVM?  Unless you need to reinitialize the whole fs, 
i.e. increase EXT4 inode count or switch to another fs, believe this is a 
perfect example of where LVM shines.

Raoul
-- 
DI (FH) Raoul Bhatia M.Sc.
E-Mail. ra...@bhatia.at
Tel. +43 699 10132530--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Best way to copy backuppc data

[Adam Pribyl]

On Sat, 3 Feb 2018, Les Mikesell wrote:


On Sat, Feb 3, 2018 at 5:50 AM, Adam Pribyl  wrote:

On Fri, 2 Feb 2018, Iturriaga Woelfel, Markus wrote:


I am running short on ideas how to copy this.


If you are copying to an identical disk you can use dd on the raw
devices.The system may be confused by the identical UUID's if you
have both drives connected when you reboot, though.


Yep, the problem is, it's live filesystem in /var mount and I'd need 
another day offline. I ended up using LVM pvmove instead of copy of 
/var/lib/backuppc only. This is not exacly what I want, but it was already 
too long without backup to spend more time copying. Maybe LVM mirror could 
be used too... anyway, this is a lesson, that copying filesystem via files 
could become impossible, which is quite interesting.



--
  Les Mikesell



Adam Pribyl

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Best way to copy backuppc data

[Les Mikesell]

On Sat, Feb 3, 2018 at 5:50 AM, Adam Pribyl  wrote:
> On Fri, 2 Feb 2018, Iturriaga Woelfel, Markus wrote:
>
>
> I am running short on ideas how to copy this.

If you are copying to an identical disk you can use dd on the raw
devices.The system may be confused by the identical UUID's if you
have both drives connected when you reboot, though.

-- 
   Les Mikesell
  lesmikes...@gmail.com

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Best way to copy backuppc data

[Holger Parplies]

Hi,

Adam Pribyl wrote on 2018-02-02 08:09:47 +0100 [[BackupPC-users] Best way to 
copy backuppc data]:
> What is to best way to copy the BackupPC data?
> [...]
> Still its taking somewhat too long... should I use a different method?

BackupPC_copyPool ... I'll send it to you on Monday. Sorry, in a hurry right
now.

Regards,
Holger

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Best way to copy backuppc data

[Alexander Moisseev via BackupPC-users]

On 2/3/2018 2:50 PM, Adam Pribyl wrote:

On Fri, 2 Feb 2018, Iturriaga Woelfel, Markus wrote:


I tried the "dump | restore" way,

dump -0 -f - /var/lib/backuppc | restore -r -f -

after few minutes:
restore: cannot write to file /tmp//rstdir1517604355: No space left on
device
   DUMP: Broken pipe
   DUMP: The ENTIRE dump is aborted.

seems restore is first writing some files to /tmp.. ok used
dump -0 -f - /var/lib/backuppc | restore -r -T /copy -f -

when it gets to the
DUMP: dumping (Pass IV) [regular files]

it just stays there for 12h with almost nothing copied.



Dump is fast because it copies file system blocks. As restore write files, it 
is slow.
You can take dumps relatively fast if you will store the dump in a file instead 
of piping it to restore.
Dump of 60G v3 cpool takes about 2 hours, but restore of this dump takes about 
1 day.

tar -C /var/lib/backuppc --one-filesystem --acls --xattrs -cf - . | tar 

-C /copy -xvf -

How does this deal with hardlinks?

I am running short on ideas how to copy this.


To get rid of hardlinks, consider to upgrade from BackupPC v3 to v4.


This seems to be yet another reason why I want to move this to RAID1 now


Another option is to move to ZFS and use `zfs send` and `zfs receive`.

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Best way to copy backuppc data

[Adam Pribyl]

On Fri, 2 Feb 2018, Iturriaga Woelfel, Markus wrote:




I used rsync to copy data including hard links:
rsync -avxHAX --progress /var/lib/backuppc /copy



Because of all the hard links, rsync gets incredibly slow and uses a
huge amount of space in /tmp when trying to copy BackupPC (v.3)


Yes this is v 3.3.0.


directories. Since it looks like you're doing this on the same system,
something like a tar pipe may work better for you.


I gave up the rsync, due to hardlinks - even thou it does the right thing, 
and was almost there.
The /tmp is fine, but the speed is 3GB per day on hardlinks, this is 
terrible.


I tried the "dump | restore" way,

dump -0 -f - /var/lib/backuppc | restore -r -f -

after few minutes:
restore: cannot write to file /tmp//rstdir1517604355: No space left on
device
  DUMP: Broken pipe
  DUMP: The ENTIRE dump is aborted.

seems restore is first writing some files to /tmp.. ok used
dump -0 -f - /var/lib/backuppc | restore -r -T /copy -f -

when it gets to the
DUMP: dumping (Pass IV) [regular files]

it just stays there for 12h with almost nothing copied.

tar -C /var/lib/backuppc --one-filesystem --acls --xattrs -cf - . | tar 

-C /copy -xvf -

How does this deal with hardlinks?

I am running short on ideas how to copy this.

This seems to be yet another reason why I want to move this to RAID1 
now


(double check that for sanity). You can achieve something similar with 

the dump/restore commands if t$


I've had some success with this. PS - removing the "verbose" part will 

likely speed up your transfer $


Markus



Thanks anyway

Adam Pribyl

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

[BackupPC-users] Best way to copy backuppc data

[Adam Pribyl]

What is to best way to copy the BackupPC data?

I used rsync to copy data including hard links:
rsync -avxHAX --progress /var/lib/backuppc /copy

on 200G backup data - it is running for 4 days, transfered 188G of 200G 
and now spending whole day on copying the cpool hashes (probably hard 
links?). Of course the backuppc itself is not running to interfere with 
the process of copy, it is running on rotational 3TB disks from one disk 
to different disk. Still its taking somewhat too long... should I use a 
different method?


Adam Pribyl




--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/