date:20181127

Re: [BackupPC-users] Find file given digest (and a decompression error)

2018-11-27 Thread Guillermo Rozas

>
> > Of course, you're right :) (although pigz failed only in 2 files out of
> > several thousands).
>
> oh well, I was wondering about that. I've yet to see such a file (and
> probably never will, because I disabled pool compression for good and
> now use btrfs' lzop filesystem-based compression), but...
>

I've found a third, all of them 6GB+ ISO. I'm starting to see a pattern :P

> BackupPC_zcat decompresses both files correctly and their checksums are
> > correct now. However, at least with one of the files there is something
> > fishy going on because the compressed version is 60KB, the decompressed
> > is 7GB!
>
> I'd bet that those two files are extremely sparse.
> There are good reasons for such a file to be generated: e.g., from a
> ddrescue run that skipped lots of bad areas on a drive, or a VM disk
> image with a recently formatted partition, or similar. On many modern
> file systems supporting sparse files, the overhead for the holes in the
> file is negligible, so it's easier from a user perspective to allocate
> the "full" file and rely on the filesystem's abilities to optimize
> storage and access.
> However, some of BackupPC's transfer methods (in particular, rsync)
> cannot treat sparse files natively, but since they compress so well,
> that's hardly an issue for transfer nor storage on the server.
>

Thanks for the nice explanation. Unfortunately in this case was a rather
more mundane reason, like me failing to properly read the number of digits
of a big number...


> The reason why I recommended pigz (unfortunately without an appropriate
> disclaimer) is that it
> - never failed on me, for the files I had around at that time, and
> - it was *magnitudes* faster than BackupPC_zcat.
>
> But I had a severely CPU-limited machine; YMMV with a more powerful CPU.
> Depending on your use case (and performance experience), it might still
> be clever to run pigz first and only run BackupPC_zcat if there is a
> mismatch. If a pigz-decompressed file matches the expected hash, I'd bet
> at approximately 1 : 2^64 that no corruption happened.
>

I'm very severely CPU-limited (Banana Pi), so this can make a huge
difference. I tested it by checking two top level cpool dirs (roughly 1/64
~ 1.5% of the pool). I compared pigz, zlib-flate and BackupPC_zcat and on
my system:
- both pigz and zlib-flate are much faster than BackupPC_zcat, they take
around a quarter of the time to check the files
- pigz is marginally faster than zlib-flate
- BackupPC_zcat puts the lower load on the CPU, zlib-flate's load is 30-35%
higher, and pigz's is a whooping 80-100% higher (pigz's load is actually
higher than 2 in this 2-core system)
- of course, BackupPC_zcat has the advantage of always working, zlib-flate
and pigz fail at the same files (very few)

With this data, I modified my script to normally run zlib-flate to check
the files, and re-check every failure with BackupPC_zcat before calling it
a real error. I think this gets the best balance between load on the system
and time spent checking the pool (I can traverse the entire pool in 32 days
with ~30 min of checking every day).

> I'll check those 2 files tonight, and hopefully
> > have a script working by the weekend.
>
> Cool! If you don't mind and are allowed to, please share here...
>

The check script is almost there, I want to verify it for a couple of days
more before sharing it. The find script seems a bit harder to code that
what I first thought :)

Cheers,
Guillermo
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Find file given digest (and a decompression error)

2018-11-27 Thread Alexander Kobel


Hi,

On 27.11.18 13:40, Guillermo Rozas wrote:

Pigz doesn't correctly support BackupPC compressed files, although
it will in some cases.  The reported error is likely the problem. 
Please use BackupPC_zcat instead and report back.



Of course, you're right :) (although pigz failed only in 2 files out of 
several thousands).


oh well, I was wondering about that. I've yet to see such a file (and 
probably never will, because I disabled pool compression for good and 
now use btrfs' lzop filesystem-based compression), but...


BackupPC_zcat decompresses both files correctly and their checksums are 
correct now. However, at least with one of the files there is something 
fishy going on because the compressed version is 60KB, the decompressed 
is 7GB!


... from what I understand, the main difference between "standard" 
gzip/pigz and BackupPC's variant is that the latter adds additional 
"sync" operations when a (e.g. sparse) file compresses way better than 
usual. In that case, BackupPC's zipping mechanism ensures that 
decompression only requires a fixed amount of memory, at the expense 
that extremely compressible data does not compress to 0.01%, but 
only to 0.001% or something. (I'm lacking the details, sorry.)


I'd bet that those two files are extremely sparse.
There are good reasons for such a file to be generated: e.g., from a 
ddrescue run that skipped lots of bad areas on a drive, or a VM disk 
image with a recently formatted partition, or similar. On many modern 
file systems supporting sparse files, the overhead for the holes in the 
file is negligible, so it's easier from a user perspective to allocate 
the "full" file and rely on the filesystem's abilities to optimize 
storage and access.
However, some of BackupPC's transfer methods (in particular, rsync) 
cannot treat sparse files natively, but since they compress so well, 
that's hardly an issue for transfer nor storage on the server.



The reason why I recommended pigz (unfortunately without an appropriate 
disclaimer) is that it

- never failed on me, for the files I had around at that time, and
- it was *magnitudes* faster than BackupPC_zcat.

But I had a severely CPU-limited machine; YMMV with a more powerful CPU.
Depending on your use case (and performance experience), it might still 
be clever to run pigz first and only run BackupPC_zcat if there is a 
mismatch. If a pigz-decompressed file matches the expected hash, I'd bet 
at approximately 1 : 2^64 that no corruption happened.



Which brings me to:

I added a guide to the Wiki to find out where a pool file is
referenced

.


That's great, thanks!


Indeed, thanks!

I'll check those 2 files tonight, and hopefully 
have a script working by the weekend.


Cool! If you don't mind and are allowed to, please share here...


Cheers,
Alex



smime.p7s
Description: S/MIME Cryptographic Signature
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] (xfer start failed: $Conf{RsyncBackupPCPath} is set to , which isn't a valid executable) (SOLVED)

2018-11-27 Thread Ed Burgstaler

Okay I finally figured out what the issue was.
Apparently, it was an rsync over ssh issue where backuppc wasn't providing
the correct ssh commands so the connection was terminated immediately.
What I had to do was edit the per host config and enable the "RsyncSshArgs
override" and add the SSH port that my server uses to connect "$sshPath -p
6188 -q -x -l root"
After making this change it is now backing up the way it should and no more
errors.
Sorry to both but sometimes it helps to think out load.

-Ed

-Original Message-
From: Ed Burgstaler  
Sent: Tuesday, November 27, 2018 12:42 PM
To: 'General list for user discussion, questions and support'

Subject: Re: [BackupPC-users] (xfer start failed: $Conf{RsyncBackupPCPath}
is set to , which isn't a valid executable)

Does it matter what rsync version is running on the client side?
I'm running rsync  version 3.1.1  protocol version 31 on my Linux client.
I can SSH to it as backuppc user no problems just seems to be the rsync that
is the problem since upgrading to rsync-bpc-3.0.9.13.

-Original Message-
From: Ari Sovijärvi 
Sent: Tuesday, November 27, 2018 10:47 AM
To: backuppc-users@lists.sourceforge.net
Subject: Re: [BackupPC-users] (xfer start failed: $Conf{RsyncBackupPCPath}
is set to , which isn't a valid executable)

> Okay I did locate the path just as you indicated and added it to the 
> config but now it gives me a different error:
> Got fatal error during xfer (rsync error: unexplained error (code 255) 
> at
> io.c(629) [Receiver=3.0.9.13])
> Not sure what that means.

See the other end's logs for clues for bad authentication, wrong share name
or something similar.


___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/



___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/



___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] (xfer start failed: $Conf{RsyncBackupPCPath} is set to , which isn't a valid executable)

2018-11-27 Thread Ed Burgstaler

Does it matter what rsync version is running on the client side?
I'm running rsync  version 3.1.1  protocol version 31 on my Linux client.
I can SSH to it as backuppc user no problems just seems to be the rsync that
is the problem since upgrading to rsync-bpc-3.0.9.13.

-Original Message-
From: Ari Sovijärvi  
Sent: Tuesday, November 27, 2018 10:47 AM
To: backuppc-users@lists.sourceforge.net
Subject: Re: [BackupPC-users] (xfer start failed: $Conf{RsyncBackupPCPath}
is set to , which isn't a valid executable)

> Okay I did locate the path just as you indicated and added it to the 
> config but now it gives me a different error:
> Got fatal error during xfer (rsync error: unexplained error (code 255) 
> at
> io.c(629) [Receiver=3.0.9.13])
> Not sure what that means.

See the other end's logs for clues for bad authentication, wrong share name
or something similar.


___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/



___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] (xfer start failed: $Conf{RsyncBackupPCPath} is set to , which isn't a valid executable)

2018-11-27 Thread Ed Burgstaler

Was all working fine prior to upgrading from 3.3.2 to version 4.3.0 so it
must be something related to that rsync-bpc file.

-Original Message-
From: Ari Sovijärvi  
Sent: Tuesday, November 27, 2018 10:47 AM
To: backuppc-users@lists.sourceforge.net
Subject: Re: [BackupPC-users] (xfer start failed: $Conf{RsyncBackupPCPath}
is set to , which isn't a valid executable)

> Okay I did locate the path just as you indicated and added it to the 
> config but now it gives me a different error:
> Got fatal error during xfer (rsync error: unexplained error (code 255) 
> at
> io.c(629) [Receiver=3.0.9.13])
> Not sure what that means.

See the other end's logs for clues for bad authentication, wrong share name
or something similar.


___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/



___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] (xfer start failed: $Conf{RsyncBackupPCPath} is set to , which isn't a valid executable)

2018-11-27 Thread Ari Sovijärvi


Okay I did locate the path just as you indicated and added it to the config
but now it gives me a different error:
Got fatal error during xfer (rsync error: unexplained error (code 255) at
io.c(629) [Receiver=3.0.9.13])
Not sure what that means.


See the other end's logs for clues for bad authentication, wrong share 
name or something similar.



___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] (xfer start failed: $Conf{RsyncBackupPCPath} is set to , which isn't a valid executable)

2018-11-27 Thread Ed Burgstaler

Oh I am on CentOS6x and my path was
 /usr/local/bin/rsync_bpc 

That is the path I added to my per host config. Not sure why the other error
now 

-Original Message-
From: Ed Burgstaler  
Sent: Tuesday, November 27, 2018 10:38 AM
To: 'General list for user discussion, questions and support'

Subject: Re: [BackupPC-users] (xfer start failed: $Conf{RsyncBackupPCPath}
is set to , which isn't a valid executable)

Okay I did locate the path just as you indicated and added it to the config
but now it gives me a different error:
Got fatal error during xfer (rsync error: unexplained error (code 255) at
io.c(629) [Receiver=3.0.9.13])
Not sure what that means.
Thank you

-Original Message-
From: Ari Sovijärvi 
Sent: Tuesday, November 27, 2018 10:33 AM
To: backuppc-users@lists.sourceforge.net
Subject: Re: [BackupPC-users] (xfer start failed: $Conf{RsyncBackupPCPath}
is set to , which isn't a valid executable)

On 27.11.2018 18.16, Ed Burgstaler wrote:
> I have just upgraded from backuppc 3.3.2 to the newest version 4.3 but 
> I now get the error below:
> (xfer start failed: $Conf{RsyncBackupPCPath} is set to , which isn't a 
> valid executable) How can I resolve this please.
> I did install the complied version of rsync-bpc-3.0.9.13 but I dont 
> know where it put it so what should the default path be if this is the 
> issue?

Locate the rsync_bpc executable and put that with its full path there. 
In my Debian packaged version it's /usr/bin/rsync_bpc


___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/



___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/



___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] (xfer start failed: $Conf{RsyncBackupPCPath} is set to , which isn't a valid executable)

2018-11-27 Thread Ed Burgstaler

Okay I did locate the path just as you indicated and added it to the config
but now it gives me a different error:
Got fatal error during xfer (rsync error: unexplained error (code 255) at
io.c(629) [Receiver=3.0.9.13]) 
Not sure what that means.
Thank you

-Original Message-
From: Ari Sovijärvi  
Sent: Tuesday, November 27, 2018 10:33 AM
To: backuppc-users@lists.sourceforge.net
Subject: Re: [BackupPC-users] (xfer start failed: $Conf{RsyncBackupPCPath}
is set to , which isn't a valid executable)

On 27.11.2018 18.16, Ed Burgstaler wrote:
> I have just upgraded from backuppc 3.3.2 to the newest version 4.3 but 
> I now get the error below:
> (xfer start failed: $Conf{RsyncBackupPCPath} is set to , which isn't a 
> valid executable) How can I resolve this please.
> I did install the complied version of rsync-bpc-3.0.9.13 but I dont 
> know where it put it so what should the default path be if this is the 
> issue?

Locate the rsync_bpc executable and put that with its full path there. 
In my Debian packaged version it's /usr/bin/rsync_bpc


___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/



___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] (xfer start failed: $Conf{RsyncBackupPCPath} is set to , which isn't a valid executable)

2018-11-27 Thread Ari Sovijärvi


On 27.11.2018 18.16, Ed Burgstaler wrote:
I have just upgraded from backuppc 3.3.2 to the newest version 4.3 but I 
now get the error below:
(xfer start failed: $Conf{RsyncBackupPCPath} is set to , which isn't a 
valid executable)

How can I resolve this please.
I did install the complied version of rsync-bpc-3.0.9.13 but I don’t 
know where it put it so what should the default path be if this is the 
issue?


Locate the rsync_bpc executable and put that with its full path there. 
In my Debian packaged version it's /usr/bin/rsync_bpc



___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

[BackupPC-users] (xfer start failed: $Conf{RsyncBackupPCPath} is set to , which isn't a valid executable)

2018-11-27 Thread Ed Burgstaler

I have just upgraded from backuppc 3.3.2 to the newest version 4.3 but I now
get the error below:

 

(xfer start failed: $Conf{RsyncBackupPCPath} is set to , which isn't a valid
executable)

 

How can I resolve this please.

 

I did install the complied version of rsync-bpc-3.0.9.13 but I don't know
where it put it so what should the default path be if this is the issue?

Thanks

__

 

Ed 

 

___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Find file given digest (and a decompression error)

2018-11-27 Thread Guillermo Rozas

>
> Pigz doesn't correctly support BackupPC compressed files, although it will
> in some cases.  The reported error is likely the problem.  Please use
> BackupPC_zcat instead and report back.
>

Of course, you're right :) (although pigz failed only in 2 files out of
several thousands).

BackupPC_zcat decompresses both files correctly and their checksums are
correct now. However, at least with one of the files there is something
fishy going on because the compressed version is 60KB, the decompressed is
7GB! Which brings me to:

I added a guide to the Wiki to find out where a pool file is referenced
> 
> .
>

That's great, thanks! I'll check those 2 files tonight, and hopefully have
a script working by the weekend.

Best regards,
Guillermo
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Find file given digest (and a decompression error)

Re: [BackupPC-users] Find file given digest (and a decompression error)

Re: [BackupPC-users] (xfer start failed: $Conf{RsyncBackupPCPath} is set to , which isn't a valid executable) (SOLVED)

Re: [BackupPC-users] (xfer start failed: $Conf{RsyncBackupPCPath} is set to , which isn't a valid executable)

Re: [BackupPC-users] (xfer start failed: $Conf{RsyncBackupPCPath} is set to , which isn't a valid executable)

Re: [BackupPC-users] (xfer start failed: $Conf{RsyncBackupPCPath} is set to , which isn't a valid executable)

Re: [BackupPC-users] (xfer start failed: $Conf{RsyncBackupPCPath} is set to , which isn't a valid executable)

Re: [BackupPC-users] (xfer start failed: $Conf{RsyncBackupPCPath} is set to , which isn't a valid executable)

Re: [BackupPC-users] (xfer start failed: $Conf{RsyncBackupPCPath} is set to , which isn't a valid executable)

[BackupPC-users] (xfer start failed: $Conf{RsyncBackupPCPath} is set to , which isn't a valid executable)

Re: [BackupPC-users] Find file given digest (and a decompression error)

11 matches

Site Navigation

Mail list logo

Footer information