Re: [BackupPC-users] BackuPC 4 hang during transfer

[Gandalf Corvotempesta]

Without using checksum, how backuppc does the pool match?

AFAIK it will use the checksum as filename but if checksum is not sent
during the transfer..

Anyway, I've seen that checksum is used only on fulls. Currently I've set a
full every 60 days, because incrementals done after a successful full are
rocket fast

Would be better, for me, to totally remove the checksum option even from
full, staying with the default rsync behavior but I don't know if this
makes a bad use of backuppc pool

Il 23 set 2017 4:33 AM, "Adam Goryachev" <
mailingli...@websitemanagers.com.au> ha scritto:

> 2017-09-22 17:24 GMT+02:00 Gandalf Corvotempesta
>> :
>>
>>> 2017-09-22 17:20 GMT+02:00 Les Mikesell :
>>>
 How does your overall CPU and RAM use look while this is happening?
 Remember, your situation is unusual in that you are competing with the
 ZFS compression activity.

>>> CPU almost idle, RAM used at 70%, due to ZFS ARC cache (50% of ram)
>>> No swap.
>>>
>>
> On 23/9/17 02:50, Gandalf Corvotempesta wrote:
>
>> Just removed "--checksum" from the BackupPC arguments.
>> Now is... FAAAST
>>
>> What i've backupped in about 40hours, now took 60 minutes.
>>
>> YES: 40 hours => 60 minutes.
>>
>> Is --checksum really needed ? (checksum is also missing from rsnapshot
>> arguments, that's why rsnapshot is rocket fast)
>>
> I can't be sure for BPC4, but maybe you need more than one full backup to
> get the checkum information available to BPC. I think on v3 you needed two
> full backups before this would happen.
> The other point to consider, is that this shows you *DID* have a
> performance issue, but you didn't seem to find it. checksum will increase
> the read load and CPU load on at least the client (and possibly BPC server
> depending on where it gets the checksum info from). So you should have seen
> that you were being limited by disk IO or CPU on either BPC server, or the
> client. I'm not sure of the memory requirement for the checksum option, but
> this too might have been an issue, especially if BPC tried to uncompress
> the file into memory. Also, all of this would trash your disk read cache on
> both systems, further increasing demands on the disks.
>
> Whether you need to use --checksum or not, will depend on if you are happy
> to potentially skip backing up some files without knowing about it until
> you need to do a restore. Of course, this is a little contrived, as it
> still requires:
> a) size doesn't change
> b) timestamp doesn't change
> c) content *does* change
> That is not a normal process, but it is the corner case that always ends
> up being the most important file ;)
>
>
>
> Regards,
> Adam
>
>
> 
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> ___
> BackupPC-users mailing list
> BackupPC-users@lists.sourceforge.net
> List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
> Wiki:http://backuppc.wiki.sourceforge.net
> Project: http://backuppc.sourceforge.net/
>
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] BackuPC 4 hang during transfer

[Adam Goryachev]

2017-09-22 17:24 GMT+02:00 Gandalf Corvotempesta
:

2017-09-22 17:20 GMT+02:00 Les Mikesell :

How does your overall CPU and RAM use look while this is happening?
Remember, your situation is unusual in that you are competing with the
ZFS compression activity.

CPU almost idle, RAM used at 70%, due to ZFS ARC cache (50% of ram)
No swap.


On 23/9/17 02:50, Gandalf Corvotempesta wrote:

Just removed "--checksum" from the BackupPC arguments.
Now is... FAAAST

What i've backupped in about 40hours, now took 60 minutes.

YES: 40 hours => 60 minutes.

Is --checksum really needed ? (checksum is also missing from rsnapshot
arguments, that's why rsnapshot is rocket fast)
I can't be sure for BPC4, but maybe you need more than one full backup 
to get the checkum information available to BPC. I think on v3 you 
needed two full backups before this would happen.
The other point to consider, is that this shows you *DID* have a 
performance issue, but you didn't seem to find it. checksum will 
increase the read load and CPU load on at least the client (and possibly 
BPC server depending on where it gets the checksum info from). So you 
should have seen that you were being limited by disk IO or CPU on either 
BPC server, or the client. I'm not sure of the memory requirement for 
the checksum option, but this too might have been an issue, especially 
if BPC tried to uncompress the file into memory. Also, all of this would 
trash your disk read cache on both systems, further increasing demands 
on the disks.


Whether you need to use --checksum or not, will depend on if you are 
happy to potentially skip backing up some files without knowing about it 
until you need to do a restore. Of course, this is a little contrived, 
as it still requires:

a) size doesn't change
b) timestamp doesn't change
c) content *does* change
That is not a normal process, but it is the corner case that always ends 
up being the most important file ;)




Regards,
Adam


--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] BackuPC 4 hang during transfer

[Gandalf Corvotempesta]

Just removed "--checksum" from the BackupPC arguments.
Now is... FAAAST

What i've backupped in about 40hours, now took 60 minutes.

YES: 40 hours => 60 minutes.

Is --checksum really needed ? (checksum is also missing from rsnapshot
arguments, that's why rsnapshot is rocket fast)

2017-09-22 17:24 GMT+02:00 Gandalf Corvotempesta
:
> 2017-09-22 17:20 GMT+02:00 Les Mikesell :
>> How does your overall CPU and RAM use look while this is happening?
>> Remember, your situation is unusual in that you are competing with the
>> ZFS compression activity.
>
> CPU almost idle, RAM used at 70%, due to ZFS ARC cache (50% of ram)
> No swap.

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] BackuPC 4 hang during transfer

[Gandalf Corvotempesta]

2017-09-22 17:20 GMT+02:00 Les Mikesell :
> How does your overall CPU and RAM use look while this is happening?
> Remember, your situation is unusual in that you are competing with the
> ZFS compression activity.

CPU almost idle, RAM used at 70%, due to ZFS ARC cache (50% of ram)
No swap.

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] BackuPC 4 hang during transfer

[Les Mikesell]

On Fri, Sep 22, 2017 at 9:41 AM, Gandalf Corvotempesta
 wrote:
> Running transfers with loglevel 4, i can see that most of slowdown are
> caused by:
>
> G bpc_file_checksum()
>
> calls.
>

How does your overall CPU and RAM use look while this is happening?
Remember, your situation is unusual in that you are competing with the
ZFS compression activity.

-- 
   Les Mikesell
 lesmikes...@gmail.com

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] BackuPC 4 hang during transfer

[Gandalf Corvotempesta]

Running transfers with loglevel 4, i can see that most of slowdown are
caused by:

G bpc_file_checksum()

calls.

2017-09-22 15:31 GMT+02:00 Gandalf Corvotempesta
:
> Also tried with "nc". I'm able to push/pull 110-120MB/s between these
> two servers.
>
> 2017-09-22 15:22 GMT+02:00 Gandalf Corvotempesta
> :
>> I've made a little bit progress.
>> Slow select calls are always network related but my net is not
>> saturated and I'm able to push 1gbp/s regurally with iperf.
>> Moreover, on client side, rsync is sending files very very fast.
>>
>> Could be a bottleneck in BackupPC network management ?
>>
>> 2017-09-22 15:12 GMT+02:00 Gandalf Corvotempesta
>> :
>>> 2017-09-21 14:37 GMT+02:00 Craig Barratt via BackupPC-users
>>> :
 I recommend running strace -p PID -T on the rsync_bpc process to see what 
 it
 is up to, and how long various system calls take.  I agree your backups
 should run much faster.
>>>
>>> Here is an extract.
>>> Some calls are really slow, more than 1 second.
>>> They seems to be select, right ? Writes seems to be ok, less than 1/10 
>>> second
>>>
>>> [pid  3453] select(4, [3], [], NULL, {tv_sec=60, tv_usec=0} 
>>> [pid  3452] read(5, "75601860.M846377P28101.xx.gu"..., 8184) = 4081 
>>> <0.35>
>>> [pid  3452] select(6, [5], [], NULL, {tv_sec=60, tv_usec=0} 
>>> [pid  3453] <... select resumed> )  = 1 (in [3], left {tv_sec=59,
>>> tv_usec=220734}) <0.779284>
>>> [pid  3453] read(3, "\374\17\0\7", 4)   = 4 <0.35>
>>> [pid  3453] select(4, [3], [], NULL, {tv_sec=60, tv_usec=0}) = 1 (in
>>> [3], left {tv_sec=59, tv_usec=98}) <0.28>
>>> [pid  3453] read(3, ".yy,S=1262266,W=1278715:2,"..., 4092) = 2896 <0.34>
>>> [pid  3453] select(4, [3], [], NULL, {tv_sec=60, tv_usec=0}) = 1 (in
>>> [3], left {tv_sec=59, tv_usec=98}) <0.29>
>>> [pid  3453] read(3, "383235089.M974673P2358.xx.gu"..., 1196) = 1196 
>>> <0.27>
>>> [pid  3453] select(7, NULL, [6], [6], {tv_sec=60, tv_usec=0}) = 1 (out
>>> [6], left {tv_sec=59, tv_usec=98}) <0.35>
>>> [pid  3453] write(6, "381086351.M234084P17834.xx.g"..., 4087) = 4087 
>>> <0.39>
>>> [pid  3452] <... select resumed> )  = 1 (in [5], left {tv_sec=59,
>>> tv_usec=220263}) <0.779785>
>>> [pid  3453] select(4, [3], [], NULL, {tv_sec=60, tv_usec=0} 
>>> [pid  3452] read(5, "381086351.M234084P17834.xx.g"..., 8184) = 4087 
>>> <0.40>
>>> [pid  3452] select(6, [5], [], NULL, {tv_sec=60, tv_usec=0} 
>>> [pid  3453] <... select resumed> )  = 1 (in [3], left {tv_sec=58,
>>> tv_usec=683238}) <1.316780>
>>> [pid  3453] read(3, "\374\17\0\7", 4)   = 4 <0.56>
>>> [pid  3453] select(4, [3], [], NULL, {tv_sec=60, tv_usec=0}) = 1 (in
>>> [3], left {tv_sec=59, tv_usec=98}) <0.29>
>>> [pid  3453] read(3,
>>> "t,S=9113,W=9346:2,S\0\231#V\10\300,\270\215\21z\360\250"..., 4092) =
>>> 4092 <0.28>
>>> [pid  3453] select(7, NULL, [6], [6], {tv_sec=60, tv_usec=0}) = 1 (out
>>> [6], left {tv_sec=59, tv_usec=98}) <0.47>
>>> [pid  3453] write(6, "45773421.M45371P9859.xx.gues"..., 4091) = 4091 
>>> <0.51>
>>> [pid  3452] <... select resumed> )  = 1 (in [5], left {tv_sec=58,
>>> tv_usec=682833}) <1.317230>
>>> [pid  3453] select(4, [3], [], NULL, {tv_sec=60, tv_usec=0} 
>>> [pid  3452] read(5, "45773421.M45371P9859.xx.gues"..., 8184) = 4091 
>>> <0.33>
>>> [pid  3452] select(6, [5], [], NULL, {tv_sec=60, tv_usec=0} 
>>> [pid  3453] <... select resumed> )  = 1 (in [3], left {tv_sec=58,
>>> tv_usec=879335}) <1.120688>
>>> [pid  3453] read(3, "\374\17\0\7", 4)   = 4 <0.57>
>>> [pid  3453] select(4, [3], [], NULL, {tv_sec=60, tv_usec=0}) = 1 (in
>>> [3], left {tv_sec=59, tv_usec=97}) <0.29>
>>> [pid  3453] read(3, ".it,S=17245,W=17685:2,S\0]CR\364\340\327Q\f"...,
>>> 4092) = 4092 <0.26>
>>> [pid  3453] select(7, NULL, [6], [6], {tv_sec=60, tv_usec=0}) = 1 (out
>>> [6], left {tv_sec=59, tv_usec=98}) <0.64>
>>> [pid  3453] write(6, "389990620.M619008P11470.xx.g"..., 4092) = 4092 
>>> <0.51>
>>> [pid  3452] <... select resumed> )  = 1 (in [5], left {tv_sec=58,
>>> tv_usec=878920}) <1.121130>
>>> [pid  3453] select(7, NULL, [6], [6], {tv_sec=60, tv_usec=0} >> ...>
>>> [pid  3452] read(5,  
>>> [pid  3453] <... select resumed> )  = 1 (out [6], left {tv_sec=59,
>>> tv_usec=98}) <0.29>
>>> [pid  3452] <... read resumed> "389990620.M619008P11470.xx.g"...,
>>> 8184) = 4092 <0.32>
>>> [pid  3453] write(6, "\333\32;^\26\252\321[:>8", 11 
>>> [pid  3452] select(6, [5], [], NULL, {tv_sec=60, tv_usec=0} 
>>> [pid  3453] <... write resumed> )   = 11 <0.30>
>>> [pid  3452] <... select resumed> )  = 1 (in [5], left {tv_sec=59,
>>> tv_usec=98}) <0.26>
>>> [pid  3453] select(4, [3], [], NULL, {tv_sec=60, tv_usec=0} 
>>> [pid  3452] read(5, "\333\32;^\26\252\321[:>8", 8184) = 11 <0.11>
>>> [pid  

Re: [BackupPC-users] BackuPC 4 hang during transfer

[Gandalf Corvotempesta]

Also tried with "nc". I'm able to push/pull 110-120MB/s between these
two servers.

2017-09-22 15:22 GMT+02:00 Gandalf Corvotempesta
:
> I've made a little bit progress.
> Slow select calls are always network related but my net is not
> saturated and I'm able to push 1gbp/s regurally with iperf.
> Moreover, on client side, rsync is sending files very very fast.
>
> Could be a bottleneck in BackupPC network management ?
>
> 2017-09-22 15:12 GMT+02:00 Gandalf Corvotempesta
> :
>> 2017-09-21 14:37 GMT+02:00 Craig Barratt via BackupPC-users
>> :
>>> I recommend running strace -p PID -T on the rsync_bpc process to see what it
>>> is up to, and how long various system calls take.  I agree your backups
>>> should run much faster.
>>
>> Here is an extract.
>> Some calls are really slow, more than 1 second.
>> They seems to be select, right ? Writes seems to be ok, less than 1/10 second
>>
>> [pid  3453] select(4, [3], [], NULL, {tv_sec=60, tv_usec=0} 
>> [pid  3452] read(5, "75601860.M846377P28101.xx.gu"..., 8184) = 4081 
>> <0.35>
>> [pid  3452] select(6, [5], [], NULL, {tv_sec=60, tv_usec=0} 
>> [pid  3453] <... select resumed> )  = 1 (in [3], left {tv_sec=59,
>> tv_usec=220734}) <0.779284>
>> [pid  3453] read(3, "\374\17\0\7", 4)   = 4 <0.35>
>> [pid  3453] select(4, [3], [], NULL, {tv_sec=60, tv_usec=0}) = 1 (in
>> [3], left {tv_sec=59, tv_usec=98}) <0.28>
>> [pid  3453] read(3, ".yy,S=1262266,W=1278715:2,"..., 4092) = 2896 <0.34>
>> [pid  3453] select(4, [3], [], NULL, {tv_sec=60, tv_usec=0}) = 1 (in
>> [3], left {tv_sec=59, tv_usec=98}) <0.29>
>> [pid  3453] read(3, "383235089.M974673P2358.xx.gu"..., 1196) = 1196 
>> <0.27>
>> [pid  3453] select(7, NULL, [6], [6], {tv_sec=60, tv_usec=0}) = 1 (out
>> [6], left {tv_sec=59, tv_usec=98}) <0.35>
>> [pid  3453] write(6, "381086351.M234084P17834.xx.g"..., 4087) = 4087 
>> <0.39>
>> [pid  3452] <... select resumed> )  = 1 (in [5], left {tv_sec=59,
>> tv_usec=220263}) <0.779785>
>> [pid  3453] select(4, [3], [], NULL, {tv_sec=60, tv_usec=0} 
>> [pid  3452] read(5, "381086351.M234084P17834.xx.g"..., 8184) = 4087 
>> <0.40>
>> [pid  3452] select(6, [5], [], NULL, {tv_sec=60, tv_usec=0} 
>> [pid  3453] <... select resumed> )  = 1 (in [3], left {tv_sec=58,
>> tv_usec=683238}) <1.316780>
>> [pid  3453] read(3, "\374\17\0\7", 4)   = 4 <0.56>
>> [pid  3453] select(4, [3], [], NULL, {tv_sec=60, tv_usec=0}) = 1 (in
>> [3], left {tv_sec=59, tv_usec=98}) <0.29>
>> [pid  3453] read(3,
>> "t,S=9113,W=9346:2,S\0\231#V\10\300,\270\215\21z\360\250"..., 4092) =
>> 4092 <0.28>
>> [pid  3453] select(7, NULL, [6], [6], {tv_sec=60, tv_usec=0}) = 1 (out
>> [6], left {tv_sec=59, tv_usec=98}) <0.47>
>> [pid  3453] write(6, "45773421.M45371P9859.xx.gues"..., 4091) = 4091 
>> <0.51>
>> [pid  3452] <... select resumed> )  = 1 (in [5], left {tv_sec=58,
>> tv_usec=682833}) <1.317230>
>> [pid  3453] select(4, [3], [], NULL, {tv_sec=60, tv_usec=0} 
>> [pid  3452] read(5, "45773421.M45371P9859.xx.gues"..., 8184) = 4091 
>> <0.33>
>> [pid  3452] select(6, [5], [], NULL, {tv_sec=60, tv_usec=0} 
>> [pid  3453] <... select resumed> )  = 1 (in [3], left {tv_sec=58,
>> tv_usec=879335}) <1.120688>
>> [pid  3453] read(3, "\374\17\0\7", 4)   = 4 <0.57>
>> [pid  3453] select(4, [3], [], NULL, {tv_sec=60, tv_usec=0}) = 1 (in
>> [3], left {tv_sec=59, tv_usec=97}) <0.29>
>> [pid  3453] read(3, ".it,S=17245,W=17685:2,S\0]CR\364\340\327Q\f"...,
>> 4092) = 4092 <0.26>
>> [pid  3453] select(7, NULL, [6], [6], {tv_sec=60, tv_usec=0}) = 1 (out
>> [6], left {tv_sec=59, tv_usec=98}) <0.64>
>> [pid  3453] write(6, "389990620.M619008P11470.xx.g"..., 4092) = 4092 
>> <0.51>
>> [pid  3452] <... select resumed> )  = 1 (in [5], left {tv_sec=58,
>> tv_usec=878920}) <1.121130>
>> [pid  3453] select(7, NULL, [6], [6], {tv_sec=60, tv_usec=0} 
>> [pid  3452] read(5,  
>> [pid  3453] <... select resumed> )  = 1 (out [6], left {tv_sec=59,
>> tv_usec=98}) <0.29>
>> [pid  3452] <... read resumed> "389990620.M619008P11470.xx.g"...,
>> 8184) = 4092 <0.32>
>> [pid  3453] write(6, "\333\32;^\26\252\321[:>8", 11 
>> [pid  3452] select(6, [5], [], NULL, {tv_sec=60, tv_usec=0} 
>> [pid  3453] <... write resumed> )   = 11 <0.30>
>> [pid  3452] <... select resumed> )  = 1 (in [5], left {tv_sec=59,
>> tv_usec=98}) <0.26>
>> [pid  3453] select(4, [3], [], NULL, {tv_sec=60, tv_usec=0} 
>> [pid  3452] read(5, "\333\32;^\26\252\321[:>8", 8184) = 11 <0.11>
>> [pid  3452] select(6, [5], [], NULL, {tv_sec=60, tv_usec=0} 
>> [pid  3453] <... select resumed> )  = 1 (in [3], left {tv_sec=58,
>> tv_usec=696881}) <1.303136>
>> [pid  3453] read(3, "\374\17\0\7", 4)   = 4 <0.28>
>> [pid  3453] select(4, [3], [], NULL, {tv_sec=60, tv_usec=0}) = 1 (in
>> [3], left {tv_sec=59, 

Re: [BackupPC-users] BackuPC 4 hang during transfer

[Gandalf Corvotempesta]

I've made a little bit progress.
Slow select calls are always network related but my net is not
saturated and I'm able to push 1gbp/s regurally with iperf.
Moreover, on client side, rsync is sending files very very fast.

Could be a bottleneck in BackupPC network management ?

2017-09-22 15:12 GMT+02:00 Gandalf Corvotempesta
:
> 2017-09-21 14:37 GMT+02:00 Craig Barratt via BackupPC-users
> :
>> I recommend running strace -p PID -T on the rsync_bpc process to see what it
>> is up to, and how long various system calls take.  I agree your backups
>> should run much faster.
>
> Here is an extract.
> Some calls are really slow, more than 1 second.
> They seems to be select, right ? Writes seems to be ok, less than 1/10 second
>
> [pid  3453] select(4, [3], [], NULL, {tv_sec=60, tv_usec=0} 
> [pid  3452] read(5, "75601860.M846377P28101.xx.gu"..., 8184) = 4081 <0.35>
> [pid  3452] select(6, [5], [], NULL, {tv_sec=60, tv_usec=0} 
> [pid  3453] <... select resumed> )  = 1 (in [3], left {tv_sec=59,
> tv_usec=220734}) <0.779284>
> [pid  3453] read(3, "\374\17\0\7", 4)   = 4 <0.35>
> [pid  3453] select(4, [3], [], NULL, {tv_sec=60, tv_usec=0}) = 1 (in
> [3], left {tv_sec=59, tv_usec=98}) <0.28>
> [pid  3453] read(3, ".yy,S=1262266,W=1278715:2,"..., 4092) = 2896 <0.34>
> [pid  3453] select(4, [3], [], NULL, {tv_sec=60, tv_usec=0}) = 1 (in
> [3], left {tv_sec=59, tv_usec=98}) <0.29>
> [pid  3453] read(3, "383235089.M974673P2358.xx.gu"..., 1196) = 1196 <0.27>
> [pid  3453] select(7, NULL, [6], [6], {tv_sec=60, tv_usec=0}) = 1 (out
> [6], left {tv_sec=59, tv_usec=98}) <0.35>
> [pid  3453] write(6, "381086351.M234084P17834.xx.g"..., 4087) = 4087 
> <0.39>
> [pid  3452] <... select resumed> )  = 1 (in [5], left {tv_sec=59,
> tv_usec=220263}) <0.779785>
> [pid  3453] select(4, [3], [], NULL, {tv_sec=60, tv_usec=0} 
> [pid  3452] read(5, "381086351.M234084P17834.xx.g"..., 8184) = 4087 <0.40>
> [pid  3452] select(6, [5], [], NULL, {tv_sec=60, tv_usec=0} 
> [pid  3453] <... select resumed> )  = 1 (in [3], left {tv_sec=58,
> tv_usec=683238}) <1.316780>
> [pid  3453] read(3, "\374\17\0\7", 4)   = 4 <0.56>
> [pid  3453] select(4, [3], [], NULL, {tv_sec=60, tv_usec=0}) = 1 (in
> [3], left {tv_sec=59, tv_usec=98}) <0.29>
> [pid  3453] read(3,
> "t,S=9113,W=9346:2,S\0\231#V\10\300,\270\215\21z\360\250"..., 4092) =
> 4092 <0.28>
> [pid  3453] select(7, NULL, [6], [6], {tv_sec=60, tv_usec=0}) = 1 (out
> [6], left {tv_sec=59, tv_usec=98}) <0.47>
> [pid  3453] write(6, "45773421.M45371P9859.xx.gues"..., 4091) = 4091 
> <0.51>
> [pid  3452] <... select resumed> )  = 1 (in [5], left {tv_sec=58,
> tv_usec=682833}) <1.317230>
> [pid  3453] select(4, [3], [], NULL, {tv_sec=60, tv_usec=0} 
> [pid  3452] read(5, "45773421.M45371P9859.xx.gues"..., 8184) = 4091 <0.33>
> [pid  3452] select(6, [5], [], NULL, {tv_sec=60, tv_usec=0} 
> [pid  3453] <... select resumed> )  = 1 (in [3], left {tv_sec=58,
> tv_usec=879335}) <1.120688>
> [pid  3453] read(3, "\374\17\0\7", 4)   = 4 <0.57>
> [pid  3453] select(4, [3], [], NULL, {tv_sec=60, tv_usec=0}) = 1 (in
> [3], left {tv_sec=59, tv_usec=97}) <0.29>
> [pid  3453] read(3, ".it,S=17245,W=17685:2,S\0]CR\364\340\327Q\f"...,
> 4092) = 4092 <0.26>
> [pid  3453] select(7, NULL, [6], [6], {tv_sec=60, tv_usec=0}) = 1 (out
> [6], left {tv_sec=59, tv_usec=98}) <0.64>
> [pid  3453] write(6, "389990620.M619008P11470.xx.g"..., 4092) = 4092 
> <0.51>
> [pid  3452] <... select resumed> )  = 1 (in [5], left {tv_sec=58,
> tv_usec=878920}) <1.121130>
> [pid  3453] select(7, NULL, [6], [6], {tv_sec=60, tv_usec=0} 
> [pid  3452] read(5,  
> [pid  3453] <... select resumed> )  = 1 (out [6], left {tv_sec=59,
> tv_usec=98}) <0.29>
> [pid  3452] <... read resumed> "389990620.M619008P11470.xx.g"...,
> 8184) = 4092 <0.32>
> [pid  3453] write(6, "\333\32;^\26\252\321[:>8", 11 
> [pid  3452] select(6, [5], [], NULL, {tv_sec=60, tv_usec=0} 
> [pid  3453] <... write resumed> )   = 11 <0.30>
> [pid  3452] <... select resumed> )  = 1 (in [5], left {tv_sec=59,
> tv_usec=98}) <0.26>
> [pid  3453] select(4, [3], [], NULL, {tv_sec=60, tv_usec=0} 
> [pid  3452] read(5, "\333\32;^\26\252\321[:>8", 8184) = 11 <0.11>
> [pid  3452] select(6, [5], [], NULL, {tv_sec=60, tv_usec=0} 
> [pid  3453] <... select resumed> )  = 1 (in [3], left {tv_sec=58,
> tv_usec=696881}) <1.303136>
> [pid  3453] read(3, "\374\17\0\7", 4)   = 4 <0.28>
> [pid  3453] select(4, [3], [], NULL, {tv_sec=60, tv_usec=0}) = 1 (in
> [3], left {tv_sec=59, tv_usec=98}) <0.27>
> [pid  3453] read(3, ".w4w.yy,S=1841,W=1902:2,S\0"..., 4092) = 2896 <0.26>
> [pid  3453] select(4, [3], [], NULL, {tv_sec=60, tv_usec=0}) = 1 (in
> [3], left {tv_sec=59, tv_usec=999866}) <0.000149>
> [pid  3453] read(3, 

Re: [BackupPC-users] BackuPC 4 hang during transfer

[Gandalf Corvotempesta]

2017-09-21 14:37 GMT+02:00 Craig Barratt via BackupPC-users
:
> I recommend running strace -p PID -T on the rsync_bpc process to see what it
> is up to, and how long various system calls take.  I agree your backups
> should run much faster.

Here is an extract.
Some calls are really slow, more than 1 second.
They seems to be select, right ? Writes seems to be ok, less than 1/10 second

[pid  3453] select(4, [3], [], NULL, {tv_sec=60, tv_usec=0} 
[pid  3452] read(5, "75601860.M846377P28101.xx.gu"..., 8184) = 4081 <0.35>
[pid  3452] select(6, [5], [], NULL, {tv_sec=60, tv_usec=0} 
[pid  3453] <... select resumed> )  = 1 (in [3], left {tv_sec=59,
tv_usec=220734}) <0.779284>
[pid  3453] read(3, "\374\17\0\7", 4)   = 4 <0.35>
[pid  3453] select(4, [3], [], NULL, {tv_sec=60, tv_usec=0}) = 1 (in
[3], left {tv_sec=59, tv_usec=98}) <0.28>
[pid  3453] read(3, ".yy,S=1262266,W=1278715:2,"..., 4092) = 2896 <0.34>
[pid  3453] select(4, [3], [], NULL, {tv_sec=60, tv_usec=0}) = 1 (in
[3], left {tv_sec=59, tv_usec=98}) <0.29>
[pid  3453] read(3, "383235089.M974673P2358.xx.gu"..., 1196) = 1196 <0.27>
[pid  3453] select(7, NULL, [6], [6], {tv_sec=60, tv_usec=0}) = 1 (out
[6], left {tv_sec=59, tv_usec=98}) <0.35>
[pid  3453] write(6, "381086351.M234084P17834.xx.g"..., 4087) = 4087 <0.39>
[pid  3452] <... select resumed> )  = 1 (in [5], left {tv_sec=59,
tv_usec=220263}) <0.779785>
[pid  3453] select(4, [3], [], NULL, {tv_sec=60, tv_usec=0} 
[pid  3452] read(5, "381086351.M234084P17834.xx.g"..., 8184) = 4087 <0.40>
[pid  3452] select(6, [5], [], NULL, {tv_sec=60, tv_usec=0} 
[pid  3453] <... select resumed> )  = 1 (in [3], left {tv_sec=58,
tv_usec=683238}) <1.316780>
[pid  3453] read(3, "\374\17\0\7", 4)   = 4 <0.56>
[pid  3453] select(4, [3], [], NULL, {tv_sec=60, tv_usec=0}) = 1 (in
[3], left {tv_sec=59, tv_usec=98}) <0.29>
[pid  3453] read(3,
"t,S=9113,W=9346:2,S\0\231#V\10\300,\270\215\21z\360\250"..., 4092) =
4092 <0.28>
[pid  3453] select(7, NULL, [6], [6], {tv_sec=60, tv_usec=0}) = 1 (out
[6], left {tv_sec=59, tv_usec=98}) <0.47>
[pid  3453] write(6, "45773421.M45371P9859.xx.gues"..., 4091) = 4091 <0.51>
[pid  3452] <... select resumed> )  = 1 (in [5], left {tv_sec=58,
tv_usec=682833}) <1.317230>
[pid  3453] select(4, [3], [], NULL, {tv_sec=60, tv_usec=0} 
[pid  3452] read(5, "45773421.M45371P9859.xx.gues"..., 8184) = 4091 <0.33>
[pid  3452] select(6, [5], [], NULL, {tv_sec=60, tv_usec=0} 
[pid  3453] <... select resumed> )  = 1 (in [3], left {tv_sec=58,
tv_usec=879335}) <1.120688>
[pid  3453] read(3, "\374\17\0\7", 4)   = 4 <0.57>
[pid  3453] select(4, [3], [], NULL, {tv_sec=60, tv_usec=0}) = 1 (in
[3], left {tv_sec=59, tv_usec=97}) <0.29>
[pid  3453] read(3, ".it,S=17245,W=17685:2,S\0]CR\364\340\327Q\f"...,
4092) = 4092 <0.26>
[pid  3453] select(7, NULL, [6], [6], {tv_sec=60, tv_usec=0}) = 1 (out
[6], left {tv_sec=59, tv_usec=98}) <0.64>
[pid  3453] write(6, "389990620.M619008P11470.xx.g"..., 4092) = 4092 <0.51>
[pid  3452] <... select resumed> )  = 1 (in [5], left {tv_sec=58,
tv_usec=878920}) <1.121130>
[pid  3453] select(7, NULL, [6], [6], {tv_sec=60, tv_usec=0} 
[pid  3452] read(5,  
[pid  3453] <... select resumed> )  = 1 (out [6], left {tv_sec=59,
tv_usec=98}) <0.29>
[pid  3452] <... read resumed> "389990620.M619008P11470.xx.g"...,
8184) = 4092 <0.32>
[pid  3453] write(6, "\333\32;^\26\252\321[:>8", 11 
[pid  3452] select(6, [5], [], NULL, {tv_sec=60, tv_usec=0} 
[pid  3453] <... write resumed> )   = 11 <0.30>
[pid  3452] <... select resumed> )  = 1 (in [5], left {tv_sec=59,
tv_usec=98}) <0.26>
[pid  3453] select(4, [3], [], NULL, {tv_sec=60, tv_usec=0} 
[pid  3452] read(5, "\333\32;^\26\252\321[:>8", 8184) = 11 <0.11>
[pid  3452] select(6, [5], [], NULL, {tv_sec=60, tv_usec=0} 
[pid  3453] <... select resumed> )  = 1 (in [3], left {tv_sec=58,
tv_usec=696881}) <1.303136>
[pid  3453] read(3, "\374\17\0\7", 4)   = 4 <0.28>
[pid  3453] select(4, [3], [], NULL, {tv_sec=60, tv_usec=0}) = 1 (in
[3], left {tv_sec=59, tv_usec=98}) <0.27>
[pid  3453] read(3, ".w4w.yy,S=1841,W=1902:2,S\0"..., 4092) = 2896 <0.26>
[pid  3453] select(4, [3], [], NULL, {tv_sec=60, tv_usec=0}) = 1 (in
[3], left {tv_sec=59, tv_usec=999866}) <0.000149>
[pid  3453] read(3, "s\311\34\344:>:74312186.M702139P19191.x3"...,
1196) = 1196 <0.31>
[pid  3453] select(7, NULL, [6], [6], {tv_sec=60, tv_usec=0}) = 1 (out
[6], left {tv_sec=59, tv_usec=98}) <0.38>
[pid  3453] write(6, "14095904.M963616P21646.xx.gu"..., 4078) = 4078 <0.48>
[pid  3452] <... select resumed> )  = 1 (in [5], left {tv_sec=58,
tv_usec=695988}) <1.304041>
[pid  3453] select(4, [3], [], NULL, {tv_sec=60, tv_usec=0} 
[pid  3452] read(5, "14095904.M963616P21646.xx.gu"..., 8184) = 4078 <0.17>
[pid  3452] select(6, [5], [], NULL, {tv_sec=60, 

Re: [BackupPC-users] BackuPC 4 hang during transfer

[Gandalf Corvotempesta]

2017-09-21 14:37 GMT+02:00 Craig Barratt via BackupPC-users
:
> Back to OP - from the rsync_bpc args you sent, it looks like you have
> XferLogLevel set to 0.  I'd recommend at least 1 or 2.  That would help
> confirm your excludes are correct (they do look pretty comprehensive, but
> you should verify the actual files being backed up match your intended
> excludes).  As earlier posters pointed out, incorrect excludes could mean
> you are trying to backup potentially very large files.

Is possible to show, somewhere, when a file is not transferred because
already exists in the pool?
rsyncd.log on source server doesn't show files not being transferred
(only logs transferred files) thus looking
at that log, bpc seems to be frozen. I'm sure that is not frozen by
stracing the process and seeing some lstat/open calls.

Would be usefull to have a sort of "rsyncd.log" on backuppc size
showing if a file is transferred or not and why is not transferred.

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] BackuPC 4 hang during transfer

[Gandalf Corvotempesta]

2017-09-21 14:37 GMT+02:00 Craig Barratt via BackupPC-users
:
> Yes, Les is right - if you change compression (on to off, or off to on) then
> the next backup will be like a first-time backup - it will be forced to a
> full and will not take advantage of any prior backup.

I know, but in my case, when i've changed compression to 0, backup was
already running, thus it was
not affected by this change.

Anyway, next backups should run a little bit faster as they don't have
to compress every file.

> Back to OP - from the rsync_bpc args you sent, it looks like you have
> XferLogLevel set to 0.  I'd recommend at least 1 or 2.  That would help
> confirm your excludes are correct (they do look pretty comprehensive, but
> you should verify the actual files being backed up match your intended
> excludes).  As earlier posters pointed out, incorrect excludes could mean
> you are trying to backup potentially very large files.

Very large file would produce some output on strace and is not my case.

> What is the version of your remote rsync?

I was on 3.1.0, but i've updated to the latest one.
Nothing changed.

> Have you confirmed you are not running short of server memory?

No memory issue, it's the first thing that i've checked.

> You could see whether ZFS is the issue by (temporarily) running a new
> BackuipPC instance with storage on a different file system (eg, ext4).

This is a new install, previously (about 15-20 days ago) i was on XFS
and even before on ext4.
ZFS is the fastest one (particularry when deleting huge directories
with tons of hardlinks made by rsnapshot)

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] BackuPC 4 hang during transfer

[Craig Barratt via BackupPC-users]

Yes, Les is right - if you change compression (on to off, or off to on)
then the next backup will be like a first-time backup - it will be forced
to a full and will not take advantage of any prior backup.

Back to OP - from the rsync_bpc args you sent, it looks like you have
XferLogLevel set to 0.  I'd recommend at least 1 or 2.  That would help
confirm your excludes are correct (they do look pretty comprehensive, but
you should verify the actual files being backed up match your intended
excludes).  As earlier posters pointed out, incorrect excludes could mean
you are trying to backup potentially very large files.

I recommend running strace -p PID -T on the rsync_bpc process to see what
it is up to, and how long various system calls take.  I agree your backups
should run much faster.

What is the version of your remote rsync?  Have you confirmed you are not
running short of server memory?

You could see whether ZFS is the issue by (temporarily) running a new
BackuipPC instance with storage on a different file system (eg, ext4).

Craig

On Wed, Sep 20, 2017 at 9:05 AM, Les Mikesell  wrote:

> On Wed, Sep 20, 2017 at 10:20 AM, Gandalf Corvotempesta
>  wrote:
> > 2017-09-20 17:15 GMT+02:00 Ray Frush :
> >> You indicate that your ZFS store does 50-70MB/s.  That's pretty slow in
> >> today's world.  I get bothered when storage is slower than a single 10K
> RPM
> >> drive (~100-120MB/sec).  I wonder how fast metadata operations are.
> >> bonnie++ benchmarks might indicate an issue here as BackupPC is metadata
> >> intensive, and has to read a lot of metadata to properly place files in
> the
> >> CPOOL.   Compare those results with other storage to gauge how well
> your ZFS
> >> is performing.   I'm not a ZFS expert.
> >
> > Yes, is not very fast but keep in mind that i'm using SATA disks.
> > But the issue is not the server performance, because all other software
> are
> > able to backup in a very short time, with the same hardware.
>
> Backuppc uses the disk much more intensively than other systems.  For
> the reasons that you want use it.   And I'd guess that zfs block-level
> compression activity would be fairly inefficient on partial blocks for
> more or less the same reasons there is a hit with raid 5.   I assume
> that your attempt to use the --inplace option reflects problems you've
> noticed with your other backup systems.  If it works, --whole-file
> might be better, given a fast LAN and slow disks.  Reconstructing a
> file from copied bits of the old and merging in the changes is pretty
> expensive.
>
>
> Also, I'm not sure anyone has experience with changing the compression
> level between runs.  If you've done that it might add overhead.
>
> --
>Les Mikesell
>  lesmikes...@gmail.com
>
> 
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> ___
> BackupPC-users mailing list
> BackupPC-users@lists.sourceforge.net
> List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
> Wiki:http://backuppc.wiki.sourceforge.net
> Project: http://backuppc.sourceforge.net/
>
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] BackuPC 4 hang during transfer

[Les Mikesell]

On Wed, Sep 20, 2017 at 10:20 AM, Gandalf Corvotempesta
 wrote:
> 2017-09-20 17:15 GMT+02:00 Ray Frush :
>> You indicate that your ZFS store does 50-70MB/s.  That's pretty slow in
>> today's world.  I get bothered when storage is slower than a single 10K RPM
>> drive (~100-120MB/sec).  I wonder how fast metadata operations are.
>> bonnie++ benchmarks might indicate an issue here as BackupPC is metadata
>> intensive, and has to read a lot of metadata to properly place files in the
>> CPOOL.   Compare those results with other storage to gauge how well your ZFS
>> is performing.   I'm not a ZFS expert.
>
> Yes, is not very fast but keep in mind that i'm using SATA disks.
> But the issue is not the server performance, because all other software are
> able to backup in a very short time, with the same hardware.

Backuppc uses the disk much more intensively than other systems.  For
the reasons that you want use it.   And I'd guess that zfs block-level
compression activity would be fairly inefficient on partial blocks for
more or less the same reasons there is a hit with raid 5.   I assume
that your attempt to use the --inplace option reflects problems you've
noticed with your other backup systems.  If it works, --whole-file
might be better, given a fast LAN and slow disks.  Reconstructing a
file from copied bits of the old and merging in the changes is pretty
expensive.


Also, I'm not sure anyone has experience with changing the compression
level between runs.  If you've done that it might add overhead.

--
   Les Mikesell
 lesmikes...@gmail.com

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] BackuPC 4 hang during transfer

[Adam Goryachev]

On 21/9/17 01:20, Gandalf Corvotempesta wrote:

2017-09-20 17:15 GMT+02:00 Ray Frush :

You indicate that your ZFS store does 50-70MB/s.  That's pretty slow in
today's world.  I get bothered when storage is slower than a single 10K RPM
drive (~100-120MB/sec).  I wonder how fast metadata operations are.
bonnie++ benchmarks might indicate an issue here as BackupPC is metadata
intensive, and has to read a lot of metadata to properly place files in the
CPOOL.   Compare those results with other storage to gauge how well your ZFS
is performing.   I'm not a ZFS expert.

Yes, is not very fast but keep in mind that i'm using SATA disks.
But the issue is not the server performance, because all other software are
able to backup in a very short time, with the same hardware.

Except it probably is or else it wouldn't take so long ;)


A 4x 1Gbps network link will look exactly like a single 1Gbps per network
channel (stream) unless you've got some really nice port aggregation
hardware that can spray data at 4Gbps across those.   As such, unless you
have parallel jobs running (multithreaded), I wouldn't expect to see any
product do better than 1Gbps from any single client in your environment.
The BackupPC server, running multiple backup jobs could see a benefit from
the bonded connection, being able to manage 4 1Gpbs streams at the same
time, under optimal conditions, which never happens.

I'm running 4 concurrent backups, with plain rsync/rsnapshot i'm able to run 8.

Except as you know, BPC demands a little more resources to do the 
backup, and running concurrent backups multiplies those demands. When 
you are close to, or slightly exceed your system capacity, then you will 
see massive decrease in overall performance. The more the system has to 
"pretend" it has capacity that it doesn't, and therefore it will do more 
swapping, or more seeking (on the HDD's) and less useful caching, then 
useful activity decreases significantly.


Find the performance bottleneck first, then you can decide how best to 
proceed, whether that's increase hardware, modify the config, or change 
the code.


Regards,
Adam

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] BackuPC 4 hang during transfer

[Adam Goryachev]

On 21/9/17 00:34, Gandalf Corvotempesta wrote:

2017-09-20 16:14 GMT+02:00 Ray Frush :

Question:   Just how big is the host you're trying to backup?  GB?  number
of files?

 From BackupPC web page: 147466.4 MB, 3465344 files.

FYI, I have a single host with 926764MB, 5327442 files.

What is the network connection between the client and the backup
server?

4x 1GbE bonded on both sides.

Remote VPN, one end is 100M the other is 40M

Full backup is about 11 hours, incr is around 8 hours, up to 10 hours.

These stats simply say "It works for me". I see you are saying "It 
doesn't work for me", but I guess my statement will help you as much as 
your statement helps you.


I would guess that you have a bottleneck/performance limitation 
somewhere in your stack, and I'd suggest finding it and then hitting it 
with something appropriate. Check CPU utilisation on the client and 
server, memory usage, swap usage, and of course bandwidth usage. Once 
you rule out all of those, then you get to the fun stuff, and start 
looking at disk performance. While everyone says it is not accurate, 
personally, I've found that "/usr/bin/iostat -dmx" and watching the util 
column is a pretty good indicator.


Apologies, I certainly haven't followed the full thread, but at this 
point, throwing your hands in the air and name calling isn't going to 
improve the situation. Either you are interested in solving the problem, 
in which case you will probably need to get your hands dirty and work 
with us to find and fix it, or you don't really care, in which case we 
don't either (because it works fine for us).


Regards,
Adam

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] BackuPC 4 hang during transfer

[Gandalf Corvotempesta]

2017-09-20 17:15 GMT+02:00 Ray Frush :
> You indicate that your ZFS store does 50-70MB/s.  That's pretty slow in
> today's world.  I get bothered when storage is slower than a single 10K RPM
> drive (~100-120MB/sec).  I wonder how fast metadata operations are.
> bonnie++ benchmarks might indicate an issue here as BackupPC is metadata
> intensive, and has to read a lot of metadata to properly place files in the
> CPOOL.   Compare those results with other storage to gauge how well your ZFS
> is performing.   I'm not a ZFS expert.

Yes, is not very fast but keep in mind that i'm using SATA disks.
But the issue is not the server performance, because all other software are
able to backup in a very short time, with the same hardware.

> 2) rsyncd vs rsync:
> When BackupPC uses the 'rsync' method, it uses ssh to start a dedicated
> rsync server on the client system with parameters picked by BackupPC
> developers.
> When you use the 'rsyncd' method,  the options on the client side were
> picked by you, and may not play well with BackupPC.  It would be easy to
> test around this by setting up backupPC to use the 'rsync' method instead
> (setting up ssh correctly of course) and seeing if you note any improvement.
> That will isolate any issues with your rsyncd configs.

Ok, I can try that.

> A 4x 1Gbps network link will look exactly like a single 1Gbps per network
> channel (stream) unless you've got some really nice port aggregation
> hardware that can spray data at 4Gbps across those.   As such, unless you
> have parallel jobs running (multithreaded), I wouldn't expect to see any
> product do better than 1Gbps from any single client in your environment.
> The BackupPC server, running multiple backup jobs could see a benefit from
> the bonded connection, being able to manage 4 1Gpbs streams at the same
> time, under optimal conditions, which never happens.

I'm running 4 concurrent backups, with plain rsync/rsnapshot i'm able to run 8.

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] BackuPC 4 hang during transfer

[Ray Frush]

Gandalf-

The server you're trying to back up doesn't seem that large.   A similarly
sized server in my environment (210GB, 3M files) does a full backup in
~80-90 minutes, and incrementals run 10-15 minutes.  Monitoring suggests
that the server never exceed 1Gbps outbound on the network connection.
This indicates to me that BackupPC is capable of doing the job, so that
leaves elements of your environment that are variables.

Here's a list that comes to mind:

1) target disk performance:
You indicate that your ZFS store does 50-70MB/s.  That's pretty slow in
today's world.  I get bothered when storage is slower than a single 10K RPM
drive (~100-120MB/sec).  I wonder how fast metadata operations are.
 bonnie++ benchmarks might indicate an issue here as BackupPC is metadata
intensive, and has to read a lot of metadata to properly place files in the
CPOOL.   Compare those results with other storage to gauge how well your
ZFS is performing.   I'm not a ZFS expert.

2) rsyncd vs rsync:
When BackupPC uses the 'rsync' method, it uses ssh to start a dedicated
rsync server on the client system with parameters picked by BackupPC
developers.
When you use the 'rsyncd' method,  the options on the client side were
picked by you, and may not play well with BackupPC.  It would be easy to
test around this by setting up backupPC to use the 'rsync' method instead
(setting up ssh correctly of course) and seeing if you note any
improvement.  That will isolate any issues with your rsyncd configs.


A 4x 1Gbps network link will look exactly like a single 1Gbps per network
channel (stream) unless you've got some really nice port aggregation
hardware that can spray data at 4Gbps across those.   As such, unless you
have parallel jobs running (multithreaded), I wouldn't expect to see any
product do better than 1Gbps from any single client in your environment.
The BackupPC server, running multiple backup jobs could see a benefit from
the bonded connection, being able to manage 4 1Gpbs streams at the same
time, under optimal conditions, which never happens.




On Wed, Sep 20, 2017 at 8:34 AM, Gandalf Corvotempesta <
gandalf.corvotempe...@gmail.com> wrote:

> 2017-09-20 16:14 GMT+02:00 Ray Frush :
> > Question:   Just how big is the host you're trying to backup?  GB?
> number
> > of files?
>
> From BackupPC web page: 147466.4 MB, 3465344 files.
>
> > What is the network connection between the client and the backup
> > server?
>
> 4x 1GbE bonded on both sides.
>
>
> > I'm curious about what it is about your environment that is making
> > it so hard to back up.
>
> It's the same with ALL servers that i'm trying to backup.
> BPC is about 12 times slower than any other tested solution.
> Obviously, same backup server, same source servers.
>
> > I believe I've mentioned my largest, hairiest server is 770GB with 6.8
> > Million files.   Full backups on that system take 8.5 hours to run.
> > Incrementals take 20-30 minutes.   I have no illusions that the
> > infrastructure I'm using to back things up is the fastest, but it's fast
> > enough for the job.
>
> The long running backup (about 38 hours, still running) is an incremental.
>
> 
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> ___
> BackupPC-users mailing list
> BackupPC-users@lists.sourceforge.net
> List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
> Wiki:http://backuppc.wiki.sourceforge.net
> Project: http://backuppc.sourceforge.net/
>



-- 
Time flies like an arrow, but fruit flies like a banana.
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] BackuPC 4 hang during transfer

[Gandalf Corvotempesta]

2017-09-20 16:52 GMT+02:00 Gandalf Corvotempesta
:
> running "zfs iostat" show about 50-70MB/s

Even more:

   capacity operationsbandwidth
poolalloc   free   read  write   read  write
--  -  -  -  -  -  -
rpool   6.66T  4.21T291393  3.44M  5.88M
rpool   6.66T  4.21T128  10.2K  14.3M   133M
rpool   6.66T  4.21T147  7.44K  16.6M   111M
rpool   6.66T  4.21T118  3.18K  7.55M  58.6M
rpool   6.66T  4.21T339  3.70K  32.1M   101M

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] BackuPC 4 hang during transfer

[Gandalf Corvotempesta]

2017-09-20 16:50 GMT+02:00 Les Mikesell :
> You mentioned using zfs with compression.  What kind of disk
> performance does that give when working with odd sized chunks?

running "zfs iostat" show about 50-70MB/s

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] BackuPC 4 hang during transfer

[Les Mikesell]

On Wed, Sep 20, 2017 at 9:34 AM, Gandalf Corvotempesta
 wrote:
>
>> I'm curious about what it is about your environment that is making
>> it so hard to back up.
>
> It's the same with ALL servers that i'm trying to backup.
> BPC is about 12 times slower than any other tested solution.
> Obviously, same backup server, same source servers.

You mentioned using zfs with compression.  What kind of disk
performance does that give when working with odd sized chunks?

-- 
  Les Mikesell
 lesmikes...@gmail.com

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] BackuPC 4 hang during transfer

[Gandalf Corvotempesta]

2017-09-20 16:14 GMT+02:00 Ray Frush :
> Question:   Just how big is the host you're trying to backup?  GB?  number
> of files?

>From BackupPC web page: 147466.4 MB, 3465344 files.

> What is the network connection between the client and the backup
> server?

4x 1GbE bonded on both sides.


> I'm curious about what it is about your environment that is making
> it so hard to back up.

It's the same with ALL servers that i'm trying to backup.
BPC is about 12 times slower than any other tested solution.
Obviously, same backup server, same source servers.

> I believe I've mentioned my largest, hairiest server is 770GB with 6.8
> Million files.   Full backups on that system take 8.5 hours to run.
> Incrementals take 20-30 minutes.   I have no illusions that the
> infrastructure I'm using to back things up is the fastest, but it's fast
> enough for the job.

The long running backup (about 38 hours, still running) is an incremental.

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] BackuPC 4 hang during transfer

[Gandalf Corvotempesta]

2017-09-20 15:10 GMT+02:00 Craig Barratt via BackupPC-users
:
> Since you have concluded what the problem is, I don't have anything
> constructive to add.

No, i've not found the exact bug. Some help is needed.
Having a backup still running from about (right now) 38 hours when the
same host is able to backup the same server with the same rsync
arguments in about 3 hours it
making me suspicious.

I know that BPC does much more things than other software. Ok, that's
clear, but I can understand something like a little slowdown (maybe
200%? 300%?), not 12 times slower. TWELVE TIMES.
It's 1200% slower

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] BackuPC 4 hang during transfer

[Ray Frush]

Question:   Just how big is the host you're trying to backup?  GB?  number
of files?  What is the network connection between the client and the backup
server?  I'm curious about what it is about your environment that is making
it so hard to back up.

I believe I've mentioned my largest, hairiest server is 770GB with 6.8
Million files.   Full backups on that system take 8.5 hours to run.
Incrementals take 20-30 minutes.   I have no illusions that the
infrastructure I'm using to back things up is the fastest, but it's fast
enough for the job.


--
Ray Frush
Colorado State University.



On Wed, Sep 20, 2017 at 12:41 AM, Gandalf Corvotempesta <
gandalf.corvotempe...@gmail.com> wrote:

> 2017-09-19 18:01 GMT+02:00 Gandalf Corvotempesta
> :
> > Removed "--inplace" from the command like and running the same backup
> > right now from BPC.
> > It's too early to be sure, but seems to go further. Let's see during the
> night.
>
> Is still running. This is ok on one side, as "--inplace" may caused
> the issue, but on the other side,
> there is something not working properly in BPC. An incremental backup
> is still running (about 80%)
> from yesterday at 17:39.
>
> rsync, rsnapshot, bacula, rdiff-backup, bareos, and borg took about 3
> hours (some a little bit more, some a little bit less) to backup this
> host in the same way (more or less, same final size of backup)
> BackupPC is taking 10 times more of any other backup software, this
> makes BPC unusable with huge hosts. An order of magnitude is totally
> unacceptable and can only mean some bugs in the code.
>
> As wrote many month ago, I think there are something not working
> properly in BPC, it's impossible that BPC deduplication is slowing
> down backups in this way.
> Also, I've removed the compression, because I'm using ZFS with native
> compression, thus BPC doesn't have to decompress, check local file,
> compress the new one and so on.
>
> And after the backup, refCnt and fsck is ran. For this server, the
> "post-backup" phase takes another hours or two.
>
> Maybe I have hardware issue on this backup server, but even all other
> backup software that i've tried are running in this server with no
> issue at all. Only BPC is slow as hell.
>
> 
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> ___
> BackupPC-users mailing list
> BackupPC-users@lists.sourceforge.net
> List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
> Wiki:http://backuppc.wiki.sourceforge.net
> Project: http://backuppc.sourceforge.net/
>



-- 
Time flies like an arrow, but fruit flies like a banana.
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] BackuPC 4 hang during transfer

[Craig Barratt via BackupPC-users]

>
> An order of magnitude is totally unacceptable and can only mean some bugs
> in the code.

Only BPC is slow as hell.


Since you have concluded what the problem is, I don't have anything
constructive to add.

Craig

On Tue, Sep 19, 2017 at 11:41 PM, Gandalf Corvotempesta <
gandalf.corvotempe...@gmail.com> wrote:

> 2017-09-19 18:01 GMT+02:00 Gandalf Corvotempesta
> :
> > Removed "--inplace" from the command like and running the same backup
> > right now from BPC.
> > It's too early to be sure, but seems to go further. Let's see during the
> night.
>
> Is still running. This is ok on one side, as "--inplace" may caused
> the issue, but on the other side,
> there is something not working properly in BPC. An incremental backup
> is still running (about 80%)
> from yesterday at 17:39.
>
> rsync, rsnapshot, bacula, rdiff-backup, bareos, and borg took about 3
> hours (some a little bit more, some a little bit less) to backup this
> host in the same way (more or less, same final size of backup)
> BackupPC is taking 10 times more of any other backup software, this
> makes BPC unusable with huge hosts. An order of magnitude is totally
> unacceptable and can only mean some bugs in the code.
>
> As wrote many month ago, I think there are something not working
> properly in BPC, it's impossible that BPC deduplication is slowing
> down backups in this way.
> Also, I've removed the compression, because I'm using ZFS with native
> compression, thus BPC doesn't have to decompress, check local file,
> compress the new one and so on.
>
> And after the backup, refCnt and fsck is ran. For this server, the
> "post-backup" phase takes another hours or two.
>
> Maybe I have hardware issue on this backup server, but even all other
> backup software that i've tried are running in this server with no
> issue at all. Only BPC is slow as hell.
>
> 
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> ___
> BackupPC-users mailing list
> BackupPC-users@lists.sourceforge.net
> List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
> Wiki:http://backuppc.wiki.sourceforge.net
> Project: http://backuppc.sourceforge.net/
>
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] BackuPC 4 hang during transfer

[Gandalf Corvotempesta]

2017-09-19 18:01 GMT+02:00 Gandalf Corvotempesta
:
> Removed "--inplace" from the command like and running the same backup
> right now from BPC.
> It's too early to be sure, but seems to go further. Let's see during the 
> night.

Is still running. This is ok on one side, as "--inplace" may caused
the issue, but on the other side,
there is something not working properly in BPC. An incremental backup
is still running (about 80%)
from yesterday at 17:39.

rsync, rsnapshot, bacula, rdiff-backup, bareos, and borg took about 3
hours (some a little bit more, some a little bit less) to backup this
host in the same way (more or less, same final size of backup)
BackupPC is taking 10 times more of any other backup software, this
makes BPC unusable with huge hosts. An order of magnitude is totally
unacceptable and can only mean some bugs in the code.

As wrote many month ago, I think there are something not working
properly in BPC, it's impossible that BPC deduplication is slowing
down backups in this way.
Also, I've removed the compression, because I'm using ZFS with native
compression, thus BPC doesn't have to decompress, check local file,
compress the new one and so on.

And after the backup, refCnt and fsck is ran. For this server, the
"post-backup" phase takes another hours or two.

Maybe I have hardware issue on this backup server, but even all other
backup software that i've tried are running in this server with no
issue at all. Only BPC is slow as hell.

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] BackuPC 4 hang during transfer

[Craig Barratt via BackupPC-users]

(please ignore this reply - I queued it up a few hours ago on a plane ride,
and it just sent now after several more useful replies to the thread)

On Tuesday, September 19, 2017, Craig Barratt <
cbarr...@users.sourceforge.net> wrote:

> Gandalf,
>
> What XferMethod are you using?
>
> Yes, it would be interesting to see the strace output for each
> BackupPC_dump process when it hangs.
>
> Craig
>
> On Tuesday, September 19, 2017, Gandalf Corvotempesta <
> gandalf.corvotempe...@gmail.com
> > wrote:
>
>> Still getting the same issue.
>> Backup for a couple of host are impossible, bpc hangs (at different
>> progresses) and doesn't continue anymore.
>>
>> No load on backup server and on source server. Simply, bpc doesn't
>> transfer.
>>
>> 2017-09-18 15:38 GMT+02:00 Gandalf Corvotempesta
>> :
>> > 2017-09-18 14:30 GMT+02:00 G.W. Haywood via BackupPC-users
>> > :
>> >> When I first used version 4 I ran into a very similar issue, there
>> >> were one or two bug-fixes which addressed it.  You have not stated
>> >> exactly what version you are using, but first make sure that all the
>> >> BPC software is up to date.
>> >
>> > I'm using the latest version: 4.1.3
>>
>> 
>> --
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>> ___
>> BackupPC-users mailing list
>> BackupPC-users@lists.sourceforge.net
>> List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
>> Wiki:http://backuppc.wiki.sourceforge.net
>> Project: http://backuppc.sourceforge.net/
>>
>
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] BackuPC 4 hang during transfer

[Craig Barratt via BackupPC-users]

Gandalf,

What XferMethod are you using?

Yes, it would be interesting to see the strace output for each
BackupPC_dump process when it hangs.

Craig

On Tuesday, September 19, 2017, Gandalf Corvotempesta <
gandalf.corvotempe...@gmail.com> wrote:

> Still getting the same issue.
> Backup for a couple of host are impossible, bpc hangs (at different
> progresses) and doesn't continue anymore.
>
> No load on backup server and on source server. Simply, bpc doesn't
> transfer.
>
> 2017-09-18 15:38 GMT+02:00 Gandalf Corvotempesta
> >:
> > 2017-09-18 14:30 GMT+02:00 G.W. Haywood via BackupPC-users
> > >:
> >> When I first used version 4 I ran into a very similar issue, there
> >> were one or two bug-fixes which addressed it.  You have not stated
> >> exactly what version you are using, but first make sure that all the
> >> BPC software is up to date.
> >
> > I'm using the latest version: 4.1.3
>
> 
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> ___
> BackupPC-users mailing list
> BackupPC-users@lists.sourceforge.net 
> List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
> Wiki:http://backuppc.wiki.sourceforge.net
> Project: http://backuppc.sourceforge.net/
>
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] BackuPC 4 hang during transfer

[Gandalf Corvotempesta]

Yes, running this right now on both: server and client.

2017-09-19 18:00 GMT+02:00 Les Mikesell :
> On Tue, Sep 19, 2017 at 10:47 AM, Gandalf Corvotempesta
>  wrote:
>> 2017-09-19 17:41 GMT+02:00 Les Mikesell :
>>> If your client rsyncd is configured to write a log file you should be
>>> able to see the invocation there.   Just guessing, I'd say there is
>>> probably something wrong in your excludes and you are wandering into
>>> the /sys, /proc, or /dev directories and hanging when you access the
>>> contents.
>>
>> rsyncd is logging, but there isn't any invocation argument.
>> Anyway, the same command line (except for the "bpc" custom arguments)
>> is used by my
>> with rsnapshot and plain "rsync" and i'm able to backup the whole
>> server without issues.
>>
>> Maybe BPC is unable to manage some kind of files that plain rsync is
>> able to handle ?
>
> A brute-force approach to debugging would be to start an strace on the
> rsyncd client soon after the backup starts and after it hangs try to
> figure out the failing operation and hope you have enough scroll back
> buffer to find where the relevant file descriptor was opened so you
> know what it was.
>
> --
> Les Mikesell
>   lesmikes...@gmail.com
>
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> ___
> BackupPC-users mailing list
> BackupPC-users@lists.sourceforge.net
> List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
> Wiki:http://backuppc.wiki.sourceforge.net
> Project: http://backuppc.sourceforge.net/

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] BackuPC 4 hang during transfer

[Gandalf Corvotempesta]

2017-09-19 16:51 GMT+02:00 Gandalf Corvotempesta
:
> /usr/local/bin/rsync_bpc --bpc-top-dir /var/backups/backuppc
> --bpc-host-name myhost --bpc-share-name everything --bpc-bkup-num 1
> --bpc-bkup-comp 3 --bpc-bkup-prevnum -1 --bpc-bkup-prevcomp -1
> --bpc-bkup-inode0 7047290 --bpc-attrib-new --bpc-log-level 0 --super
> --recursive --protect-args --numeric-ids --perms --owner --group -D
> --times --links --hard-links --delete --delete-excluded --partial
> --log-format=log: %o %i %B %8U,%8G %9l %f%L --stats
> --block-size=131072 --inplace --timeout=72000
> --password-file=/var/backups/backuppc/pc/myhost/.rsyncdpw24363
> --exclude=var/backups/* --exclude=admin_backups/*
> --exclude=reseller_backups/* --exclude=user_backups/* --exclude=tmp/*
> --exclude=proc/* --exclude=sys/* --exclude=media/* --exclude=mnt/*
> --exclude=tmp/* --exclude=wp-content/cache/object/*
> --exclude=wp-content/cache/page_enhanced/*
> --exclude=wp-content/cache/db/*
> --exclude=usr/local/directadmin/data/tickets/* --exclude=var/cache/*
> --exclude=var/log/directadmin/* --exclude=var/log/lastlog
> --exclude=var/log/rsync* --exclude=var/log/bacula/*
> --exclude=var/log/ntpstats --exclude=var/lib/mlocate
> --exclude=var/lib/mysql/* --exclude=var/lib/apt/lists/*
> --exclude=var/cache/apt/archives/* --exclude=usr/local/php55/sockets/*
> --exclude=var/run/* --exclude=var/spool/exim/*
> backuppc@myhost::everything /


Removed "--inplace" from the command like and running the same backup
right now from BPC.
It's too early to be sure, but seems to go further. Let's see during the night.

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] BackuPC 4 hang during transfer

[Les Mikesell]

On Tue, Sep 19, 2017 at 10:47 AM, Gandalf Corvotempesta
 wrote:
> 2017-09-19 17:41 GMT+02:00 Les Mikesell :
>> If your client rsyncd is configured to write a log file you should be
>> able to see the invocation there.   Just guessing, I'd say there is
>> probably something wrong in your excludes and you are wandering into
>> the /sys, /proc, or /dev directories and hanging when you access the
>> contents.
>
> rsyncd is logging, but there isn't any invocation argument.
> Anyway, the same command line (except for the "bpc" custom arguments)
> is used by my
> with rsnapshot and plain "rsync" and i'm able to backup the whole
> server without issues.
>
> Maybe BPC is unable to manage some kind of files that plain rsync is
> able to handle ?

A brute-force approach to debugging would be to start an strace on the
rsyncd client soon after the backup starts and after it hangs try to
figure out the failing operation and hope you have enough scroll back
buffer to find where the relevant file descriptor was opened so you
know what it was.

-- 
Les Mikesell
  lesmikes...@gmail.com

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] BackuPC 4 hang during transfer

[Gandalf Corvotempesta]

2017-09-19 17:41 GMT+02:00 Les Mikesell :
> If your client rsyncd is configured to write a log file you should be
> able to see the invocation there.   Just guessing, I'd say there is
> probably something wrong in your excludes and you are wandering into
> the /sys, /proc, or /dev directories and hanging when you access the
> contents.

rsyncd is logging, but there isn't any invocation argument.
Anyway, the same command line (except for the "bpc" custom arguments)
is used by my
with rsnapshot and plain "rsync" and i'm able to backup the whole
server without issues.

Maybe BPC is unable to manage some kind of files that plain rsync is
able to handle ?

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] BackuPC 4 hang during transfer

[Les Mikesell]

On Tue, Sep 19, 2017 at 9:51 AM, Gandalf Corvotempesta
 wrote:
> I can't get rsync command from the client system, as "ps aux" doesn't
> show the command invocation by the server.

If your client rsyncd is configured to write a log file you should be
able to see the invocation there.   Just guessing, I'd say there is
probably something wrong in your excludes and you are wandering into
the /sys, /proc, or /dev directories and hanging when you access the
contents.

-- 
   Les Mikesell
lesmikes...@gmail.com

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] BackuPC 4 hang during transfer

[Ray Frush]

Gandalf-

Hopefully, someone with more rsyncd experience can step in and help.   I
haven't used the rsyncd method for 3-4 years, and don't have any current
examples to help you with.

Good luck!
--
Ray Frush

On Tue, Sep 19, 2017 at 9:32 AM, Gandalf Corvotempesta <
gandalf.corvotempe...@gmail.com> wrote:

> rsyncd is running on all servers, as i'm able to backup them properly
> with plain rsync or rsnapshot.
> Only BPC is freezing.
>
> I don't use SSH at all, i'm directly connecting to rsyncd via rsync.
>
> 2017-09-19 17:19 GMT+02:00 Ray Frush :
> > Gandalf-
> >
> > It looks like you're using the "rsyncd" method vs "rsync" is that
> correct?
> > I don't have experience using the 'rsyncd' method, so my ability to
> continue
> > troubleshooting ends here.  The main thing that jumps out at me is to
> check
> > that rsyncd is actually running on your clients, and that you can connect
> > from the backuppc server using the command you found above.
> >
> >
> > I use the 'rsync' method, and the rest of my answer below is predicated
> on
> > that scheme:
> >
> > I kicked off a backup and did a 'ps -elf | grep backuppc' to get these
> from
> > the BackupPC server:
> >
> > 1) the BackupPC_dump command
> >
> > backuppc  9603  3682  0 09:05 ?00:00:00 /usr/bin/perl
> > /usr/local/BackupPC/bin/BackupPC_dump -i isast201
> >
> > 2) the local rsync_bpc instanc:
> >
> > backuppc  9606  9603  8 09:05 ?00:00:03 /usr/local/bin/rsync_bpc
> > --bpc-top-dir /mnt/backups/BackupPC --bpc-host-name isast201
> > --bpc-share-name / --bpc-bkup-num 118 --bpc-bkup-comp 3
> --bpc-bkup-prevnum
> > 117 --bpc-bkup-prevcomp 3 --bpc-bkup-inode0 203221 --bpc-attrib-new
> > --bpc-log-level 1 -e /usr/bin/ssh -l root --rsync-path=/usr/bin/rsync
> > --super --recursive --protect-args --numeric-ids --perms --owner --group
> -D
> > --times --links --hard-links --delete --partial --log-format=log: %o %i
> %B
> > %8U,%8G %9l %f%L --stats --iconv=utf8,UTF-8 --timeout=72000
> --exclude=stuff
> > isast201:/ /
> >
> > 3) the ssh command initiated by rsync_bpc to the client to initiate the
> > server:  THIS IS THE IMPORTANT ONE to test next:
> >
> > backuppc  9607  9606  1 09:05 ?00:00:00 /usr/bin/ssh -l root
> > isast201 /usr/bin/rsync --server --sender -slHogDtpre.iLsf --iconv=UTF-8
> >
> > 4) The active portion of process 9606 above:
> >
> > backuppc  9608  9606  0 09:05 ?00:00:00 /usr/local/bin/rsync_bpc
> > --bpc-top-dir /mnt/backups/BackupPC --bpc-host-name isast201
> > --bpc-share-name / --bpc-bkup-num 118 --bpc-bkup-comp 3
> --bpc-bkup-prevnum
> > 117 --bpc-bkup-prevcomp 3 --bpc-bkup-inode0 203221 --bpc-attrib-new
> > --bpc-log-level 1 -e /usr/bin/ssh -l root --rsync-path=/usr/bin/rsync
> > --super --recursive --protect-args --numeric-ids --perms --owner --group
> -D
> > --times --links --hard-links --delete --partial --log-format=log: %o %i
> %B
> > %8U,%8G %9l %f%L --stats --iconv=utf8,UTF-8 --timeout=72000
> --exclude=stuff
> > isast201:/ /
> >
> >
> > In my example, I have setup ssh keys to allow the BackupPC user to access
> > the clients.
> >
> >
> >
> > On Tue, Sep 19, 2017 at 8:51 AM, Gandalf Corvotempesta
> >  wrote:
> >>
> >> I can't get rsync command from the client system, as "ps aux" doesn't
> >> show the command invocation by the server.
> >> BackupPC is running the following:
> >>
> >> /usr/bin/perl /usr/local/backuppc/bin/BackupPC_dump -i myhost
> >>
> >> spawing two identical processes:
> >>
> >> /usr/local/bin/rsync_bpc --bpc-top-dir /var/backups/backuppc
> >> --bpc-host-name myhost --bpc-share-name everything --bpc-bkup-num 1
> >> --bpc-bkup-comp 3 --bpc-bkup-prevnum -1 --bpc-bkup-prevcomp -1
> >> --bpc-bkup-inode0 7047290 --bpc-attrib-new --bpc-log-level 0 --super
> >> --recursive --protect-args --numeric-ids --perms --owner --group -D
> >> --times --links --hard-links --delete --delete-excluded --partial
> >> --log-format=log: %o %i %B %8U,%8G %9l %f%L --stats
> >> --block-size=131072 --inplace --timeout=72000
> >> --password-file=/var/backups/backuppc/pc/myhost/.rsyncdpw24363
> >> --exclude=var/backups/* --exclude=admin_backups/*
> >> --exclude=reseller_backups/* --exclude=user_backups/* --exclude=tmp/*
> >> --exclude=proc/* --exclude=sys/* --exclude=media/* --exclude=mnt/*
> >> --exclude=tmp/* --exclude=wp-content/cache/object/*
> >> --exclude=wp-content/cache/page_enhanced/*
> >> --exclude=wp-content/cache/db/*
> >> --exclude=usr/local/directadmin/data/tickets/* --exclude=var/cache/*
> >> --exclude=var/log/directadmin/* --exclude=var/log/lastlog
> >> --exclude=var/log/rsync* --exclude=var/log/bacula/*
> >> --exclude=var/log/ntpstats --exclude=var/lib/mlocate
> >> --exclude=var/lib/mysql/* --exclude=var/lib/apt/lists/*
> >> --exclude=var/cache/apt/archives/* --exclude=usr/local/php55/sockets/*
> >> --exclude=var/run/* --exclude=var/spool/exim/*
> >> backuppc@myhost::everything /
> >>
> >>
> >>
> >> standard rsync works.
> >> rsnapshot 

Re: [BackupPC-users] BackuPC 4 hang during transfer

[Gandalf Corvotempesta]

rsyncd is running on all servers, as i'm able to backup them properly
with plain rsync or rsnapshot.
Only BPC is freezing.

I don't use SSH at all, i'm directly connecting to rsyncd via rsync.

2017-09-19 17:19 GMT+02:00 Ray Frush :
> Gandalf-
>
> It looks like you're using the "rsyncd" method vs "rsync" is that correct?
> I don't have experience using the 'rsyncd' method, so my ability to continue
> troubleshooting ends here.  The main thing that jumps out at me is to check
> that rsyncd is actually running on your clients, and that you can connect
> from the backuppc server using the command you found above.
>
>
> I use the 'rsync' method, and the rest of my answer below is predicated on
> that scheme:
>
> I kicked off a backup and did a 'ps -elf | grep backuppc' to get these from
> the BackupPC server:
>
> 1) the BackupPC_dump command
>
> backuppc  9603  3682  0 09:05 ?00:00:00 /usr/bin/perl
> /usr/local/BackupPC/bin/BackupPC_dump -i isast201
>
> 2) the local rsync_bpc instanc:
>
> backuppc  9606  9603  8 09:05 ?00:00:03 /usr/local/bin/rsync_bpc
> --bpc-top-dir /mnt/backups/BackupPC --bpc-host-name isast201
> --bpc-share-name / --bpc-bkup-num 118 --bpc-bkup-comp 3 --bpc-bkup-prevnum
> 117 --bpc-bkup-prevcomp 3 --bpc-bkup-inode0 203221 --bpc-attrib-new
> --bpc-log-level 1 -e /usr/bin/ssh -l root --rsync-path=/usr/bin/rsync
> --super --recursive --protect-args --numeric-ids --perms --owner --group -D
> --times --links --hard-links --delete --partial --log-format=log: %o %i %B
> %8U,%8G %9l %f%L --stats --iconv=utf8,UTF-8 --timeout=72000 --exclude=stuff
> isast201:/ /
>
> 3) the ssh command initiated by rsync_bpc to the client to initiate the
> server:  THIS IS THE IMPORTANT ONE to test next:
>
> backuppc  9607  9606  1 09:05 ?00:00:00 /usr/bin/ssh -l root
> isast201 /usr/bin/rsync --server --sender -slHogDtpre.iLsf --iconv=UTF-8
>
> 4) The active portion of process 9606 above:
>
> backuppc  9608  9606  0 09:05 ?00:00:00 /usr/local/bin/rsync_bpc
> --bpc-top-dir /mnt/backups/BackupPC --bpc-host-name isast201
> --bpc-share-name / --bpc-bkup-num 118 --bpc-bkup-comp 3 --bpc-bkup-prevnum
> 117 --bpc-bkup-prevcomp 3 --bpc-bkup-inode0 203221 --bpc-attrib-new
> --bpc-log-level 1 -e /usr/bin/ssh -l root --rsync-path=/usr/bin/rsync
> --super --recursive --protect-args --numeric-ids --perms --owner --group -D
> --times --links --hard-links --delete --partial --log-format=log: %o %i %B
> %8U,%8G %9l %f%L --stats --iconv=utf8,UTF-8 --timeout=72000 --exclude=stuff
> isast201:/ /
>
>
> In my example, I have setup ssh keys to allow the BackupPC user to access
> the clients.
>
>
>
> On Tue, Sep 19, 2017 at 8:51 AM, Gandalf Corvotempesta
>  wrote:
>>
>> I can't get rsync command from the client system, as "ps aux" doesn't
>> show the command invocation by the server.
>> BackupPC is running the following:
>>
>> /usr/bin/perl /usr/local/backuppc/bin/BackupPC_dump -i myhost
>>
>> spawing two identical processes:
>>
>> /usr/local/bin/rsync_bpc --bpc-top-dir /var/backups/backuppc
>> --bpc-host-name myhost --bpc-share-name everything --bpc-bkup-num 1
>> --bpc-bkup-comp 3 --bpc-bkup-prevnum -1 --bpc-bkup-prevcomp -1
>> --bpc-bkup-inode0 7047290 --bpc-attrib-new --bpc-log-level 0 --super
>> --recursive --protect-args --numeric-ids --perms --owner --group -D
>> --times --links --hard-links --delete --delete-excluded --partial
>> --log-format=log: %o %i %B %8U,%8G %9l %f%L --stats
>> --block-size=131072 --inplace --timeout=72000
>> --password-file=/var/backups/backuppc/pc/myhost/.rsyncdpw24363
>> --exclude=var/backups/* --exclude=admin_backups/*
>> --exclude=reseller_backups/* --exclude=user_backups/* --exclude=tmp/*
>> --exclude=proc/* --exclude=sys/* --exclude=media/* --exclude=mnt/*
>> --exclude=tmp/* --exclude=wp-content/cache/object/*
>> --exclude=wp-content/cache/page_enhanced/*
>> --exclude=wp-content/cache/db/*
>> --exclude=usr/local/directadmin/data/tickets/* --exclude=var/cache/*
>> --exclude=var/log/directadmin/* --exclude=var/log/lastlog
>> --exclude=var/log/rsync* --exclude=var/log/bacula/*
>> --exclude=var/log/ntpstats --exclude=var/lib/mlocate
>> --exclude=var/lib/mysql/* --exclude=var/lib/apt/lists/*
>> --exclude=var/cache/apt/archives/* --exclude=usr/local/php55/sockets/*
>> --exclude=var/run/* --exclude=var/spool/exim/*
>> backuppc@myhost::everything /
>>
>>
>>
>> standard rsync works.
>> rsnapshot works too (i'm using rsnapshot to backup this host, as BPC
>> freeze)
>>
>> 2017-09-19 16:42 GMT+02:00 Ray Frush :
>> > Gandalf-
>> >
>> > As a troubleshooting step, collect the actual running rsync commands
>> > from
>> > the client system, and from the BackupPC server (found in the Xferlog).
>> > Post them here to get a wider audience.
>> >
>> > Try running an rsync manually  using the same parameters, and see if it
>> > works.  My guess is not, and there is a misconfiguration that will leap
>> > out
>> 

Re: [BackupPC-users] BackuPC 4 hang during transfer

[Ray Frush]

Gandalf-

It looks like you're using the "rsyncd" method vs "rsync" is that correct?
  I don't have experience using the 'rsyncd' method, so my ability to
continue troubleshooting ends here.  The main thing that jumps out at me is
to check that rsyncd is actually running on your clients, and that you can
connect from the backuppc server using the command you found above.


I use the 'rsync' method, and the rest of my answer below is predicated on
that scheme:

I kicked off a backup and did a 'ps -elf | grep backuppc' to get these from
the BackupPC server:

1) the BackupPC_dump command

backuppc  9603  3682  0 09:05 ?00:00:00 /usr/bin/perl
/usr/local/BackupPC/bin/BackupPC_dump -i isast201

2) the local rsync_bpc instanc:

backuppc  9606  9603  8 09:05 ?00:00:03 /usr/local/bin/rsync_bpc
--bpc-top-dir /mnt/backups/BackupPC --bpc-host-name isast201
--bpc-share-name / --bpc-bkup-num 118 --bpc-bkup-comp 3 --bpc-bkup-prevnum
117 --bpc-bkup-prevcomp 3 --bpc-bkup-inode0 203221 --bpc-attrib-new
--bpc-log-level 1 -e /usr/bin/ssh -l root --rsync-path=/usr/bin/rsync
--super --recursive --protect-args --numeric-ids --perms --owner --group -D
--times --links --hard-links --delete --partial --log-format=log: %o %i %B
%8U,%8G %9l %f%L --stats --iconv=utf8,UTF-8 --timeout=72000 --exclude=stuff
isast201:/ /

3) the ssh command initiated by rsync_bpc to the client to initiate the
server:  THIS IS THE IMPORTANT ONE to test next:

backuppc  9607  9606  1 09:05 ?00:00:00 /usr/bin/ssh -l root
isast201 /usr/bin/rsync --server --sender -slHogDtpre.iLsf --iconv=UTF-8

4) The active portion of process 9606 above:

backuppc  9608  9606  0 09:05 ?00:00:00 /usr/local/bin/rsync_bpc
--bpc-top-dir /mnt/backups/BackupPC --bpc-host-name isast201
--bpc-share-name / --bpc-bkup-num 118 --bpc-bkup-comp 3 --bpc-bkup-prevnum
117 --bpc-bkup-prevcomp 3 --bpc-bkup-inode0 203221 --bpc-attrib-new
--bpc-log-level 1 -e /usr/bin/ssh -l root --rsync-path=/usr/bin/rsync
--super --recursive --protect-args --numeric-ids --perms --owner --group -D
--times --links --hard-links --delete --partial --log-format=log: %o %i %B
%8U,%8G %9l %f%L --stats --iconv=utf8,UTF-8 --timeout=72000 --exclude=stuff
isast201:/ /


In my example, I have setup ssh keys to allow the BackupPC user to access
the clients.



On Tue, Sep 19, 2017 at 8:51 AM, Gandalf Corvotempesta <
gandalf.corvotempe...@gmail.com> wrote:

> I can't get rsync command from the client system, as "ps aux" doesn't
> show the command invocation by the server.
> BackupPC is running the following:
>
> /usr/bin/perl /usr/local/backuppc/bin/BackupPC_dump -i myhost
>
> spawing two identical processes:
>
> /usr/local/bin/rsync_bpc --bpc-top-dir /var/backups/backuppc
> --bpc-host-name myhost --bpc-share-name everything --bpc-bkup-num 1
> --bpc-bkup-comp 3 --bpc-bkup-prevnum -1 --bpc-bkup-prevcomp -1
> --bpc-bkup-inode0 7047290 --bpc-attrib-new --bpc-log-level 0 --super
> --recursive --protect-args --numeric-ids --perms --owner --group -D
> --times --links --hard-links --delete --delete-excluded --partial
> --log-format=log: %o %i %B %8U,%8G %9l %f%L --stats
> --block-size=131072 --inplace --timeout=72000
> --password-file=/var/backups/backuppc/pc/myhost/.rsyncdpw24363
> --exclude=var/backups/* --exclude=admin_backups/*
> --exclude=reseller_backups/* --exclude=user_backups/* --exclude=tmp/*
> --exclude=proc/* --exclude=sys/* --exclude=media/* --exclude=mnt/*
> --exclude=tmp/* --exclude=wp-content/cache/object/*
> --exclude=wp-content/cache/page_enhanced/*
> --exclude=wp-content/cache/db/*
> --exclude=usr/local/directadmin/data/tickets/* --exclude=var/cache/*
> --exclude=var/log/directadmin/* --exclude=var/log/lastlog
> --exclude=var/log/rsync* --exclude=var/log/bacula/*
> --exclude=var/log/ntpstats --exclude=var/lib/mlocate
> --exclude=var/lib/mysql/* --exclude=var/lib/apt/lists/*
> --exclude=var/cache/apt/archives/* --exclude=usr/local/php55/sockets/*
> --exclude=var/run/* --exclude=var/spool/exim/*
> backuppc@myhost::everything /
>
>
>
> standard rsync works.
> rsnapshot works too (i'm using rsnapshot to backup this host, as BPC
> freeze)
>
> 2017-09-19 16:42 GMT+02:00 Ray Frush :
> > Gandalf-
> >
> > As a troubleshooting step, collect the actual running rsync commands from
> > the client system, and from the BackupPC server (found in the Xferlog).
> > Post them here to get a wider audience.
> >
> > Try running an rsync manually  using the same parameters, and see if it
> > works.  My guess is not, and there is a misconfiguration that will leap
> out
> > at you as you work through this.
> >
> > I had to do the same thing when I was doing an initial install.
> >
> > --
> > Ray Frush
> > Colorado State University.
> >
> > On Tue, Sep 19, 2017 at 2:52 AM, Gandalf Corvotempesta
> >  wrote:
> >>
> >> Still getting the same issue.
> >> Backup for a couple of host are impossible, bpc hangs (at different
> >> progresses) and doesn't 

Re: [BackupPC-users] BackuPC 4 hang during transfer

[Gandalf Corvotempesta]

I can't get rsync command from the client system, as "ps aux" doesn't
show the command invocation by the server.
BackupPC is running the following:

/usr/bin/perl /usr/local/backuppc/bin/BackupPC_dump -i myhost

spawing two identical processes:

/usr/local/bin/rsync_bpc --bpc-top-dir /var/backups/backuppc
--bpc-host-name myhost --bpc-share-name everything --bpc-bkup-num 1
--bpc-bkup-comp 3 --bpc-bkup-prevnum -1 --bpc-bkup-prevcomp -1
--bpc-bkup-inode0 7047290 --bpc-attrib-new --bpc-log-level 0 --super
--recursive --protect-args --numeric-ids --perms --owner --group -D
--times --links --hard-links --delete --delete-excluded --partial
--log-format=log: %o %i %B %8U,%8G %9l %f%L --stats
--block-size=131072 --inplace --timeout=72000
--password-file=/var/backups/backuppc/pc/myhost/.rsyncdpw24363
--exclude=var/backups/* --exclude=admin_backups/*
--exclude=reseller_backups/* --exclude=user_backups/* --exclude=tmp/*
--exclude=proc/* --exclude=sys/* --exclude=media/* --exclude=mnt/*
--exclude=tmp/* --exclude=wp-content/cache/object/*
--exclude=wp-content/cache/page_enhanced/*
--exclude=wp-content/cache/db/*
--exclude=usr/local/directadmin/data/tickets/* --exclude=var/cache/*
--exclude=var/log/directadmin/* --exclude=var/log/lastlog
--exclude=var/log/rsync* --exclude=var/log/bacula/*
--exclude=var/log/ntpstats --exclude=var/lib/mlocate
--exclude=var/lib/mysql/* --exclude=var/lib/apt/lists/*
--exclude=var/cache/apt/archives/* --exclude=usr/local/php55/sockets/*
--exclude=var/run/* --exclude=var/spool/exim/*
backuppc@myhost::everything /



standard rsync works.
rsnapshot works too (i'm using rsnapshot to backup this host, as BPC freeze)

2017-09-19 16:42 GMT+02:00 Ray Frush :
> Gandalf-
>
> As a troubleshooting step, collect the actual running rsync commands from
> the client system, and from the BackupPC server (found in the Xferlog).
> Post them here to get a wider audience.
>
> Try running an rsync manually  using the same parameters, and see if it
> works.  My guess is not, and there is a misconfiguration that will leap out
> at you as you work through this.
>
> I had to do the same thing when I was doing an initial install.
>
> --
> Ray Frush
> Colorado State University.
>
> On Tue, Sep 19, 2017 at 2:52 AM, Gandalf Corvotempesta
>  wrote:
>>
>> Still getting the same issue.
>> Backup for a couple of host are impossible, bpc hangs (at different
>> progresses) and doesn't continue anymore.
>>
>> No load on backup server and on source server. Simply, bpc doesn't
>> transfer.
>>
>> 2017-09-18 15:38 GMT+02:00 Gandalf Corvotempesta
>> :
>> > 2017-09-18 14:30 GMT+02:00 G.W. Haywood via BackupPC-users
>> > :
>> >> When I first used version 4 I ran into a very similar issue, there
>> >> were one or two bug-fixes which addressed it.  You have not stated
>> >> exactly what version you are using, but first make sure that all the
>> >> BPC software is up to date.
>> >
>> > I'm using the latest version: 4.1.3
>>
>>
>> --
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>> ___
>> BackupPC-users mailing list
>> BackupPC-users@lists.sourceforge.net
>> List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
>> Wiki:http://backuppc.wiki.sourceforge.net
>> Project: http://backuppc.sourceforge.net/
>
>
>
>
> --
> Time flies like an arrow, but fruit flies like a banana.
>
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> ___
> BackupPC-users mailing list
> BackupPC-users@lists.sourceforge.net
> List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
> Wiki:http://backuppc.wiki.sourceforge.net
> Project: http://backuppc.sourceforge.net/
>

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] BackuPC 4 hang during transfer

[Ray Frush]

Gandalf-

As a troubleshooting step, collect the actual running rsync commands from
the client system, and from the BackupPC server (found in the Xferlog).
Post them here to get a wider audience.

Try running an rsync manually  using the same parameters, and see if it
works.  My guess is not, and there is a misconfiguration that will leap out
at you as you work through this.

I had to do the same thing when I was doing an initial install.

--
Ray Frush
Colorado State University.

On Tue, Sep 19, 2017 at 2:52 AM, Gandalf Corvotempesta <
gandalf.corvotempe...@gmail.com> wrote:

> Still getting the same issue.
> Backup for a couple of host are impossible, bpc hangs (at different
> progresses) and doesn't continue anymore.
>
> No load on backup server and on source server. Simply, bpc doesn't
> transfer.
>
> 2017-09-18 15:38 GMT+02:00 Gandalf Corvotempesta
> :
> > 2017-09-18 14:30 GMT+02:00 G.W. Haywood via BackupPC-users
> > :
> >> When I first used version 4 I ran into a very similar issue, there
> >> were one or two bug-fixes which addressed it.  You have not stated
> >> exactly what version you are using, but first make sure that all the
> >> BPC software is up to date.
> >
> > I'm using the latest version: 4.1.3
>
> 
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> ___
> BackupPC-users mailing list
> BackupPC-users@lists.sourceforge.net
> List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
> Wiki:http://backuppc.wiki.sourceforge.net
> Project: http://backuppc.sourceforge.net/
>



-- 
Time flies like an arrow, but fruit flies like a banana.
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] BackuPC 4 hang during transfer

[Gandalf Corvotempesta]

Still getting the same issue.
Backup for a couple of host are impossible, bpc hangs (at different
progresses) and doesn't continue anymore.

No load on backup server and on source server. Simply, bpc doesn't transfer.

2017-09-18 15:38 GMT+02:00 Gandalf Corvotempesta
:
> 2017-09-18 14:30 GMT+02:00 G.W. Haywood via BackupPC-users
> :
>> When I first used version 4 I ran into a very similar issue, there
>> were one or two bug-fixes which addressed it.  You have not stated
>> exactly what version you are using, but first make sure that all the
>> BPC software is up to date.
>
> I'm using the latest version: 4.1.3

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] BackuPC 4 hang during transfer

[Gandalf Corvotempesta]

2017-09-18 14:30 GMT+02:00 G.W. Haywood via BackupPC-users
:
> When I first used version 4 I ran into a very similar issue, there
> were one or two bug-fixes which addressed it.  You have not stated
> exactly what version you are using, but first make sure that all the
> BPC software is up to date.

I'm using the latest version: 4.1.3

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] BackuPC 4 hang during transfer

[G.W. Haywood via BackupPC-users]

Hi there,

On Mon, 18 Sep 2017, Gandalf Corvotempesta wrote:


I'm tryiig to backup 2 hosts with BPC4 but bpc seems to be frozen
during transfer.  [...] nothing happens from about 3 hours.


When I first used version 4 I ran into a very similar issue, there
were one or two bug-fixes which addressed it.  You have not stated
exactly what version you are using, but first make sure that all the
BPC software is up to date.

https://sourceforge.net/p/backuppc/mailman/message/35769986/

--

73,
Ged.

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

[BackupPC-users] BackuPC 4 hang during transfer

[Gandalf Corvotempesta]

Hi,
I'm tryiig to backup 2 hosts with BPC4 but bpc seems to be frozen
during transfer.

I'm stracing both processes (the main bpc process and both transfer
process) but nothing
happens from about 3 hours.

Any clue?

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/