Re: [BackupPC-users] Issue with multiple restores
On Wed, Jun 17, 2015 at 11:29 AM, Andy Taylor andy.tay...@sift.com wrote: It's not something I had considered for this use case as the files in question are uploaded site files (for example, images) as opposed to site code, and can sometimes total up to 10GB+. I'm assuming you thought I was talking about code files which is why you suggested a VCS? Subversion will handle any type of files - maybe not quite as efficiently as rsync but it will still just send the differences during updates and commits, so it would certainly work. The down side is that each workspace keeps a hidden duplicate copy of the current checkout and it is next to impossible to ever remove old revisions from the central repository history so it can grow very large over time. -- Les Mikesell lesmikes...@gmail.com -- ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Issue with multiple restores
Hi Holger/Les,
Thank you for the suggestions. We ruled out it being a corrupt attrib file
by running a full backup. In the end we implemented a script which carries
out the following:
1. Exports the latest backup of a folder to the BackupPC server via
BackupPC_tarCreate
2. Rsyncs that folder to our network file share server
3. Deletes the restore from the BackupPC server
This seemed faster than doing a tar directly over SSH, and seems to be
working well for us so far. The 'big picture' is that our developers use
the restored files on the network file share server as copies of live
website files, so when they are developing they can be sure they are
looking at relatively up-to-date copies of the files which are on our live
servers. We could have rsynced those files directly from the live server,
but using BackupPC saves on duplication and has the added benefit of
testing our backups.
Thanks again for the help.
Andy
On 23 May 2015 at 12:57, Holger Parplies wb...@parplies.de wrote:
Hi,
Les Mikesell wrote on 2015-05-22 11:53:10 -0500 [Re: [BackupPC-users]
Issue with multiple restores]:
On Fri, May 22, 2015 at 11:17 AM, Andy Taylor andy.tay...@sift.com
wrote:
Thank you for the advice Les. I compiled rsync 2.6.9 and tried a
restore
with it. It seems to hang now once it's got past the last file of the
restore - I've run strace against the rsync processes and I can see it
close
the last file and then just hang, with this sort of output:
[pid 15625] select(4, [3], [], NULL, {53, 660732} unfinished ...
[pid 15624] select(7, [3 6], [], NULL, NULL unfinished ...
[pid 15662] select(1, [0], [], NULL, {6, 915513}) = 0 (Timeout)
[pid 15662] select(1, [0], [], NULL, {60, 0} unfinished ...
[pid 15625] ... select resumed ) = 0 (Timeout)
[pid 15625] select(4, [3], [], NULL, {60, 0} unfinished ...
[pid 15662] ... select resumed ) = 0 (Timeout)
Do you know what that might indicate about the restore?
perhaps a corrupt attrib file, as in: one end of the connection is
expecting
[much] more data than the other end sends. This would seem to fit in with
your
error message from newer rsync versions as well as the fact that it only
happens for one host. Have you tried forcing a full backup for that host?
[...]
What's the big picture here?
Whenever I revisit this thread, I immediately think BackupPC fuse
module?,
but I haven't used it myself yet, so I can't give you any detail. But it
might
well be something to look into - providing it fits into the big picture
you
haven't given ;-).
Regards,
Holger
--
One dashboard for servers and applications across Physical-Virtual-Cloud
Widest out-of-the-box monitoring support with 50+ applications
Performance metrics, stats and reports that give you Actionable Insights
Deep dive visibility with transaction tracing using APM Insight.
http://ad.doubleclick.net/ddm/clk/290420510;117567292;y
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
--
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Issue with multiple restores
On Wed, Jun 17, 2015 at 8:01 AM, Andy Taylor andy.tay...@sift.com wrote: The 'big picture' is that our developers use the restored files on the network file share server as copies of live website files, so when they are developing they can be sure they are looking at relatively up-to-date copies of the files which are on our live servers. We could have rsynced those files directly from the live server, but using BackupPC saves on duplication and has the added benefit of testing our backups. Have you considered using a central version control system like subversion for this instead? You would need to have a process to ensure that the live system was always updated from a tagged/known revision, but then updating all the development workspaces to stay in sync becomes easier and more efficient. -- Les Mikesell lesmikes...@gmail.com -- ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Issue with multiple restores
It's not something I had considered for this use case as the files in question are uploaded site files (for example, images) as opposed to site code, and can sometimes total up to 10GB+. I'm assuming you thought I was talking about code files which is why you suggested a VCS? On 17 June 2015 at 15:28, Les Mikesell lesmikes...@gmail.com wrote: On Wed, Jun 17, 2015 at 8:01 AM, Andy Taylor andy.tay...@sift.com wrote: The 'big picture' is that our developers use the restored files on the network file share server as copies of live website files, so when they are developing they can be sure they are looking at relatively up-to-date copies of the files which are on our live servers. We could have rsynced those files directly from the live server, but using BackupPC saves on duplication and has the added benefit of testing our backups. Have you considered using a central version control system like subversion for this instead? You would need to have a process to ensure that the live system was always updated from a tagged/known revision, but then updating all the development workspaces to stay in sync becomes easier and more efficient. -- Les Mikesell lesmikes...@gmail.com -- ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/ -- ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Issue with multiple restores
Hi,
Les Mikesell wrote on 2015-05-22 11:53:10 -0500 [Re: [BackupPC-users] Issue
with multiple restores]:
On Fri, May 22, 2015 at 11:17 AM, Andy Taylor andy.tay...@sift.com wrote:
Thank you for the advice Les. I compiled rsync 2.6.9 and tried a restore
with it. It seems to hang now once it's got past the last file of the
restore - I've run strace against the rsync processes and I can see it close
the last file and then just hang, with this sort of output:
[pid 15625] select(4, [3], [], NULL, {53, 660732} unfinished ...
[pid 15624] select(7, [3 6], [], NULL, NULL unfinished ...
[pid 15662] select(1, [0], [], NULL, {6, 915513}) = 0 (Timeout)
[pid 15662] select(1, [0], [], NULL, {60, 0} unfinished ...
[pid 15625] ... select resumed ) = 0 (Timeout)
[pid 15625] select(4, [3], [], NULL, {60, 0} unfinished ...
[pid 15662] ... select resumed ) = 0 (Timeout)
Do you know what that might indicate about the restore?
perhaps a corrupt attrib file, as in: one end of the connection is expecting
[much] more data than the other end sends. This would seem to fit in with your
error message from newer rsync versions as well as the fact that it only
happens for one host. Have you tried forcing a full backup for that host?
[...]
What's the big picture here?
Whenever I revisit this thread, I immediately think BackupPC fuse module?,
but I haven't used it myself yet, so I can't give you any detail. But it might
well be something to look into - providing it fits into the big picture you
haven't given ;-).
Regards,
Holger
--
One dashboard for servers and applications across Physical-Virtual-Cloud
Widest out-of-the-box monitoring support with 50+ applications
Performance metrics, stats and reports that give you Actionable Insights
Deep dive visibility with transaction tracing using APM Insight.
http://ad.doubleclick.net/ddm/clk/290420510;117567292;y
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Issue with multiple restores
Thank you for the advice Les. I compiled rsync 2.6.9 and tried a restore
with it. It seems to hang now once it's got past the last file of the
restore - I've run strace against the rsync processes and I can see it
close the last file and then just hang, with this sort of output:
[pid 15625] select(4, [3], [], NULL, {53, 660732} unfinished ...
[pid 15624] select(7, [3 6], [], NULL, NULL unfinished ...
[pid 15662] select(1, [0], [], NULL, {6, 915513}) = 0 (Timeout)
[pid 15662] select(1, [0], [], NULL, {60, 0} unfinished ...
[pid 15625] ... select resumed ) = 0 (Timeout)
[pid 15625] select(4, [3], [], NULL, {60, 0} unfinished ...
[pid 15662] ... select resumed ) = 0 (Timeout)
Do you know what that might indicate about the restore?
With regards to tar, I don't care much about the bandwidth, but the speed
of rsync would be very useful for the restores. I'll have a look at how
fast tar is over SSH.
Thanks again,
Andy
On 21 May 2015 at 17:25, Les Mikesell lesmikes...@gmail.com wrote:
On Thu, May 21, 2015 at 11:08 AM, Andy Taylor andy.tay...@sift.com
wrote:
After a bit more digging I've managed to get a bit more information out
of the logs:
2015/05/21 15:15:09 [14010] Invalid remainder length 467096064 [receiver]
2015/05/21 15:15:09 [14010] rsync error: protocol incompatibility (code
2) at io.c(1352) [receiver=3.0.9]
2015/05/21 15:15:09 [14010] [receiver] _exit_cleanup(code=2, file=io.c,
line=1352): about to call exit(2)
2015/05/21 15:15:09 [14010] rsync: connection unexpectedly closed
(1022021 bytes received so far) [generator]
2015/05/21 15:15:09 [14010] rsync error: error in rsync protocol data
stream (code 12) at io.c(605) [generator=3.0.9]
2015/05/21 15:15:09 [14010] [generator] _exit_cleanup(code=12,
file=io.c, line=605): about to call exit(12)
Does anyone have any idea what this might be? It seems the protocol
incompatibility error can sometimes be down to the version of rsync you are
using, but I have tried a few different versions with no success.
Did you try going down to a 2.x version of rsync? Newer versions are
supposed to negotiate the protocol level but it looks like that isn't
happening in this situation. But, unless you care about the
bandwidth used, you could just script something around the command
line BackupPC_tarCreate tool piped though ssh to a remote extract
instead of using rysnc for this at all.
--
Les Mikesell
lesmikes...@gmail.com
--
One dashboard for servers and applications across Physical-Virtual-Cloud
Widest out-of-the-box monitoring support with 50+ applications
Performance metrics, stats and reports that give you Actionable Insights
Deep dive visibility with transaction tracing using APM Insight.
http://ad.doubleclick.net/ddm/clk/290420510;117567292;y
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
--
One dashboard for servers and applications across Physical-Virtual-Cloud
Widest out-of-the-box monitoring support with 50+ applications
Performance metrics, stats and reports that give you Actionable Insights
Deep dive visibility with transaction tracing using APM Insight.
http://ad.doubleclick.net/ddm/clk/290420510;117567292;y___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Issue with multiple restores
On Fri, May 22, 2015 at 11:17 AM, Andy Taylor andy.tay...@sift.com wrote:
Thank you for the advice Les. I compiled rsync 2.6.9 and tried a restore
with it. It seems to hang now once it's got past the last file of the
restore - I've run strace against the rsync processes and I can see it close
the last file and then just hang, with this sort of output:
[pid 15625] select(4, [3], [], NULL, {53, 660732} unfinished ...
[pid 15624] select(7, [3 6], [], NULL, NULL unfinished ...
[pid 15662] select(1, [0], [], NULL, {6, 915513}) = 0 (Timeout)
[pid 15662] select(1, [0], [], NULL, {60, 0} unfinished ...
[pid 15625] ... select resumed ) = 0 (Timeout)
[pid 15625] select(4, [3], [], NULL, {60, 0} unfinished ...
[pid 15662] ... select resumed ) = 0 (Timeout)
Do you know what that might indicate about the restore?
No idea - but it looks like there are still open fd's at this point.
With regards to tar, I don't care much about the bandwidth, but the speed of
rsync would be very useful for the restores. I'll have a look at how fast
tar is over SSH.
It will probably end up being about the same as if you do the restore
from scratch instead of on top of an existing tree. What's the big
picture here? That is, why not just let people restore files directly
from backuppc as needed? Are you trying to emulate a version control
system with the ability to do diffs, etc.?
--
Les Mikesell
lesmikes...@gmail.com
--
One dashboard for servers and applications across Physical-Virtual-Cloud
Widest out-of-the-box monitoring support with 50+ applications
Performance metrics, stats and reports that give you Actionable Insights
Deep dive visibility with transaction tracing using APM Insight.
http://ad.doubleclick.net/ddm/clk/290420510;117567292;y
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Issue with multiple restores
After a bit more digging I've managed to get a bit more information out of the logs: 2015/05/21 15:15:09 [14010] Invalid remainder length 467096064 [receiver] 2015/05/21 15:15:09 [14010] rsync error: protocol incompatibility (code 2) at io.c(1352) [receiver=3.0.9] 2015/05/21 15:15:09 [14010] [receiver] _exit_cleanup(code=2, file=io.c, line=1352): about to call exit(2) 2015/05/21 15:15:09 [14010] rsync: connection unexpectedly closed (1022021 bytes received so far) [generator] 2015/05/21 15:15:09 [14010] rsync error: error in rsync protocol data stream (code 12) at io.c(605) [generator=3.0.9] 2015/05/21 15:15:09 [14010] [generator] _exit_cleanup(code=12, file=io.c, line=605): about to call exit(12) Does anyone have any idea what this might be? It seems the protocol incompatibility error can sometimes be down to the version of rsync you are using, but I have tried a few different versions with no success. *Andy Taylor* Operations Engineer Bridge House, 48-52 Baldwin Street, Bristol, BS1 1QB direct:+44 (0)1179158634 | mob: +44 (0)7730 019316 *www.sift.com http://www.sift.com* | digital thinking Sift Ltd is registered in England and Wales no. 3230061 On 14 May 2015 at 16:09, Andy Taylor andy.tay...@sift.com wrote: Hello, I'm currently looking at regularly restoring files out of BackupPC to a central location on our network for some of our users to have as a reference. I'm going to run this restore on a daily basis to ensure that the files are up to date. This seems to work fine for all of our hosts except for one. The first restore works fine, but any subsequent restores fail right at the end with this error: Remote[1]: Invalid remainder length 467096064 [receiver] exiting after signal PIPE restore failed: aborted by signal=PIPE It only happens on this one host, and it only seems to happen at the very end of the restore. Has anyone seen anything like this before? The only way I've managed to get around it is by adding --ignore-existing or --whole-file as extra RsyncRestoreArgs. This would seem to imply to me that rsync is failing when it tries to compare/overwrite a file on the receiver, but I'm not really sure. Thanks, Andy -- One dashboard for servers and applications across Physical-Virtual-Cloud Widest out-of-the-box monitoring support with 50+ applications Performance metrics, stats and reports that give you Actionable Insights Deep dive visibility with transaction tracing using APM Insight. http://ad.doubleclick.net/ddm/clk/290420510;117567292;y___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Issue with multiple restores
On Thu, May 21, 2015 at 11:08 AM, Andy Taylor andy.tay...@sift.com wrote: After a bit more digging I've managed to get a bit more information out of the logs: 2015/05/21 15:15:09 [14010] Invalid remainder length 467096064 [receiver] 2015/05/21 15:15:09 [14010] rsync error: protocol incompatibility (code 2) at io.c(1352) [receiver=3.0.9] 2015/05/21 15:15:09 [14010] [receiver] _exit_cleanup(code=2, file=io.c, line=1352): about to call exit(2) 2015/05/21 15:15:09 [14010] rsync: connection unexpectedly closed (1022021 bytes received so far) [generator] 2015/05/21 15:15:09 [14010] rsync error: error in rsync protocol data stream (code 12) at io.c(605) [generator=3.0.9] 2015/05/21 15:15:09 [14010] [generator] _exit_cleanup(code=12, file=io.c, line=605): about to call exit(12) Does anyone have any idea what this might be? It seems the protocol incompatibility error can sometimes be down to the version of rsync you are using, but I have tried a few different versions with no success. Did you try going down to a 2.x version of rsync? Newer versions are supposed to negotiate the protocol level but it looks like that isn't happening in this situation. But, unless you care about the bandwidth used, you could just script something around the command line BackupPC_tarCreate tool piped though ssh to a remote extract instead of using rysnc for this at all. -- Les Mikesell lesmikes...@gmail.com -- One dashboard for servers and applications across Physical-Virtual-Cloud Widest out-of-the-box monitoring support with 50+ applications Performance metrics, stats and reports that give you Actionable Insights Deep dive visibility with transaction tracing using APM Insight. http://ad.doubleclick.net/ddm/clk/290420510;117567292;y ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
[BackupPC-users] Issue with multiple restores
Hello, I'm currently looking at regularly restoring files out of BackupPC to a central location on our network for some of our users to have as a reference. I'm going to run this restore on a daily basis to ensure that the files are up to date. This seems to work fine for all of our hosts except for one. The first restore works fine, but any subsequent restores fail right at the end with this error: Remote[1]: Invalid remainder length 467096064 [receiver] exiting after signal PIPE restore failed: aborted by signal=PIPE It only happens on this one host, and it only seems to happen at the very end of the restore. Has anyone seen anything like this before? The only way I've managed to get around it is by adding --ignore-existing or --whole-file as extra RsyncRestoreArgs. This would seem to imply to me that rsync is failing when it tries to compare/overwrite a file on the receiver, but I'm not really sure. Thanks, Andy -- One dashboard for servers and applications across Physical-Virtual-Cloud Widest out-of-the-box monitoring support with 50+ applications Performance metrics, stats and reports that give you Actionable Insights Deep dive visibility with transaction tracing using APM Insight. http://ad.doubleclick.net/ddm/clk/290420510;117567292;y___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
