Hi Stephen, Hi all, Thanks for your feedback and sharing your experience. Here some clarification on my side.
*** Regarding BPC being not reliable. I don't deny that BPC works very well in many situations, and can sustain heavy load etc. But in my case it was not the flawless setup I imagined at first. Of course, the main problem is the small amount of memory. Looking in the dmesg logs, I could spot regularly OOM messages, the kernel killing the backuppc_dump process, etc. Now it is a bit unfair of me blaming BPC when actually the main culprit is the lack of memory. But the thing is that BPC is quite unhelpful in these situations. Server logs are mostly useless, no timestamps, and there is no attempt to restart again, but instead BPC goes into a long process of counting references, etc, meaning most of the server time is spent in (apparently) unproductive tasks. Again, the main culprit is the platform, and actually BPC never lost any data (afaik), and always recovered somehow. Still, there are some traces of corruption in the db (like warning about reference being equal to -1 instead of 0), indicating that maybe BPC is not atomic. *** Regarding the 256MB requirement. Admittedly this is very demanding and very far off most feedbacks I've seen on the net. Stephen's setup of a recycled Dell PE 1950 (8 cores, 16 GB RAM) seems more typical than my poor lacie-cloudbox with 256MB. But when I monitor memory usage, for instance when doing a full backup of 600k-file client, BPC dump/rsync processes consume consistently around 100MB memory (htop/smem). Looking at rsync page, they say that rsync 3.0+ should consume 100bytes/file, so a total of 60MB for rsync. So I don't see any blocking point why BPC would not fit in a 256-MB memory budget. Of course it will be slower, but it must work. This WE I again spent some time tuning down the Lacie-Cloudbox, stripping away all useless processes, like those hungry python stuff. Now, in idle, the Lacie has 200MB free physical + 200MB free swap space, and in that setup BPC worked for 2 days w/o crash doing a full 55GB, 650k files backup, in 2 hours, 7.5MB/s (almost no changes, hence the very high speed of course). For now I disabled all my machines but a few, and will enable the remaining ones one by one. I have good hope that it will work again. Now, my wishes would be to restore some other services, but this will likely require increasing the swap space. *** Regarding BPC being slow I only give BPC 256MB, so I should expect too much regarding performance. That I fully agree. However when I say it is slow, I mean it is slow even if I take into account that fact. Transfer speed is ok-ish; it uses rsync at its best, which requires some heavy processing sometimes. But I don't understand why it needs so much time for the remaining tasks (ref counting, etc). I'm actually convinced (perhaps naively) that this can be significantly improved. See further down. *** Regarding "trashing" rsync + @kosowsky.org about designing a totally new backup program My statement was... too brutal I guess ;-) I fully agree with Stephen's comment. And I don't want to create a new program from scratch. rsync is one of the best open-source sync program. Unison and duplicity are basically using rsync internally. I do think however that BPC can be significantly improved. - Flatten the backup hierarchy Initially BPC was a "mere" wrapper around rsync. First duplicating a complete hierarchy with hard-links, then rsync'ing over it. It had the advantage of simplicity but is very slow to maintain, and impossible to duplicate. Now the trend in 4.0alpha was to move to a custom C implementation of rsync, where hierarchy only stores attrib files. I think that we can improve the maintenance phase further (ref counting, backup deletion...) by flattening this structure into a single linear file, and by listing once for all the references in a given backup, possibly with caching of references per directory. Directory entries would be more like git objects, attaching a name to a reference along with some metadata. This means integrating further with the inner working of rsync. It would be fully compliant with rsync from the client side. But refcounting and backup deletion should then be equivalent to sorting and finding duplicate/unique entries, which can be very efficient. Even on my Lacie sorting a 600k-line file with 32B random hash entries takes only a couple seconds. - Client-side sync Sure, this must be an optional feature, and I agree this is not the priority. Many clients will still simply run rsyncd or rsync/ssh. But the client-side sync would allow to detect hard links more efficiently. It will also decrease memory usage on the server (see rsync faq). Then it opens up a whole new set of optimization, delta-diff on multiple files... *** Regarding writing in C Ok, I'm not a perl fan. But I agree, it is useful for stuff where performance does not matter, for website interface, etc. But I would rewrite in C the ref counting part and similar. Kind regards, Michaël ------------------------------------------------------------------------------ Site24x7 APM Insight: Get Deep Visibility into Application Performance APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month Monitor end-to-end web transactions and take corrective actions now Troubleshoot faster and improve end-user experience. Signup Now! http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140 _______________________________________________ BackupPC-devel mailing list BackupPC-devel@lists.sourceforge.net List: https://lists.sourceforge.net/lists/listinfo/backuppc-devel Wiki: http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
