Using rsync, make and some smart log file processing using logpro I was able to bidirectionally sync over 250 fossils every two minutes between three sites. It worked very well for two years and then we switched to all sites using ssh. If details on this would be useful let me know and I'll write it up. BTW by using logpro we get notified only when things go wrong but you can easily use some awk, sad perl or whatever to do the same thing. On Aug 9, 2014 3:46 AM, <[email protected]> wrote:
> 'Lo. > > First of all, because this doesn't get said enough: Thanks for fossil! > Been a happy user for three years now, and I love the approach taken to > UI simplicity and the focus on reliability and integrity. I have no > idea how I coped before having an integrated ticket system. > > However, one aspect of fossil is really starting to become a problem: > Syncing large numbers of repositories. > > I'm using what I believe is probably a common setup: I have a server on > my private network here containing the "canonical" repositories for all > my projects. I have clones of these repositories on all machines here, > and the repositories on the server are backed up nightly. I then have > public-facing repositories on io7m.com which are basically read-only > public mirrors of my own private repositores (I have no contributors): > > http://fossil.io7m.com > > The io7m.com site is hosted by site5.com, using their standard shared > web hosting account. I have ssh access, but not root access. I cannot > run any long-running processes on it. > > As the number of repositories has grown, this has become a real > problem. I first started using cron to sync repositories. After this > became unwieldly (due to the number of repositories and the way cron > would send me email every time fossil said anything), I ended up > writing a simple program to sync all repositories at staggered, random > intervals, sending email only if any sync failed. The servers at site5 > appear to implement some sort of rate limiting: If you open too many > ssh connections in too short a time, they'll throttle and close > connections. In order to try to reduce the load on their side, I've > ended up using longer and longer staggered intervals (to the point > where I'm basically syncing any one repository less frequently than > hourly). > > Now that the number of (public) repositories has grown to 32, even this > has become a problem: Due to the number of repositories, I'm still > opening too many connections to the server, and the number of > repositories is only going to grow. All I can do is keep trying to > increase the upper bound on the sync interval. > > I believe the "sync periodically" approach is fundamentally wasteful, > and scales increasingly poorly when the number of repositories grows. > I'm basically burning through gigabytes of bandwidth (32 repositories, > 24 hours a day soon adds up) even though I'm generally only committing > to one repository at a time, and nobody is making any changes to > repositories on the io7m.com side. Perhaps 95% of syncs are entirely > pointless! > > What are my options here? I'd much rather sync on demand: When a > repository on my private server receives any artifacts, it should then > sync with the io7m.com side. This would massively reduce my own > bandwidth usage and would give the io7m.com servers a break. Is there > any way to achieve this, currently? > > Regards, > Mark > _______________________________________________ > fossil-users mailing list > [email protected] > http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users >
_______________________________________________ fossil-users mailing list [email protected] http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
