If you pull from aaf git (git.a.o) or github, you are not using git-svn at all, bypassing the actual git-svn problem.
Check out https://github.com/apache/infrastructure-puppet/tree/deployment/modules/git_mirror_asf for what we use, specifically the update-mirror.sh script. That is was svn2gitupdate runs when a pubsub event happens. On Dec 8, 2015 1:00 PM, "Upayavira" <u...@odoko.co.uk> wrote: > Here's what I've just got on the Infra hipchat channel: > > The ASF has a tool, svn2gitupdate[1], which I presume uses git-svn, which > fails periodically. When it does fail, it takes with it all other ASF > projects that make use of the same tool, until an admin can intervene and > restart things. > > When it fails, it OOMs, and blocks all disk activity. > > If someone wanted to reproduce this issue, you could: > * create a 4Gb VM > * Install svn2gitupdate from [1] > * Clone the Lucene git repo from ASF git or github > * Run the tool repeatedly until it fails > - it is the pull from SVN that fails, not the push to git, so we don't > need a remote git server > > The other option is just switching to Git. Now, given the issue is with > reading from SVN, not writing to Git, Infrastructure *would* be able to > give us a decent SVN->Git export - even if they had to rerun the process a > number of times, this would be acceptable as a one-off task. > > So it seems we have two options: > 1) Set up a VM and debug reading from SVN > 2) Just migrate to Git and be done with it. > > Thoughts? Volunteers? > > Upayavira > > [1] > https://svn.apache.org/repos/infra/infrastructure/trunk/projects/git/svn2gitupdate/ > > > On Tue, Dec 8, 2015, at 08:49 PM, Geoffrey Corey wrote: > > If you do that, then the changes do not sync to github, and there's a 99% > chance that the next time a change is seen by the mirroring process (or by > the hourly cron that updates all the svn->git mirrors) the same memory leak > would happen. > > On Tue, Dec 8, 2015 at 12:40 PM, Scott Blum <dragonsi...@gmail.com> wrote: > > Dumb question, but searching around suggests that git-svn can be killed > and then resumed with `git svn fetch`. Shouldn't that resolve any > process-level memory leak? > > On Fri, Dec 4, 2015 at 3:57 PM, Michael McCandless < > luc...@mikemccandless.com> wrote: > > Hello devs, > > The infra team has notified us (Lucene/Solr) that in 26 days our > git-svn mirror will be turned off, because running it consumes too > many system resources, affecting other projects, apparently because of > a memory leak in git-svn. > > Does anyone know of a link to this git-svn issue? Is it a known > issue? If there's something simple we can do (remove old jars from > our svn history, remove old branches), maybe we can sidestep the issue > and infra will allow it to keep running? > > Or maybe someone in the Lucene/Solr dev community with prior > experience with git-svn could volunteer to play with it to see if > there's a viable solution, maybe with command-line options e.g. to > only mirror specific branches (trunk, 5.x)? > > Or maybe it's time for us to switch to git, but there are problems > there too, e.g. we are currently missing large parts of our svn > history from the mirror now and it's not clear whether that would be > fixed if we switched: > https://issues.apache.org/jira/browse/INFRA-10828 Also, because we > used to add JAR files to svn, the "git clone" would likely take > several GBs unless we remove those JARs from our history. > > Or if anyone has any other ideas, we should explore them, because > otherwise in 26 days there will be no more updates to the git mirror > of Lucene and Solr sources... > > Thanks, > > Mike McCandless > > http://blog.mikemccandless.com > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org > > >
