Re: Regression and failure to clone/fetch with new code Re: git-svn performance

2014-10-30 Thread Eric Wong
Hin-Tak Leung ht...@users.sourceforge.net wrote: Here is the data dumper info . I tried the dumper code on the R repo as well, and saw that against the virtual box repo, there is one curious difference - $self-{last_rev} is a string rather than a number. I tried hacking around doing $x += 0;

Re: differences between old clone and new Re: git-svn performance

2014-10-30 Thread Eric Wong
Hin-Tak Leung ht...@users.sourceforge.net wrote: That's quite straight-forward, I think - except for the recent burst (I am essentially adapting the git 2.1.0 release shipped by the upcoming fedora 21 scheduled for christmas) I tend to update to the latest fedora release about a week or

Re: Regression and failure to clone/fetch with new code Re: git-svn performance

2014-10-30 Thread Hin-Tak Leung
-- On Thu, Oct 30, 2014 08:46 GMT Eric Wong wrote: Hin-Tak Leung ht...@users.sourceforge.net wrote: Here is the data dumper info . I tried the dumper code on the R repo as well, and saw that against the virtual box repo, there is one curious difference -

Re: Regression and failure to clone/fetch with new code Re: git-svn performance

2014-10-30 Thread Hin-Tak Leung
(apology for the empty message again - sticky fingers in smart phone...) -- On Thu, Oct 30, 2014 08:46 GMT Eric Wong wrote: Thanks, I'm not able to reproduce the issue, but can you try the following? diff --git a/perl/Git/SVN/Ra.pm b/perl/Git/SVN/Ra.pm index

Re: Regression and failure to clone/fetch with new code Re: git-svn performance

2014-10-29 Thread Eric Wong
Hin-Tak Leung ht...@users.sourceforge.net wrote: Hi, I patched my system git with the recent git-svn improvements, and just use it for general use; so theses are the patches, against 2.1.0. 0001-git-svn-only-look-at-the-new-parts-of-svn-mergeinfo.patch

Re: Regression and failure to clone/fetch with new code Re: git-svn performance

2014-10-29 Thread Hin-Tak Leung
is a missing merge in at R-2-14-branch@58454 , and two extra merges at djm-parseRd@46925 and djm-parseRd@46906 . On Wed, 29/10/14, Eric Wong normalper...@yhbt.net wrote: Subject: Re: Regression and failure to clone/fetch with new code Re: git-svn

Re: differences between old clone and new Re: git-svn performance

2014-10-29 Thread Eric Wong
Hin-Tak Leung ht...@users.sourceforge.net wrote: On Tue, 28/10/14, Eric Wong normalper...@yhbt.net wrote: So both merges are correct, but we lose one, and gain one? I'll try to check more closely tomorrow.  Can you point out the exact revisions in the R repo?  Thanks. The

Re: Regression and failure to clone/fetch with new code Re: git-svn performance

2014-10-29 Thread Eric Wong
Hin-Tak Leung ht...@users.sourceforge.net wrote: Argh, sorry. I thought I included the info but I didn't. Thanks. I'll try a different version of svn later. What do you think were missing in my e-mails? I was skimming and missed the part about Debian packages :) -- To unsubscribe from this

Re: differences between old clone and new Re: git-svn performance

2014-10-29 Thread Hin-Tak Leung
On Thu, 30/10/14, Eric Wong normalper...@yhbt.net wrote: The missing merge on branch R-2-14-branch is: commit 93af4d4cc3a5e0039944dd4e340d26995be8a252 Merge: 121990f 6ff1b87 Author: ripley ripley@00db46b3-68df-0310-9c12-caf00c1e9a41

Re: Regression and failure to clone/fetch with new code Re: git-svn performance

2014-10-29 Thread Hin-Tak Leung
Here is the data dumper info . I tried the dumper code on the R repo as well, and saw that against the virtual box repo, there is one curious difference - $self-{last_rev} is a string rather than a number. I tried hacking around doing $x += 0; to coerce last_rev to a number at various places but

Re: differences between old clone and new Re: git-svn performance

2014-10-28 Thread Eric Wong
Hin-Tak Leung ht...@users.sourceforge.net wrote: To compare the old clone with the new, I did: git branch -r | sort | xargs -n 1 git log --decorate=full -n 1 It turned out other than the empty vs 3 word commit messages about two years ago on trunk (which are inherited in all the newer

Re: Anomaly with the new code - Re: git-svn performance

2014-10-28 Thread Eric Wong
Hin-Tak Leung ht...@users.sourceforge.net wrote: Eric Wong normalper...@yhbt.net wrote: Which SVN version are you using?  I'm cloning (currently on r373xx) https://svn.r-project.org/R using --stdlayout and unable to see memory growth of the git-svn Perl process beyond 40M (on a 32-bit

Regression and failure to clone/fetch with new code Re: git-svn performance

2014-10-28 Thread Hin-Tak Leung
Hi, I patched my system git with the recent git-svn improvements, and just use it for general use; so theses are the patches, against 2.1.0. 0001-git-svn-only-look-at-the-new-parts-of-svn-mergeinfo.patch 0002-git-svn-only-look-at-the-root-path-for-svn-mergeinfo.patch

Re: differences between old clone and new Re: git-svn performance

2014-10-28 Thread Hin-Tak Leung
On Tue, 28/10/14, Eric Wong normalper...@yhbt.net wrote: So both merges are correct, but we lose one, and gain one? I'll try to check more closely tomorrow.  Can you point out the exact revisions in the R repo?  Thanks. The missing merge on branch R-2-14-branch is: commit

Re: Anomaly with the new code - Re: git-svn performance

2014-10-27 Thread Eric Wong
Hin-Tak Leung ht...@users.sourceforge.net wrote: On Sat, Oct 25, 2014 00:34 BST Eric Wong wrote: 0006 is insufficient and incompatible with older SVN. I pushed git-svn: reload RA every log-window-size (commit dfa72fdb96befbd790f623bb2909a347176753c2) instead which saves much more memory: it

Re: Anomaly with the new code - Re: git-svn performance

2014-10-27 Thread Eric Wong
Eric Wong normalper...@yhbt.net wrote: Which SVN version are you using? I'm cloning (currently on r373xx) https://svn.r-project.org/R using --stdlayout and unable to see memory growth of the git-svn Perl process beyond 40M (on a 32-bit system). git-svn hit 45M and took 11:44 to finish. My

Re: Anomaly with the new code - Re: git-svn performance

2014-10-27 Thread Hin-Tak Leung
-- On Mon, Oct 27, 2014 06:38 GMT Eric Wong wrote: Which SVN version are you using?  I'm cloning (currently on r373xx) https://svn.r-project.org/R using --stdlayout and unable to see memory growth of the git-svn Perl process beyond 40M (on a 32-bit system). I also

Re: Anomaly with the new code - Re: git-svn performance

2014-10-27 Thread Hin-Tak Leung
-- On Mon, Oct 27, 2014 16:56 GMT Eric Wong wrote: Eric Wong normalper...@yhbt.net wrote: Which SVN version are you using?  I'm cloning (currently on r373xx) https://svn.r-project.org/R using --stdlayout and unable to see memory growth of the git-svn Perl process

differences between old clone and new Re: git-svn performance

2014-10-27 Thread Hin-Tak Leung
To compare the old clone with the new, I did: git branch -r | sort | xargs -n 1 git log --decorate=full -n 1 It turned out other than the empty vs 3 word commit messages about two years ago on trunk (which are inherited in all the newer branches), there are two other groups of differences. One

Re: git-svn performance

2014-10-25 Thread Eric Wong
Hin-Tak Leung ht...@users.sourceforge.net wrote: the old didn't missing a revision - just a revision 'message' - blank instead of 3 words, above the git svn id. I supppse it is possible some power problem or etc caused this. I'll check the other branches as well, and possibly clone again to be

Re: Anomaly with the new code - Re: git-svn performance

2014-10-25 Thread Eric Wong
Hin-Tak Leung ht...@users.sourceforge.net wrote: On Sat, Oct 25, 2014 00:34 BST Eric Wong wrote: Hin-Tak Leung ht...@users.sourceforge.net wrote: 0006-git-svn-clear-global-SVN-pool-between-get_log-invoca.patch 0006 is insufficient and incompatible with older SVN. I pushed git-svn:

Re: git-svn performance

2014-10-25 Thread Eric Wong
Hin-Tak Leung ht...@users.sourceforge.net wrote: btw, git svn seems to disallow single word commit messages (or is it a svn config?). i found that i could not do git svn dcommit, when i had merely did git commit -m 'typos', for example, for an svn repo i have write access to. (I don't have

single-word commit messages - Re: git-svn performance

2014-10-25 Thread Hin-Tak Leung
On Sat, 25/10/14, Eric Wong normalper...@yhbt.net wrote: Probably an SVN hook preventing it.  git-svn test cases such as t/t9118-git-svn-funky-branch-names.sh do single word commits. Thanks - I see indeed - at least that clears that up. -- To

Re: git-svn performance

2014-10-25 Thread Hin-Tak Leung
-- On Sat, Oct 25, 2014 01:02 BST Eric Wong wrote: Hin-Tak Leung ht...@users.sourceforge.net wrote: Comparing trunk of old and new, I see one difference - One short commit message is missing in the *old* (the Add checkPoFiles etc. part) and so all the sha1

Re: Anomaly with the new code - Re: git-svn performance

2014-10-25 Thread Hin-Tak Leung
-- On Sat, Oct 25, 2014 00:34 BST Eric Wong wrote: Hin-Tak Leung ht...@users.sourceforge.net wrote: I keep tabs of a particular svn repository over many years and run git svn fetch --all every few days. So that's the old clone. Since this discussion started, I

Re: git-svn performance

2014-10-25 Thread Hin-Tak Leung
-- On Sat, Oct 25, 2014 06:32 BST Eric Wong wrote: Hin-Tak Leung ht...@users.sourceforge.net wrote: the old didn't missing a revision - just a revision 'message' - blank instead of 3 words, above the git svn id. I supppse it is possible some power problem or etc

Anomaly with the new code - Re: git-svn performance

2014-10-24 Thread Hin-Tak Leung
I keep tabs of a particular svn repository over many years and run git svn fetch --all every few days. So that's the old clone. Since this discussion started, I made a new one with git 2.1.0 patched with the first two patches below, a couple of weeks ago. And I ran 'git svn fetch --all' on both

Re: Anomaly with the new code - Re: git-svn performance

2014-10-24 Thread Eric Wong
Hin-Tak Leung ht...@users.sourceforge.net wrote: I keep tabs of a particular svn repository over many years and run git svn fetch --all every few days. So that's the old clone. Since this discussion started, I made a new one with git 2.1.0 patched with the first two patches below, a couple of

Re: git-svn performance

2014-10-24 Thread Eric Wong
Hin-Tak Leung ht...@users.sourceforge.net wrote: Comparing trunk of old and new, I see one difference - One short commit message is missing in the *old* (the Add checkPoFiles etc. part) and so all the sha1 afterwards differed. Is that an old bug that's fixed and therefore I should throw away

Re: git-svn performance

2014-10-22 Thread Hin-Tak Leung
-- On Tue, Oct 21, 2014 10:00 BST Eric Wong wrote: Jakob Stoklund Olesen stokl...@2pi.dk wrote: Yes, but I think you can remove cached_mergeinfo_rev too. Thanks, pushed the patch at the bottom, too. Also started working on some memory reductions here:

Re: git-svn performance

2014-10-21 Thread Eric Wong
Jakob Stoklund Olesen stokl...@2pi.dk wrote: Yes, but I think you can remove cached_mergeinfo_rev too. Thanks, pushed the patch at the bottom, too. Also started working on some memory reductions here: http://mid.gmane.org/20141021033912.ga27...@dcvr.yhbt.net But there seem to be more problems

Re: git-svn performance

2014-10-20 Thread Jakob Stoklund Olesen
On Oct 19, 2014, at 18:16, Eric Wong normalper...@yhbt.net wrote: Jakob Stoklund Olesen stokl...@2pi.dk wrote: If cached_mergeinfo is using too much memory, you can probably drop that cache entirely. IIRC, it didn't give that much of a speed up. I am surprised that it is using a lot of

Re: git-svn performance

2014-10-19 Thread Fabian Schmied
On Sun, Oct 19, 2014 at 2:32 AM, Eric Wong normalper...@yhbt.net wrote: Fabian Schmied fabian.schm...@gmail.com wrote: Hi, I'm currently migrating an SVN repository to Git using git-svn (Git for Windows 1.8.3-preview20130601), and I'm experiencing severe performance problems with git svn

Re: git-svn performance

2014-10-19 Thread Jakob Stoklund Olesen
On Oct 18, 2014, at 19:33, Eric Wong normalper...@yhbt.net wrote: Eric Wong normalper...@yhbt.net wrote: This reduces hash lookups for looking up cache data and will simplify tying data to disk in the next commit. I considered the following, but GDBM might not be readily available on

Re: git-svn performance

2014-10-19 Thread Eric Wong
Jakob Stoklund Olesen stokl...@2pi.dk wrote: If cached_mergeinfo is using too much memory, you can probably drop that cache entirely. IIRC, it didn't give that much of a speed up. I am surprised that it is using a lot of memory, though. There is only one entry per SVN branch. Something like

Re: git-svn performance

2014-10-18 Thread Eric Wong
Fabian Schmied fabian.schm...@gmail.com wrote: Hi, I'm currently migrating an SVN repository to Git using git-svn (Git for Windows 1.8.3-preview20130601), and I'm experiencing severe performance problems with git svn fetch. Commits to the SVN trunk are fetched very fast (a few seconds or so

Re: git-svn performance

2014-10-18 Thread Eric Wong
Eric Wong normalper...@yhbt.net wrote: Hin-Tak (Cc-ed) reported good improvements with them, but also a large memory increase: This might reduce the pathname and internal hash overheads: 8--- From: Eric Wong normalper...@yhbt.net Date: Sun, 19 Oct

Re: git-svn performance

2014-10-18 Thread Eric Wong
Eric Wong normalper...@yhbt.net wrote: This reduces hash lookups for looking up cache data and will simplify tying data to disk in the next commit. I considered the following, but GDBM might not be readily available on non-POSIX platforms. I think the other problem is the existing caches are

git-svn performance

2014-10-17 Thread Fabian Schmied
Hi, I'm currently migrating an SVN repository to Git using git-svn (Git for Windows 1.8.3-preview20130601), and I'm experiencing severe performance problems with git svn fetch. Commits to the SVN trunk are fetched very fast (a few seconds or so per SVN revision), but commits to some branches

[PATCH v2] improve git svn performance

2014-01-22 Thread manjian2006
From: manjian2006 manjian2...@gmail.com * perl/Git/SVN.pm Modified according to Eric Wong normalper...@yhbt.net Hi, I'm interested in this. How much did performance improve by (and how many revisions is the repository) Our svn server are built in a LAN,15152 revisions.Not optimized git-svn

Re: [PATCH v2] improve git svn performance

2014-01-22 Thread Eric Wong
manjian2...@gmail.com wrote: * perl/Git/SVN.pm Modified according to Eric Wong normalper...@yhbt.net Hi, I'm interested in this. How much did performance improve by (and how many revisions is the repository) Our svn server are built in a LAN,15152 revisions.Not optimized git-svn used

Re: [PATCH] improve git svn performance

2014-01-21 Thread Eric Wong
manjian2...@gmail.com wrote: From: linzj li...@ucweb.com I am trying to improve git svn's performance according to some profiling data.As the data showed,_rev_list subroutine and rebuild subroutine are consuming a large proportion of time.So I improve _rev_list's performance by

[PATCH] improve git svn performance Hi, I am trying to improve git svn's performance according to some profiling data.As the data showed,_rev_list subroutine and rebuild subroutine are consuming a l

2014-01-19 Thread manjian2006
From: linzj li...@ucweb.com --- perl/Git/SVN.pm | 63 - 1 file changed, 49 insertions(+), 14 deletions(-) diff --git a/perl/Git/SVN.pm b/perl/Git/SVN.pm index 5273ee8..3cd1c8f 100644 --- a/perl/Git/SVN.pm +++ b/perl/Git/SVN.pm @@ -1599,27

[PATCH] improve git svn performance

2014-01-19 Thread manjian2006
From: linzj li...@ucweb.com Hi, I am trying to improve git svn's performance according to some profiling data.As the data showed,_rev_list subroutine and rebuild subroutine are consuming a large proportion of time.So I improve _rev_list's performance by memoize its results,and avoid