Re: [fossil-users] Scalability, a single file commit and lots of disk reads

2011-09-30 Thread Joerg Sonnenberger
On Fri, Sep 30, 2011 at 10:25:28AM -0400, Jeff Slutter wrote: > On 9/29/2011 2:12 PM, Joerg Sonnenberger wrote: > >What Operating System is that on? There might be a limit to the number > >of filesystem objects that can be cached and your tree just large enough > >to not fit into it. Another thing

Re: [fossil-users] Scalability, a single file commit and lots of disk reads

2011-09-30 Thread Jeff Slutter
On 9/29/2011 2:12 PM, Joerg Sonnenberger wrote: What Operating System is that on? There might be a limit to the number of filesystem objects that can be cached and your tree just large enough to not fit into it. Another thing to try is forcing the _FOSSIL_ file into cache (e.g. cat _FOSSIL_> /de

Re: [fossil-users] Scalability, a single file commit and lots of disk reads

2011-09-29 Thread Jeff Slutter
On 9/29/2011 2:12 PM, Joerg Sonnenberger wrote: What Operating System is that on? There might be a limit to the number of filesystem objects that can be cached and your tree just large enough to not fit into it. Another thing to try is forcing the _FOSSIL_ file into cache (e.g. cat _FOSSIL_> /d

Re: [fossil-users] Scalability, a single file commit and lots of disk reads

2011-09-29 Thread Joerg Sonnenberger
On Thu, Sep 29, 2011 at 11:31:19AM -0400, Jeff Slutter wrote: > There seems to be a minimum time of 6 seconds for my operations of > status, changes, and commit, and it would make sense that they all > have to do the same work at some point (that would be 'finding out > what files have changed') W

Re: [fossil-users] Scalability, a single file commit and lots of disk reads

2011-09-29 Thread Stephan Beal
On Thu, Sep 29, 2011 at 5:31 PM, Jeff Slutter wrote: > I don't know if that "6 seconds" can be improved on, but I am definitely > much happier than I was yesterday. > We all love success stories! Keep 'em coming! :) And thanks for having the patience to try to get to the bottom of the problem,

Re: [fossil-users] Scalability, a single file commit and lots of disk reads

2011-09-29 Thread Jeff Slutter
Some good news... I came in to work, disabled repo-cksum, on the copy of the repository at work and tested again. Single file commit took 6 seconds. I made a number of changes to files (11 files total, a collection of edits, adds and removes) and did a fossil commit (without specifying files o

Re: [fossil-users] Scalability, a single file commit and lots of disk reads

2011-09-29 Thread Stephan Beal
On Thu, Sep 29, 2011 at 3:18 PM, Jeff Slutter wrote: > Add was sub 1 second > Commit took 59 seconds > A few weeks ago someone posted about horrible performance in his BSD Ports repo - many tens of thousands of files. Richard explained (though i cannot find the post at the moment) something abou

Re: [fossil-users] Scalability, a single file commit and lots of disk reads

2011-09-29 Thread Jeff Slutter
On 9/29/2011 5:23 AM, Stephan Beal wrote: > > This _might_ be relevant, might not: > > (from within checkout folder root, on the Windows machine) fossil set repo-cksum off fossil set mtime-changes on dir * /s > test.txt fossil add test.txt fossil commit -m "test" test.txt Add was sub 1 second Com

Re: [fossil-users] Scalability, a single file commit and lots of disk reads

2011-09-29 Thread Dmitry Chestnykh
> To open the repository to a new checkout it took Fossil about 26 > minutes. Roughly 13 minutes extracting the files into the directory, and > then 13 minutes of ... doing something, before it came back. > The equivalent command in Mercurial ("hg update null" to reset the > checkout then the timed

Re: [fossil-users] Scalability, a single file commit and lots of disk reads

2011-09-29 Thread Dmitry Chestnykh
On Sep 29, 2011, at 5:49 AM, Jeff Slutter wrote: > Initial 'add' of all the files took roughly 10 minutes. For comparison, > Mercurial took less than a minute. Let's start with this and work on low-hanging fruits. I have created ~7 tiny files (no directories). Adding all files takes 16 sec (

Re: [fossil-users] Scalability, a single file commit and lots of disk reads

2011-09-29 Thread Stephan Beal
On Thu, Sep 29, 2011 at 6:16 AM, Jeff Slutter wrote: > Interesting... > > I failed to mention in my post that my version of fossil was from 'trunk' > sometime this afternoon, build with MSVC 2008. I also made one minor change > to fix handling for repos > 2gig (MSVC build version only...patch wa

Re: [fossil-users] Scalability, a single file commit and lots of disk reads

2011-09-29 Thread Konstantin Khomoutov
On Thu, 29 Sep 2011 08:15:15 + ala...@snell-pym.org.uk wrote: > Just a thought - is there some virus-scanning software involved, that > feels a need to scan every file opened? The OP got the same results on Ubuntu which supposedy is not infested with antivirus software. ___

Re: [fossil-users] Scalability, a single file commit and lots of disk reads

2011-09-29 Thread alaric
: Reply-To: fossil-users@lists.fossil-scm.org Subject: Re: [fossil-users] Scalability, a single file commit and lots of disk reads On Thu, Sep 29, 2011 at 01:45:29PM +0800, mjbmik...@gmail.com wrote: > It is the Windows version. > > I'm currently in the process of commiting a new 0 b

Re: [fossil-users] Scalability, a single file commit and lots of disk reads

2011-09-29 Thread LluĂ­s Batlle i Rossell
On Thu, Sep 29, 2011 at 01:45:29PM +0800, mjbmik...@gmail.com wrote: > It is the Windows version. > > I'm currently in the process of commiting a new 0 byte file to an existing > 2GB repo and Windows task manager says that the fossil process has read >3GB > of data since I issued the commit comm

Re: [fossil-users] Scalability, a single file commit and lots of disk reads

2011-09-29 Thread Jeff Slutter
I downloaded the Windows 1.18 version (supposedly build with mingw) from the website and tested it getting the same results as my previous post's timings ('delete' mode and 'wal' mode). Using Sysinternals' Process Monitor it was clear that fossil was reading all the way through the repository seve

Re: [fossil-users] Scalability, a single file commit and lots of disk reads

2011-09-28 Thread mjbmikeb2
It is the Windows version. I'm currently in the process of commiting a new 0 byte file to an existing 2GB repo and Windows task manager says that the fossil process has read >3GB of data since I issued the commit command several minutes ago, and it's still running. First it read the the entire

Re: [fossil-users] Scalability, a single file commit and lots of disk reads

2011-09-28 Thread Jeff Slutter
Interesting... I failed to mention in my post that my version of fossil was from 'trunk' sometime this afternoon, build with MSVC 2008. I also made one minor change to fix handling for repos > 2gig (MSVC build version only...patch was sent to drh). Now I will have to build fossil.exe tomorrow usi

Re: [fossil-users] Scalability, a single file commit and lots of disk reads

2011-09-28 Thread Heinrich Huss
Are you using the Windows Version of fossil which Richard had build with VC? I had a similar issue. It went away after I have switched to a fossil version build with mingw and gcc. You can clone the fossil repository for test. With this repository it should work really snappy. Chers Hein A

Re: [fossil-users] Scalability, a single file commit and lots of disk reads

2011-09-28 Thread Jeff Slutter
This is pretty relevant to my day today because I sat out to load test Fossil to make sure it will be a good fit for our future projects. I took our existing source and asset folders (so, not importing from Perforce, just taking the current 'head') of our recent project and put them into Fossil. F

Re: [fossil-users] Scalability, a single file commit and lots of disk reads

2011-09-28 Thread Mike Buckler
Switching to wal mode hasn't made any difference. There is still a huge amount of disk read activity. ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Scalability, a single file commit and lots of disk reads

2011-09-28 Thread Konstantin Khomoutov
On Wed, 28 Sep 2011 23:40:11 +0800 Mike Buckler wrote: > I have noticed that fossil reads the entire database file (in 1024 > byte increments) several times during the commit process, even when > committing a single file locally with no remote server. > While this doesn't matter for small repos

Re: [fossil-users] Scalability, a single file commit and lots of disk reads

2011-09-28 Thread Joshua Paine
On 9/28/2011 11:40 AM, Mike Buckler wrote: my test repo at 40 MB is border line unusable even when run from a fast solid state disk. I have a 1 GB repo that has some irritating lag on my netbook with 5400 RPM drive. But I regularly use 4 repos between 30 and 90 MB and they're all quite snappy.

[fossil-users] Scalability, a single file commit and lots of disk reads

2011-09-28 Thread Mike Buckler
I have noticed that fossil reads the entire database file (in 1024 byte increments) several times during the commit process, even when committing a single file locally with no remote server. While this doesn't matter for small repos (<10 MB), my test repo at 40 MB is border line unusable even w