Re: [fossil-users] Scalability, a single file commit and lots of disk reads

2011-09-30 Thread Jeff Slutter
On 9/29/2011 2:12 PM, Joerg Sonnenberger wrote: What Operating System is that on? There might be a limit to the number of filesystem objects that can be cached and your tree just large enough to not fit into it. Another thing to try is forcing the _FOSSIL_ file into cache (e.g. cat _FOSSIL_

Re: [fossil-users] Scalability, a single file commit and lots of disk reads

2011-09-30 Thread Joerg Sonnenberger
On Fri, Sep 30, 2011 at 10:25:28AM -0400, Jeff Slutter wrote: On 9/29/2011 2:12 PM, Joerg Sonnenberger wrote: What Operating System is that on? There might be a limit to the number of filesystem objects that can be cached and your tree just large enough to not fit into it. Another thing to try

Re: [fossil-users] Scalability, a single file commit and lots of disk reads

2011-09-29 Thread Jeff Slutter
I downloaded the Windows 1.18 version (supposedly build with mingw) from the website and tested it getting the same results as my previous post's timings ('delete' mode and 'wal' mode). Using Sysinternals' Process Monitor it was clear that fossil was reading all the way through the repository

Re: [fossil-users] Scalability, a single file commit and lots of disk reads

2011-09-29 Thread Lluís Batlle i Rossell
On Thu, Sep 29, 2011 at 01:45:29PM +0800, mjbmik...@gmail.com wrote: It is the Windows version. I'm currently in the process of commiting a new 0 byte file to an existing 2GB repo and Windows task manager says that the fossil process has read 3GB of data since I issued the commit command

Re: [fossil-users] Scalability, a single file commit and lots of disk reads

2011-09-29 Thread alaric
Just a thought - is there some virus-scanning software involved, that feels a need to scan every file opened? Sent from my BlackBerry® wireless device -Original Message- From: Lluís Batlle i Rossell virik...@gmail.com Sender: fossil-users-boun...@lists.fossil-scm.org Date: Thu, 29 Sep

Re: [fossil-users] Scalability, a single file commit and lots of disk reads

2011-09-29 Thread Konstantin Khomoutov
On Thu, 29 Sep 2011 08:15:15 + ala...@snell-pym.org.uk wrote: Just a thought - is there some virus-scanning software involved, that feels a need to scan every file opened? The OP got the same results on Ubuntu which supposedy is not infested with antivirus software.

Re: [fossil-users] Scalability, a single file commit and lots of disk reads

2011-09-29 Thread Stephan Beal
On Thu, Sep 29, 2011 at 6:16 AM, Jeff Slutter j...@slutter.com wrote: Interesting... I failed to mention in my post that my version of fossil was from 'trunk' sometime this afternoon, build with MSVC 2008. I also made one minor change to fix handling for repos 2gig (MSVC build version

Re: [fossil-users] Scalability, a single file commit and lots of disk reads

2011-09-29 Thread Dmitry Chestnykh
To open the repository to a new checkout it took Fossil about 26 minutes. Roughly 13 minutes extracting the files into the directory, and then 13 minutes of ... doing something, before it came back. The equivalent command in Mercurial (hg update null to reset the checkout then the timed hg

Re: [fossil-users] Scalability, a single file commit and lots of disk reads

2011-09-29 Thread Stephan Beal
On Thu, Sep 29, 2011 at 3:18 PM, Jeff Slutter j...@slutter.com wrote: Add was sub 1 second Commit took 59 seconds A few weeks ago someone posted about horrible performance in his BSD Ports repo - many tens of thousands of files. Richard explained (though i cannot find the post at the moment)

Re: [fossil-users] Scalability, a single file commit and lots of disk reads

2011-09-29 Thread Jeff Slutter
Some good news... I came in to work, disabled repo-cksum, on the copy of the repository at work and tested again. Single file commit took 6 seconds. I made a number of changes to files (11 files total, a collection of edits, adds and removes) and did a fossil commit (without specifying files

Re: [fossil-users] Scalability, a single file commit and lots of disk reads

2011-09-29 Thread Stephan Beal
On Thu, Sep 29, 2011 at 5:31 PM, Jeff Slutter j...@slutter.com wrote: I don't know if that 6 seconds can be improved on, but I am definitely much happier than I was yesterday. We all love success stories! Keep 'em coming! :) And thanks for having the patience to try to get to the bottom of

Re: [fossil-users] Scalability, a single file commit and lots of disk reads

2011-09-29 Thread Joerg Sonnenberger
On Thu, Sep 29, 2011 at 11:31:19AM -0400, Jeff Slutter wrote: There seems to be a minimum time of 6 seconds for my operations of status, changes, and commit, and it would make sense that they all have to do the same work at some point (that would be 'finding out what files have changed') What

Re: [fossil-users] Scalability, a single file commit and lots of disk reads

2011-09-29 Thread Jeff Slutter
On 9/29/2011 2:12 PM, Joerg Sonnenberger wrote: What Operating System is that on? There might be a limit to the number of filesystem objects that can be cached and your tree just large enough to not fit into it. Another thing to try is forcing the _FOSSIL_ file into cache (e.g. cat _FOSSIL_

[fossil-users] Scalability, a single file commit and lots of disk reads

2011-09-28 Thread Mike Buckler
I have noticed that fossil reads the entire database file (in 1024 byte increments) several times during the commit process, even when committing a single file locally with no remote server. While this doesn't matter for small repos (10 MB), my test repo at 40 MB is border line unusable even

Re: [fossil-users] Scalability, a single file commit and lots of disk reads

2011-09-28 Thread Joshua Paine
On 9/28/2011 11:40 AM, Mike Buckler wrote: my test repo at 40 MB is border line unusable even when run from a fast solid state disk. I have a 1 GB repo that has some irritating lag on my netbook with 5400 RPM drive. But I regularly use 4 repos between 30 and 90 MB and they're all quite

Re: [fossil-users] Scalability, a single file commit and lots of disk reads

2011-09-28 Thread Mike Buckler
Switching to wal mode hasn't made any difference. There is still a huge amount of disk read activity. ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Scalability, a single file commit and lots of disk reads

2011-09-28 Thread Jeff Slutter
This is pretty relevant to my day today because I sat out to load test Fossil to make sure it will be a good fit for our future projects. I took our existing source and asset folders (so, not importing from Perforce, just taking the current 'head') of our recent project and put them into Fossil.

Re: [fossil-users] Scalability, a single file commit and lots of disk reads

2011-09-28 Thread Heinrich Huss
Are you using the Windows Version of fossil which Richard had build with VC? I had a similar issue. It went away after I have switched to a fossil version build with mingw and gcc. You can clone the fossil repository for test. With this repository it should work really snappy. Chers Hein

Re: [fossil-users] Scalability, a single file commit and lots of disk reads

2011-09-28 Thread Jeff Slutter
Interesting... I failed to mention in my post that my version of fossil was from 'trunk' sometime this afternoon, build with MSVC 2008. I also made one minor change to fix handling for repos 2gig (MSVC build version only...patch was sent to drh). Now I will have to build fossil.exe tomorrow