Re: [fossil-users] Scalability (WAS: something else)

2014-09-03 Thread Stephan Beal
On Tue, Sep 2, 2014 at 11:26 PM, sky5w...@gmail.com wrote: ​ (2) Fossil's purpose is to be able to recreate historical versions of the project - exactly. It cannot do that if historical images have been deleted. I understand the purity intended, but continue to be frustrated by it. :) I

[fossil-users] Scalability (WAS: something else)

2014-09-02 Thread Stephan Beal
On Tue, Sep 2, 2014 at 4:18 PM, sky5w...@gmail.com wrote: My reservation being scalability of large repo support. While I am unaffected, those professionals charged with release and maintenance of large code bases look past Fossil and its SQLite core. Questions: Will Fossil ever seek to

Re: [fossil-users] Scalability (WAS: something else)

2014-09-02 Thread Ron W
On Tue, Sep 2, 2014 at 10:27 AM, Stephan Beal sgb...@googlemail.com wrote: but by very large source control i envision something akin to git's octopus model, reaching fractally out into the universe Fossil uses the octopus model as well. I just don't know of any projects using Fossil that

Re: [fossil-users] Scalability (WAS: something else)

2014-09-02 Thread Warren Young
On 9/2/2014 08:27, Stephan Beal wrote: On Tue, Sep 2, 2014 at 4:18 PM, sky5w...@gmail.com mailto:sky5w...@gmail.com wrote: Will Fossil ever seek to address very large source control? Fossil's main target is sqlite (it's a cyclic relationship), and in my humble (but quite fallible) opinion

Re: [fossil-users] Scalability (WAS: something else)

2014-09-02 Thread Joerg Sonnenberger
On Tue, Sep 02, 2014 at 02:45:13PM -0600, Warren Young wrote: Fossil currently wants to do a cryptographically strong checksum on every version of every graphic file I've ever created on every checkin. Consequently, a checkin takes several seconds here. There was a recent proposal that you

Re: [fossil-users] Scalability (WAS: something else)

2014-09-02 Thread Warren Young
On 9/2/2014 14:47, Joerg Sonnenberger wrote: On Tue, Sep 02, 2014 at 02:45:13PM -0600, Warren Young wrote: Fossil currently wants to do a cryptographically strong checksum on every version of every graphic file I've ever created on every checkin. Consequently, a checkin takes several seconds

Re: [fossil-users] Scalability (WAS: something else)

2014-09-02 Thread sky5walk
While disabling checksums helps with speed http://www.fossil-scm.org/index.html/help?cmd=settings It does not help with redundant binary images in the repo. For that, you have to shun and rebuild. If you could flag a file as Keep latest only, that would be less painless. I don't mind the artifact

Re: [fossil-users] Scalability (WAS: something else)

2014-09-02 Thread Richard Hipp
On Tue, Sep 2, 2014 at 5:04 PM, Warren Young war...@etr-usa.com wrote: On 9/2/2014 14:47, Joerg Sonnenberger wrote: On Tue, Sep 02, 2014 at 02:45:13PM -0600, Warren Young wrote: Fossil currently wants to do a cryptographically strong checksum on every version of every graphic file I've ever

Re: [fossil-users] Scalability (WAS: something else)

2014-09-02 Thread Richard Hipp
On Tue, Sep 2, 2014 at 5:07 PM, sky5w...@gmail.com wrote: While disabling checksums helps with speed http://www.fossil-scm.org/index.html/help?cmd=settings It does not help with redundant binary images in the repo. For that, you have to shun and rebuild. If you could flag a file as Keep

Re: [fossil-users] Scalability (WAS: something else)

2014-09-02 Thread Warren Young
On 9/2/2014 15:07, sky5w...@gmail.com wrote: If you could flag a file as Keep latest only, that would be less painless. That wouldn't work for me. I want the past versions of the image. [*] The branch I made of the web app three years ago won't run right with the current bitmaps. The new

Re: [fossil-users] Scalability (WAS: something else)

2014-09-02 Thread Warren Young
On 9/2/2014 15:11, Richard Hipp wrote: (1) Fossil *does* store binary files as diffs from their predecessor, if they are sufficiently similar (that is, if the diff is smaller than the file itself). the problem is that with compressed images, changing a single pixel can potentially change most

Re: [fossil-users] Scalability (WAS: something else)

2014-09-02 Thread sky5walk
​ (2) Fossil's purpose is to be able to recreate historical versions of the project - exactly. It cannot do that if historical images have been deleted. I understand the purity intended, but continue to be frustrated by it. :) I merely seek an automated way within Fossil to manage garbage.

[fossil-users] Scalability limits

2014-02-07 Thread Rich Neswold
Hello, I first want to say what a terrific version control manager Fossil is! I took my first serious look at Fossil last week and have already converted a few of my personal projects away from 'git'. The built-in bug tracker and wiki are genius touches! Thank you, Fossil community, for your

Re: [fossil-users] Scalability limits

2014-02-07 Thread Stephan Beal
On Fri, Feb 7, 2014 at 4:33 PM, Rich Neswold rich.nesw...@gmail.com wrote: I don't have any question; I just thought I'd document my experiences. Thanks for your feedback! IMO (possibly a minority opinion), Fossil has never aspired to host repos quite as large as those. i remember the pkgsrc

Re: [fossil-users] Scalability limits

2014-02-07 Thread Ron Wilson
On Fri, Feb 7, 2014 at 11:17 AM, Stephan Beal sgb...@googlemail.com wrote: On Fri, Feb 7, 2014 at 4:33 PM, Rich Neswold rich.nesw...@gmail.comwrote: I don't have any question; I just thought I'd document my experiences. Thanks for your feedback! IMO (possibly a minority opinion), Fossil

Re: [fossil-users] Scalability limits

2014-02-07 Thread Gour
On Fri, 7 Feb 2014 18:40:32 +0100 Stephan Beal sgb...@googlemail.com wrote: It would be really cool to see someone implement their own SCM based on fossil's core artifact model and their own db back-end, though. What about Monotone? Linus was looking at it, but it was too slow at that time.

Re: [fossil-users] Scalability limits

2014-02-07 Thread Lluís Batlle i Rossell
On Fri, Feb 07, 2014 at 07:39:37PM +0100, Gour wrote: On Fri, 7 Feb 2014 18:40:32 +0100 Stephan Beal sgb...@googlemail.com wrote: It would be really cool to see someone implement their own SCM based on fossil's core artifact model and their own db back-end, though. What about Monotone?

Re: [fossil-users] Scalability limits

2014-02-07 Thread Gour
On Fri, 7 Feb 2014 20:32:56 +0100 Lluís Batlle i Rossell vi...@viric.name wrote: It was a bug of monotone, that slowness. Fixed, for what I remember. Yeah, too bad. Otherwise we wouldn't see git. :-) But monotone works on sqlite, if the deal is sqlite. Right, but I see Monotone's influence

Re: [fossil-users] Scalability limits

2014-02-07 Thread Joerg Sonnenberger
On Fri, Feb 07, 2014 at 05:17:23PM +0100, Stephan Beal wrote: i'd be interested in seeing the output of 'dbstat' on your repo, except that it could take some time for it to finish generating its output (so don't feel obligated to try it). Here's the info for the current fossil core repo:

Re: [fossil-users] Scalability limits

2014-02-07 Thread Stephan Beal
On Fri, Feb 7, 2014 at 6:15 PM, Ron Wilson ronw.m...@gmail.com wrote: I am guessing this is a limitation of SQLite, which is designed to be light. It would be interesting to see how Fossil would perform when plugged in to, for example, PostgreSQL, MariaSQL or other heavy duty SQL server. Of

Re: [fossil-users] Scalability limits

2014-02-07 Thread Rich Neswold
On Fri, Feb 7, 2014 at 10:17 AM, Stephan Beal sgb...@googlemail.com wrote: i'd be interested in seeing the output of 'dbstat' on your repo, except that it could take some time for it to finish generating its output (so don't feel obligated to try it). Here's the info for the current fossil core

Re: [fossil-users] Scalability limits

2014-02-07 Thread Stephan Beal
On Fri, Feb 7, 2014 at 9:11 PM, Joerg Sonnenberger jo...@britannica.bec.dewrote: On Fri, Feb 07, 2014 at 05:17:23PM +0100, Stephan Beal wrote: i'd be interested in seeing the output of 'dbstat' on your repo, except that it could take some time for it to finish generating its output (so

Re: [fossil-users] Scalability limits

2014-02-07 Thread Arnel Legaspi
On 2/8/2014 5:19 AM, Stephan Beal wrote: It would be really cool to see someone implement their own SCM based on fossil's core artifact model and their own db back-end, though. It would likely require a complete re-implementation, not just rewriting most of the SQL. Wasn't Veracity

Re: [fossil-users] Scalability limits

2014-02-07 Thread Nico Williams
On Fri, Feb 7, 2014 at 11:40 AM, Stephan Beal sgb...@googlemail.com wrote: On Fri, Feb 7, 2014 at 6:15 PM, Ron Wilson ronw.m...@gmail.com wrote: I am guessing this is a limitation of SQLite, which is designed to be light. It would be interesting to see how Fossil would perform when plugged in

Re: [fossil-users] Scalability

2012-03-17 Thread Gour
On Sat, 17 Mar 2012 00:44:24 +0100 Jan Danielsson jan.m.daniels...@gmail.com wrote: At first I thought it was a problem with the server being overloaded, but when I finally got all of it downloaded to one of my servers, I tried to pull it to other systems from there, but I'm running into

Re: [fossil-users] Scalability

2012-03-17 Thread Lluís Batlle i Rossell
On Sat, Mar 17, 2012 at 12:44:24AM +0100, Jan Danielsson wrote: Joerg's NetBSD repository suddenly grew from ~2.5G to over 6GB. As it has grown, I've been having increasing problems pulling the latest changes. I started getting the database is locked errors and (more often) fossil: server

Re: [fossil-users] Scalability, a single file commit and lots of disk reads

2011-09-30 Thread Jeff Slutter
On 9/29/2011 2:12 PM, Joerg Sonnenberger wrote: What Operating System is that on? There might be a limit to the number of filesystem objects that can be cached and your tree just large enough to not fit into it. Another thing to try is forcing the _FOSSIL_ file into cache (e.g. cat _FOSSIL_

Re: [fossil-users] Scalability, a single file commit and lots of disk reads

2011-09-30 Thread Joerg Sonnenberger
On Fri, Sep 30, 2011 at 10:25:28AM -0400, Jeff Slutter wrote: On 9/29/2011 2:12 PM, Joerg Sonnenberger wrote: What Operating System is that on? There might be a limit to the number of filesystem objects that can be cached and your tree just large enough to not fit into it. Another thing to try

Re: [fossil-users] Scalability, a single file commit and lots of disk reads

2011-09-29 Thread Jeff Slutter
I downloaded the Windows 1.18 version (supposedly build with mingw) from the website and tested it getting the same results as my previous post's timings ('delete' mode and 'wal' mode). Using Sysinternals' Process Monitor it was clear that fossil was reading all the way through the repository

Re: [fossil-users] Scalability, a single file commit and lots of disk reads

2011-09-29 Thread Lluís Batlle i Rossell
On Thu, Sep 29, 2011 at 01:45:29PM +0800, mjbmik...@gmail.com wrote: It is the Windows version. I'm currently in the process of commiting a new 0 byte file to an existing 2GB repo and Windows task manager says that the fossil process has read 3GB of data since I issued the commit command

Re: [fossil-users] Scalability, a single file commit and lots of disk reads

2011-09-29 Thread alaric
2011 09:55:45 To: fossil-users@lists.fossil-scm.org Reply-To: fossil-users@lists.fossil-scm.org Subject: Re: [fossil-users] Scalability, a single file commit and lots of disk reads On Thu, Sep 29, 2011 at 01:45:29PM +0800, mjbmik...@gmail.com wrote: It is the Windows version. I'm currently

Re: [fossil-users] Scalability, a single file commit and lots of disk reads

2011-09-29 Thread Konstantin Khomoutov
On Thu, 29 Sep 2011 08:15:15 + ala...@snell-pym.org.uk wrote: Just a thought - is there some virus-scanning software involved, that feels a need to scan every file opened? The OP got the same results on Ubuntu which supposedy is not infested with antivirus software.

Re: [fossil-users] Scalability, a single file commit and lots of disk reads

2011-09-29 Thread Stephan Beal
On Thu, Sep 29, 2011 at 6:16 AM, Jeff Slutter j...@slutter.com wrote: Interesting... I failed to mention in my post that my version of fossil was from 'trunk' sometime this afternoon, build with MSVC 2008. I also made one minor change to fix handling for repos 2gig (MSVC build version

Re: [fossil-users] Scalability, a single file commit and lots of disk reads

2011-09-29 Thread Dmitry Chestnykh
To open the repository to a new checkout it took Fossil about 26 minutes. Roughly 13 minutes extracting the files into the directory, and then 13 minutes of ... doing something, before it came back. The equivalent command in Mercurial (hg update null to reset the checkout then the timed hg

Re: [fossil-users] Scalability, a single file commit and lots of disk reads

2011-09-29 Thread Stephan Beal
On Thu, Sep 29, 2011 at 3:18 PM, Jeff Slutter j...@slutter.com wrote: Add was sub 1 second Commit took 59 seconds A few weeks ago someone posted about horrible performance in his BSD Ports repo - many tens of thousands of files. Richard explained (though i cannot find the post at the moment)

Re: [fossil-users] Scalability, a single file commit and lots of disk reads

2011-09-29 Thread Jeff Slutter
Some good news... I came in to work, disabled repo-cksum, on the copy of the repository at work and tested again. Single file commit took 6 seconds. I made a number of changes to files (11 files total, a collection of edits, adds and removes) and did a fossil commit (without specifying files

Re: [fossil-users] Scalability, a single file commit and lots of disk reads

2011-09-29 Thread Stephan Beal
On Thu, Sep 29, 2011 at 5:31 PM, Jeff Slutter j...@slutter.com wrote: I don't know if that 6 seconds can be improved on, but I am definitely much happier than I was yesterday. We all love success stories! Keep 'em coming! :) And thanks for having the patience to try to get to the bottom of

Re: [fossil-users] Scalability, a single file commit and lots of disk reads

2011-09-29 Thread Joerg Sonnenberger
On Thu, Sep 29, 2011 at 11:31:19AM -0400, Jeff Slutter wrote: There seems to be a minimum time of 6 seconds for my operations of status, changes, and commit, and it would make sense that they all have to do the same work at some point (that would be 'finding out what files have changed') What

Re: [fossil-users] Scalability, a single file commit and lots of disk reads

2011-09-29 Thread Jeff Slutter
On 9/29/2011 2:12 PM, Joerg Sonnenberger wrote: What Operating System is that on? There might be a limit to the number of filesystem objects that can be cached and your tree just large enough to not fit into it. Another thing to try is forcing the _FOSSIL_ file into cache (e.g. cat _FOSSIL_

[fossil-users] Scalability, a single file commit and lots of disk reads

2011-09-28 Thread Mike Buckler
I have noticed that fossil reads the entire database file (in 1024 byte increments) several times during the commit process, even when committing a single file locally with no remote server. While this doesn't matter for small repos (10 MB), my test repo at 40 MB is border line unusable even

Re: [fossil-users] Scalability, a single file commit and lots of disk reads

2011-09-28 Thread Joshua Paine
On 9/28/2011 11:40 AM, Mike Buckler wrote: my test repo at 40 MB is border line unusable even when run from a fast solid state disk. I have a 1 GB repo that has some irritating lag on my netbook with 5400 RPM drive. But I regularly use 4 repos between 30 and 90 MB and they're all quite

Re: [fossil-users] Scalability, a single file commit and lots of disk reads

2011-09-28 Thread Mike Buckler
Switching to wal mode hasn't made any difference. There is still a huge amount of disk read activity. ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Scalability, a single file commit and lots of disk reads

2011-09-28 Thread Jeff Slutter
This is pretty relevant to my day today because I sat out to load test Fossil to make sure it will be a good fit for our future projects. I took our existing source and asset folders (so, not importing from Perforce, just taking the current 'head') of our recent project and put them into Fossil.

Re: [fossil-users] Scalability, a single file commit and lots of disk reads

2011-09-28 Thread Heinrich Huss
Are you using the Windows Version of fossil which Richard had build with VC? I had a similar issue. It went away after I have switched to a fossil version build with mingw and gcc. You can clone the fossil repository for test. With this repository it should work really snappy. Chers Hein

Re: [fossil-users] Scalability, a single file commit and lots of disk reads

2011-09-28 Thread Jeff Slutter
Interesting... I failed to mention in my post that my version of fossil was from 'trunk' sometime this afternoon, build with MSVC 2008. I also made one minor change to fix handling for repos 2gig (MSVC build version only...patch was sent to drh). Now I will have to build fossil.exe tomorrow