Re: [fossil-users] Scalability, a single file commit and lots of disk reads

2011-09-29 Thread Jeff Slutter
I downloaded the Windows 1.18 version (supposedly build with mingw) from the website and tested it getting the same results as my previous post's timings ('delete' mode and 'wal' mode). Using Sysinternals' Process Monitor it was clear that fossil was reading all the way through the repository

Re: [fossil-users] Scalability, a single file commit and lots of disk reads

2011-09-29 Thread Lluís Batlle i Rossell
On Thu, Sep 29, 2011 at 01:45:29PM +0800, mjbmik...@gmail.com wrote: It is the Windows version. I'm currently in the process of commiting a new 0 byte file to an existing 2GB repo and Windows task manager says that the fossil process has read 3GB of data since I issued the commit command

Re: [fossil-users] is a rebuild in http mode possible?

2011-09-29 Thread Dmitry Chestnykh
On Sep 29, 2011, at 1:40 AM, Richard Hipp wrote: In the past there have been issues with output that rebuild sometimes generates. I haven't pressed that button lately, so I don't know if those issues are currently fixed or not. As of [2cd21f8dc2] (trunk: [ce354d0a9f]) there shouldn't be

Re: [fossil-users] Scalability, a single file commit and lots of disk reads

2011-09-29 Thread alaric
Just a thought - is there some virus-scanning software involved, that feels a need to scan every file opened? Sent from my BlackBerry® wireless device -Original Message- From: Lluís Batlle i Rossell virik...@gmail.com Sender: fossil-users-boun...@lists.fossil-scm.org Date: Thu, 29 Sep

Re: [fossil-users] Scalability, a single file commit and lots of disk reads

2011-09-29 Thread Konstantin Khomoutov
On Thu, 29 Sep 2011 08:15:15 + ala...@snell-pym.org.uk wrote: Just a thought - is there some virus-scanning software involved, that feels a need to scan every file opened? The OP got the same results on Ubuntu which supposedy is not infested with antivirus software.

Re: [fossil-users] Scalability, a single file commit and lots of disk reads

2011-09-29 Thread Stephan Beal
On Thu, Sep 29, 2011 at 6:16 AM, Jeff Slutter j...@slutter.com wrote: Interesting... I failed to mention in my post that my version of fossil was from 'trunk' sometime this afternoon, build with MSVC 2008. I also made one minor change to fix handling for repos 2gig (MSVC build version

[fossil-users] Square brackets in filenames

2011-09-29 Thread Dmitry Chestnykh
What's the rationale for disallowing [ and ] in filenames? Are there any systems that don't like them (maybe FAT)? Or maybe this is due to collision with [wiki/ci links]? From file_is_simple_pathname(const char *z): ** * Does not contain any of these characters in the path: \*[]?

Re: [fossil-users] Scalability, a single file commit and lots of disk reads

2011-09-29 Thread Dmitry Chestnykh
To open the repository to a new checkout it took Fossil about 26 minutes. Roughly 13 minutes extracting the files into the directory, and then 13 minutes of ... doing something, before it came back. The equivalent command in Mercurial (hg update null to reset the checkout then the timed hg

Re: [fossil-users] Square brackets in filenames

2011-09-29 Thread Richard Hipp
On Thu, Sep 29, 2011 at 5:27 AM, Dmitry Chestnykh dmi...@codingrobots.comwrote: What's the rationale for disallowing [ and ] in filenames? Are there any systems that don't like them (maybe FAT)? Or maybe this is due to collision with [wiki/ci links]? Special characters like this are a

Re: [fossil-users] Square brackets in filenames

2011-09-29 Thread Benoit Mortgat
Actually, FAT and NTFS both support brackets in file names. I ran into this issue of brackets when trying to version uncompressed Office 2007 documents (docx is just a zip that contains xml files, including a [Content-Types].xml at the root). On Thu, Sep 29, 2011 at 13:43, Richard Hipp

Re: [fossil-users] Scalability, a single file commit and lots of disk reads

2011-09-29 Thread Stephan Beal
On Thu, Sep 29, 2011 at 3:18 PM, Jeff Slutter j...@slutter.com wrote: Add was sub 1 second Commit took 59 seconds A few weeks ago someone posted about horrible performance in his BSD Ports repo - many tens of thousands of files. Richard explained (though i cannot find the post at the moment)

Re: [fossil-users] Scalability, a single file commit and lots of disk reads

2011-09-29 Thread Jeff Slutter
Some good news... I came in to work, disabled repo-cksum, on the copy of the repository at work and tested again. Single file commit took 6 seconds. I made a number of changes to files (11 files total, a collection of edits, adds and removes) and did a fossil commit (without specifying files

Re: [fossil-users] Scalability, a single file commit and lots of disk reads

2011-09-29 Thread Stephan Beal
On Thu, Sep 29, 2011 at 5:31 PM, Jeff Slutter j...@slutter.com wrote: I don't know if that 6 seconds can be improved on, but I am definitely much happier than I was yesterday. We all love success stories! Keep 'em coming! :) And thanks for having the patience to try to get to the bottom of

[fossil-users] Protection against timing attacks

2011-09-29 Thread Dmitry Chestnykh
Dear archeologists, Please review changes in the following branch: http://www.fossil-scm.org/index.html/timeline?r=dmitry-security The more eyes the better, as it touches login code. I try to protect against timing attacks on login, cookies, and sync by using the constant-time comparison

Re: [fossil-users] Scalability, a single file commit and lots of disk reads

2011-09-29 Thread Joerg Sonnenberger
On Thu, Sep 29, 2011 at 11:31:19AM -0400, Jeff Slutter wrote: There seems to be a minimum time of 6 seconds for my operations of status, changes, and commit, and it would make sense that they all have to do the same work at some point (that would be 'finding out what files have changed') What

Re: [fossil-users] Protection against timing attacks

2011-09-29 Thread Stephan Beal
On Thu, Sep 29, 2011 at 8:00 PM, Dmitry Chestnykh dmi...@codingrobots.comwrote: The more eyes the better, as it touches login code. ...COMPARE(A, PASSWORD) returns FALSE in 0.1 msec, but COMPARE(P, PASSWORD) returns FALSE in 0.3 msec, because it did two comparisons: Given

Re: [fossil-users] Protection against timing attacks

2011-09-29 Thread Lluís Batlle i Rossell
On Thu, Sep 29, 2011 at 08:15:55PM +0200, Stephan Beal wrote: On Thu, Sep 29, 2011 at 8:00 PM, Dmitry Chestnykh dmi...@codingrobots.comwrote: The more eyes the better, as it touches login code. ...COMPARE(A, PASSWORD) returns FALSE in 0.1 msec, but COMPARE(P, PASSWORD)

Re: [fossil-users] Scalability, a single file commit and lots of disk reads

2011-09-29 Thread Jeff Slutter
On 9/29/2011 2:12 PM, Joerg Sonnenberger wrote: What Operating System is that on? There might be a limit to the number of filesystem objects that can be cached and your tree just large enough to not fit into it. Another thing to try is forcing the _FOSSIL_ file into cache (e.g. cat _FOSSIL_

Re: [fossil-users] Wanted: adventurous JSON fans to take a test spin

2011-09-29 Thread David Bovill
Thanks for this - I'm definitely going to need it, and will try to make time to do some tests next week :) On 27 September 2011 06:12, Stephan Beal sgb...@googlemail.com wrote: Hi, all! With almost 90kb of new functionality and 21 pages of draft spec docs[1], the JSON branch has reached a

[fossil-users] possible bug + performance hit in timeline_cmd()

2011-09-29 Thread Stephan Beal
Hiya, core dev(s), In timeline_cmd(), is there a reason that: db_find_and_open_repository(0, 0); showfilesFlag = find_option(showfiles,f, 0)!=0; db_find_and_open_repository(0, 0); db_find_and_open_repository() is called twice, or is that a mistake? That can cause db_verify_schema() to be

Re: [fossil-users] Wanted: adventurous JSON fans to take a test spin

2011-09-29 Thread Stephan Beal
On Thu, Sep 29, 2011 at 8:45 PM, David Bovill da...@architex.tv wrote: Thanks for this - I'm definitely going to need it, and will try to make time to do some tests next week :) If you're looking for a staring point/demo, see ajax/index.html and friends. Any and all feedback is welcomed.

Re: [fossil-users] Protection against timing attacks

2011-09-29 Thread Dmitry Chestnykh
On Sep 29, 2011, at 20:15 , Stephan Beal wrote: Given the relatively high overhead fossil has when it opens a db or runs a query, and network latency, i cannot imagine that someone could accurately time the difference of a memcmp() operation on 8 or 10 bytes. The number of factors involved

Re: [fossil-users] Protection against timing attacks

2011-09-29 Thread Lluís Batlle i Rossell
On Thu, Sep 29, 2011 at 09:07:37PM +0200, Stephan Beal wrote: On Thu, Sep 29, 2011 at 8:57 PM, Dmitry Chestnykh dmi...@codingrobots.comwrote: I posted a link about this concern: http://rdist.root.org/2010/01/07/timing-independent-array-comparison/ So why not simply add the following

Re: [fossil-users] Protection against timing attacks

2011-09-29 Thread Dmitry Chestnykh
So why not simply add the following logic to server mode: A) fetch config option add-random-sleep (integer, default=0) B) if ((A)0) AND user is nobody, sleep for random 1..(A) ms. (This attack would seem to be useless for anyone but the nobody user. If you're logged in, you've got your

Re: [fossil-users] Protection against timing attacks

2011-09-29 Thread Stephan Beal
On Thu, Sep 29, 2011 at 8:57 PM, Dmitry Chestnykh dmi...@codingrobots.comwrote: I posted a link about this concern: http://rdist.root.org/2010/01/07/timing-independent-array-comparison/ So why not simply add the following logic to server mode: A) fetch config option add-random-sleep

Re: [fossil-users] Protection against timing attacks

2011-09-29 Thread Stephan Beal
2011/9/29 Lluís Batlle i Rossell virik...@gmail.com Well, if someone has studied those attacks, and the code by dmitry looks fine and works based on some heavier studies than our first thoughts on that, let's use it. No? i can't object, i just think it's paranoid :). -- - stephan

Re: [fossil-users] Wildcards not working on windows

2011-09-29 Thread Jan Danielsson
On 09/26/11 15:02, Richard Hipp wrote: Richard seem to have released 1.19 binary for Windows compiled with Visual Studio, while 1.18 has been compiled with MinGW. Correct. I didn't realized there was a difference. Should I make a point of always building future Fossil releases using