Re: Journal2 patches + timings + notes
On Tue, Dec 2, 2008 at 4:27 AM, Michael Stone [EMAIL PROTECTED] wrote: I also did a bit of timing on my XO, (see the 'timing' branch), wherein I learned, _anecdotally_ (i.e. with sample-size n=1) that: * startup time seems okay, but the initial query takes about 10s to return results. I have about 300 MB of data indexed under /home/olpc, plus a bit of other junk. (~/.pinot is ~ 35 MB). * it takes around 4.2 seconds to fetch and display each new result * the standard deviation in my sample was large, e.g. about 3.3 seconds. * 19 results were fetched. What is the size of teh data indexed there? Working with git internals I quickly realised that there are some good trivial strategies (ab)using the fs performance on linux that are simpler and faster than using an indexing library as I've done in the past. On a slow fs like jffs2, fancy indexers can be even worse than the cheapskates fs techniques because mmapping may suck on jffs2 but (ab)using directories doesn't... cheers, m -- [EMAIL PROTECTED] [EMAIL PROTECTED] -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: Journal2 patches + timings + notes
On Tue, Dec 2, 2008 at 1:27 AM, Michael Stone [EMAIL PROTECTED] wrote: * startup time seems okay, but the initial query takes about 10s to return results. I have about 300 MB of data indexed under /home/olpc, plus a bit of other junk. (~/.pinot is ~ 35 MB). Yes, that's because pinot is not actually returning incremental results yet; it is doing the entire search, sorting it chronologically, and *then* returning the first N. * it takes around 4.2 seconds to fetch and display each new result See above. The subsequent searches have a hot cache, at least, but they are not actually incremental. * With today's code, the search system is effectively unusable while the indexer is hard at work. * While I have not yet measured it carefully, it seemed to me that full indexing of a basic 767 system + activities _on XO_ may have taken as long as several hours. See above; also the plan is to index /home, not /. * With some, but not all queries, I noticed that the query failed because pinot failed to reply to journal2's blocking message sends within the usual dbus timeout of (30? 60?) seconds. I have no idea what was happening inside pinot. See above. P.S. - Thanks for the stimulating vacation entertainment. You're welcome! Thanks for the patches! Want to work some on pinot? --scott -- ( http://cscott.net/ ) ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Journal2 patches + timings + notes
Scott, I spent some more time with Journal2 today and implemented two minor features; namely, * display of the estimated no. of results - (which helps deal with the display of empty result-sets) * incremental display of results - (which makes it easier to time things). Please review and consider merging my patches, which can be found in http://dev.laptop.org/git/users/mstone/journal2. (That repo contains your journal2 code but none of the big pdfs or screencasts you made since these are very time-consuming to download onto an XO.) I also did a bit of timing on my XO, (see the 'timing' branch), wherein I learned, _anecdotally_ (i.e. with sample-size n=1) that: * startup time seems okay, but the initial query takes about 10s to return results. I have about 300 MB of data indexed under /home/olpc, plus a bit of other junk. (~/.pinot is ~ 35 MB). * it takes around 4.2 seconds to fetch and display each new result * the standard deviation in my sample was large, e.g. about 3.3 seconds. * 19 results were fetched. Notes: * I did not take care to ensure that the indexer had quiesced before my test run. (However, I don't think it was working hard since it did the bulk of the indexing yesterday; today, it only had to deal files changed by my editing and by the test run itself.) * With today's code, the search system is effectively unusable while the indexer is hard at work. * While I have not yet measured it carefully, it seemed to me that full indexing of a basic 767 system + activities _on XO_ may have taken as long as several hours. * With some, but not all queries, I noticed that the query failed because pinot failed to reply to journal2's blocking message sends within the usual dbus timeout of (30? 60?) seconds. I have no idea what was happening inside pinot. Michael P.S. - Thanks for the stimulating vacation entertainment. ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
