Re: madvise and gregs hallucinations

2012-12-11 Thread Greg Bowyer
On 12/11/2012 11:59 AM, Yonik Seeley wrote: > On Tue, Dec 11, 2012 at 2:32 PM, Greg Bowyer wrote: >> Yes the index can fit in ram on the boxes I am testing with - Its the >> main rationale for sharding to make sure that we can hold an index in >> ram at all times. >> >> MADV_WILLNEED might be rath

Re: madvise and gregs hallucinations

2012-12-11 Thread Yonik Seeley
On Tue, Dec 11, 2012 at 2:32 PM, Greg Bowyer wrote: > Yes the index can fit in ram on the boxes I am testing with - Its the > main rationale for sharding to make sure that we can hold an index in > ram at all times. > > MADV_WILLNEED might be rather bad if the index is bigger than ram > (something

Re: madvise and gregs hallucinations

2012-12-11 Thread Greg Bowyer
Yes the index can fit in ram on the boxes I am testing with - Its the main rationale for sharding to make sure that we can hold an index in ram at all times. MADV_WILLNEED might be rather bad if the index is bigger than ram (something to test maybe) On 12/11/2012 02:40 AM, Michael McCandless wrot

RE: madvise and gregs hallucinations

2012-12-11 Thread Uwe Schindler
; To: lucene-...@apache.org > Subject: madvise and gregs hallucinations > > Since its too long (and has too much HTML and pictures and such forth) for > the mailing list I have a more detailed write up here > > http://people.apache.org/~gbowyer/madvise-perf/index.html > >

Re: madvise and gregs hallucinations

2012-12-11 Thread Michael McCandless
Yes, do keep digging! darklaunch sounds like a very useful tool :) And big bonus points because it's written in Python, heh. The results are incredible. In this test is there plenty of RAM to hold the index? Mike McCandless http://blog.mikemccandless.com On Tue, Dec 11, 2012 at 12:33 AM, Greg

Re: madvise and gregs hallucinations

2012-12-10 Thread Dawid Weiss
Keep digging, this is interesting ;) D. On Tue, Dec 11, 2012 at 6:33 AM, Greg Bowyer wrote: > Since its too long (and has too much HTML and pictures and such forth) > for the mailing list I have a more detailed write up here > > http://people.apache.org/~gbowyer/madvise-perf/index.html > > Howev

Re: madvise and gregs hallucinations

2012-12-10 Thread Shawn Heisey
On 12/10/2012 10:33 PM, Greg Bowyer wrote: Since its too long (and has too much HTML and pictures and such forth) for the mailing list I have a more detailed write up here http://people.apache.org/~gbowyer/madvise-perf/index.html Then again this might be my own personal insanity .. :S If

madvise and gregs hallucinations

2012-12-10 Thread Greg Bowyer
Since its too long (and has too much HTML and pictures and such forth) for the mailing list I have a more detailed write up here http://people.apache.org/~gbowyer/madvise-perf/index.html However the short version. At $DAYJOB as part of moving to lucene 4.0 I have been looking at what we can chan