I just did that so I could read it. :) I'll leave it up until Glen resends or posts it somewhere... http://www.casscostello.com/?page_id=28
On Tue, Apr 15, 2008 at 5:18 PM, Ian Holsman <[EMAIL PROTECTED]> wrote: > Hi Glen. > can you resend this in plain text? > or put the HTML up on a server somewhere and point to it with a brief > summary in the post? > I'd love to look and read it, all those tags are making me go blind. > > > Glen Newton wrote: > > > <benchmark> > > <ul> > > <p> > > <b>Hardware Environment</b><br/> > > <li><i>Dedicated machine for indexing</i>: yes</li> > > <li><i>CPU</i>: Dual processor dual core Xeon CPU 3.00GHz; > > hyperthreading ON for 8 virtual cores</li> > > <li><i>RAM</i>: 8GB</li> > > <li><i>Drive configuration</i>: Dell EMC AX150 storage array fibre > > channel</li> > > </p> > > <p> > > <b>Software environment</b><br/> > > <li><i>Lucene Version</i>: 2.3.1</li> > > <li><i>Java Version</i>: Java(TM) SE Runtime Environment (build > > 1.6.0_02-b05)</li> > > <li><i>Java VM</i>: Java HotSpot(TM) 64-Bit Server VM (build > > 1.6.0_02-b05, mixed mode)</li> > > <li><i>OS Version</i>: Linux OpenSUSE 10.2 (64-bit X86-64)</li> > > <li><i>Location of index</i>: Filesystem, on attached storage</li> > > </p> > > <p> > > <b>Lucene indexing variables</b><br/> > > <li><i>Number of source documents</i>: 6,404,464</li> > > <li><i>Total filesize of source documents</i>: 141GB; Note that this > > is only the full-text: the metadata (title, author(s), abstract, > > keywords, journal name) are in addition to this</li> > > <li><i>Average filesize of source documents</i>: > > 22KB + metadata (see above)</li> > > <li><i>Source documents storage location</i>: Where are the documents > > being indexed located? > > Filesystem</li> > > <li><i>File type of source documents</i>: text (PDFs converted to > > text then gzipped)</li> > > <li><i>Parser(s) used, if any</i>: None, but files GZIPed & had to > > be un-gziped by Java application which also did indexing</li> > > <li><i>Analyzer(s) used</i>: StandardAnalyzer</li> > > <li><i>Number of fields per document</i>: 24</li> > > <li><i>Type of fields</i>: all text; 20 stored; 3 of indexed > > tokenized with term vector (full-text [not stored], title, abstract); > > 10 stored with no parsing; </li> > > <li><i>Index persistence</i>: FSDirectory</li> > > <li><i>Index size</i>: 83GB</li> > > <li><i>Number of terms</i>: 143,298,010</li> > > </p> > > <p> > > <b>Figures</b><br/> > > <li><i>Time taken (in ms/s as an average of at least 3 indexing > > runs)</i>: 20.5 hours</li> > > <li><i>Time taken / 1000 docs indexed</i>: 11.5 seconds </li> > > <li><i>Memory consumption</i>: -Xms4000m -Xmx6000m</li> > > <li><i>Query speed</i>: average time a query takes, type > > of queries (e.g. simple one-term query, phrase query), > > not measuring any overhead outside Lucene</li> > > </p> > > <p> > > <b>Notes</b><br/> > > <li><i>Notes</i>: > > <ul> > > <li> > > These are journal articles, so the additional fields besides > > the > > full-text are bibliographic metadata, such as title, authors, > > abstract, keywords, journal name, volume, issue, start page, year. > > </li> > > <li>Java command line directives: -XX:+AggressiveOpts > > -XX:+ScavengeBeforeFullGC -XX:-UseParallelGC -server -Xms4000m > > -Xmx6000m > > </li> > > <li>Highly multithreaded & pipelined architecture using > > java.util.concurrent.ThreadPoolExecutor > > </li> > > <li>File system file reading and Un-gzip performed multithreaded > > </li> > > <li>Eight separate parallel IndexWriters are fed by the pipeline > > (creation of Document objects occurs in parallel with 64 threads), > > merged at end into single index. Each parallel index had slightly > > different RAM_BUFFER_SIZE_MB (64, 67, 70, 73, 76, 79, 83, 85 MB > > respectively), so that flushing wouldn't all happen at the same time. > > </li> > > <li> > > Contact: glen DOT newton AT nrc-cnrc DOT gc DOT ca > > </li> > > > > </ul> > > </li> > > </p> > > </ul> > > </benchmark> > > > > > > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > -- Lego timeline: http://cache.gizmodo.com/assets/resources/2008/01/lego-brick4-timeline.jpg
