It takes roughly 6 hours for me to index a Gig of data. The benchmarks take
quite a bit less if I'm reading it correctly. I'll try out the
StringBuffer/Builder and let you know. Thanks for the quick response and if
you have any more suggestions please let me know.
--JP
On 8/11/07, karl wettin <[EMAIL PROTECTED]> wrote:
>
> How much slower than anticipated is it?
>
> I would start by using a StringBuffer/Builder rather than appending
> (immutable) strings to each other.
>
>
> 11 aug 2007 kl. 19.05 skrev John Paul Sondag:
>
> > Hi,
> >
> > I was hoping that maybe you guys could see if I'm somehow indexing
> > inefficiently. I'm putting relevant parts of my code below. I've
> > looked at
> > the "benchmarks" page on Lucene and my indexing time is taking a
> > substantial
> > amount of time more than what I see posted. I'm not sure when I
> > should call
> > flush() ( I saw that I should be doing that on the
> > ImproveIndexingSpeed
> > page). I'd really appreciate any advice.
> >
> > Here's my code:
> >
> > File directory = new File( "/mounts/falcon5/disks/0/tcheng3/
> > Dataset");
> > File[] theFiles = directory.listFiles();
> >
> > //go through each file inside the directory and index it
> > for(int curFile = 0; curFile < theFiles.length; curFile++)
> > {
> > File fin=theFiles[curFile];
> >
> > //open up the file
> > FileInputStream inf = new FileInputStream(fin);
> > InputStreamReader isr = new InputStreamReader(inf,
> > "US-ASCII");
> > BufferedReader in = new BufferedReader(isr);
> > String text="";
> > String docid="";
> >
> > while (true) {
> >
> > //read in the file one line at a time, and act accordingly
> > String line = in.readLine();
> > if (line == null) { break;}
> >
> > if (line.startsWith("<DOC>") ) {
> > //get docID
> > line = in.readLine();
> > String tempStr = line.substring(8,line.length());
> > int pos = tempStr.indexOf(' ');
> > docid = tempStr.substring(0,pos);
> > }else if (line.startsWith("</DOC>")) {
> >
> > Document doc = new Document();
> >
> > doc.add(new Field("contents",text,
> > Field.Store.NO,
> > Field.Index.TOKENIZED, Field.TermVector.WITH_POSITIONS ));
> > doc.add(new Field("DocID",docid, Field.Store.YES,
> > Field.Index.NO));
> > writer.addDocument(doc);
> > text="";
> > } else {
> > text = text + "\n" + line;
> > }
> > }
> >
> > }
> >
> >
> > int numIndexed = writer.docCount();
> >
> > writer.optimize();
> > writer.close();
> >
> >
> > Thanks,
> >
> > --JP
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
>
>
