Hi: During some test of Lucene Domain Index (http://docs.google.com/View?id=ddgw7sjp_54fgj9kg) with big data sources we found an exception caused for calling Directory.fieldLength() method on non existing file. FSDirectory implements this method as: /** Returns the length in bytes of a file in the directory. */ public long fileLength(String name) { ensureOpen(); File file = new File(directory, name); return file.length(); }
According to JDK1.5 calling to File constructor causes a file creation without throwing an exception: http://java.sun.com/j2se/1.5.0/docs/api/java/io/File.html#File(java.lang.String, java.lang.String) But either RAMDirectory nor OJVMDirectory do this: RAMDirectory: /** Returns the length in bytes of a file in the directory. * @throws IOException if the file does not exist */ public final long fileLength(String name) throws IOException { ensureOpen(); RAMFile file; synchronized (this) { file = (RAMFile)fileMap.get(name); } if (file==null) throw new FileNotFoundException(name); return file.getLength(); } If OJVMDirectory throws an exception if a file doesn't exist it causes that the IndexWriter fail to do the job, here the stack trace: IW 3 [Root Thread]: DW: RAM: now flush @ usedMB=15.001 allocMB=15.001 deletesMB=0 triggerMB=15 IW 3 [Root Thread]: flush: segment=_0 docStoreSegment=_0 docStoreOffset=0 flushDocs=true flushDeletes=false flushDocStores=false numDocs=109169 numBufDelTerms=0 IW 3 [Root Thread]: index before flush IW 3 [Root Thread]: DW: flush postings as segment _0 numDocs=109169 *** 2010-03-11 17:27:15.696 IW 3 [Root Thread]: DW: docWriter: now abort IW 3 [Root Thread]: hit exception flushing segment _0 IFD [Root Thread]: refresh [prefix=_0]: removing newly created unreferenced file "_0.tii" IFD [Root Thread]: delete "_0.tii" IFD [Root Thread]: refresh [prefix=_0]: removing newly created unreferenced file "_0.fnm" IFD [Root Thread]: delete "_0.fnm" IFD [Root Thread]: refresh [prefix=_0]: removing newly created unreferenced file "_0.fdx" IFD [Root Thread]: delete "_0.fdx" IFD [Root Thread]: refresh [prefix=_0]: removing newly created unreferenced file "_0.fdt" IFD [Root Thread]: delete "_0.fdt" IFD [Root Thread]: refresh [prefix=_0]: removing newly created unreferenced file "_0.prx" IFD [Root Thread]: delete "_0.prx" IFD [Root Thread]: refresh [prefix=_0]: removing newly created unreferenced file "_0.nrm" IFD [Root Thread]: delete "_0.nrm" IFD [Root Thread]: refresh [prefix=_0]: removing newly created unreferenced file "_0.frq" IFD [Root Thread]: delete "_0.frq" IFD [Root Thread]: refresh [prefix=_0]: removing newly created unreferenced file "_0.tis" IFD [Root Thread]: delete "_0.tis" Mar 11, 2010 5:27:15 PM org.apache.lucene.indexer.LuceneDomainIndex ODCIIndexCreate SEVERE: failed to create index: cannot verify file: _0.fdx. Reason: Exhausted Resultset Mar 11, 2010 5:27:15 PM org.apache.lucene.indexer.LuceneDomainIndex ODCIIndexCreate FINER: THROW java.io.IOException: cannot verify file: _0.fdx. Reason: Exhausted Resultset at org.apache.lucene.store.OJVMDirectory.fileLength(OJVMDirectory.java:633) at org.apache.lucene.index.SegmentInfo.sizeInBytes(SegmentInfo.java:271) at org.apache.lucene.index.DocumentsWriter.flush(DocumentsWriter.java:593) at org.apache.lucene.index.IndexWriter.doFlushInternal(IndexWriter.java:4311) at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:4209) at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:4200) at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:2497) at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:2451) at org.apache.lucene.indexer.TableIndexer.index(TableIndexer.java:374) at org.apache.lucene.indexer.LuceneDomainIndex.ODCIIndexCreate(LuceneDomainIndex.java:568) IW 3 [Root Thread]: now flush at close IW 3 [Root Thread]: flush: segment=null docStoreSegment=null docStoreOffset=0 flushDocs=false flushDeletes=true flushDocStores=false numDocs=0 numBufDelTerms=0 IW 3 [Root Thread]: index before flush IW 3 [Root Thread]: CMS: now merge IW 3 [Root Thread]: CMS: index: IW 3 [Root Thread]: CMS: no more merges pending; now return IW 3 [Root Thread]: now call final commit() IW 3 [Root Thread]: startCommit(): start sizeInBytes=0 IW 3 [Root Thread]: startCommit index= changeCount=1 IW 3 [Root Thread]: done all syncs IW 3 [Root Thread]: commit: pendingCommit != null IW 3 [Root Thread]: commit: wrote segments file "segments_2" IFD [Root Thread]: now checkpoint "segments_2" [0 segments ; isCommit = true] IFD [Root Thread]: deleteCommits: now decRef commit "segments_1" IFD [Root Thread]: delete "segments_1" IW 3 [Root Thread]: commit: done IW 3 [Root Thread]: at close: Which is the correct behavior for this method? We changed OJVMDirectory.fileLength() method to returns 0 if no file exists instead of throwing an exception and IndexWriter works properly, but I am not complete sure if this is the correct implementation. Here OJVMDirectory implementation: /** * Returns the length of a file in the directory. */ public long fileLength(String name) throws IOException { PreparedStatement stmt = null; ResultSet rs = null; try { stmt = conn.prepareStatement(StringUtils.replace(getFileLengthStmt,"%IDX%",this.prefix)); stmt.setString(1, name); rs = stmt.executeQuery(); if (rs.next()) return rs.getLong(1); else return 0; } catch (SQLException e) { throw new IOException("cannot verify file: " + name + ". Reason: " + e.getMessage()); } finally { OJVMUtil.closeDbResources(stmt, rs); } } Best regards, Marcelo -- Marcelo F. Ochoa http://marceloochoa.blogspot.com/ http://mochoa.sites.exa.unicen.edu.ar/ ______________ Want to integrate Lucene and Oracle? http://marceloochoa.blogspot.com/2007/09/running-lucene-inside-your-oracle-jvm.html Is Oracle 11g REST ready? http://marceloochoa.blogspot.com/2008/02/is-oracle-11g-rest-ready.html --------------------------------------------------------------------- To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org For additional commands, e-mail: java-dev-h...@lucene.apache.org
