[
https://issues.apache.org/jira/browse/LUCENE-1192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12572449#action_12572449
]
Michael McCandless commented on LUCENE-1192:
--------------------------------------------
This actually isn't quite a valid use of Lucene's APIs, because the
ramIndexWriter is actively changing the files in the RAMDirectory.
This worked in past versions of Lucene because merges were done
synchronously. Now, with background merges by default, a merge can
complete and commit changes to the RAMDirectory while
addIndexesNoOptimize is running. The addIndexesNoOptimize method
requires a "static" Directory, where no writer is actively making
changes.
There are a few ways you could get this to work:
* Sync all concurrent merges before calling addIndexesNoOptimize, by
calling ((ConcurrentMergeScheduler)
ramIndexWriter.getMergeScheduler()).sync()
* Switch back to SerialMergeScheduler
* Close the ramIndexWriter before passing its directory to
addIndexesNoOptimize. In this case you need to separately hold
the underlying RAMDirectory since you can't call
IndexWriter.getDirectory after close.
> FileNotFound exception on adding a RAMDirectory to an IndexWriter
> -----------------------------------------------------------------
>
> Key: LUCENE-1192
> URL: https://issues.apache.org/jira/browse/LUCENE-1192
> Project: Lucene - Java
> Issue Type: Bug
> Affects Versions: 2.3.1
> Environment: || Description || Value ||
> | java -version | {noformat}java version "1.5.0_14"
> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_14-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_14-b03, mixed mode)
> {noformat} |
> | JVM settings | {noformat}-server -verbose:gc -Xloggc:gc.log
> -XX:+PrintGCApplicationStoppedTime
> -XX:NewRatio=2 -XX:ParallelGCThreads=4 -Xms6000m -Xmx6000m{noformat} |
> | uname -a | {noformat}Linux pdbindexer2 2.6.22.14-72.fc6 #1
> SMP Wed Nov 21 14:10:25 EST 2007 x86_64 x86_64 x86_64 GNU/Linux{noformat} |
> | CPU | Dual Core AMD Opteron(tm) Processor 280 |
> | Total available memory | 8G |
> Reporter: Alex Falca
>
> Hi,
> I'm getting FileNotFoundException on adding RAMDirectory to an IndexWriter
> with following stack trace:
> {noformat}
> java.io.FileNotFoundException: _8.fnm
> at
> org.apache.lucene.store.RAMDirectory.openInput(RAMDirectory.java:234)
> at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:57)
> at
> org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:298)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:262)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:221)
> at
> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3099)
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:2834)
> at
> org.apache.lucene.index.IndexWriter.copyExternalSegments(IndexWriter.java:2263)
> at
> org.apache.lucene.index.IndexWriter.addIndexesNoOptimize(IndexWriter.java:2238)
> {noformat}
> Same code works fine with Lucene 2.2.0. Basically I'm trying to index a huge
> number of documents (~21mln) having following workflow:
> 1. First documents are added to and index which use a RAMDirectory as a
> storage
> 2. Once a given condition occurs RAM based index writer is flushed and his
> directory is being added to another index writer, which use a FSDirectory as
> a storage.
> Sample code:
> {code}
> IndexWriter mainIndexWriter = new IndexWriter(indexDir, new
> JapaneseAnalyzer(), true);
> IndexWriter ramIndexWriter = new IndexWriter(new RAMDirectory(), new
> JapaneseAnalyzer(), true);
> mainIndexWriter.setMergeFactor(10);
> mainIndexWriter.setUseCompoundFile(false);
> ramIndexWriter.setUseCompoundFile(false);
> // Here some code for adding documents to RAM index writer
> ...................................................................................................................
> // method where I'm flushing from memory to disk and getting exception
> ramIndexWriter.flush();
> mainIndexWriter.addIndexesNoOptimize(new Directory[]
> {ramIndexWriter.getDirectory()});
> ramIndexWriter.close();
> ramIndexWriter = new IndexWriter(new RAMDirectory(), new JapaneseAnalyzer(),
> true);
> ramIndexWriter.setUseCompoundFile(false);
> System.gc();
> {code}
> I understand that I should be using new features of IndexWriter like flushing
> by RAM Usage, but wanna see if there any bugs in my case. Possible I'm facing
> issue described in LUCENE-1175, but need a confirmation on that.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
