I've seen a couple of other people reporting similar issues when switching to the latest 1.3 tree. However, usually the problems come from lack of synchronization of multiple threads doing reads and writes against the index. But it's definetely possible that the compound index format makes this issue more visible or that it has its own bugs. In my testing and in the initial testing of the Lucene dev team when we were working on the compound index format we didn't see any such issues, but of course that could just mean that we didn't look in the right place.
While I'm fairly certain our app fulfills all the locking requirements for lucene, it's possible that we've missed something as well. I'll go through the code again to see if there is any place I allow multiple threads access improperly.
Anyway, it would be extremely helpful if you could produce a self-contained program that demonstrates this behavior. I know that this can be hard to impossible with concurrency-related issues, but still.
I'll see what I can do.
Also, what OS is this running under? This issue is related to filesystem locking behavior which is different between Unix and Windows.
Red Hat Linux release 9 (Shrike) under their 2.4.20-24.9smp kernel and Sun's 1.4.2_03 SDK. We did disable lucene's file based locking mechanism since our app is not setup to allow multiple different JVM's to write to a single index (one index per JVM) and we've had no end of issues with the file based locking. This was done by just setting FSDirectory.DISABLE_LOCKS to true;
Does this happen every time or does it happen only under heavy load, etc.?
What is the system doing when this occurs (i.e. indexing and searching at the same time or just searching etc)?
Unfortunately, I have no answers for any of these questions. The best I can say is that the load on the server is usually very low (load average of 0.03) and the box itself has fast scsi based disks (IO is much faster for instance than my local development machine). The only other pertinent pience of info that I can think of is that this instance of the application was setup to update the index with new and updated content once every 10 minutes.
Regards,
Bruce Ritchie
Bruce Ritchie wrote:
We've started using the new compound indexing functionality available in lucene 1.3 final however we've been seeing an issue that I haven't quite resolved and was hoping that people on this list might be able to lend help on.
The stack trace of the issue is:
2004.01.13 18:15 [com.jivesoftware.forum.database.DbQuery.executeQuery(DbQuery.java:402)]
java.io.FileNotFoundException: /var/search/BfUfO9s1/_qb5.fnm (No such file or directory)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:204)
at org.apache.lucene.store.FSInputStream$Descriptor.<init>(FSDirectory.java:392)
at org.apache.lucene.store.FSInputStream.<init>(FSDirectory.java:421)
at org.apache.lucene.store.FSDirectory.openFile(FSDirectory.java:294)
at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:80)
at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:138)
at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:120)
at org.apache.lucene.index.IndexReader$1.doBody(IndexReader.java:122)
at org.apache.lucene.store.Lock$With.run(Lock.java:148)
at org.apache.lucene.index.IndexReader.open(IndexReader.java:111)
at com.jivesoftware.forum.database.DbSearchManager.getSearcher(DbSearchManager.java:897)
...
The error seems obvious enough from a quick glance of the above code - the file was been deleted. However, since we are using compound indexing there should be a /var/search/BfUfO9s1/_qb5.cfs file, however that file does not exist. The only files in the search directory are:
-bash-2.05b$ ls -l total 6908 -rw-r--r-- 1 resin js 4 Jan 13 18:14 deletable -rw-r--r-- 1 resin js 7029055 Jan 13 10:03 _qau.cfs -rw-r--r-- 1 resin js 9 Jan 13 18:14 _qb5.del -rw-r--r-- 1 resin js 16060 Jan 13 18:14 _qb7.cfs -rw-r--r-- 1 resin js 34 Jan 13 18:14 segments
From what I can tell it seems to me that the segments are somehow getting out of sync with what is actually on the filesystem somehow.
Any suggestions are where to start looking for the cause of this bug?
Regards,
Bruce Ritchie
--------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
-- AOL - bruceritchie101 ICQ - 9929791 MSN - [EMAIL PROTECTED] http://www.jivesoftware.com/
smime.p7s
Description: S/MIME Cryptographic Signature
