lucene-??? files: are they ever deleted after? was: locking problems
Hi, From earlier postings I understand that when lucene opens an IndexWriter or executes a number of specific methods on an IndexReader that a lock is created in .../tomcat/temp. My question is. Are these files ever deleted after something has gone wrong? For instance, when the server is restarted or something similar? If not, is there any way to figure out where the lock came from? e.g a timestamp as part of the filename? BTW: thanks for the earlier responses, I have a working solution now.. Aad -- Aad Nales [EMAIL PROTECTED], +31-(0)6 54 207 340 - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: lucene-??? files: are they ever deleted after? was: locking problems
Aad, note that .../topic/temp is not always correct. If you use Lucene outside of Tomcat, the lock files will be stored elsewhere. If IndexWriter/Readers are properly handled, they will clean up lock files after themselves. If something breaks (e.g. IOException because you are out of disk space or because something kills the process), lock files will remain and you will need to clean them up yourself. Checking whether these lock files are current or stale is something you'll have to do in your application, I'm afraid, possibly using their time stamps to determine this. Otis --- Aad Nales [EMAIL PROTECTED] wrote: Hi, From earlier postings I understand that when lucene opens an IndexWriter or executes a number of specific methods on an IndexReader that a lock is created in .../tomcat/temp. My question is. Are these files ever deleted after something has gone wrong? For instance, when the server is restarted or something similar? If not, is there any way to figure out where the lock came from? e.g a timestamp as part of the filename? BTW: thanks for the earlier responses, I have a working solution now.. Aad -- Aad Nales [EMAIL PROTECTED], +31-(0)6 54 207 340 - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: lucene-??? files: are they ever deleted after? was: locking problems
Once again, Thanks a bundle. I have added a check at server startup, cheers, Aad -Original Message- From: Otis Gospodnetic [mailto:[EMAIL PROTECTED] Sent: Monday, 11 October, 2004 15:09 To: Lucene Users List Subject: Re: lucene-??? files: are they ever deleted after? was: locking problems Aad, note that .../topic/temp is not always correct. If you use Lucene outside of Tomcat, the lock files will be stored elsewhere. If IndexWriter/Readers are properly handled, they will clean up lock files after themselves. If something breaks (e.g. IOException because you are out of disk space or because something kills the process), lock files will remain and you will need to clean them up yourself. Checking whether these lock files are current or stale is something you'll have to do in your application, I'm afraid, possibly using their time stamps to determine this. Otis --- Aad Nales [EMAIL PROTECTED] wrote: Hi, From earlier postings I understand that when lucene opens an IndexWriter or executes a number of specific methods on an IndexReader that a lock is created in .../tomcat/temp. My question is. Are these files ever deleted after something has gone wrong? For instance, when the server is restarted or something similar? If not, is there any way to figure out where the lock came from? e.g a timestamp as part of the filename? BTW: thanks for the earlier responses, I have a working solution now.. Aad -- Aad Nales [EMAIL PROTECTED], +31-(0)6 54 207 340 - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
locking problems
Based on discussions in this group I figure that I should 'cache' IndexSearchers and IndexReaders, which i did. I have build an IndexSearcherPool and an IndexReaderPool. Both seem to work fine (although I am still testing). However, whenever I use these I can not create an IndexWriter. The thread fails and generates a timeout on org.apache.lucene.store.Lock.obtain (1.3.1) in line 97. Can somebody help me to figure out with what actions these locks are obtained? I have been reading all faq's on the subject but failed to understand the following: 1. can I have one or multiple searchers open when I open a writer? 2. can I have one or multiple readers open when I open a writer? And if not. I am writing an application that does regular updates on the index what kind of strategy could you advise? Should I use ResourcePooling at all? TIA, Aad Nales -- Aad Nales [EMAIL PROTECTED], +31-(0)6 54 207 340 - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: locking problems
Aad Nales wrote: 1. can I have one or multiple searchers open when I open a writer? 2. can I have one or multiple readers open when I open a writer? Yes, with one caveat: if you've called the IndexReader methods delete(), undelete() or setNorm() then you may not open an IndexWriter until you've closed that IndexReader instance. In general, only a single object may modify an index at once, but many may access it simultaneously in a read-only manner, including while it is modified. Indexes are modified by either an IndexWriter or by the IndexReader methods delete(), undelete() and setNorm(). Typically an application which modifies and searches simultaneously should keep the following open: 1. A single IndexReader instance used for all searches, perhaps opened via an IndexSearcher. Periodically, as the index changes, this is discarded, and replaced with a new instance. 2. Either: a. An IndexReader to delete documents. b. An IndexWriter to add documents; or So an updating thread might open (2a), delete old documents, close it, then open (2b) add new documents, perhaps optimize, then close. At this point, when the index has been updated (1) can be discarded and replaced with a new instance. Typically the old instance of (1) is not explicitly closed, rather the garbage collector closes it when the last thread searching it completes. Doug - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Locking problems with NFS
Dear Lucene users, we are experiencing some difficulties in using Lucene with a NFS filesystem. Basically, locking seems not to work properly, since it appears that attempted concurring writing on the index (from different VMs) are not blocked, and this often causes the index to be corrupted. I found a reference to this topic in the archives: http://issues.apache.org/eyebrowse/[EMAIL PROTECTED] he.orgmsgId=784959 [Doug Cutting] So long as all access is read-only, there should not be a problem. Keep in mind however that lock files are known to not work correctly over NFS. Does anybody have a solution for this problem? Does anyone have some suggestions about the implementation of some alternative approach for locking, in order to make Lucene compatible with NFS? We are using 1.4rc2. Thanks in advance, Francesco - Francesco Bellomi Use truth to show illusion, and illusion to show truth. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Locking problems with NFS
Francesco Bellomi wrote: we are experiencing some difficulties in using Lucene with a NFS filesystem. Basically, locking seems not to work properly, since it appears that attempted concurring writing on the index (from different VMs) are not blocked, and this often causes the index to be corrupted. Does anybody have a solution for this problem? Does anyone have some suggestions about the implementation of some alternative approach for locking, in order to make Lucene compatible with NFS? I just found the following article, which might point the way: http://www.spinnaker.de/linux/nfs-locking.html FSDirectory.lock currently implements what this article calls dotlocking. One can implement fcntl() locking using nio, but this requires a 1.4 JVM, which Lucene does not (yet). http://java.sun.com/j2se/1.4.2/docs/api/java/nio/channels/FileChannel.html#tryLock() A couple of alternatives: 1. One could implement a 1.4-only subclass of FSDirectory in the Sandbox that implements dotlocking plus fcntl() locking. 2. We could add this directly into FSDirectory so that it will only compile in 1.4+ JDKs, but that it will run in earlier JDKs by only performing fcntl() locking when in a 1.4+ JVM. Some folks would complain. Thoughts? Doug - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Locking problems with NFS
Doug, thanks for your suggestions! I will implement your solution and make it public. We already use a 1.4 JVM, so we have no problems with that. The only problem is that, as lucene 1.4rc2, FSDirectory is 'final'; I would go for the subclassing solution if you can make it not final; otherwise, if someone has good arguments for the single 'fat' FSDirectory solution, just let me know it. In fact, a third architectural approach would be to define an API for pluggable lock implementations: IMHO that would be more robust to changes in the Directory interface and implementations. Thanks again! Francesco Doug Cutting [EMAIL PROTECTED] wrote: Francesco Bellomi wrote: we are experiencing some difficulties in using Lucene with a NFS filesystem. Basically, locking seems not to work properly, since it appears that attempted concurring writing on the index (from different VMs) are not blocked, and this often causes the index to be corrupted. Does anybody have a solution for this problem? Does anyone have some suggestions about the implementation of some alternative approach for locking, in order to make Lucene compatible with NFS? I just found the following article, which might point the way: http://www.spinnaker.de/linux/nfs-locking.html FSDirectory.lock currently implements what this article calls dotlocking. One can implement fcntl() locking using nio, but this requires a 1.4 JVM, which Lucene does not (yet). http://java.sun.com/j2se/1.4.2/docs/api/java/nio/channels/FileChannel.html#t ryLock() A couple of alternatives: 1. One could implement a 1.4-only subclass of FSDirectory in the Sandbox that implements dotlocking plus fcntl() locking. 2. We could add this directly into FSDirectory so that it will only compile in 1.4+ JDKs, but that it will run in earlier JDKs by only performing fcntl() locking when in a 1.4+ JVM. Some folks would complain. Thoughts? Doug - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - Francesco Bellomi Use truth to show illusion, and illusion to show truth. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Locking problems with NFS
Francesco Bellomi wrote: The only problem is that, as lucene 1.4rc2, FSDirectory is 'final'. Please submit a patch to lucene-dev to make FSDirectory non-final. In fact, a third architectural approach would be to define an API for pluggable lock implementations: IMHO that would be more robust to changes in the Directory interface and implementations. If you have a proposal, submit a patch to lucene-dev to make locking pluggable. FYI, I just ran across another NFS-safe locking approach: #In NFS, the O_CREAT|O_EXCL isn't guaranteed to be atomic. #So we create a temp file that is probably unique in space #and time ($folder.lock.$time.$pid.$host). #Then we use link to create the real lock file. Since link #is atomic across nfs, this works. This is from: http://search.cpan.org/src/MARKOV/Mail-Box-2.053/lib/Mail/Box/Locker/NFS.pm Unfortunately links are not supported by Java's io APIs, so I think the dotlock+fcntl() is probably best for Lucene. Doug - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
