Locking problems with NFS
Dear Lucene users, we are experiencing some difficulties in using Lucene with a NFS filesystem. Basically, locking seems not to work properly, since it appears that attempted concurring writing on the index (from different VMs) are not blocked, and this often causes the index to be corrupted. I found a reference to this topic in the archives: http://issues.apache.org/eyebrowse/[EMAIL PROTECTED] he.orgmsgId=784959 [Doug Cutting] So long as all access is read-only, there should not be a problem. Keep in mind however that lock files are known to not work correctly over NFS. Does anybody have a solution for this problem? Does anyone have some suggestions about the implementation of some alternative approach for locking, in order to make Lucene compatible with NFS? We are using 1.4rc2. Thanks in advance, Francesco - Francesco Bellomi Use truth to show illusion, and illusion to show truth. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Locking problems with NFS
Doug, thanks for your suggestions! I will implement your solution and make it public. We already use a 1.4 JVM, so we have no problems with that. The only problem is that, as lucene 1.4rc2, FSDirectory is 'final'; I would go for the subclassing solution if you can make it not final; otherwise, if someone has good arguments for the single 'fat' FSDirectory solution, just let me know it. In fact, a third architectural approach would be to define an API for pluggable lock implementations: IMHO that would be more robust to changes in the Directory interface and implementations. Thanks again! Francesco Doug Cutting [EMAIL PROTECTED] wrote: Francesco Bellomi wrote: we are experiencing some difficulties in using Lucene with a NFS filesystem. Basically, locking seems not to work properly, since it appears that attempted concurring writing on the index (from different VMs) are not blocked, and this often causes the index to be corrupted. Does anybody have a solution for this problem? Does anyone have some suggestions about the implementation of some alternative approach for locking, in order to make Lucene compatible with NFS? I just found the following article, which might point the way: http://www.spinnaker.de/linux/nfs-locking.html FSDirectory.lock currently implements what this article calls dotlocking. One can implement fcntl() locking using nio, but this requires a 1.4 JVM, which Lucene does not (yet). http://java.sun.com/j2se/1.4.2/docs/api/java/nio/channels/FileChannel.html#t ryLock() A couple of alternatives: 1. One could implement a 1.4-only subclass of FSDirectory in the Sandbox that implements dotlocking plus fcntl() locking. 2. We could add this directly into FSDirectory so that it will only compile in 1.4+ JDKs, but that it will run in earlier JDKs by only performing fcntl() locking when in a 1.4+ JVM. Some folks would complain. Thoughts? Doug - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - Francesco Bellomi Use truth to show illusion, and illusion to show truth. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Vector - LinkedList for performance reasons...
I agree that synchronization in Vector is a waste of time if it isn't required, but I'm not sure if LinkedList is a better (faster) choice than ArrayList. I think only a profiler could tell. Francesco Kevin A. Burton [EMAIL PROTECTED] wrote: I'm looking at a lot of the code in Lucene... I assume Vector is used for legacy reasons. In an upcoming version I think it might make sense to migrate to using a LinkedList... since Vector has to do an array copy when it's exhausted. It's also synchronized which kind of sucks... I'm seeing this being used in a lot of tight loops so things might be sped up a bit by using Collections ... Kevin -- Please reply using PGP: http://peerfear.org/pubkey.asc NewsMonster - http://www.newsmonster.org/ Dean in 2004! - http://blog.deanforamerica.com/ Kevin A. Burton, Location - San Francisco, CA, Cell - 415.595.9965 AIM/YIM - sfburtonator, Web - http://peerfear.org/ GPG fingerprint: 4D20 40A0 C734 307E C7B4 DCAA 0303 3AC5 BD9D 7C4D IRC - freenode.net #infoanarchy | #p2p-hackers | #newsmonster - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - Francesco Bellomi Use truth to show illusion, and illusion to show truth. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Lucene version 1.3.
I'm using 1.3RC2 in production right now. Francesco Scott Smith [EMAIL PROTECTED] wrote: If you had to be production in January, would you be using 1.3RC2 or 1.2? - Francesco Bellomi Use truth to show illusion, and illusion to show truth. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: java.nio.channels.FileLock
Hi, Ronald Muller [EMAIL PROTECTED] wrote: What is the advantage of using a FileLock object instead of the way Lucene does it? (I do not see it) FileLock makes use of the underlying locking mechanism of the OS that is more robust, whereas I think current Lucene implementation has some problems if the JVM dies unexpectedly, because some locks may be left behind, and subsequent Lucene processes may find the index locked even if the process that originally created the lock no more exists. Also, I believe (but I'm not sure) that FileLock could be more efficient, because it does not require the creation of a file in the file system. Also note an mportant limitation: File locks are held on behalf of the entire Java virtual machine. They are not suitable for controlling access to a file by multiple threads within the same virtual machine. I think a reliable locking mechanism should combine local synchronization mechanisms for thread-safety in the same VM, with file locking for concurrency safety between multiple VM. Also, I noticed that 1.3 RC2 puts locks in java.io.tmpdir, which initialized as the default system directory for temporary files. I think this default can cause some troubles, if two processes running in different machines are using the same index in a shared file system, because the two machines have different (local) default temp dir. Francesco - Francesco Bellomi Use truth to show illusion, and illusion to show truth. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
