OK, I did some digging on this ... using native locks has been
discussed before:
* In this thread (2.5 yrs ago), to get locking across machines, a
pluggable locking framework was created. This looks very clean as
it would allow, eg, single JVM instances of Lucene to use an
in-process lock w/ FSDirectory instead of filesystem based locks.
Locking is then independent of Directory implemenation in use:
http://www.gossamer-threads.com/lists/lucene/java-dev/24329?search_string=pluggable%20lock;#24329
This was then opened as an issue with nice patch:
http://issues.apache.org/jira/browse/LUCENE-305
* Also 2.5 yrs ago, this bug (separately) was opened, which looks
like a starvation issue (because there's no scheduling of lock
acquisitions), with a patch was that uses native locks:
http://issues.apache.org/jira/browse/LUCENE-307
* Then in this thread (2 yrs ago), native locks were explored as a
way to make locking over NFS work, but apparently (??) didn't
reach closure because the approach seemed to hang over NFS:
http://www.gossamer-threads.com/lists/lucene/java-dev/22464?search_string=FileLock;#22464
* There in this thread (1.5 yrs ago) it seems this was the original
driver for switching from 1.3 to 1.4 (but then didn't happen):
http://www.gossamer-threads.com/lists/lucene/java-dev/25034?search_string=FileLock;#25034
It seems like:
* Locking is a sore point now. Just googling on the lock obtain
timed out produces quite a few results.
* Most of these are probably from hard shutdowns (jvm crashed or was
killed, power lost to machine, etc) leaving lock files in the
filesystem, which native locking would automatically prevent.
* Some of these cases are probably the thread starvation case
(LUCENE-307) maybe (speculating...) from frequent interleaving of
deleting docs via IndexReader adding docs via IndexWriter. With
the right Lock class we can resolve this.
* Remote locking is a relatively common need but not supported now
(and native locking and/or pluggable locking w/ a DB or other
implementation, should be able to resolve).
I think the best course of action here is:
* Let's first try to get the patches in LUCENE-305 updated to
current HEAD sources and then committed? This can be done
independent of native locks, by having the current locking
implementation be the default Lock class for starters.
* Next, create a native locking class that extends Lock. Borrow /
copy / be inspired from the examples above (and also from Hadoop /
Jackrabbit and others). Try to resolve other issues (the
starvation issue above; using wait/notify instead of sleep /
poll). Add this to core or contrib, and at some point make it the
default Lock implementation for FSDirectory.
Does that sound right? If so, how to nudge this forward? There was
thread recently on voting for bugs to fix. I vote +1 for LUCENE-305!
I'll volunteer to update LUCENE-305 to current HEAD unless original
author (or someone else) wants to?
Mike
Yonik Seeley [EMAIL PROTECTED]
06/21/2006 11:53 AM
Please respond to
java-dev@lucene.apache.org
To
java-dev@lucene.apache.org
cc
java-user@lucene.apache.org
Subject
Re: using lucene Lock inter-jvm
On 6/21/06, Michael McCandless [EMAIL PROTECTED] wrote:
Does anyone know of any reasons not to switch Lucene's FSDirectory
locking
to the java.nio.channels.FileLock? EG, are there any performance issues
that people are aware of? It's available since Java 1.4.
Good question Michael, no reason that I know of... I think its
probably just that no one has revisited the issue since Lucene moved
to 1.4
-Yonik
http://incubator.apache.org/solr Solr, the open-source Lucene search
server
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
