Re: using lucene Lock inter-jvm
: I think the best course of action here is: : : * Let's first try to get the patches in LUCENE-305 updated to : current HEAD sources and then committed? This can be done : independent of native locks, by having the current locking : implementation be the default Lock class for starters. : : * Next, create a native locking class that extends Lock. Borrow / : copy / be inspired from the examples above (and also from Hadoop / : Jackrabbit and others). Try to resolve other issues (the : starvation issue above; using wait/notify instead of sleep / : poll). Add this to core or contrib, and at some point make it the : default Lock implementation for FSDirectory. Your plan sounds good to me, but i'm not really a Threading/Locking expert. : Does that sound right? If so, how to nudge this forward? There was : thread recently on voting for bugs to fix. I vote +1 for LUCENE-305! Just to clarify: Jira has a built in voting mechanism. Anyone can create a Jira account, load up an issue, and then click the "Vote for it" link in the left nav. This automagically causes it to show up on the "Popular Issues" page for the project... http://issues.apache.org/jira/browse/LUCENE?report=com.atlassian.jira.plugin.system.project:popularissues-panel -Hoss - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: using lucene Lock inter-jvm
OK, I did some digging on this ... using native locks has been discussed before: * In this thread (2.5 yrs ago), to get locking across machines, a pluggable locking framework was created. This looks very clean as it would allow, eg, "single JVM" instances of Lucene to use an in-process lock w/ FSDirectory instead of filesystem based locks. Locking is then independent of Directory implemenation in use: http://www.gossamer-threads.com/lists/lucene/java-dev/24329?search_string=pluggable%20lock;#24329 This was then opened as an issue with nice patch: http://issues.apache.org/jira/browse/LUCENE-305 * Also 2.5 yrs ago, this bug (separately) was opened, which looks like a starvation issue (because there's no "scheduling" of lock acquisitions), with a patch was that uses native locks: http://issues.apache.org/jira/browse/LUCENE-307 * Then in this thread (2 yrs ago), native locks were explored as a way to make locking over NFS work, but apparently (??) didn't reach closure because the approach seemed to hang over NFS: http://www.gossamer-threads.com/lists/lucene/java-dev/22464?search_string=FileLock;#22464 * There in this thread (1.5 yrs ago) it seems this was the original driver for switching from 1.3 to 1.4 (but then didn't happen): http://www.gossamer-threads.com/lists/lucene/java-dev/25034?search_string=FileLock;#25034 It seems like: * Locking is a sore point now. Just googling on the "lock obtain timed out" produces quite a few results. * Most of these are probably from hard shutdowns (jvm crashed or was killed, power lost to machine, etc) leaving lock files in the filesystem, which native locking would automatically prevent. * Some of these cases are probably the thread starvation case (LUCENE-307) maybe (speculating...) from frequent interleaving of deleting docs via IndexReader & adding docs via IndexWriter. With the right Lock class we can resolve this. * Remote locking is a relatively common need but not supported now (and native locking and/or pluggable locking w/ a DB or other implementation, should be able to resolve). I think the best course of action here is: * Let's first try to get the patches in LUCENE-305 updated to current HEAD sources and then committed? This can be done independent of native locks, by having the current locking implementation be the default Lock class for starters. * Next, create a native locking class that extends Lock. Borrow / copy / be inspired from the examples above (and also from Hadoop / Jackrabbit and others). Try to resolve other issues (the starvation issue above; using wait/notify instead of sleep / poll). Add this to core or contrib, and at some point make it the default Lock implementation for FSDirectory. Does that sound right? If so, how to nudge this forward? There was thread recently on voting for bugs to fix. I vote +1 for LUCENE-305! I'll volunteer to update LUCENE-305 to current HEAD unless original author (or someone else) wants to? Mike "Yonik Seeley" <[EMAIL PROTECTED]> 06/21/2006 11:53 AM Please respond to java-dev@lucene.apache.org To java-dev@lucene.apache.org cc java-user@lucene.apache.org Subject Re: using lucene Lock inter-jvm On 6/21/06, Michael McCandless <[EMAIL PROTECTED]> wrote: > Does anyone know of any reasons not to switch Lucene's FSDirectory locking > to the java.nio.channels.FileLock? EG, are there any performance issues > that people are aware of? It's available since Java 1.4. Good question Michael, no reason that I know of... I think its probably just that no one has revisited the issue since Lucene moved to 1.4 -Yonik http://incubator.apache.org/solr Solr, the open-source Lucene search server - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: using lucene Lock inter-jvm
On 6/21/06, Michael McCandless <[EMAIL PROTECTED]> wrote: Does anyone know of any reasons not to switch Lucene's FSDirectory locking to the java.nio.channels.FileLock? EG, are there any performance issues that people are aware of? It's available since Java 1.4. Good question Michael, no reason that I know of... I think its probably just that no one has revisited the issue since Lucene moved to 1.4 -Yonik http://incubator.apache.org/solr Solr, the open-source Lucene search server - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: using lucene Lock inter-jvm
OK, the root cause was found here (passing "true" as 2nd arg to
FSDirectory.getDirectory(...)) -- that was causing the directory (and lock
file) to be removed.
But I think the questions below of java.io.File.createNewFile vs
java.nio.channels.FileLock for locking are still important / relevant.
Mike
Michael McCandless/Lexington/[EMAIL PROTECTED]
06/21/2006 11:00 AM
Please respond to
java-user@lucene.apache.org
To
java-user@lucene.apache.org
cc
java-dev@lucene.apache.org
Subject
Re: using lucene Lock inter-jvm
CC'ing java-dev to talk about details of locking.
I can reproduce this on Windows XP, Java 1.4.2: two separate JVMs are able
to get the Lock at the same time.
The code looks correct to me.
Strangely, if I make a separate standalone test that just uses
java.io.File.createNewFile directly, it works correctly (this is what
FSDirectory.makeLock uses). If I hardwire the lock filename inside
FSDirectory.makeLock, it also fails, unless I use a very short filename,
then it seems to work. Something intermittent is going on.
Then in looking at the docs for java.io.File.createNewfile():
http://java.sun.com/j2se/1.5.0/docs/api/java/io/File.html#createNewFile()
There is this spooky comment:
Note: this method should not be used for file-locking, as the
resulting protocol cannot be made to work reliably. The FileLock facility
should be used instead.
Finally, it looks like Hadoop's LocalFileSystem class is already using
FileLock's. One benefit of FileLocks is if JVM crashes, the OS should
handle removing the lock correctly. I know this has been an issue in the
past with "commit lock timeout" errors due to the lock file remaining in
the filesystem, with the current approach.
Does anyone know of any reasons not to switch Lucene's FSDirectory locking
to the java.nio.channels.FileLock? EG, are there any performance issues
that people are aware of? It's available since Java 1.4.
Mike
jm <[EMAIL PROTECTED]>
06/20/2006 01:19 PM
Please respond to
java-user@lucene.apache.org
To
java-user@lucene.apache.org
cc
Subject
using lucene Lock inter-jvm
Hi,
I am trying to peruse lucene's Lock for my own purposes, I need to
lock several java processes and I thought I could reuse the Lock
stuff. I understand lucene locks work across jvm.
But I cannot make it work. I tried to reproduce my problem in a small
class:
public class SysLock {
private static final Logger logger = Logger.getLogger(SysLock.class);
private int id;
public SysLock(int i) {
id = i;
}
//public static void main(String[] args) throws Exception {
//System.setProperty("org.apache.lucene.lockDir",
"C:\\temp\\todel");
//SysLock l1 = new SysLock(1);
//SysLock l2 = new SysLock(2);
//
//TransferThread t = l1.new TransferThread(l1);
//t.start();
//TransferThread t2 = l2.new TransferThread(l2);
//t2.start();
//
//logger.info("Finished.");
//}
public static void main(String[] args) throws Exception {
System.setProperty("org.apache.lucene.lockDir",
"C:\\temp\\todel");
SysLock l1 = new SysLock(new Date().getSeconds());
TransferThread t = l1.new TransferThread(l1);
t.start();
logger.info("Finished.");
}
private void forever() throws IOException {
FSDirectory directory = FSDirectory.getDirectory("C:\\temp\\a",
true);
try {
new Lock.With(directory.makeLock("COMMIT_LOCK_NAME"),
COMMIT_LOCK_TIMEOUT) {
public Object doBody() throws IOException {
while (true) {
System.out.println("i'm " + id);
try {
Thread.sleep(2000);
}
catch (InterruptedException e) {
e.printStackTrace();
}
}
}
}.run();
}
catch (Exception e) {
System.out.println(id + " could not get lock");
}
}
class TransferThread extends Thread {
public TransferThread(SysLock sl) {
this.sl = sl;
}
public void run() {
try {
sl.forever();
}
catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
private SysLock sl;
}
}
When I run the main() that is commented (that is, the lock works with
two threads in the same jvm) it works ok, the second TransferThread
cannot get the lock.
But when I run the uncommented main() twice, both processes adquire a
lock, even if only one lock file exists in the lockdir. Something I am
missing probably
Re: using lucene Lock inter-jvm
CC'ing java-dev to talk about details of locking.
I can reproduce this on Windows XP, Java 1.4.2: two separate JVMs are able
to get the Lock at the same time.
The code looks correct to me.
Strangely, if I make a separate standalone test that just uses
java.io.File.createNewFile directly, it works correctly (this is what
FSDirectory.makeLock uses). If I hardwire the lock filename inside
FSDirectory.makeLock, it also fails, unless I use a very short filename,
then it seems to work. Something intermittent is going on.
Then in looking at the docs for java.io.File.createNewfile():
http://java.sun.com/j2se/1.5.0/docs/api/java/io/File.html#createNewFile()
There is this spooky comment:
Note: this method should not be used for file-locking, as the
resulting protocol cannot be made to work reliably. The FileLock facility
should be used instead.
Finally, it looks like Hadoop's LocalFileSystem class is already using
FileLock's. One benefit of FileLocks is if JVM crashes, the OS should
handle removing the lock correctly. I know this has been an issue in the
past with "commit lock timeout" errors due to the lock file remaining in
the filesystem, with the current approach.
Does anyone know of any reasons not to switch Lucene's FSDirectory locking
to the java.nio.channels.FileLock? EG, are there any performance issues
that people are aware of? It's available since Java 1.4.
Mike
jm <[EMAIL PROTECTED]>
06/20/2006 01:19 PM
Please respond to
java-user@lucene.apache.org
To
java-user@lucene.apache.org
cc
Subject
using lucene Lock inter-jvm
Hi,
I am trying to peruse lucene's Lock for my own purposes, I need to
lock several java processes and I thought I could reuse the Lock
stuff. I understand lucene locks work across jvm.
But I cannot make it work. I tried to reproduce my problem in a small
class:
public class SysLock {
private static final Logger logger = Logger.getLogger(SysLock.class);
private int id;
public SysLock(int i) {
id = i;
}
//public static void main(String[] args) throws Exception {
//System.setProperty("org.apache.lucene.lockDir",
"C:\\temp\\todel");
//SysLock l1 = new SysLock(1);
//SysLock l2 = new SysLock(2);
//
//TransferThread t = l1.new TransferThread(l1);
//t.start();
//TransferThread t2 = l2.new TransferThread(l2);
//t2.start();
//
//logger.info("Finished.");
//}
public static void main(String[] args) throws Exception {
System.setProperty("org.apache.lucene.lockDir",
"C:\\temp\\todel");
SysLock l1 = new SysLock(new Date().getSeconds());
TransferThread t = l1.new TransferThread(l1);
t.start();
logger.info("Finished.");
}
private void forever() throws IOException {
FSDirectory directory = FSDirectory.getDirectory("C:\\temp\\a",
true);
try {
new Lock.With(directory.makeLock("COMMIT_LOCK_NAME"),
COMMIT_LOCK_TIMEOUT) {
public Object doBody() throws IOException {
while (true) {
System.out.println("i'm " + id);
try {
Thread.sleep(2000);
}
catch (InterruptedException e) {
e.printStackTrace();
}
}
}
}.run();
}
catch (Exception e) {
System.out.println(id + " could not get lock");
}
}
class TransferThread extends Thread {
public TransferThread(SysLock sl) {
this.sl = sl;
}
public void run() {
try {
sl.forever();
}
catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
private SysLock sl;
}
}
When I run the main() that is commented (that is, the lock works with
two threads in the same jvm) it works ok, the second TransferThread
cannot get the lock.
But when I run the uncommented main() twice, both processes adquire a
lock, even if only one lock file exists in the lockdir. Something I am
missing probably
Many thanks
javi
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
