Re: Lucene shouldn't use java.io.tmpdir
On Monday 12 July 2004 09:04, Morus Walter wrote: > Lucene might work around this by creating a directory in java.io.tmpdir > setting apropriate permission (can that be done with java os > independently?) and put the lock there. But if everybody can delete your lock files, that would be a security problem. Deleting stale locks isn't a problem, but how would one decide if a lock is stale? Regards Daniel -- http://www.danielnaber.de - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Lucene shouldn't use java.io.tmpdir
Doug Cutting writes: > Armbrust, Daniel C. wrote: > > The problem I ran into the other day with the new lock location is that Person A > > had started an index, ran into problems, erased the index and asked me to look at > > it. I tried to rebuild the index (in the same place on a Solaris machine) and > > found out that A) - her locks still existed, B) - I didn't have a clue where it > > put the locks on the Solaris machine (since no full path was given with the error > > - has this been fixed?) and C) - I didn't have permission to remove her locks. > > I think these problems have been fixed. When an index is created, all > old locks are first removed. And when a lock cannot be obtained, it's > full pathname is printed. Can you replicate this with 1.4-final? > Hmm. If user A creates a lock in /tmp and lucene crashes leaving the lock, user B won't be able to remove the lock (unless B is root) since /tmp usually has permissions drwxrwxrwt 12 root root 8192 Jul 12 08:50 tmp/ were the 't' means that normal users may delete only their own files (at least on linux and IIRC solaris). Or did I miss something? Lucene might work around this by creating a directory in java.io.tmpdir setting apropriate permission (can that be done with java os independently?) and put the lock there. Morus - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Lucene shouldn't use java.io.tmpdir
Armbrust, Daniel C. wrote: The problem I ran into the other day with the new lock location is that Person A had started an index, ran into problems, erased the index and asked me to look at it. I tried to rebuild the index (in the same place on a Solaris machine) and found out that A) - her locks still existed, B) - I didn't have a clue where it put the locks on the Solaris machine (since no full path was given with the error - has this been fixed?) and C) - I didn't have permission to remove her locks. I think these problems have been fixed. When an index is created, all old locks are first removed. And when a lock cannot be obtained, it's full pathname is printed. Can you replicate this with 1.4-final? I think the locks should go back in the index, and we should fall back or give an option to put them elsewhere for the case of the read-only index. Changing the lock location is risky. Code which writes an index would not be required to alter the lock location, but code which reads it would be. This can easily lead to uncoordinated access. So it is best if the default lock location works well in most cases. We try to use a temporary directory writable by all users, and attempt to handle situations like those you describe above. Please tell me if you continue to have problems with locking. Thanks, Doug - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Lucene shouldn't use java.io.tmpdir
On Friday 09 July 2004 16:15, Armbrust, Daniel C. wrote: > (since no full path was given with the error - has this been fixed?) and C) That's fixed in Lucene 1.4. > I think the locks should go back in the index, and we should fall back or > give an option to put them elsewhere for the case of the read-only index. There's already a Java system property that let's you specify the lock directory. Regards Daniel - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Lucene shouldn't use java.io.tmpdir
The problem I ran into the other day with the new lock location is that Person A had started an index, ran into problems, erased the index and asked me to look at it. I tried to rebuild the index (in the same place on a Solaris machine) and found out that A) - her locks still existed, B) - I didn't have a clue where it put the locks on the Solaris machine (since no full path was given with the error - has this been fixed?) and C) - I didn't have permission to remove her locks. I think the locks should go back in the index, and we should fall back or give an option to put them elsewhere for the case of the read-only index. Dan - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Lucene shouldn't use java.io.tmpdir
Doug Cutting wrote: Kevin A. Burton wrote: This is why I think it makes more sense to use our own java.io.tmpdir to be on the safe side. I think the bug is that Tomcat changes java.io.tmpdir. I thought that the point of the system property java.io.tmpdir was to have a portable name for /tmp on unix, c:\windows\tmp on Windows, etc. Tomcat breaks that. So must Lucene have its own way of finding the platform-specific temporary directory that everyone can write to? Perhaps, but it seems a shame, since Java already has a standard mechanism for this, which Tomcat abuses... I've seen this done in other places as well. I think Weblogic did/does it. I'm wondering what some of these big EJB containsers use which is why I brought this up. I'm not sure the problem is just with Tomcat. Kevin -- Please reply using PGP. http://peerfear.org/pubkey.asc NewsMonster - http://www.newsmonster.org/ Kevin A. Burton, Location - San Francisco, CA, Cell - 415.595.9965 AIM/YIM - sfburtonator, Web - http://peerfear.org/ GPG fingerprint: 5FB2 F3E2 760E 70A8 6174 D393 E84D 8D04 99F1 4412 IRC - freenode.net #infoanarchy | #p2p-hackers | #newsmonster - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Lucene shouldn't use java.io.tmpdir
Kevin A. Burton wrote: This is why I think it makes more sense to use our own java.io.tmpdir to be on the safe side. I think the bug is that Tomcat changes java.io.tmpdir. I thought that the point of the system property java.io.tmpdir was to have a portable name for /tmp on unix, c:\windows\tmp on Windows, etc. Tomcat breaks that. So must Lucene have its own way of finding the platform-specific temporary directory that everyone can write to? Perhaps, but it seems a shame, since Java already has a standard mechanism for this, which Tomcat abuses... Doug - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Lucene shouldn't use java.io.tmpdir
Otis Gospodnetic wrote: Hey Kevin, Not sure if you're aware of it, but you can specify the lock dir, so in your example, both JVMs could use the exact same lock dir, as long as you invoke the VMs with the same params. Most people won't do this or won't even understand WHY they need to do this :-/. You shouldn't be writing the same index with more than 1 IndexWriter though (not sure if this was just a bad example or a real scenario). Yes... I realize that you shouldn't use more than one IndexWriter. That was the point. The locks are to prevent this from happening. If one were to accidentally do this the locks would be in different directories and our IndexWriter would corrupt the index. This is why I think it makes more sense to use our own java.io.tmpdir to be on the safe side. -- Please reply using PGP. http://peerfear.org/pubkey.asc NewsMonster - http://www.newsmonster.org/ Kevin A. Burton, Location - San Francisco, CA, Cell - 415.595.9965 AIM/YIM - sfburtonator, Web - http://peerfear.org/ GPG fingerprint: 5FB2 F3E2 760E 70A8 6174 D393 E84D 8D04 99F1 4412 IRC - freenode.net #infoanarchy | #p2p-hackers | #newsmonster - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Lucene shouldn't use java.io.tmpdir
Hey Kevin, Not sure if you're aware of it, but you can specify the lock dir, so in your example, both JVMs could use the exact same lock dir, as long as you invoke the VMs with the same params. You shouldn't be writing the same index with more than 1 IndexWriter though (not sure if this was just a bad example or a real scenario). Otis --- "Kevin A. Burton" <[EMAIL PROTECTED]> wrote: > As per 1.3 (or was it 1.4) Lucene migrated to using java.iot.tmpdir > to > store the locks for the index. > > While under most situations this is save a lot of application servers > > change java.io.tmpdir at runtime. > > Tomcat is a good example. Within Tomcat this property is set to > TOMCAT_HOME/temp.. > > Under this situation if I were to create two IndexWriters within two > VMs > and try to write to the same index the index would get corrupted if > one > Lucene instance was within Tomcat and the other was within a standard > VM. > > I think we should consider either: > > 1. Using out own tmpdir property based on the given OS. > 2. Go back to the old mechanism of storing the locks within the index > > basedir (if it's not readonly). > > Thoughts? > > -- > > Please reply using PGP. > > http://peerfear.org/pubkey.asc > > NewsMonster - http://www.newsmonster.org/ > > Kevin A. Burton, Location - San Francisco, CA, Cell - 415.595.9965 >AIM/YIM - sfburtonator, Web - http://peerfear.org/ > GPG fingerprint: 5FB2 F3E2 760E 70A8 6174 D393 E84D 8D04 99F1 4412 > IRC - freenode.net #infoanarchy | #p2p-hackers | #newsmonster > > > - > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Lucene shouldn't use java.io.tmpdir
As per 1.3 (or was it 1.4) Lucene migrated to using java.iot.tmpdir to store the locks for the index. While under most situations this is save a lot of application servers change java.io.tmpdir at runtime. Tomcat is a good example. Within Tomcat this property is set to TOMCAT_HOME/temp.. Under this situation if I were to create two IndexWriters within two VMs and try to write to the same index the index would get corrupted if one Lucene instance was within Tomcat and the other was within a standard VM. I think we should consider either: 1. Using out own tmpdir property based on the given OS. 2. Go back to the old mechanism of storing the locks within the index basedir (if it's not readonly). Thoughts? -- Please reply using PGP. http://peerfear.org/pubkey.asc NewsMonster - http://www.newsmonster.org/ Kevin A. Burton, Location - San Francisco, CA, Cell - 415.595.9965 AIM/YIM - sfburtonator, Web - http://peerfear.org/ GPG fingerprint: 5FB2 F3E2 760E 70A8 6174 D393 E84D 8D04 99F1 4412 IRC - freenode.net #infoanarchy | #p2p-hackers | #newsmonster - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
