[jira] [Updated] (LUCENE-6507) NativeFSLock.close() can invalidate other locks
[ https://issues.apache.org/jira/browse/LUCENE-6507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael McCandless updated LUCENE-6507: --- Attachment: LUCENE-6507-410x.patch So ... here's a 4.10.x backport patch, but it was kinda messy: lots of conflicts because we've basically already rewritten locking once in 5.x. I stuck with java.io APIs (File) instead of converting to NIO.2 apis (Path). I also back-ported AssertingLock to MockDirectoryWrapper. This patch breaks NativeFSLockFactory.clearLock: its impl relied on this when I close I nuke any other locks behavior, and I had to remove one test case that in facets module that was doing this. The API is deprecated (gone in 5.x) but still feels wrong to break it on such an old bugfix branch... Net/net this is a biggish change, and I don't think we should backport this to 4.10.x: this branch is very old now, and this change is a too risky. NativeFSLock.close() can invalidate other locks --- Key: LUCENE-6507 URL: https://issues.apache.org/jira/browse/LUCENE-6507 Project: Lucene - Core Issue Type: Bug Reporter: Simon Willnauer Priority: Blocker Fix For: 4.10.5, 5.2 Attachments: LUCENE-6507-410x.patch, LUCENE-6507.patch, LUCENE-6507.patch, LUCENE-6507.patch, LUCENE-6507.patch, LUCENE-6507.patch, LUCENE-6507.patch, LUCENE-6507.patch, LUCENE-6507.patch, LUCENE-6507.patch, LUCENE-6507.patch, LUCENE-6507.patch the lock API in Lucene is super trappy since the lock that we return form this API must first be obtained and if we can't obtain it the lock should not be closed since we might ie. close the underlying channel in the NativeLock case which releases all lock for this file on some operating systems. I think the makeLock method should try to obtain and only return a lock if we successfully obtained it. Not sure if it's possible everywhere but we should at least make the documentation clear here. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-6507) NativeFSLock.close() can invalidate other locks
[ https://issues.apache.org/jira/browse/LUCENE-6507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael McCandless updated LUCENE-6507: --- Attachment: LUCENE-6507.patch Another iteration: * Also throw exc on double obtain to HdfsLockFactory * Put back accidental test change in my last patch * Other minor cleanups I think patch is ready; that's a good catch in VerifyingLockFactory: it should NOT be trusting the LockFactory's isLocked impl... NativeFSLock.close() can invalidate other locks --- Key: LUCENE-6507 URL: https://issues.apache.org/jira/browse/LUCENE-6507 Project: Lucene - Core Issue Type: Bug Reporter: Simon Willnauer Priority: Blocker Fix For: 4.10.5, 5.2 Attachments: LUCENE-6507.patch, LUCENE-6507.patch, LUCENE-6507.patch, LUCENE-6507.patch, LUCENE-6507.patch, LUCENE-6507.patch, LUCENE-6507.patch, LUCENE-6507.patch, LUCENE-6507.patch, LUCENE-6507.patch, LUCENE-6507.patch the lock API in Lucene is super trappy since the lock that we return form this API must first be obtained and if we can't obtain it the lock should not be closed since we might ie. close the underlying channel in the NativeLock case which releases all lock for this file on some operating systems. I think the makeLock method should try to obtain and only return a lock if we successfully obtained it. Not sure if it's possible everywhere but we should at least make the documentation clear here. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-6507) NativeFSLock.close() can invalidate other locks
[ https://issues.apache.org/jira/browse/LUCENE-6507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael McCandless updated LUCENE-6507: --- Attachment: LUCENE-6507.patch New patch, fixing SingleInstanceLF to not set obtained to false if you try to obtain it twice, plus a failing test. I also had to fix/relax MockDirectoryWrapper.AssertingLock's behavior if you call .obtain twice on a single lock ... it was clearing its obtained member, but I don't think it should. NativeFSLock.close() can invalidate other locks --- Key: LUCENE-6507 URL: https://issues.apache.org/jira/browse/LUCENE-6507 Project: Lucene - Core Issue Type: Bug Reporter: Simon Willnauer Priority: Blocker Fix For: 4.10.5, 5.2 Attachments: LUCENE-6507.patch, LUCENE-6507.patch, LUCENE-6507.patch, LUCENE-6507.patch, LUCENE-6507.patch, LUCENE-6507.patch, LUCENE-6507.patch, LUCENE-6507.patch, LUCENE-6507.patch the lock API in Lucene is super trappy since the lock that we return form this API must first be obtained and if we can't obtain it the lock should not be closed since we might ie. close the underlying channel in the NativeLock case which releases all lock for this file on some operating systems. I think the makeLock method should try to obtain and only return a lock if we successfully obtained it. Not sure if it's possible everywhere but we should at least make the documentation clear here. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-6507) NativeFSLock.close() can invalidate other locks
[ https://issues.apache.org/jira/browse/LUCENE-6507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Muir updated LUCENE-6507: Attachment: LUCENE-6507.patch Updated patch removing my changes to SimpleFSLockFactory.isLocked() I didn't mean to change the semantics for this _totally unnecessary method_ (unused by lucene). Of course, no tests fail either way, and this is bogus unnecessary stuff in our locking api. Its a search engine library, not a filelocking library.. IndexWriter.isLocked needs to die, like, as fast as possible, as well as Lock.isLocked. We cant even get the basics right, i dont know why we have stupid methods like this. NativeFSLock.close() can invalidate other locks --- Key: LUCENE-6507 URL: https://issues.apache.org/jira/browse/LUCENE-6507 Project: Lucene - Core Issue Type: Bug Reporter: Simon Willnauer Priority: Blocker Fix For: 4.10.5, 5.2 Attachments: LUCENE-6507.patch, LUCENE-6507.patch, LUCENE-6507.patch, LUCENE-6507.patch, LUCENE-6507.patch, LUCENE-6507.patch, LUCENE-6507.patch, LUCENE-6507.patch the lock API in Lucene is super trappy since the lock that we return form this API must first be obtained and if we can't obtain it the lock should not be closed since we might ie. close the underlying channel in the NativeLock case which releases all lock for this file on some operating systems. I think the makeLock method should try to obtain and only return a lock if we successfully obtained it. Not sure if it's possible everywhere but we should at least make the documentation clear here. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-6507) NativeFSLock.close() can invalidate other locks
[ https://issues.apache.org/jira/browse/LUCENE-6507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael McCandless updated LUCENE-6507: --- Attachment: LUCENE-6507.patch bq. IMO we should deliver an exception if you do this. Good idea, I changed it to throw LockObtainFailedExc if you (stupidly) try to call .obtain twice on a single instance, and added test cases for the 3 core LockFactory impls (minus NoLockFactory). NativeFSLock.close() can invalidate other locks --- Key: LUCENE-6507 URL: https://issues.apache.org/jira/browse/LUCENE-6507 Project: Lucene - Core Issue Type: Bug Reporter: Simon Willnauer Priority: Blocker Fix For: 4.10.5, 5.2 Attachments: LUCENE-6507.patch, LUCENE-6507.patch, LUCENE-6507.patch, LUCENE-6507.patch, LUCENE-6507.patch, LUCENE-6507.patch, LUCENE-6507.patch, LUCENE-6507.patch, LUCENE-6507.patch, LUCENE-6507.patch the lock API in Lucene is super trappy since the lock that we return form this API must first be obtained and if we can't obtain it the lock should not be closed since we might ie. close the underlying channel in the NativeLock case which releases all lock for this file on some operating systems. I think the makeLock method should try to obtain and only return a lock if we successfully obtained it. Not sure if it's possible everywhere but we should at least make the documentation clear here. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-6507) NativeFSLock.close() can invalidate other locks
[ https://issues.apache.org/jira/browse/LUCENE-6507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Muir updated LUCENE-6507: Fix Version/s: 5.2 4.10.5 NativeFSLock.close() can invalidate other locks --- Key: LUCENE-6507 URL: https://issues.apache.org/jira/browse/LUCENE-6507 Project: Lucene - Core Issue Type: Bug Reporter: Simon Willnauer Fix For: 4.10.5, 5.2 Attachments: LUCENE-6507.patch, LUCENE-6507.patch, LUCENE-6507.patch, LUCENE-6507.patch the lock API in Lucene is super trappy since the lock that we return form this API must first be obtained and if we can't obtain it the lock should not be closed since we might ie. close the underlying channel in the NativeLock case which releases all lock for this file on some operating systems. I think the makeLock method should try to obtain and only return a lock if we successfully obtained it. Not sure if it's possible everywhere but we should at least make the documentation clear here. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-6507) NativeFSLock.close() can invalidate other locks
[ https://issues.apache.org/jira/browse/LUCENE-6507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Muir updated LUCENE-6507: Summary: NativeFSLock.close() can invalidate other locks (was: Directory#makeLock is trappy (it does not aquire lock, although its name implies) + NativeFSLock.close() has side effects) NativeFSLock.close() can invalidate other locks --- Key: LUCENE-6507 URL: https://issues.apache.org/jira/browse/LUCENE-6507 Project: Lucene - Core Issue Type: Bug Reporter: Simon Willnauer Attachments: LUCENE-6507.patch, LUCENE-6507.patch, LUCENE-6507.patch, LUCENE-6507.patch the lock API in Lucene is super trappy since the lock that we return form this API must first be obtained and if we can't obtain it the lock should not be closed since we might ie. close the underlying channel in the NativeLock case which releases all lock for this file on some operating systems. I think the makeLock method should try to obtain and only return a lock if we successfully obtained it. Not sure if it's possible everywhere but we should at least make the documentation clear here. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-6507) NativeFSLock.close() can invalidate other locks
[ https://issues.apache.org/jira/browse/LUCENE-6507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Muir updated LUCENE-6507: Priority: Blocker (was: Major) NativeFSLock.close() can invalidate other locks --- Key: LUCENE-6507 URL: https://issues.apache.org/jira/browse/LUCENE-6507 Project: Lucene - Core Issue Type: Bug Reporter: Simon Willnauer Priority: Blocker Fix For: 4.10.5, 5.2 Attachments: LUCENE-6507.patch, LUCENE-6507.patch, LUCENE-6507.patch, LUCENE-6507.patch the lock API in Lucene is super trappy since the lock that we return form this API must first be obtained and if we can't obtain it the lock should not be closed since we might ie. close the underlying channel in the NativeLock case which releases all lock for this file on some operating systems. I think the makeLock method should try to obtain and only return a lock if we successfully obtained it. Not sure if it's possible everywhere but we should at least make the documentation clear here. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-6507) NativeFSLock.close() can invalidate other locks
[ https://issues.apache.org/jira/browse/LUCENE-6507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Muir updated LUCENE-6507: Attachment: LUCENE-6507.patch Updated patch. Fixes a test bug in AssertingLock, it can't assert on close either, for the NoLockFactory case. e.g. TestIndexWriter.testNoSegmentFile use this intentionally to create more than one IW on the same dir... this should be cleaned up later, I don't understand why tests need to do that. NativeFSLock.close() can invalidate other locks --- Key: LUCENE-6507 URL: https://issues.apache.org/jira/browse/LUCENE-6507 Project: Lucene - Core Issue Type: Bug Reporter: Simon Willnauer Priority: Blocker Fix For: 4.10.5, 5.2 Attachments: LUCENE-6507.patch, LUCENE-6507.patch, LUCENE-6507.patch, LUCENE-6507.patch, LUCENE-6507.patch the lock API in Lucene is super trappy since the lock that we return form this API must first be obtained and if we can't obtain it the lock should not be closed since we might ie. close the underlying channel in the NativeLock case which releases all lock for this file on some operating systems. I think the makeLock method should try to obtain and only return a lock if we successfully obtained it. Not sure if it's possible everywhere but we should at least make the documentation clear here. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-6507) NativeFSLock.close() can invalidate other locks
[
https://issues.apache.org/jira/browse/LUCENE-6507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Muir updated LUCENE-6507:
Attachment: LUCENE-6507.patch
Updated patch, fixing SimpleFSLock.close() to be safe as well.
It passed before because tests never use SimpleFSLock. I fixed them to
randomize between Native and Simple in all tests.
This uncovers some new stuff, like this:
{noformat}
[junit4] Suite: org.apache.lucene.index.TestCrashCausesCorruptIndex
[junit4] 2 NOTE: reproduce with: ant test
-Dtestcase=TestCrashCausesCorruptIndex -Dtests.method=testCrashCorruptsIndexing
-Dtests.seed=8DBA6BF430E29A48 -Dtests.locale=no_NO_NY
-Dtests.timezone=SystemV/PST8 -Dtests.asserts=true
-Dtests.file.encoding=US-ASCII
[junit4] ERROR 1.37s |
TestCrashCausesCorruptIndex.testCrashCorruptsIndexing
[junit4] Throwable #1:
org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out:
org.apache.lucene.store.MockDirectoryWrapper$AssertingLock@17e1509f
[junit4]at
__randomizedtesting.SeedInfo.seed([8DBA6BF430E29A48:FA3DFBC572D35AA3]:0)
[junit4]at org.apache.lucene.store.Lock.obtain(Lock.java:89)
[junit4]at
org.apache.lucene.index.IndexWriter.init(IndexWriter.java:775)
[junit4]at
org.apache.lucene.index.TestCrashCausesCorruptIndex.indexAfterRestart(TestCrashCausesCorruptIndex.java:98)
[junit4]at
org.apache.lucene.index.TestCrashCausesCorruptIndex.testCrashCorruptsIndexing(TestCrashCausesCorruptIndex.java:50)
[junit4]at java.lang.Thread.run(Thread.java:745)
[junit4] 2 NOTE: leaving temporary files on disk at:
/home/rmuir/workspace/trunk-iw/lucene/build/core/test/J0/temp/lucene.index.TestCrashCausesCorruptIndex
8DBA6BF430E29A48-001
[junit4] 2 NOTE: test params are:
codec=HighCompressionCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=HIGH_COMPRESSION,
chunkSize=32683, maxDocsPerChunk=8, blockSize=6),
termVectorsFormat=CompressingTermVectorsFormat(compressionMode=HIGH_COMPRESSION,
chunkSize=32683, blockSize=6)),
sim=RandomSimilarityProvider(queryNorm=false,coord=crazy): {text=DFR GL1},
locale=no_NO_NY, timezone=SystemV/PST8
[junit4] 2 NOTE: Linux 3.13.0-49-generic amd64/Oracle Corporation
1.8.0_45 (64-bit)/cpus=8,threads=1,free=208117928,total=253231104
[junit4] 2 NOTE: All tests run in this JVM: [TestCrashCausesCorruptIndex]
[junit4] Completed [1/1] in 1.55s, 1 test, 1 error FAILURES!
{noformat}
I have not looked into that one yet.
NativeFSLock.close() can invalidate other locks
---
Key: LUCENE-6507
URL: https://issues.apache.org/jira/browse/LUCENE-6507
Project: Lucene - Core
Issue Type: Bug
Reporter: Simon Willnauer
Priority: Blocker
Fix For: 4.10.5, 5.2
Attachments: LUCENE-6507.patch, LUCENE-6507.patch, LUCENE-6507.patch,
LUCENE-6507.patch, LUCENE-6507.patch, LUCENE-6507.patch
the lock API in Lucene is super trappy since the lock that we return form
this API must first be obtained and if we can't obtain it the lock should not
be closed since we might ie. close the underlying channel in the NativeLock
case which releases all lock for this file on some operating systems. I think
the makeLock method should try to obtain and only return a lock if we
successfully obtained it. Not sure if it's possible everywhere but we should
at least make the documentation clear here.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-6507) NativeFSLock.close() can invalidate other locks
[ https://issues.apache.org/jira/browse/LUCENE-6507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Muir updated LUCENE-6507: Attachment: LUCENE-6507.patch Here is a simpler approach for this bugfix. Don't randomize lockfactory in all tests, just in simon's new one. Defer randomization across all tests to https://issues.apache.org/jira/browse/LUCENE-6509, where we can prevent test bugs. I don't want to totally destabilize tests in this way right now. NativeFSLock.close() can invalidate other locks --- Key: LUCENE-6507 URL: https://issues.apache.org/jira/browse/LUCENE-6507 Project: Lucene - Core Issue Type: Bug Reporter: Simon Willnauer Priority: Blocker Fix For: 4.10.5, 5.2 Attachments: LUCENE-6507.patch, LUCENE-6507.patch, LUCENE-6507.patch, LUCENE-6507.patch, LUCENE-6507.patch, LUCENE-6507.patch, LUCENE-6507.patch the lock API in Lucene is super trappy since the lock that we return form this API must first be obtained and if we can't obtain it the lock should not be closed since we might ie. close the underlying channel in the NativeLock case which releases all lock for this file on some operating systems. I think the makeLock method should try to obtain and only return a lock if we successfully obtained it. Not sure if it's possible everywhere but we should at least make the documentation clear here. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
