[jira] [Commented] (LUCENE-3147) MockDirectoryWrapper should track open file handles of IndexOutput too
[
https://issues.apache.org/jira/browse/LUCENE-3147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13040774#comment-13040774
]
Michael McCandless commented on LUCENE-3147:
{quote}
bq. But: why do we ever abort a full flush...? It looks like it happens
because one of the DWPTs hits an exc while flushing...?
I think this is the way to go here. if one DWPT can not flush why would be
continue flushing the remaining ones. You will need to call rollback anyway or
reindex all documents since the last commit. Its unnecessary to flush those
DWPT since we are already lost the failed segment.
{quote}
In theory, if disk space freed up, the other DWPTs could flush? Ie the failure
of one DWPT to flush should only have to above that one DWPT?
But I agree, I think it's OK if we leave it this way for now... you're right
that the app is going to have to do its own recovery anyway. Just feels like
we shouldn't be discarding healthy DWPTs... but I think it's minor.
MockDirectoryWrapper should track open file handles of IndexOutput too
--
Key: LUCENE-3147
URL: https://issues.apache.org/jira/browse/LUCENE-3147
Project: Lucene - Java
Issue Type: Test
Components: general/test
Reporter: Shai Erera
Assignee: Shai Erera
Priority: Minor
Fix For: 3.2, 4.0
Attachments: LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch,
LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch,
LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch,
LUCENE-3147.patch, LUCENE-3147.patch
MockDirectoryWrapper currently tracks open file handles of IndexInput only.
Therefore IO files that are not closed do not fail our tests, which can then
lead to test directories fail to delete on Windows. We should make sure all
open files are tracked and if they are left open, fail the test. I'll attach
a patch shortly.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3147) MockDirectoryWrapper should track open file handles of IndexOutput too
[ https://issues.apache.org/jira/browse/LUCENE-3147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13040794#comment-13040794 ] Michael McCandless commented on LUCENE-3147: Nice catch in closeMergeReaders -- the changes there look great. Much less code dup now :) You need to svn add MockLockFWWrapper.java. MockDirectoryWrapper should track open file handles of IndexOutput too -- Key: LUCENE-3147 URL: https://issues.apache.org/jira/browse/LUCENE-3147 Project: Lucene - Java Issue Type: Test Components: general/test Reporter: Shai Erera Assignee: Shai Erera Priority: Minor Fix For: 3.2, 4.0 Attachments: LUCENE-3147-3x.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch MockDirectoryWrapper currently tracks open file handles of IndexInput only. Therefore IO files that are not closed do not fail our tests, which can then lead to test directories fail to delete on Windows. We should make sure all open files are tracked and if they are left open, fail the test. I'll attach a patch shortly. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3147) MockDirectoryWrapper should track open file handles of IndexOutput too
[ https://issues.apache.org/jira/browse/LUCENE-3147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13040805#comment-13040805 ] Shai Erera commented on LUCENE-3147: Also, why do we call checkpoint() only if suppressWarnings=true and not only if anyChanges=true? Why do we care about suppressWarnings or not? From what I read, the two are equivalent in that context, b/c drop=!suppressWarnings which means anyChanges=false whenever drop=true ... It's weird about MLFWrapper - when I 'svn stat' it appears w/ A + and not ?. Yet svn diff MLFW yields no diff ... If I svn ci, I see that it's included in the list of files ... strange. MockDirectoryWrapper should track open file handles of IndexOutput too -- Key: LUCENE-3147 URL: https://issues.apache.org/jira/browse/LUCENE-3147 Project: Lucene - Java Issue Type: Test Components: general/test Reporter: Shai Erera Assignee: Shai Erera Priority: Minor Fix For: 3.2, 4.0 Attachments: LUCENE-3147-3x.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch MockDirectoryWrapper currently tracks open file handles of IndexInput only. Therefore IO files that are not closed do not fail our tests, which can then lead to test directories fail to delete on Windows. We should make sure all open files are tracked and if they are left open, fail the test. I'll attach a patch shortly. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3147) MockDirectoryWrapper should track open file handles of IndexOutput too
[ https://issues.apache.org/jira/browse/LUCENE-3147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13040809#comment-13040809 ] Michael McCandless commented on LUCENE-3147: bq. Also, why do we call checkpoint() only if suppressWarnings=true and not only if anyChanges=true? Good question! My guess is the checkpoint() can be removed? We call checkpoint up above after closeMergeReaders unconditionally... bq. It's weird about MLFWrapper - when I 'svn stat' it appears w/ A + and not ?. Yet svn diff MLFW yields no diff ... If I svn ci, I see that it's included in the list of files ... strange. Ahh yes OK. This is from svn merge I think? Like svn diff won't show it... maybe there's some option to svn diff to show these added due to merge changes? MockDirectoryWrapper should track open file handles of IndexOutput too -- Key: LUCENE-3147 URL: https://issues.apache.org/jira/browse/LUCENE-3147 Project: Lucene - Java Issue Type: Test Components: general/test Reporter: Shai Erera Assignee: Shai Erera Priority: Minor Fix For: 3.2, 4.0 Attachments: LUCENE-3147-3x.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch MockDirectoryWrapper currently tracks open file handles of IndexInput only. Therefore IO files that are not closed do not fail our tests, which can then lead to test directories fail to delete on Windows. We should make sure all open files are tracked and if they are left open, fail the test. I'll attach a patch shortly. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3147) MockDirectoryWrapper should track open file handles of IndexOutput too
[ https://issues.apache.org/jira/browse/LUCENE-3147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13040819#comment-13040819 ] Shai Erera commented on LUCENE-3147: ok I removed the call to checkpoint(), and do not track anyChanges anymore. I tried svn diff --force, but still nothing. And yes, it's an svn merge that I did. I guess we don't post patches after backport very often, so not a biggy. I'll commit shortly. MockDirectoryWrapper should track open file handles of IndexOutput too -- Key: LUCENE-3147 URL: https://issues.apache.org/jira/browse/LUCENE-3147 Project: Lucene - Java Issue Type: Test Components: general/test Reporter: Shai Erera Assignee: Shai Erera Priority: Minor Fix For: 3.2, 4.0 Attachments: LUCENE-3147-3x.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch MockDirectoryWrapper currently tracks open file handles of IndexInput only. Therefore IO files that are not closed do not fail our tests, which can then lead to test directories fail to delete on Windows. We should make sure all open files are tracked and if they are left open, fail the test. I'll attach a patch shortly. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3147) MockDirectoryWrapper should track open file handles of IndexOutput too
[
https://issues.apache.org/jira/browse/LUCENE-3147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13040823#comment-13040823
]
Shai Erera commented on LUCENE-3147:
Hmm ... I think checkpoint() is needed after all. After I committed to trunk, I
ran tests again on 3x before commit, and TestIndexWriter.testNoWaitClose failed:
{noformat}
[junit] Testsuite: org.apache.lucene.index.TestIndexWriter
[junit] Testcase: testNoWaitClose(org.apache.lucene.index.TestIndexWriter):
FAILED
[junit] IndexFileDeleter doesn't know about file _a6_1.del
[junit] junit.framework.AssertionFailedError: IndexFileDeleter doesn't know
about file _a6_1.del
[junit] at
org.apache.lucene.util.LuceneTestCase$LuceneTestCaseRunner.runChild(LuceneTestCase.java:1207)
[junit] at
org.apache.lucene.util.LuceneTestCase$LuceneTestCaseRunner.runChild(LuceneTestCase.java:1125)
[junit] at
org.apache.lucene.index.IndexWriter.filesExist(IndexWriter.java:4421)
[junit] at
org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:4469)
[junit] at
org.apache.lucene.index.IndexWriter.prepareCommit(IndexWriter.java:3354)
[junit] at
org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3425)
[junit] at
org.apache.lucene.index.IndexWriter.closeInternal(IndexWriter.java:1870)
[junit] at
org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1813)
[junit] at
org.apache.lucene.index.TestIndexWriter.testNoWaitClose(TestIndexWriter.java:1515)
[junit]
[junit]
[junit] Tests run: 68, Failures: 1, Errors: 0, Time elapsed: 24.884 sec
[junit]
[junit] - Standard Error -
[junit] NOTE: reproduce with: ant test -Dtestcase=TestIndexWriter
-Dtestmethod=testNoWaitClose
-Dtests.seed=8584244356995777356:6228535864161192995
[junit] NOTE: test params are: locale=da_DK, timezone=America/Menominee
[junit] NOTE: all tests run in this JVM:
[junit] [TestClassicAnalyzer, TestPerFieldAnalzyerWrapper,
TestCharTermAttributeImpl, TestNumberTools, TestConcurrentMergeScheduler,
TestFilterIndexReader, TestIndexWriter]
[junit] NOTE: Windows 7 6.1 build 7600 amd64/IBM Corporation 1.6.0
(64-bit)/cpus=2,threads=3,free=1805128,total=34830336
[junit] - ---
[junit] TEST org.apache.lucene.index.TestIndexWriter FAILED
{noformat}
Reproduced it on 3x, not on trunk. closeMergeReaders is called from
commitMerge, which indeed calls checkpoint() afterwards unconditionally, but
also from mergeMiddle, in its finally block if (!success). And the exception
suggests that IFD does not know about a certain file, and checkpoint() calls
deleter.refresh. So I'll revert my change to trunk and call checkpoint() if
anyChanges + suppressExceptions. I don't want to change previous behavior.
MockDirectoryWrapper should track open file handles of IndexOutput too
--
Key: LUCENE-3147
URL: https://issues.apache.org/jira/browse/LUCENE-3147
Project: Lucene - Java
Issue Type: Test
Components: general/test
Reporter: Shai Erera
Assignee: Shai Erera
Priority: Minor
Fix For: 3.2, 4.0
Attachments: LUCENE-3147-3x.patch, LUCENE-3147.patch,
LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch,
LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch,
LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch
MockDirectoryWrapper currently tracks open file handles of IndexInput only.
Therefore IO files that are not closed do not fail our tests, which can then
lead to test directories fail to delete on Windows. We should make sure all
open files are tracked and if they are left open, fail the test. I'll attach
a patch shortly.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3147) MockDirectoryWrapper should track open file handles of IndexOutput too
[ https://issues.apache.org/jira/browse/LUCENE-3147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13040828#comment-13040828 ] Michael McCandless commented on LUCENE-3147: Ahh OK it is in fact necessary. It's because closeMergeReaders is also called from the finally clause when there is an exception (hence the suppressExcs=true), and checkpoint is NOT then called from that location, so we do need to call it inside here. Alternatively we could always call it inside closeMergeReaders, but then not above (inside commitMerge()). Or we can leave it be for now :) MockDirectoryWrapper should track open file handles of IndexOutput too -- Key: LUCENE-3147 URL: https://issues.apache.org/jira/browse/LUCENE-3147 Project: Lucene - Java Issue Type: Test Components: general/test Reporter: Shai Erera Assignee: Shai Erera Priority: Minor Fix For: 3.2, 4.0 Attachments: LUCENE-3147-3x.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch MockDirectoryWrapper currently tracks open file handles of IndexInput only. Therefore IO files that are not closed do not fail our tests, which can then lead to test directories fail to delete on Windows. We should make sure all open files are tracked and if they are left open, fail the test. I'll attach a patch shortly. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3147) MockDirectoryWrapper should track open file handles of IndexOutput too
[ https://issues.apache.org/jira/browse/LUCENE-3147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13040831#comment-13040831 ] Shai Erera commented on LUCENE-3147: I prefer that it's always called outside of closeMergeReaders. That's practically what happens already (checkpoint() is either called outside or inside in confusing code), so why not make it explicit and remove the confusing code? MockDirectoryWrapper should track open file handles of IndexOutput too -- Key: LUCENE-3147 URL: https://issues.apache.org/jira/browse/LUCENE-3147 Project: Lucene - Java Issue Type: Test Components: general/test Reporter: Shai Erera Assignee: Shai Erera Priority: Minor Fix For: 3.2, 4.0 Attachments: LUCENE-3147-3x.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch MockDirectoryWrapper currently tracks open file handles of IndexInput only. Therefore IO files that are not closed do not fail our tests, which can then lead to test directories fail to delete on Windows. We should make sure all open files are tracked and if they are left open, fail the test. I'll attach a patch shortly. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3147) MockDirectoryWrapper should track open file handles of IndexOutput too
[ https://issues.apache.org/jira/browse/LUCENE-3147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13040833#comment-13040833 ] Michael McCandless commented on LUCENE-3147: Well... the problem is, mergeMiddle isn't sync'd? But, we could wrap the call to closeMergeReaders in sync block, and put the checkpoint() inside that? MockDirectoryWrapper should track open file handles of IndexOutput too -- Key: LUCENE-3147 URL: https://issues.apache.org/jira/browse/LUCENE-3147 Project: Lucene - Java Issue Type: Test Components: general/test Reporter: Shai Erera Assignee: Shai Erera Priority: Minor Fix For: 3.2, 4.0 Attachments: LUCENE-3147-3x.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch MockDirectoryWrapper currently tracks open file handles of IndexInput only. Therefore IO files that are not closed do not fail our tests, which can then lead to test directories fail to delete on Windows. We should make sure all open files are tracked and if they are left open, fail the test. I'll attach a patch shortly. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3147) MockDirectoryWrapper should track open file handles of IndexOutput too
[ https://issues.apache.org/jira/browse/LUCENE-3147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13040618#comment-13040618 ] Michael McCandless commented on LUCENE-3147: But even after fixing that, I still see *.tiv files left open... I'll dig. MockDirectoryWrapper should track open file handles of IndexOutput too -- Key: LUCENE-3147 URL: https://issues.apache.org/jira/browse/LUCENE-3147 Project: Lucene - Java Issue Type: Test Components: general/test Reporter: Shai Erera Assignee: Shai Erera Priority: Minor Fix For: 3.2, 4.0 Attachments: LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch MockDirectoryWrapper currently tracks open file handles of IndexInput only. Therefore IO files that are not closed do not fail our tests, which can then lead to test directories fail to delete on Windows. We should make sure all open files are tracked and if they are left open, fail the test. I'll attach a patch shortly. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3147) MockDirectoryWrapper should track open file handles of IndexOutput too
[ https://issues.apache.org/jira/browse/LUCENE-3147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13040617#comment-13040617 ] Michael McCandless commented on LUCENE-3147: OK one problem is ThrottledIndexOutput's close method. Right now it sleeps and then closes, but this must be done in a try/finally because the sleep could throw ThreadInterrupt. MockDirectoryWrapper should track open file handles of IndexOutput too -- Key: LUCENE-3147 URL: https://issues.apache.org/jira/browse/LUCENE-3147 Project: Lucene - Java Issue Type: Test Components: general/test Reporter: Shai Erera Assignee: Shai Erera Priority: Minor Fix For: 3.2, 4.0 Attachments: LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch MockDirectoryWrapper currently tracks open file handles of IndexInput only. Therefore IO files that are not closed do not fail our tests, which can then lead to test directories fail to delete on Windows. We should make sure all open files are tracked and if they are left open, fail the test. I'll attach a patch shortly. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3147) MockDirectoryWrapper should track open file handles of IndexOutput too
[ https://issues.apache.org/jira/browse/LUCENE-3147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13040623#comment-13040623 ] Robert Muir commented on LUCENE-3147: - adding the try-finally to vargapwriter gets you past that, but now there is a new one: ant test -Dtestcase=TestDocumentWriter -Dtestmethod=testAddDocument -Dtests.seed=6194875654219552577:595815136781814876 -Dtests.codec=MockRandom MockDirectoryWrapper should track open file handles of IndexOutput too -- Key: LUCENE-3147 URL: https://issues.apache.org/jira/browse/LUCENE-3147 Project: Lucene - Java Issue Type: Test Components: general/test Reporter: Shai Erera Assignee: Shai Erera Priority: Minor Fix For: 3.2, 4.0 Attachments: LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch MockDirectoryWrapper currently tracks open file handles of IndexInput only. Therefore IO files that are not closed do not fail our tests, which can then lead to test directories fail to delete on Windows. We should make sure all open files are tracked and if they are left open, fail the test. I'll attach a patch shortly. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3147) MockDirectoryWrapper should track open file handles of IndexOutput too
[ https://issues.apache.org/jira/browse/LUCENE-3147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13040624#comment-13040624 ] Michael McCandless commented on LUCENE-3147: Also, VariableGapTermsIndexWriter.close needs to use try/finally. MockDirectoryWrapper should track open file handles of IndexOutput too -- Key: LUCENE-3147 URL: https://issues.apache.org/jira/browse/LUCENE-3147 Project: Lucene - Java Issue Type: Test Components: general/test Reporter: Shai Erera Assignee: Shai Erera Priority: Minor Fix For: 3.2, 4.0 Attachments: LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch MockDirectoryWrapper currently tracks open file handles of IndexInput only. Therefore IO files that are not closed do not fail our tests, which can then lead to test directories fail to delete on Windows. We should make sure all open files are tracked and if they are left open, fail the test. I'll attach a patch shortly. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3147) MockDirectoryWrapper should track open file handles of IndexOutput too
[ https://issues.apache.org/jira/browse/LUCENE-3147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13040629#comment-13040629 ] Robert Muir commented on LUCENE-3147: - ant test -Dtestcase=TestIndexWriterDelete -Dtestmethod=testUpdatesOnDiskFull -Dtests.seed=-505532335460902096:-6988290248406972759 MockDirectoryWrapper should track open file handles of IndexOutput too -- Key: LUCENE-3147 URL: https://issues.apache.org/jira/browse/LUCENE-3147 Project: Lucene - Java Issue Type: Test Components: general/test Reporter: Shai Erera Assignee: Shai Erera Priority: Minor Fix For: 3.2, 4.0 Attachments: LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch MockDirectoryWrapper currently tracks open file handles of IndexInput only. Therefore IO files that are not closed do not fail our tests, which can then lead to test directories fail to delete on Windows. We should make sure all open files are tracked and if they are left open, fail the test. I'll attach a patch shortly. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3147) MockDirectoryWrapper should track open file handles of IndexOutput too
[ https://issues.apache.org/jira/browse/LUCENE-3147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13040642#comment-13040642 ] Robert Muir commented on LUCENE-3147: - ant test -Dtestcase=TestIndexWriterWithThreads -Dtestmethod=testImmediateDiskFullWithThreads -Dtests.seed=3427070915583133981:-5436728766623164583 MockDirectoryWrapper should track open file handles of IndexOutput too -- Key: LUCENE-3147 URL: https://issues.apache.org/jira/browse/LUCENE-3147 Project: Lucene - Java Issue Type: Test Components: general/test Reporter: Shai Erera Assignee: Shai Erera Priority: Minor Fix For: 3.2, 4.0 Attachments: LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch MockDirectoryWrapper currently tracks open file handles of IndexInput only. Therefore IO files that are not closed do not fail our tests, which can then lead to test directories fail to delete on Windows. We should make sure all open files are tracked and if they are left open, fail the test. I'll attach a patch shortly. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3147) MockDirectoryWrapper should track open file handles of IndexOutput too
[ https://issues.apache.org/jira/browse/LUCENE-3147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13040645#comment-13040645 ] Robert Muir commented on LUCENE-3147: - thanks simon, i'm gonna take your patch and make a throwaway branch now, since we are all trying to merge various fixes. MockDirectoryWrapper should track open file handles of IndexOutput too -- Key: LUCENE-3147 URL: https://issues.apache.org/jira/browse/LUCENE-3147 Project: Lucene - Java Issue Type: Test Components: general/test Reporter: Shai Erera Assignee: Shai Erera Priority: Minor Fix For: 3.2, 4.0 Attachments: LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch MockDirectoryWrapper currently tracks open file handles of IndexInput only. Therefore IO files that are not closed do not fail our tests, which can then lead to test directories fail to delete on Windows. We should make sure all open files are tracked and if they are left open, fail the test. I'll attach a patch shortly. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3147) MockDirectoryWrapper should track open file handles of IndexOutput too
[ https://issues.apache.org/jira/browse/LUCENE-3147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13040648#comment-13040648 ] Robert Muir commented on LUCENE-3147: - https://svn.apache.org/repos/asf/lucene/dev/branches/leaky3147 MockDirectoryWrapper should track open file handles of IndexOutput too -- Key: LUCENE-3147 URL: https://issues.apache.org/jira/browse/LUCENE-3147 Project: Lucene - Java Issue Type: Test Components: general/test Reporter: Shai Erera Assignee: Shai Erera Priority: Minor Fix For: 3.2, 4.0 Attachments: LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch MockDirectoryWrapper currently tracks open file handles of IndexInput only. Therefore IO files that are not closed do not fail our tests, which can then lead to test directories fail to delete on Windows. We should make sure all open files are tracked and if they are left open, fail the test. I'll attach a patch shortly. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3147) MockDirectoryWrapper should track open file handles of IndexOutput too
[ https://issues.apache.org/jira/browse/LUCENE-3147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13040650#comment-13040650 ] Robert Muir commented on LUCENE-3147: - probably the same fail as before, preflexrw again: ant test -Dtestcase=TestConcurrentMergeScheduler -Dtestmethod=testFlushExceptions -Dtests.seed=-1522558944672603540:-5713175873482685091 MockDirectoryWrapper should track open file handles of IndexOutput too -- Key: LUCENE-3147 URL: https://issues.apache.org/jira/browse/LUCENE-3147 Project: Lucene - Java Issue Type: Test Components: general/test Reporter: Shai Erera Assignee: Shai Erera Priority: Minor Fix For: 3.2, 4.0 Attachments: LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch MockDirectoryWrapper currently tracks open file handles of IndexInput only. Therefore IO files that are not closed do not fail our tests, which can then lead to test directories fail to delete on Windows. We should make sure all open files are tracked and if they are left open, fail the test. I'll attach a patch shortly. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3147) MockDirectoryWrapper should track open file handles of IndexOutput too
[ https://issues.apache.org/jira/browse/LUCENE-3147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13040651#comment-13040651 ] Michael McCandless commented on LUCENE-3147: Aha! Thanks Simon :) IOUtils had to be fixed to accept ? extends Closeable, not just Closeable. MockDirectoryWrapper should track open file handles of IndexOutput too -- Key: LUCENE-3147 URL: https://issues.apache.org/jira/browse/LUCENE-3147 Project: Lucene - Java Issue Type: Test Components: general/test Reporter: Shai Erera Assignee: Shai Erera Priority: Minor Fix For: 3.2, 4.0 Attachments: LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch MockDirectoryWrapper currently tracks open file handles of IndexInput only. Therefore IO files that are not closed do not fail our tests, which can then lead to test directories fail to delete on Windows. We should make sure all open files are tracked and if they are left open, fail the test. I'll attach a patch shortly. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3147) MockDirectoryWrapper should track open file handles of IndexOutput too
[ https://issues.apache.org/jira/browse/LUCENE-3147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13040653#comment-13040653 ] Robert Muir commented on LUCENE-3147: - after svn up, i still got another preflex fail: ant test -Dtestcase=TestIndexWriterExceptions -Dtestmethod=testOptimizeExceptions -Dtests.seed=-4125144081554890699:-9213228330712396498 MockDirectoryWrapper should track open file handles of IndexOutput too -- Key: LUCENE-3147 URL: https://issues.apache.org/jira/browse/LUCENE-3147 Project: Lucene - Java Issue Type: Test Components: general/test Reporter: Shai Erera Assignee: Shai Erera Priority: Minor Fix For: 3.2, 4.0 Attachments: LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch MockDirectoryWrapper currently tracks open file handles of IndexInput only. Therefore IO files that are not closed do not fail our tests, which can then lead to test directories fail to delete on Windows. We should make sure all open files are tracked and if they are left open, fail the test. I'll attach a patch shortly. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3147) MockDirectoryWrapper should track open file handles of IndexOutput too
[ https://issues.apache.org/jira/browse/LUCENE-3147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13040669#comment-13040669 ] Robert Muir commented on LUCENE-3147: - I put a fail() in the branch in case a test cannot delete its directory. running tests in a loop on windows and linux: so far things look pretty good, except I hit a situation where windows can't delete a tempdir in TestIndexReader... inside the directory is two simpletext segments and write lock... MockDirectoryWrapper should track open file handles of IndexOutput too -- Key: LUCENE-3147 URL: https://issues.apache.org/jira/browse/LUCENE-3147 Project: Lucene - Java Issue Type: Test Components: general/test Reporter: Shai Erera Assignee: Shai Erera Priority: Minor Fix For: 3.2, 4.0 Attachments: LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch MockDirectoryWrapper currently tracks open file handles of IndexInput only. Therefore IO files that are not closed do not fail our tests, which can then lead to test directories fail to delete on Windows. We should make sure all open files are tracked and if they are left open, fail the test. I'll attach a patch shortly. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3147) MockDirectoryWrapper should track open file handles of IndexOutput too
[ https://issues.apache.org/jira/browse/LUCENE-3147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13040707#comment-13040707 ] Robert Muir commented on LUCENE-3147: - OK, I think all the windows-only errors are resolved now from LUCENE-3152. MockDirectoryWrapper should track open file handles of IndexOutput too -- Key: LUCENE-3147 URL: https://issues.apache.org/jira/browse/LUCENE-3147 Project: Lucene - Java Issue Type: Test Components: general/test Reporter: Shai Erera Assignee: Shai Erera Priority: Minor Fix For: 3.2, 4.0 Attachments: LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch MockDirectoryWrapper currently tracks open file handles of IndexInput only. Therefore IO files that are not closed do not fail our tests, which can then lead to test directories fail to delete on Windows. We should make sure all open files are tracked and if they are left open, fail the test. I'll attach a patch shortly. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3147) MockDirectoryWrapper should track open file handles of IndexOutput too
[
https://issues.apache.org/jira/browse/LUCENE-3147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13040708#comment-13040708
]
Michael McCandless commented on LUCENE-3147:
I'm still hitting an intermittent fail:
{noformat}
NOTE: reproduce with: ant test -Dtestcase=TestIndexWriterWithThreads
-Dtestmethod=testIOExceptionDuringWriteSegmentWithThreads
-Dtests.seed=-317321914391759:573268695851991855
java.lang.RuntimeException: MockDirectoryWrapper: cannot close: there are still
open files: {_h.fdt=1, _h.tvd=1, _h.tvf=1, _h.fdx=1, _h.tvx=1}
at
org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:454)
at
org.apache.lucene.index.TestIndexWriterWithThreads._testMultipleThreadsFailure(TestIndexWriterWithThreads.java:279)
at
org.apache.lucene.index.TestIndexWriterWithThreads.testIOExceptionDuringWriteSegmentWithThreads(TestIndexWriterWithThreads.java:407)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:48)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
at
org.apache.lucene.util.LuceneTestCase$LuceneTestCaseRunner.runChild(LuceneTestCase.java:1334)
at
org.apache.lucene.util.LuceneTestCase$LuceneTestCaseRunner.runChild(LuceneTestCase.java:1252)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
at org.junit.runners.Suite.runChild(Suite.java:128)
at org.junit.runners.Suite.runChild(Suite.java:24)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
at org.junit.runner.JUnitCore.run(JUnitCore.java:157)
at org.junit.runner.JUnitCore.run(JUnitCore.java:136)
at org.junit.runner.JUnitCore.run(JUnitCore.java:117)
at org.junit.runner.JUnitCore.runMain(JUnitCore.java:98)
at org.junit.runner.JUnitCore.runMainAndExit(JUnitCore.java:53)
at org.junit.runner.JUnitCore.main(JUnitCore.java:45)
Caused by: java.lang.RuntimeException: unclosed IndexOutput: _h.tvf
at
org.apache.lucene.store.MockDirectoryWrapper.addFileHandle(MockDirectoryWrapper.java:393)
at
org.apache.lucene.store.MockDirectoryWrapper.createOutput(MockDirectoryWrapper.java:369)
at
org.apache.lucene.index.TermVectorsTermsWriter.initTermVectorsWriter(TermVectorsTermsWriter.java:98)
at
org.apache.lucene.index.TermVectorsTermsWriter.finishDocument(TermVectorsTermsWriter.java:123)
at org.apache.lucene.index.TermsHash.finishDocument(TermsHash.java:138)
at
org.apache.lucene.index.DocInverter.finishDocument(DocInverter.java:95)
at
org.apache.lucene.index.DocFieldProcessor.finishDocument(DocFieldProcessor.java:293)
at
org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:246)
at
org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:368)
at
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1469)
at
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1441)
at
[jira] [Commented] (LUCENE-3147) MockDirectoryWrapper should track open file handles of IndexOutput too
[ https://issues.apache.org/jira/browse/LUCENE-3147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13040727#comment-13040727 ] Simon Willnauer commented on LUCENE-3147: - bq. I'm still hitting an intermittent fail: I fixed the last failure mike reported on the branch. If we abort a flushAll and a excp. is thrown in the abort method we were not closing the remaining dwpt in the queues. bq. But: why do we ever abort a full flush...? It looks like it happens because one of the DWPTs hits an exc while flushing...? I think this is the way to go here. if one DWPT can not flush why would be continue flushing the remaining ones. You will need to call rollback anyway or reindex all documents since the last commit. Its unnecessary to flush those DWPT since we are already lost the failed segment. MockDirectoryWrapper should track open file handles of IndexOutput too -- Key: LUCENE-3147 URL: https://issues.apache.org/jira/browse/LUCENE-3147 Project: Lucene - Java Issue Type: Test Components: general/test Reporter: Shai Erera Assignee: Shai Erera Priority: Minor Fix For: 3.2, 4.0 Attachments: LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch MockDirectoryWrapper currently tracks open file handles of IndexInput only. Therefore IO files that are not closed do not fail our tests, which can then lead to test directories fail to delete on Windows. We should make sure all open files are tracked and if they are left open, fail the test. I'll attach a patch shortly. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3147) MockDirectoryWrapper should track open file handles of IndexOutput too
[ https://issues.apache.org/jira/browse/LUCENE-3147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13040749#comment-13040749 ] Shai Erera commented on LUCENE-3147: Committed revision 1128830 to trunk. I will now backport to 3x. MockDirectoryWrapper should track open file handles of IndexOutput too -- Key: LUCENE-3147 URL: https://issues.apache.org/jira/browse/LUCENE-3147 Project: Lucene - Java Issue Type: Test Components: general/test Reporter: Shai Erera Assignee: Shai Erera Priority: Minor Fix For: 3.2, 4.0 Attachments: LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch MockDirectoryWrapper currently tracks open file handles of IndexInput only. Therefore IO files that are not closed do not fail our tests, which can then lead to test directories fail to delete on Windows. We should make sure all open files are tracked and if they are left open, fail the test. I'll attach a patch shortly. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3147) MockDirectoryWrapper should track open file handles of IndexOutput too
[
https://issues.apache.org/jira/browse/LUCENE-3147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13040083#comment-13040083
]
Simon Willnauer commented on LUCENE-3147:
-
bq. Short summary - we're leaking handles . Good news - the changes to
MockDirWrapper catches all of them !
its awesome that you tracked it down!
some minor things:
for readability:
{code}
if (segnOutput != null) segnOutput.close();
// should be
if (segnOutput != null) {
segnOutput.close();
}
{code}
I think you should also do that in other places. Those one line if statements
are hard to read and could lead to errors if somebody changes the code later.
In FixedGapTermsIndexWriter you do:
{code}
} finally {
if (success) {
out.close();
} else {
IOUtils.closeSafelyNoException(out);
}
}
{code}
I wonder if we should rather add a boolean to IOUtils.closeSafely(boolean
suppressException ,Closable... objs) to make situations like that simply a
one-liner that way you could do IOUtils.closeSafely(!success, out)
After all I think you found lots of places where we leak file handles :)
nice
MockDirectoryWrapper should track open file handles of IndexOutput too
--
Key: LUCENE-3147
URL: https://issues.apache.org/jira/browse/LUCENE-3147
Project: Lucene - Java
Issue Type: Test
Components: general/test
Reporter: Shai Erera
Assignee: Shai Erera
Priority: Minor
Fix For: 3.2, 4.0
Attachments: LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch
MockDirectoryWrapper currently tracks open file handles of IndexInput only.
Therefore IO files that are not closed do not fail our tests, which can then
lead to test directories fail to delete on Windows. We should make sure all
open files are tracked and if they are left open, fail the test. I'll attach
a patch shortly.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3147) MockDirectoryWrapper should track open file handles of IndexOutput too
[ https://issues.apache.org/jira/browse/LUCENE-3147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13040189#comment-13040189 ] Shai Erera commented on LUCENE-3147: bq. In BlockTermsWriter, the success boolean is never set to true! Good catch. Fixed ! bq. for readability: Sure, I'll change bq. I wonder if we should rather add a boolean to IOUtils.closeSafely I didn't want to do that since it's public API, and I intend to backport these changes to 3x. But I can add another closeSafely variant as you suggest, and have the current one call it w/ false? Also, I think that closeSafely isn't really safe :). If RuntimeException is thrown, the code will blow away, not closing all Closeables. We need to fix that as well. (I'm not sure, but I think some Faliure objects throw RuntimeException during close() ... MockDirectoryWrapper should track open file handles of IndexOutput too -- Key: LUCENE-3147 URL: https://issues.apache.org/jira/browse/LUCENE-3147 Project: Lucene - Java Issue Type: Test Components: general/test Reporter: Shai Erera Assignee: Shai Erera Priority: Minor Fix For: 3.2, 4.0 Attachments: LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch MockDirectoryWrapper currently tracks open file handles of IndexInput only. Therefore IO files that are not closed do not fail our tests, which can then lead to test directories fail to delete on Windows. We should make sure all open files are tracked and if they are left open, fail the test. I'll attach a patch shortly. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3147) MockDirectoryWrapper should track open file handles of IndexOutput too
[ https://issues.apache.org/jira/browse/LUCENE-3147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13040295#comment-13040295 ] Uwe Schindler commented on LUCENE-3147: --- Can we also add a closeSafely others taking IterableCloseable instead of varargs/array? When you fix more leaks, there may be more places where collections may be used, so it would be a nice addition. MockDirectoryWrapper should track open file handles of IndexOutput too -- Key: LUCENE-3147 URL: https://issues.apache.org/jira/browse/LUCENE-3147 Project: Lucene - Java Issue Type: Test Components: general/test Reporter: Shai Erera Assignee: Shai Erera Priority: Minor Fix For: 3.2, 4.0 Attachments: LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch MockDirectoryWrapper currently tracks open file handles of IndexInput only. Therefore IO files that are not closed do not fail our tests, which can then lead to test directories fail to delete on Windows. We should make sure all open files are tracked and if they are left open, fail the test. I'll attach a patch shortly. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3147) MockDirectoryWrapper should track open file handles of IndexOutput too
[ https://issues.apache.org/jira/browse/LUCENE-3147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13040348#comment-13040348 ] Simon Willnauer commented on LUCENE-3147: - bq. Can we also add a closeSafely others taking IterableCloseable instead of varargs/array? When you fix more leaks, there may be more places where collections may be used, so it would be a nice addition. +1 bq. Simon, I will fix IOUtils as you suggest, just noticed it's @internal. I'll do that after tests stop failing at least once on my machine cool thanks, calling it with false from the existing API is what I had in mind. thanks! MockDirectoryWrapper should track open file handles of IndexOutput too -- Key: LUCENE-3147 URL: https://issues.apache.org/jira/browse/LUCENE-3147 Project: Lucene - Java Issue Type: Test Components: general/test Reporter: Shai Erera Assignee: Shai Erera Priority: Minor Fix For: 3.2, 4.0 Attachments: LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch MockDirectoryWrapper currently tracks open file handles of IndexInput only. Therefore IO files that are not closed do not fail our tests, which can then lead to test directories fail to delete on Windows. We should make sure all open files are tracked and if they are left open, fail the test. I'll attach a patch shortly. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3147) MockDirectoryWrapper should track open file handles of IndexOutput too
[ https://issues.apache.org/jira/browse/LUCENE-3147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13040550#comment-13040550 ] Shai Erera commented on LUCENE-3147: Another thing to clarify, everyone can reproduce these bugs now, even on Linux. MockDirectoryWrapper should track open file handles of IndexOutput too -- Key: LUCENE-3147 URL: https://issues.apache.org/jira/browse/LUCENE-3147 Project: Lucene - Java Issue Type: Test Components: general/test Reporter: Shai Erera Assignee: Shai Erera Priority: Minor Fix For: 3.2, 4.0 Attachments: LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch MockDirectoryWrapper currently tracks open file handles of IndexInput only. Therefore IO files that are not closed do not fail our tests, which can then lead to test directories fail to delete on Windows. We should make sure all open files are tracked and if they are left open, fail the test. I'll attach a patch shortly. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3147) MockDirectoryWrapper should track open file handles of IndexOutput too
[
https://issues.apache.org/jira/browse/LUCENE-3147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13039658#comment-13039658
]
Shai Erera commented on LUCENE-3147:
Here's another one:
{noformat}
NOTE: reproduce with: ant test -Dtestcase=TestIndexWriterExceptions
-Dtestmethod=testExceptionsDuringCommit
-Dtests.seed=-7460760853320570890:8937985421698447215
NOTE: reproduce with: ant test -Dtestcase=TestIndexWriterExceptions
-Dtestmethod=testOptimizeExceptions
-Dtests.seed=-7460760853320570890:8014438941838172914
NOTE: test params are: codec=PreFlex, locale=ar_IQ, timezone=SystemV/HST10
NOTE: all tests run in this JVM:
[TestIndexWriterExceptions]
NOTE: Windows 7 6.1 build 7600 amd64/IBM Corporation 1.6.0
(64-bit)/cpus=2,threads=4,free=8136368,total=39485440
{noformat}
This one actually has a FIXME which I think after we resolve why the segments_1
is not closed, we can reinstate the assert too.
MockDirectoryWrapper should track open file handles of IndexOutput too
--
Key: LUCENE-3147
URL: https://issues.apache.org/jira/browse/LUCENE-3147
Project: Lucene - Java
Issue Type: Test
Components: general/test
Reporter: Shai Erera
Assignee: Shai Erera
Priority: Minor
Fix For: 3.2, 4.0
Attachments: LUCENE-3147.patch
MockDirectoryWrapper currently tracks open file handles of IndexInput only.
Therefore IO files that are not closed do not fail our tests, which can then
lead to test directories fail to delete on Windows. We should make sure all
open files are tracked and if they are left open, fail the test. I'll attach
a patch shortly.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3147) MockDirectoryWrapper should track open file handles of IndexOutput too
[ https://issues.apache.org/jira/browse/LUCENE-3147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13039697#comment-13039697 ] Simon Willnauer commented on LUCENE-3147: - Shai I have a couple of comments * first good catch! :) * reenable the FIXME is awesome * in DefaultSegmentInfosWriter you might want to change the try / catch blocks to a simple try / finally if (!succes) out.close(); for simplicity * in MockDirectoryWrapper#addFileHandle(io, name) you create an exception that always says Unclosed IndexInput which should say IndexOutput if the handle is an index output. I think this could be very confusing if you run into bugs there. simon MockDirectoryWrapper should track open file handles of IndexOutput too -- Key: LUCENE-3147 URL: https://issues.apache.org/jira/browse/LUCENE-3147 Project: Lucene - Java Issue Type: Test Components: general/test Reporter: Shai Erera Assignee: Shai Erera Priority: Minor Fix For: 3.2, 4.0 Attachments: LUCENE-3147.patch, LUCENE-3147.patch MockDirectoryWrapper currently tracks open file handles of IndexInput only. Therefore IO files that are not closed do not fail our tests, which can then lead to test directories fail to delete on Windows. We should make sure all open files are tracked and if they are left open, fail the test. I'll attach a patch shortly. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3147) MockDirectoryWrapper should track open file handles of IndexOutput too
[ https://issues.apache.org/jira/browse/LUCENE-3147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13040002#comment-13040002 ] Robert Muir commented on LUCENE-3147: - bq. and it fails on _e_0.doc left open. Is that the DocValues file (not sure if it was landed on trunk or not). This is the doc deltas file from SepCodec MockDirectoryWrapper should track open file handles of IndexOutput too -- Key: LUCENE-3147 URL: https://issues.apache.org/jira/browse/LUCENE-3147 Project: Lucene - Java Issue Type: Test Components: general/test Reporter: Shai Erera Assignee: Shai Erera Priority: Minor Fix For: 3.2, 4.0 Attachments: LUCENE-3147.patch, LUCENE-3147.patch MockDirectoryWrapper currently tracks open file handles of IndexInput only. Therefore IO files that are not closed do not fail our tests, which can then lead to test directories fail to delete on Windows. We should make sure all open files are tracked and if they are left open, fail the test. I'll attach a patch shortly. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3147) MockDirectoryWrapper should track open file handles of IndexOutput too
[ https://issues.apache.org/jira/browse/LUCENE-3147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13040074#comment-13040074 ] Uwe Schindler commented on LUCENE-3147: --- Hey, looks really cool to catch all that stuff. One thing: In BlockTermsWriter, the success boolean is never set to true! MockDirectoryWrapper should track open file handles of IndexOutput too -- Key: LUCENE-3147 URL: https://issues.apache.org/jira/browse/LUCENE-3147 Project: Lucene - Java Issue Type: Test Components: general/test Reporter: Shai Erera Assignee: Shai Erera Priority: Minor Fix For: 3.2, 4.0 Attachments: LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch MockDirectoryWrapper currently tracks open file handles of IndexInput only. Therefore IO files that are not closed do not fail our tests, which can then lead to test directories fail to delete on Windows. We should make sure all open files are tracked and if they are left open, fail the test. I'll attach a patch shortly. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
