[jira] [Commented] (LUCENE-2762) Don't leak deleted open file handles with pooled readers
[
https://issues.apache.org/jira/browse/LUCENE-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13041075#comment-13041075
]
Josef commented on LUCENE-2762:
---
{quote}
Unless: Josef, are you regularly hitting exceptions somewhere?
{quote}
No, we don't get any exceptions. I'll do some testing with our Glassfish
integration and also make sure, my test scenario matches the concrete
application integration. Hope this will isolate our problem. I'll report back
here.
Don't leak deleted open file handles with pooled readers
Key: LUCENE-2762
URL: https://issues.apache.org/jira/browse/LUCENE-2762
Project: Lucene - Java
Issue Type: Bug
Affects Versions: 2.9.4, 3.0.3, 3.1, 4.0
Reporter: Michael McCandless
Assignee: Michael McCandless
Fix For: 2.9.4, 3.0.3, 3.1, 4.0
Attachments: LUCENE-2762.patch
If you have CFS enabled today, and pooling is enabled (either directly
or because you've pulled an NRT reader), IndexWriter will hold open
SegmentReaders against the non-CFS format of each merged segment.
So even if you close all NRT readers you've pulled from the writer,
you'll still see file handles open against files that have been
deleted.
This count will not grow unbounded, since it's limited by the number
of segments in the index, but it's still a serious problem since the
app had turned off CFS in the first place presumably to avoid risk of
too-many-open-files. It's also bad because it ties up disk space
since these files would otherwise be deleted.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-2762) Don't leak deleted open file handles with pooled readers
[ https://issues.apache.org/jira/browse/LUCENE-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13040789#comment-13040789 ] Shai Erera commented on LUCENE-2762: Could this be related to all the leaked handles we fixed at LUCENE-3147? Don't leak deleted open file handles with pooled readers Key: LUCENE-2762 URL: https://issues.apache.org/jira/browse/LUCENE-2762 Project: Lucene - Java Issue Type: Bug Affects Versions: 2.9.4, 3.0.3, 3.1, 4.0 Reporter: Michael McCandless Assignee: Michael McCandless Fix For: 2.9.4, 3.0.3, 3.1, 4.0 Attachments: LUCENE-2762.patch If you have CFS enabled today, and pooling is enabled (either directly or because you've pulled an NRT reader), IndexWriter will hold open SegmentReaders against the non-CFS format of each merged segment. So even if you close all NRT readers you've pulled from the writer, you'll still see file handles open against files that have been deleted. This count will not grow unbounded, since it's limited by the number of segments in the index, but it's still a serious problem since the app had turned off CFS in the first place presumably to avoid risk of too-many-open-files. It's also bad because it ties up disk space since these files would otherwise be deleted. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-2762) Don't leak deleted open file handles with pooled readers
[ https://issues.apache.org/jira/browse/LUCENE-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13040795#comment-13040795 ] Michael McCandless commented on LUCENE-2762: bq. Could this be related to all the leaked handles we fixed at LUCENE-3147? I think that's unlikely? Ie, the leaks we are fixing in LUCENE-3147 are all cases where an exception was hit at some point? Unless: Josef, are you regularly hitting exceptions somewhere? Don't leak deleted open file handles with pooled readers Key: LUCENE-2762 URL: https://issues.apache.org/jira/browse/LUCENE-2762 Project: Lucene - Java Issue Type: Bug Affects Versions: 2.9.4, 3.0.3, 3.1, 4.0 Reporter: Michael McCandless Assignee: Michael McCandless Fix For: 2.9.4, 3.0.3, 3.1, 4.0 Attachments: LUCENE-2762.patch If you have CFS enabled today, and pooling is enabled (either directly or because you've pulled an NRT reader), IndexWriter will hold open SegmentReaders against the non-CFS format of each merged segment. So even if you close all NRT readers you've pulled from the writer, you'll still see file handles open against files that have been deleted. This count will not grow unbounded, since it's limited by the number of segments in the index, but it's still a serious problem since the app had turned off CFS in the first place presumably to avoid risk of too-many-open-files. It's also bad because it ties up disk space since these files would otherwise be deleted. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-2762) Don't leak deleted open file handles with pooled readers
[ https://issues.apache.org/jira/browse/LUCENE-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13038998#comment-13038998 ] Josef commented on LUCENE-2762: --- Thanks for your response. I'll try to extract a small test mimicking the behavior. And thanks for pointing out incRef / decRef and threading issue. I am aware of that and the relevant code is synchronized Don't leak deleted open file handles with pooled readers Key: LUCENE-2762 URL: https://issues.apache.org/jira/browse/LUCENE-2762 Project: Lucene - Java Issue Type: Bug Affects Versions: 2.9.4, 3.0.3, 3.1, 4.0 Reporter: Michael McCandless Assignee: Michael McCandless Fix For: 2.9.4, 3.0.3, 3.1, 4.0 Attachments: LUCENE-2762.patch If you have CFS enabled today, and pooling is enabled (either directly or because you've pulled an NRT reader), IndexWriter will hold open SegmentReaders against the non-CFS format of each merged segment. So even if you close all NRT readers you've pulled from the writer, you'll still see file handles open against files that have been deleted. This count will not grow unbounded, since it's limited by the number of segments in the index, but it's still a serious problem since the app had turned off CFS in the first place presumably to avoid risk of too-many-open-files. It's also bad because it ties up disk space since these files would otherwise be deleted. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-2762) Don't leak deleted open file handles with pooled readers
[ https://issues.apache.org/jira/browse/LUCENE-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13039139#comment-13039139 ] Josef commented on LUCENE-2762: --- I have extracted the relevant code into a small Java application, but I am not able to reproduce the observed behavior with it. It might have to do with our integration and usage of Lucene in the Glassfish application server. I'll go hunting there no. Any advice with such integration? Thanks for your help anyway! Don't leak deleted open file handles with pooled readers Key: LUCENE-2762 URL: https://issues.apache.org/jira/browse/LUCENE-2762 Project: Lucene - Java Issue Type: Bug Affects Versions: 2.9.4, 3.0.3, 3.1, 4.0 Reporter: Michael McCandless Assignee: Michael McCandless Fix For: 2.9.4, 3.0.3, 3.1, 4.0 Attachments: LUCENE-2762.patch If you have CFS enabled today, and pooling is enabled (either directly or because you've pulled an NRT reader), IndexWriter will hold open SegmentReaders against the non-CFS format of each merged segment. So even if you close all NRT readers you've pulled from the writer, you'll still see file handles open against files that have been deleted. This count will not grow unbounded, since it's limited by the number of segments in the index, but it's still a serious problem since the app had turned off CFS in the first place presumably to avoid risk of too-many-open-files. It's also bad because it ties up disk space since these files would otherwise be deleted. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-2762) Don't leak deleted open file handles with pooled readers
[ https://issues.apache.org/jira/browse/LUCENE-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13039174#comment-13039174 ] Michael McCandless commented on LUCENE-2762: Hmm I don't know anything about Glassfish... but it's interesting your separate tool couldn't repro the problem. Maybe re-scrutinize to verify your standalone tool truly matches how the app is using Lucene? Don't leak deleted open file handles with pooled readers Key: LUCENE-2762 URL: https://issues.apache.org/jira/browse/LUCENE-2762 Project: Lucene - Java Issue Type: Bug Affects Versions: 2.9.4, 3.0.3, 3.1, 4.0 Reporter: Michael McCandless Assignee: Michael McCandless Fix For: 2.9.4, 3.0.3, 3.1, 4.0 Attachments: LUCENE-2762.patch If you have CFS enabled today, and pooling is enabled (either directly or because you've pulled an NRT reader), IndexWriter will hold open SegmentReaders against the non-CFS format of each merged segment. So even if you close all NRT readers you've pulled from the writer, you'll still see file handles open against files that have been deleted. This count will not grow unbounded, since it's limited by the number of segments in the index, but it's still a serious problem since the app had turned off CFS in the first place presumably to avoid risk of too-many-open-files. It's also bad because it ties up disk space since these files would otherwise be deleted. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-2762) Don't leak deleted open file handles with pooled readers
[
https://issues.apache.org/jira/browse/LUCENE-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13038533#comment-13038533
]
Josef commented on LUCENE-2762:
---
Since I am still observing file references on deleted index files, I have a
quick question regarding this issue:
We are using Lucene 3.0.3 with standard configuration and near real-time
reader, as the following pseudo code shows:
{code}
...
// initially obtaining and referencing a near-real time searcher
IndexSearcher currentSearcher = new IndexSearcher(writer.getReader())
...
// subsequently obtaining a new near-real time searcher if necessary
if (!currentSearcher.getIndexReader().isCurrent()) {
IndexReader newReader = currentSearcher.getIndexReader().reopen();
IndexSearcher newSearcher = new IndexSearcher(newReader);
// release old searcher (by decreasing reference)
currentSearcher.getIndexReader().decRef();
currentSearcher = newSearcher;
}
...
{code}
After running the application for a while with index updates and doing new
searcher using the newly obtained IndexSearcher, we noticed that JVM still
holds references to already deleted index files.
For example:
{code}
java20742 xxx 394r REG8,6 366 3412376
/home/xxx/Desktop/mp.home/dev/index/Artist/_fc.cfs (deleted)
java20742 xxx 398r REG8,6 366 3412375
/home/xxx/Desktop/mp.home/dev/index/Artist/_fq.cfs
java20742 xxx 401uw REG8,60 3412333
/home/xxx/Desktop/mp.home/dev/index/Artist/write.lock
java20742 xxx 415r REG8,6 128 3412349
/home/xxx/Desktop/mp.home/dev/index/Artist/_fp.tis
java20742 xxx 416r REG8,6 366 3412341
/home/xxx/Desktop/mp.home/dev/index/Artist/_fd.cfs (deleted)
java20742 xxx 417r REG8,6 366 3412344
/home/xxx/Desktop/mp.home/dev/index/Artist/_fe.cfs (deleted)
java20742 xxx 418r REG8,6 71 3412356
/home/xxx/Desktop/mp.home/dev/index/Artist/_fb.tis (deleted)
java20742 xxx 424r REG8,67 3412362
/home/xxx/Desktop/mp.home/dev/index/Artist/_fb.frq (deleted)
java20742 xxx 425r REG8,67 3412363
/home/xxx/Desktop/mp.home/dev/index/Artist/_fb.prx (deleted)
java20742 xxx 426r REG8,6 23 3412351
/home/xxx/Desktop/mp.home/dev/index/Artist/_fb.fdt (deleted)
java20742 xxx 427r REG8,6 12 3412352
/home/xxx/Desktop/mp.home/dev/index/Artist/_fb.fdx (deleted)
java20742 xxx 428r REG8,6 10 3412365
/home/xxx/Desktop/mp.home/dev/index/Artist/_fb.nrm (deleted)
java20742 xxx 429r REG8,6 21 3412357
/home/xxx/Desktop/mp.home/dev/index/Artist/_fp.frq
java20742 xxx 432r REG8,6 21 3412358
/home/xxx/Desktop/mp.home/dev/index/Artist/_fp.prx
java20742 xxx 433r REG8,6 61 3412347
/home/xxx/Desktop/mp.home/dev/index/Artist/_fp.fdt
java20742 xxx 434r REG8,6 28 3412348
/home/xxx/Desktop/mp.home/dev/index/Artist/_fp.fdx
java20742 xxx 445r REG8,6 22 3412360
/home/xxx/Desktop/mp.home/dev/index/Artist/_fp.nrm
{code}
The application reaches the limit of maximum number of open files and then
stops.
Are we doing anything wrong here or does the bug still exist in version 3.0.3?
Any advice is welcome.
Thanks!
Don't leak deleted open file handles with pooled readers
Key: LUCENE-2762
URL: https://issues.apache.org/jira/browse/LUCENE-2762
Project: Lucene - Java
Issue Type: Bug
Affects Versions: 2.9.4, 3.0.3, 3.1, 4.0
Reporter: Michael McCandless
Assignee: Michael McCandless
Fix For: 2.9.4, 3.0.3, 3.1, 4.0
Attachments: LUCENE-2762.patch
If you have CFS enabled today, and pooling is enabled (either directly
or because you've pulled an NRT reader), IndexWriter will hold open
SegmentReaders against the non-CFS format of each merged segment.
So even if you close all NRT readers you've pulled from the writer,
you'll still see file handles open against files that have been
deleted.
This count will not grow unbounded, since it's limited by the number
of segments in the index, but it's still a serious problem since the
app had turned off CFS in the first place presumably to avoid risk of
too-many-open-files. It's also bad because it ties up disk space
since these files would otherwise be deleted.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (LUCENE-2762) Don't leak deleted open file handles with pooled readers
[ https://issues.apache.org/jira/browse/LUCENE-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13038629#comment-13038629 ] Michael McCandless commented on LUCENE-2762: Hmmm... the fix here was backported to 3.0.x, so it should be fixed in 3.0.3. The code fragment looks correct, but your usage of decRef (not close) likely means you have other places that also incRef the reader? Are you sure all incRefs are balanced by a decRef? Are you sure you really have leaking file handles and not just too many open files? EG what mergeFactor are you using...? Don't leak deleted open file handles with pooled readers Key: LUCENE-2762 URL: https://issues.apache.org/jira/browse/LUCENE-2762 Project: Lucene - Java Issue Type: Bug Affects Versions: 2.9.4, 3.0.3, 3.1, 4.0 Reporter: Michael McCandless Assignee: Michael McCandless Fix For: 2.9.4, 3.0.3, 3.1, 4.0 Attachments: LUCENE-2762.patch If you have CFS enabled today, and pooling is enabled (either directly or because you've pulled an NRT reader), IndexWriter will hold open SegmentReaders against the non-CFS format of each merged segment. So even if you close all NRT readers you've pulled from the writer, you'll still see file handles open against files that have been deleted. This count will not grow unbounded, since it's limited by the number of segments in the index, but it's still a serious problem since the app had turned off CFS in the first place presumably to avoid risk of too-many-open-files. It's also bad because it ties up disk space since these files would otherwise be deleted. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-2762) Don't leak deleted open file handles with pooled readers
[ https://issues.apache.org/jira/browse/LUCENE-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13038847#comment-13038847 ] Michael McCandless commented on LUCENE-2762: We have a test, TestNRTThreads, that stress-tests NRT to try to catch issues like this. I've backported to 3.0.x, and was able to run it for 30 minutes with max 1024 file handles, successfully... so if there is a file handle leak, this test failed to uncover it. Not sure what's up... can you try to boil down your test case to a small test showing the problem? Note that when you incRef the searcher you have to do that sync'd somehow with your reopen logic to prevent the final decRef from running at the same time. (I don't think that's related to the file handle issue but it's important to get right else you have a thread hazard). Don't leak deleted open file handles with pooled readers Key: LUCENE-2762 URL: https://issues.apache.org/jira/browse/LUCENE-2762 Project: Lucene - Java Issue Type: Bug Affects Versions: 2.9.4, 3.0.3, 3.1, 4.0 Reporter: Michael McCandless Assignee: Michael McCandless Fix For: 2.9.4, 3.0.3, 3.1, 4.0 Attachments: LUCENE-2762.patch If you have CFS enabled today, and pooling is enabled (either directly or because you've pulled an NRT reader), IndexWriter will hold open SegmentReaders against the non-CFS format of each merged segment. So even if you close all NRT readers you've pulled from the writer, you'll still see file handles open against files that have been deleted. This count will not grow unbounded, since it's limited by the number of segments in the index, but it's still a serious problem since the app had turned off CFS in the first place presumably to avoid risk of too-many-open-files. It's also bad because it ties up disk space since these files would otherwise be deleted. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] Commented: (LUCENE-2762) Don't leak deleted open file handles with pooled readers
[ https://issues.apache.org/jira/browse/LUCENE-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12933051#action_12933051 ] Michael McCandless commented on LUCENE-2762: So with this patch, we now build the CFS for a merged segment before adding that segment to the segment infos. This is important, to prevent an NRT reader from opening the pre-CFS version, thus tying open the files, using up extra disk space, and leaking deleted-but-open files even once all NRT readers are closed. But, unfortunately, this means the worst-case temporary peak free disk space required when using CFS has gone up... this worst case is hit if you 1) open an existing index, 2) call optimize on it, 3) the index needs more than 1 merge to become optimized, and 4) on the final merge of that optimize just after it's built the CFS but hasn't yet committed it to the segment infos. At that point you have 1X due to starting segments (which cannot be deleted until commit), another 1X due to the segments created by the prior merge (now being merged), another 1X by the newly merged single segment, and a final 1X from the final CFS. In this worst case that means we require 3X of your index size in temporary space. In other cases we use less disk space (the NRT case). And of course if CFS is off there's no change to the temp disk space. I've noted this in the javadocs and will add to CHANGES... But... I think we should improve our default MP. First, maybe we should set a maxMergeMB by default? Because immense merges cause all sorts of problems, and, likely are not going to impact search perf. Second, I think if a newly merged segment will be more than X% of the index, I think we should leave it in non-compound-file format even if useCompoundFile is enabled... I think there's a separate issue open somewhere for that 2nd one. Don't leak deleted open file handles with pooled readers Key: LUCENE-2762 URL: https://issues.apache.org/jira/browse/LUCENE-2762 Project: Lucene - Java Issue Type: Bug Affects Versions: 2.9.4, 3.0.3, 3.1, 4.0 Reporter: Michael McCandless Assignee: Michael McCandless Attachments: LUCENE-2762.patch If you have CFS enabled today, and pooling is enabled (either directly or because you've pulled an NRT reader), IndexWriter will hold open SegmentReaders against the non-CFS format of each merged segment. So even if you close all NRT readers you've pulled from the writer, you'll still see file handles open against files that have been deleted. This count will not grow unbounded, since it's limited by the number of segments in the index, but it's still a serious problem since the app had turned off CFS in the first place presumably to avoid risk of too-many-open-files. It's also bad because it ties up disk space since these files would otherwise be deleted. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] Commented: (LUCENE-2762) Don't leak deleted open file handles with pooled readers
[
https://issues.apache.org/jira/browse/LUCENE-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12933070#action_12933070
]
Jason Rutherglen commented on LUCENE-2762:
--
{quote}I think we should improve our default MP. First, maybe we should set a
maxMergeMB by default?{quote}
That's a good idea, however would we set an absolute size or a size relative to
the aggregate size of the index? I'm using 5 GB in production as otherwise I'm
not sure the merge cost is worth the potential performance improvement, ie,
long merges adversely affects indexing performance.
Don't leak deleted open file handles with pooled readers
Key: LUCENE-2762
URL: https://issues.apache.org/jira/browse/LUCENE-2762
Project: Lucene - Java
Issue Type: Bug
Affects Versions: 2.9.4, 3.0.3, 3.1, 4.0
Reporter: Michael McCandless
Assignee: Michael McCandless
Attachments: LUCENE-2762.patch
If you have CFS enabled today, and pooling is enabled (either directly
or because you've pulled an NRT reader), IndexWriter will hold open
SegmentReaders against the non-CFS format of each merged segment.
So even if you close all NRT readers you've pulled from the writer,
you'll still see file handles open against files that have been
deleted.
This count will not grow unbounded, since it's limited by the number
of segments in the index, but it's still a serious problem since the
app had turned off CFS in the first place presumably to avoid risk of
too-many-open-files. It's also bad because it ties up disk space
since these files would otherwise be deleted.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
