[
https://issues.apache.org/jira/browse/LUCENE-4069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13418314#comment-13418314
]
Mark Harwood commented on LUCENE-4069:
--------------------------------------
One more remaining issue before I commit which has appeared sporadically and
looks to be consistently raised by this test:
ant test -Dtestcase=TestIndexWriterCommit
-Dtests.method=testCommitThreadSafety -Dtests.seed=EA320250471B75AE
-Dtests.slow=true -Dtests.postingsformat=TestBloomFilteredLucene40Postings
-Dtests.locale=no -Dtests.timezone=Europe/Belfast
-Dtests.file.encoding=ISO-8859-1
The error it produces is this:
[junit4:junit4] > Caused by: java.lang.IllegalStateException: Missing
file:_9_TestBloomFilteredLucene40Postings_0.blm
[junit4:junit4] > at
org.apache.lucene.codecs.bloom.BloomFilteringPostingsFormat$BloomFilteredFieldsProducer.<init>(BloomFilteringPostingsFormat.java:175)
[junit4:junit4] > at
org.apache.lucene.codecs.bloom.BloomFilteringPostingsFormat.fieldsProducer(BloomFilteringPostingsFormat.java:156)
MockDirectoryWrapper looks to be randomly deleting files (probably my "blm"
file shown above) to simulate the effects of crashes.
Presumably I am doing the "right thing" in always throwing an exception if the
.blm file is missing? The alternative would be to silently ignore the missing
file which seems undesirable.
IF MDW is intended to only delete uncommitted files I'm not sure how we end up
in a scenario where BloomPF is being asked to open the uncommitted segment?
> Segment-level Bloom filters for a 2 x speed up on rare term searches
> --------------------------------------------------------------------
>
> Key: LUCENE-4069
> URL: https://issues.apache.org/jira/browse/LUCENE-4069
> Project: Lucene - Java
> Issue Type: Improvement
> Components: core/index
> Affects Versions: 3.6, 4.0-ALPHA
> Reporter: Mark Harwood
> Priority: Minor
> Fix For: 4.0
>
> Attachments: BloomFilterPostingsBranch4x.patch,
> LUCENE-4069-tryDeleteDocument.patch, LUCENE-4203.patch,
> MHBloomFilterOn3.6Branch.patch, PKLookupUpdatePerfTest.java,
> PKLookupUpdatePerfTest.java, PKLookupUpdatePerfTest.java,
> PKLookupUpdatePerfTest.java, PrimaryKeyPerfTest40.java
>
>
> An addition to each segment which stores a Bloom filter for selected fields
> in order to give fast-fail to term searches, helping avoid wasted disk access.
> Best suited for low-frequency fields e.g. primary keys on big indexes with
> many segments but also speeds up general searching in my tests.
> Overview slideshow here:
> http://www.slideshare.net/MarkHarwood/lucene-bloomfilteredsegments
> Benchmarks based on Wikipedia content here: http://goo.gl/X7QqU
> Patch based on 3.6 codebase attached.
> There are no 3.6 API changes currently - to play just add a field with "_blm"
> on the end of the name to invoke special indexing/querying capability.
> Clearly a new Field or schema declaration(!) would need adding to APIs to
> configure the service properly.
> Also, a patch for Lucene4.0 codebase introducing a new PostingsFormat
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
