[
https://issues.apache.org/jira/browse/LUCENE-2975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13008504#comment-13008504
]
Robert Muir commented on LUCENE-2975:
-------------------------------------
again it looks like we are bit by the MultiMMapIndexInput:
For background MMapDirectory has two types of IndexInputs it uses, the
MMapIndexInput, and the MultiMMapIndexInput (only used for large files larger
than the chunk size, which for Uwe defaults to Integer.MAX_VALUE).
In the last release no unit tests even tested this MultiMmapII, we added a test
and found a bug already in this chunking (LUCENE-2627), but I think we need to
do more.
Ideally our tests, via MockDirectoryWrapper, would sometimes force the use of
this MultiMMapDirectory. The idea is to set a low chunk size to force this
type of stuff. We might have to add a nasty instanceof check or similar to
MockIndexInput to ensure we don't create too many mappings.
And the second problem would be, if its an integer overflow issue, then we need
a standalone 'slow' @Nightly-only test to ensure this case works.
> MMapDirectory on chunk size boundaries broken
> ---------------------------------------------
>
> Key: LUCENE-2975
> URL: https://issues.apache.org/jira/browse/LUCENE-2975
> Project: Lucene - Java
> Issue Type: Bug
> Affects Versions: 3.1
> Reporter: Uwe Schindler
> Priority: Blocker
>
> When testing the 3.1-RC1 made by Yonik on the PANGAEA (www.pangaea.de)
> productive system I figured out that suddenly on a large segment (about 5
> GiB) some stored fiels suddenly produce a strange deflate decompression
> problem (CompressionTools) although the stored fields are no longer pre-3.0
> compressed. It seems that the header of the stored field is read incorrectly
> at the buffer boundary in MultiMMapDir and then FieldsReader just incorrectly
> detects a deflate-compressed field (CompressionTools).
> The error occurs reproducible on CheckIndex with MMapDirectory, but not with
> NIODir or SimpleDir. The FDT file of that segment is 2.6 GiB, on Solaris the
> chunk size is Integer.MAX_VALUE, so we have 2 MultiMMap IndexInputs.
> Robert and me have the index ready as a tar file, we will do tests on our
> local machines and hopefully solve the bug, maybe introduced by Robert's
> recent changes to MMap.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
