Hi.
Hey, Bill. It's been a long time!
I've got a Lucene application that's been in use for about two years. Some users are using Lucene 1.2, some 1.3, and some are moving to 1.4. The indices seem to behave differently under each version. I'd like to add code to my application that checks the current user's index version against the version of Lucene that they are using, and automatically re-indexes their files if necessary. However, I can't figure out how to tell the version, from the index files.
Prior to 1.4, there were no format numbers in the index. These are being added, file-by-file, as we change file formats. As you've discovered, there is currently no public API to obtain the format number of an index. Also, the formats of different files are revved at different times, so there may not be a single format number for the entire index. (Perhaps we should remedy this, by, e.g., always revving the "segments" version whenever any file changes format.)
The documentation on the file formats, at
http://jakarta.apache.org/lucene/docs/fileformats.html, directs me to
the "segments" file. However, when I look at a version 1.3 segments
file, it seems to bear little relationship to the format described in
fileformats.html.
Have a look at the version of fileformats.html that shipped with 1.3. You can find this by browsing CVS, looking for the 1.3-final tag. But let me do it for you:
http://cvs.apache.org/viewcvs.cgi/*checkout*/jakarta-lucene/docs/fileformats.html?rev=1.15
According to CVS tags, that describes both the 1.3 and 1.2 index file formats.
But the part of fileformats.html dealing with the
segments file contains no "compatibility notes", so I assume it hasn't
changed since 1.3.
I wrote the bit about "compatibility notes" when I first documented file formats, and then promptly forgot about it. So, until someone contributes them, there are no compatibility notes. Sorry.
Even if it had, what's the idea of using -1 as the format number for 1.4?
The idea is to promptly break 1.3 and 1.2 code which tries to read the index. Those versions of Lucene don't check format numbers (because there were none). Positive values would give unpredictable errors. A negative value causes an immediate failure.
So, anyone know a way to tell the difference between the various versions of the index files? Crufty hacks welcome :-).
The first four bytes of the "segments" file will mostly do the trick. If it is zero or positive, then the index is a 1.2 or 1.3 index. If it is -2, then it's a 1.4-final or later index.
There was a change in formats between 1.2 and 1.3, with no format number change. This was in 1.3 RC1 (note #12 in CHANGES.txt). The semantics of each byte in norm files (.f[0-9]) changed. In 1.3 each byte represented 0.0-255.0 on a linear scale. In 1.3 and later they're eight-bit floats (three-bit mantissa, five-bit exponent, no sign bit). The net result is that if you use a 1.2 index with 1.3 or later then the correct documents will be returned, but scores and rankings will be wacky.
With the exception of this last bit, 1.4 should be able to correctly handle indexes from earlier releases. Please report if this is not the case.
Cheers,
Doug
--------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
