Doug Cutting wrote:
Having a single, huge patch, implementing all the functionality, seems to be very difficult to maintain thru Bugzilla. So i would suggest to split the whole implementation in maybe 3 different steps.Bernhard Messer wrote:
a few month ago, there was a very interesting discussion about field compression and the possibility to store binary field values within a lucene document. Regarding to this topic, Drew Farris came up with a patch to add the necessary functionality. I ran all the necessary tests on his implementation and didn't find one problem. So the original implementation from Drew could now be enhanced to compress the binary field data (maybe even the text fields if they are stored only) before writing to disc. I made some simple statistical measurements using the java.util.zip package for data compression. Enabling it, we could save about 40% data when compressing plain text files with a size from 1KB to 4KB. If there is still some interest, we could first try to update the patch, because it's outdated due to several changes within the Fields class. After finishing that, compression could be added to the updated version of the patch.
I like this patch and support upgrading it and adding it to Lucene.
1) updating the binary field patch and add it to lucene
2) making FieldsReader and FieldsWriter more readable using private static finals and add compression
3) additional thoughts about compressing whole documents instead of single fields.
I'm totally with you. Compressing string values would make sense if the length reaches a certain size (the same for byte[]). This limit is something we have to figure out, what the minimum size of a compression candidate has to be. During my tests, i saw that everything up to 100 bytes is a perfect candidate for compression. But there is much more work to do in that area.I imagine a public API like:
public static final class Store {
[ ... ]
public static final COMPRESS = new Store(); }
new Field(String, byte[]) // stored, not compressed or indexed new Field(String, byte[], Store)
Also, in Field.java, perhaps we could replace:
String stringValue; Reader readerValue; byte[] binaryValue;
with:
Object value;
And in FieldsReader.java and FieldsWriter.java, some package-private constants would make the code more readable, like:
static final int FieldWriter.IS_TOKENIZED = 1; static final int FieldWriter.IS_BINARY = 2; static final int FieldWriter.IS_COMPRESSED = 4;
Note that it makes sense to compress non-binary values. One could use String.getBytes("UTF-8") and compress that.
I wonder if it might make more sense to compress entire document records, rather than individual fields. This would probably do better when documents have lots of short text fields, as is not uncommon, and would also minimize the fixed compression/decompression setup costs (i.e., inflator/deflator allocations). We could instead add a "isCompressed" flag to Document, and then, in Field{Reader,Writer}, store a bit per document indicating whether it is compressed. Document records could first be serialized uncompressed to a buffer which is then compressed and written. Thoughts?Interesting idea. I think this strongly depends on the fields, the options they have and at least their values. Would it make sense to compress a field which is tokenized and indexed but not stored ? My be we could think on some kind of algorithm, checking the document fields setting and decide if it is a candidate for compression. Just a thought ;-)
Doug
--------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
--------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
