[ https://issues.apache.org/jira/browse/LUCENE-4161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13417054#comment-13417054 ]
Adrien Grand commented on LUCENE-4161: -------------------------------------- Hi! I think {{PackedInts.Format.PACKED.nblocks(bitsPerValue, n)}} should help. It returns the number of {{long}} blocks required to store {{n}} {{bitsPerValue}}-bits values. On your pfor branch, I'm afraid instantiating a PackedInts.ReaderIterator for every block would hurt performance since blocks only contain 128 values. Maybe the easiest way to go would be to try to replace calls to {{PackedIntsDecompress.decode*}} with calls to {{PackedInts.BulkOperation.get/set}}. These methods are not exposed yet and only work with longs but if it looks good to you, I should be able to come up with something in the next few days... > Make PackedInts usable by codecs > -------------------------------- > > Key: LUCENE-4161 > URL: https://issues.apache.org/jira/browse/LUCENE-4161 > Project: Lucene - Java > Issue Type: Improvement > Components: core/store > Reporter: Adrien Grand > Assignee: Adrien Grand > Priority: Minor > Fix For: 4.0-ALPHA > > Attachments: LUCENE-4161.patch, LUCENE-4161.patch > > > Some codecs might be interested in using > PackedInts.{Writer,Reader,ReaderIterator} to read and write fixed-size values > efficiently. > The problem is that the serialization format is self contained, and always > writes the name of the codec, its version, its number of bits per value and > its format. For example, if you want to use packed ints to store your > postings list, this is a lot of overhead (at least ~60 bytes per term, in > case you only use one Writer per term, more otherwise). > Users should be able to externalize the storage of metadata to save space. > For example, to use PackedInts to store a postings list, one should be able > to store the codec name, its version and the number of bits per doc in the > header of the terms+postings list instead of having to write it once (or > more!) per term. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org