Re: Compressing docValues with variable length bytes[] by block of 16k ?

2015-08-09 Thread Robert Muir
That makes no sense at all, it would make it slow as shit. I am tired of repeating this: Don't use BINARY docvalues Don't use BINARY docvalues Don't use BINARY docvalues Use types like SORTED/SORTED_SET which will compress the term dictionary and make use of ordinals in your application instead.

Re: Compressing docValues with variable length bytes[] by block of 16k ?

2015-08-09 Thread Arjen van der Meijden
On 9-8-2015 16:22, Toke Eskildsen wrote: Robert Muir rcm...@gmail.com wrote: I am tired of repeating this: Don't use BINARY docvalues Don't use BINARY docvalues Don't use BINARY docvalues Use types like SORTED/SORTED_SET which will compress the term dictionary and make use of ordinals in

Re: Compressing docValues with variable length bytes[] by block of 16k ?

2015-08-09 Thread Olivier Binda
On 08/09/2015 06:29 PM, Uwe Schindler wrote: Hi, My values are unique and equal to the number of documents, They have varying sizes, say at least 10 bytes and may be a lot bigger (say 4kbytes) I don't share, index or sort them. I don't do grouping/faceting either I only want to store,

RE: Compressing docValues with variable length bytes[] by block of 16k ?

2015-08-09 Thread Uwe Schindler
Hi, My values are unique and equal to the number of documents, They have varying sizes, say at least 10 bytes and may be a lot bigger (say 4kbytes) I don't share, index or sort them. I don't do grouping/faceting either I only want to store, retrieve and traverse those values Then use

Re: Compressing docValues with variable length bytes[] by block of 16k ?

2015-08-09 Thread Toke Eskildsen
Arjen van der Meijden acmmail...@tweakers.net wrote: On 9-8-2015 16:22, Toke Eskildsen wrote: Maybe you could update the JavaDoc for that field to warn against using it? It (probably) depends on the contents of the values. That was my impression too, but we both seem to be second-guessing

Re: Compressing docValues with variable length bytes[] by block of 16k ?

2015-08-09 Thread Olivier Binda
On 08/09/2015 04:55 PM, Arjen van der Meijden wrote: On 9-8-2015 16:22, Toke Eskildsen wrote: Robert Muir rcm...@gmail.com wrote: I am tired of repeating this: Don't use BINARY docvalues Don't use BINARY docvalues Don't use BINARY docvalues Use types like SORTED/SORTED_SET which will compress

Re: Compressing docValues with variable length bytes[] by block of 16k ?

2015-08-09 Thread Toke Eskildsen
Robert Muir rcm...@gmail.com wrote: I am tired of repeating this: Don't use BINARY docvalues Don't use BINARY docvalues Don't use BINARY docvalues Use types like SORTED/SORTED_SET which will compress the term dictionary and make use of ordinals in your application instead. This seems

Compressing docValues with variable length bytes[] by block of 16k ?

2015-08-08 Thread Olivier Binda
Greetings are there any plans to implement compression of the variable length bites[] binary doc Values, say in blocks of 16k like for stored values ? my .cfs file goes from 2MB to like 400k when I zip it Best regards, Olivier On 08/08/2015 02:32 PM, jamie wrote: Greetings Our app