[
https://issues.apache.org/jira/browse/LUCENE-6841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14960523#comment-14960523
]
Adrien Grand commented on LUCENE-6841:
--------------------------------------
This issue is a bit tricky, because we have some users wishing that stored
fields compression was more aggressive, and other users wishing that
compression was less aggressive. But on the other hand, we want to support as
few options as possible in order to keep backward compatibility manageable (we
already have 2 and certainly want to avoid 3).
Are you sure that the issue is with lz4 and not I/O? The method that performs
decompression performs I/O and actual decompression at the same time, so from
the output of a profiler it could easily look like the issue is with lz4 while
it actually is with I/O.
Also what kind of workload do you have (number of docs, size of individual
docs, number of docs retrieved per page)? Any chance that your index fits in
memory entirely?
> LZ4 compression using too much CPU time
> ---------------------------------------
>
> Key: LUCENE-6841
> URL: https://issues.apache.org/jira/browse/LUCENE-6841
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/codecs
> Affects Versions: 5.3.1
> Environment: Linux, Java 8
> Reporter: Karl von Randow
>
> I am using Lucene for search indexing, including storing a large number of
> small fields, and some larger plain text fields, and searching using both
> exact matches and analyzed queries.
> LZ4 (specifically the decompress method) is using nearly exactly 50% of the
> application's CPU time.
> It seems to me that LZ4 is inappropriate for my use case. I note that I can
> choose BEST_SPEED or BEST_COMPRESSION.
> Would it be palatable to add a NO_COMPRESSION option, or some way to pick and
> choose which fields get compressed? Perhaps a minimum length of a field could
> be specified before it's compressed? I'm not sure if that's possible.
> If this approach, or similar is palatable, I would be happy to contribute a
> patch (or to consume and test a patch).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
