[jira] [Commented] (LUCENE-8689) Boolean DocValues Codec Implementation
[ https://issues.apache.org/jira/browse/LUCENE-8689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16794897#comment-16794897 ] Toke Eskildsen commented on LUCENE-8689: [~ivan.mamontov] as [~jpountz] suggests, the random access DV-issue has been generally addressed in Lucene/Solr 8. Still, looking at your code, I'll wager that your specialized implementation is faster for searching and a lot faster for indexing. Maybe I'm misunderstanding something, but it seems that the case of a document not having a boolean value defined is not handled? All lookups will give either true or false, which runs contrary to the other DV-types? This could be handled by having either a mirror-bitset for existence or using 2 bits/value, both of which adds complexity and slows down processing. > Boolean DocValues Codec Implementation > -- > > Key: LUCENE-8689 > URL: https://issues.apache.org/jira/browse/LUCENE-8689 > Project: Lucene - Core > Issue Type: Improvement > Components: core/codecs >Reporter: Ivan Mamontov >Priority: Minor > Labels: patch, performance > Attachments: LUCENE-8689.patch, SynteticDocValuesBench70.java, > results2.png > > > To avoid issues where some products become available/unavailable at some > point in time after being out-of-stock, e-commerce search system designers > need to embed up-to-date information about inventory availability right into > the search engines. Key requirement is to be able to accurately filter out > unavailable products and use availability as one of ranking signals. However, > keeping availability data up-to-date is a non-trivial task. Straightforward > implementation based on a partial updates of Lucene documents causes Solr > cache trashing with negatively affected query performance and resource > utilization. > As an alternative solution we can use DocValues and build-in in-place > updates where field values can be independently updated without touching > inverted index, and while filtering by DocValues is a bit slower, overall > performance gain is better. However existing long based docValues are not > sufficiently optimized for carrying boolean inventory availability data: > * All DocValues queries are internally rewritten into > org.apache.lucene.search.DocValuesNumbersQuery which is based on direct > iteration over all column values and typically much slower than using > TermsQuery. > * On every commit/merge codec has to iterate over DocValues a couple times > in order to choose the best compression algorithm suitable for given data. As > a result for 4K fields and 3M max doc merge takes more than 10 minutes > This issue is intended to solve these limitations via special bitwise doc > values format that uses internal representation of > org.apache.lucene.util.FixedBitSet in order to store indexed values and load > them at search time as a simple long array without additional decoding. There > are several reasons for this: > * At index time encoding is super fast without superfluous iterations over > all values to choose the best compression algorithm suitable for given data. > * At query time decoding is also simple and fast, no GC pressure and extra > steps > * Internal representation allows to perform random access in constant time > Limitations are: > * Does not support non boolean fields > * Boolean fields must be represented as long values 1 for true and 0 for > false > * Current implementation does not support advanced bit set formats like > org.apache.lucene.util.SparseFixedBitSet or > org.apache.lucene.util.RoaringDocIdSet > In order to evaluate performance gain I've wrote a simple JMH based benchmark > [^SynteticDocValuesBench70.java] which allows to estimate a relative cost of > DF filters. This benchmark creates 2 000 000 documents with 5 boolean columns > with different density, where 10, 35, 50, 60 and 90 is an amount of documents > with value 1. Each method tries to enumerate over all values in synthetic > store field in all available ways: > * baseline – in almost all cases Solr uses FixedBitSet in filter cache to > keep store availability. This test just iterates over all bits. > * docValuesRaw – iterates over all values of DV column, the same code is > used in "post filtering", sorting and faceting. > * docValuesNumbersQuery – iterates over all values produced by query/filter > store:1, actually there is the only query implementation for DV based fields > - DocValuesNumbersQuery. This means that Lucene rewrites all term, range and > filter queries for non indexed
Re: Improving DocValue sorting performance
On Mon, 2019-02-11 at 13:42 +0100, Daniel Penning wrote: > I think sorting and searching on docvalues can benefit from similar > optimizations like impacts do when scoring is used. By storing the > minimum and maximum value for blocks of docvalues it would be > possible to skip over sets of documents that don't contain any values > in the relevant range. Nice idea: The docID-indexed blocks are handled by IndexedDISI, where the blocks represents 65536 docs each. It would be simple enough to enrich these with min & max, although it would mean 16 bytes extra for each block, where the current minimum is 0 bytes + 8 bytes for the jump table entry. It could be made into an option of course... As Alan suggests, the jump tables would hold the values for maximum skippiness. I unsure how much API-change it would require for the values to be used. Adding a method such as nextDocIDWhereNumericMightBeHigherThan() seems awfully specialized and might require knowledge of the block size used inside of IndexedDISI, which is implementation specific. I do like the general idea of "can we somehow skip ahead?". Another sore point if how much this would statistically help. What are the chances that a block can be skipped at all? Of course it depends on the distribution of the values themselves, so sorting descending on a long tail distribution might work very well, whereas a more uniform distribution would result in no speed-up. A date field in a time-series corpus, indexed in chronological order, would work extremely well sorting ascending and extremely poorly descending (possible it's the other way around. It's getting late here). So yeah, it is definitely technically possible and a very interesting idea. And maybe very specialized? Cc: to Daniel as the thread is a month old. - Toke Eskildsen, Royal Danish Library - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8585) Create jump-tables for DocValues at index-time
[
https://issues.apache.org/jira/browse/LUCENE-8585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16791049#comment-16791049
]
Toke Eskildsen commented on LUCENE-8585:
I need to check my mail filters to discover why I did not see this before now.
I apologize and thank [~romseygeek] and [~jpountz] for handling it.
> Create jump-tables for DocValues at index-time
> --
>
> Key: LUCENE-8585
> URL: https://issues.apache.org/jira/browse/LUCENE-8585
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
>Affects Versions: 8.0
> Reporter: Toke Eskildsen
>Priority: Minor
> Labels: performance
> Fix For: 8.0
>
> Attachments: LUCENE-8585.patch, LUCENE-8585.patch,
> make_patch_lucene8585.sh
>
> Time Spent: 10.5h
> Remaining Estimate: 0h
>
> As noted in LUCENE-7589, lookup of DocValues should use jump-tables to avoid
> long iterative walks. This is implemented in LUCENE-8374 at search-time
> (first request for DocValues from a field in a segment), with the benefit of
> working without changes to existing Lucene 7 indexes and the downside of
> introducing a startup time penalty and a memory overhead.
> As discussed in LUCENE-8374, the codec should be updated to create these
> jump-tables at index time. This eliminates the segment-open time & memory
> penalties, with the potential downside of increasing index-time for DocValues.
> The three elements of LUCENE-8374 should be transferable to index-time
> without much alteration of the core structures:
> * {{IndexedDISI}} block offset and index skips: A {{long}} (64 bits) for
> every 65536 documents, containing the offset of the block in 33 bits and the
> index (number of set bits) up to the block in 31 bits.
> It can be build sequentially and should be stored as a simple sequence of
> consecutive longs for caching of lookups.
> As it is fairly small, relative to document count, it might be better to
> simply memory cache it?
> * {{IndexedDISI}} DENSE (> 4095, < 65536 set bits) blocks: A {{short}} (16
> bits) for every 8 {{longs}} (512 bits) for a total of 256 bytes/DENSE_block.
> Each {{short}} represents the number of set bits up to right before the
> corresponding sub-block of 512 docIDs.
> The \{{shorts}} can be computed sequentially or when the DENSE block is
> flushed (probably the easiest). They should be stored as a simple sequence of
> consecutive shorts for caching of lookups, one logically independent sequence
> for each DENSE block. The logical position would be one sequence at the start
> of every DENSE block.
> Whether it is best to read all the 16 {{shorts}} up front when a DENSE block
> is accessed or whether it is best to only read any individual {{short}} when
> needed is not clear at this point.
> * Variable Bits Per Value: A {{long}} (64 bits) for every 16384 numeric
> values. Each {{long}} holds the offset to the corresponding block of values.
> The offsets can be computed sequentially and should be stored as a simple
> sequence of consecutive {{longs}} for caching of lookups.
> The vBPV-offsets has the largest space overhead og the 3 jump-tables and a
> lot of the 64 bits in each long are not used for most indexes. They could be
> represented as a simple {{PackedInts}} sequence or {{MonotonicLongValues}},
> with the downsides of a potential lookup-time overhead and the need for doing
> the compression after all offsets has been determined.
> I have no experience with the codec-parts responsible for creating
> index-structures. I'm quite willing to take a stab at this, although I
> probably won't do much about it before January 2019. Should anyone else wish
> to adopt this JIRA-issue or co-work on it, I'll be happy to share.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8585) Create jump-tables for DocValues at index-time
[
https://issues.apache.org/jira/browse/LUCENE-8585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766949#comment-16766949
]
Toke Eskildsen commented on LUCENE-8585:
I have no outstanding issues, so resolving is fine.
Further performance-oriented testing form my part has been delayed due to
outside factors. There might well be further performance-oriented tweaks that
would be beneficial, but as a first implementation, I consider LUCENE-8585 to
be done. For the potential performance-tweaks, I will open another issue for
later Lucene/solr releases.
> Create jump-tables for DocValues at index-time
> --
>
> Key: LUCENE-8585
> URL: https://issues.apache.org/jira/browse/LUCENE-8585
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
>Affects Versions: 8.0
> Reporter: Toke Eskildsen
>Priority: Minor
> Labels: performance
> Fix For: 8.0
>
> Attachments: LUCENE-8585.patch, LUCENE-8585.patch,
> make_patch_lucene8585.sh
>
> Time Spent: 10.5h
> Remaining Estimate: 0h
>
> As noted in LUCENE-7589, lookup of DocValues should use jump-tables to avoid
> long iterative walks. This is implemented in LUCENE-8374 at search-time
> (first request for DocValues from a field in a segment), with the benefit of
> working without changes to existing Lucene 7 indexes and the downside of
> introducing a startup time penalty and a memory overhead.
> As discussed in LUCENE-8374, the codec should be updated to create these
> jump-tables at index time. This eliminates the segment-open time & memory
> penalties, with the potential downside of increasing index-time for DocValues.
> The three elements of LUCENE-8374 should be transferable to index-time
> without much alteration of the core structures:
> * {{IndexedDISI}} block offset and index skips: A {{long}} (64 bits) for
> every 65536 documents, containing the offset of the block in 33 bits and the
> index (number of set bits) up to the block in 31 bits.
> It can be build sequentially and should be stored as a simple sequence of
> consecutive longs for caching of lookups.
> As it is fairly small, relative to document count, it might be better to
> simply memory cache it?
> * {{IndexedDISI}} DENSE (> 4095, < 65536 set bits) blocks: A {{short}} (16
> bits) for every 8 {{longs}} (512 bits) for a total of 256 bytes/DENSE_block.
> Each {{short}} represents the number of set bits up to right before the
> corresponding sub-block of 512 docIDs.
> The \{{shorts}} can be computed sequentially or when the DENSE block is
> flushed (probably the easiest). They should be stored as a simple sequence of
> consecutive shorts for caching of lookups, one logically independent sequence
> for each DENSE block. The logical position would be one sequence at the start
> of every DENSE block.
> Whether it is best to read all the 16 {{shorts}} up front when a DENSE block
> is accessed or whether it is best to only read any individual {{short}} when
> needed is not clear at this point.
> * Variable Bits Per Value: A {{long}} (64 bits) for every 16384 numeric
> values. Each {{long}} holds the offset to the corresponding block of values.
> The offsets can be computed sequentially and should be stored as a simple
> sequence of consecutive {{longs}} for caching of lookups.
> The vBPV-offsets has the largest space overhead og the 3 jump-tables and a
> lot of the 64 bits in each long are not used for most indexes. They could be
> represented as a simple {{PackedInts}} sequence or {{MonotonicLongValues}},
> with the downsides of a potential lookup-time overhead and the need for doing
> the compression after all offsets has been determined.
> I have no experience with the codec-parts responsible for creating
> index-structures. I'm quite willing to take a stab at this, although I
> probably won't do much about it before January 2019. Should anyone else wish
> to adopt this JIRA-issue or co-work on it, I'll be happy to share.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8585) Create jump-tables for DocValues at index-time
[
https://issues.apache.org/jira/browse/LUCENE-8585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748610#comment-16748610
]
Toke Eskildsen commented on LUCENE-8585:
Skipping the method call makes sense, [~jpountz], thanks. The synthetic
accessor on private methods is new to me (I have now read up on it and
understand the problem), so thanks for the enlightment - there's some older
code elsewhere I have to re-visit with that in mind.
> Create jump-tables for DocValues at index-time
> --
>
> Key: LUCENE-8585
> URL: https://issues.apache.org/jira/browse/LUCENE-8585
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
>Affects Versions: 8.0
> Reporter: Toke Eskildsen
>Priority: Minor
> Labels: performance
> Attachments: LUCENE-8585.patch, LUCENE-8585.patch,
> make_patch_lucene8585.sh
>
> Time Spent: 10.5h
> Remaining Estimate: 0h
>
> As noted in LUCENE-7589, lookup of DocValues should use jump-tables to avoid
> long iterative walks. This is implemented in LUCENE-8374 at search-time
> (first request for DocValues from a field in a segment), with the benefit of
> working without changes to existing Lucene 7 indexes and the downside of
> introducing a startup time penalty and a memory overhead.
> As discussed in LUCENE-8374, the codec should be updated to create these
> jump-tables at index time. This eliminates the segment-open time & memory
> penalties, with the potential downside of increasing index-time for DocValues.
> The three elements of LUCENE-8374 should be transferable to index-time
> without much alteration of the core structures:
> * {{IndexedDISI}} block offset and index skips: A {{long}} (64 bits) for
> every 65536 documents, containing the offset of the block in 33 bits and the
> index (number of set bits) up to the block in 31 bits.
> It can be build sequentially and should be stored as a simple sequence of
> consecutive longs for caching of lookups.
> As it is fairly small, relative to document count, it might be better to
> simply memory cache it?
> * {{IndexedDISI}} DENSE (> 4095, < 65536 set bits) blocks: A {{short}} (16
> bits) for every 8 {{longs}} (512 bits) for a total of 256 bytes/DENSE_block.
> Each {{short}} represents the number of set bits up to right before the
> corresponding sub-block of 512 docIDs.
> The \{{shorts}} can be computed sequentially or when the DENSE block is
> flushed (probably the easiest). They should be stored as a simple sequence of
> consecutive shorts for caching of lookups, one logically independent sequence
> for each DENSE block. The logical position would be one sequence at the start
> of every DENSE block.
> Whether it is best to read all the 16 {{shorts}} up front when a DENSE block
> is accessed or whether it is best to only read any individual {{short}} when
> needed is not clear at this point.
> * Variable Bits Per Value: A {{long}} (64 bits) for every 16384 numeric
> values. Each {{long}} holds the offset to the corresponding block of values.
> The offsets can be computed sequentially and should be stored as a simple
> sequence of consecutive {{longs}} for caching of lookups.
> The vBPV-offsets has the largest space overhead og the 3 jump-tables and a
> lot of the 64 bits in each long are not used for most indexes. They could be
> represented as a simple {{PackedInts}} sequence or {{MonotonicLongValues}},
> with the downsides of a potential lookup-time overhead and the need for doing
> the compression after all offsets has been determined.
> I have no experience with the codec-parts responsible for creating
> index-structures. I'm quite willing to take a stab at this, although I
> probably won't do much about it before January 2019. Should anyone else wish
> to adopt this JIRA-issue or co-work on it, I'll be happy to share.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8585) Create jump-tables for DocValues at index-time
[
https://issues.apache.org/jira/browse/LUCENE-8585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16747934#comment-16747934
]
Toke Eskildsen commented on LUCENE-8585:
Thank you for cleaning up my mess, [~erickerickson]. I have cherry-picked it
along with the original LUCENE-8585 commit from {{master}} to {{branch_8x}},
which I hope is the correct action.
I consider LUCENE-8585 shipped and will experiment with different performance
tests for larger-scale shards in the coming weeks. I will keep an eye on
nightly benchmarks (Mike's and Elastic's) to check for regressions.
> Create jump-tables for DocValues at index-time
> --
>
> Key: LUCENE-8585
> URL: https://issues.apache.org/jira/browse/LUCENE-8585
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
>Affects Versions: 8.0
> Reporter: Toke Eskildsen
>Priority: Minor
> Labels: performance
> Attachments: LUCENE-8585.patch, LUCENE-8585.patch,
> make_patch_lucene8585.sh
>
> Time Spent: 10.5h
> Remaining Estimate: 0h
>
> As noted in LUCENE-7589, lookup of DocValues should use jump-tables to avoid
> long iterative walks. This is implemented in LUCENE-8374 at search-time
> (first request for DocValues from a field in a segment), with the benefit of
> working without changes to existing Lucene 7 indexes and the downside of
> introducing a startup time penalty and a memory overhead.
> As discussed in LUCENE-8374, the codec should be updated to create these
> jump-tables at index time. This eliminates the segment-open time & memory
> penalties, with the potential downside of increasing index-time for DocValues.
> The three elements of LUCENE-8374 should be transferable to index-time
> without much alteration of the core structures:
> * {{IndexedDISI}} block offset and index skips: A {{long}} (64 bits) for
> every 65536 documents, containing the offset of the block in 33 bits and the
> index (number of set bits) up to the block in 31 bits.
> It can be build sequentially and should be stored as a simple sequence of
> consecutive longs for caching of lookups.
> As it is fairly small, relative to document count, it might be better to
> simply memory cache it?
> * {{IndexedDISI}} DENSE (> 4095, < 65536 set bits) blocks: A {{short}} (16
> bits) for every 8 {{longs}} (512 bits) for a total of 256 bytes/DENSE_block.
> Each {{short}} represents the number of set bits up to right before the
> corresponding sub-block of 512 docIDs.
> The \{{shorts}} can be computed sequentially or when the DENSE block is
> flushed (probably the easiest). They should be stored as a simple sequence of
> consecutive shorts for caching of lookups, one logically independent sequence
> for each DENSE block. The logical position would be one sequence at the start
> of every DENSE block.
> Whether it is best to read all the 16 {{shorts}} up front when a DENSE block
> is accessed or whether it is best to only read any individual {{short}} when
> needed is not clear at this point.
> * Variable Bits Per Value: A {{long}} (64 bits) for every 16384 numeric
> values. Each {{long}} holds the offset to the corresponding block of values.
> The offsets can be computed sequentially and should be stored as a simple
> sequence of consecutive {{longs}} for caching of lookups.
> The vBPV-offsets has the largest space overhead og the 3 jump-tables and a
> lot of the 64 bits in each long are not used for most indexes. They could be
> represented as a simple {{PackedInts}} sequence or {{MonotonicLongValues}},
> with the downsides of a potential lookup-time overhead and the need for doing
> the compression after all offsets has been determined.
> I have no experience with the codec-parts responsible for creating
> index-structures. I'm quite willing to take a stab at this, although I
> probably won't do much about it before January 2019. Should anyone else wish
> to adopt this JIRA-issue or co-work on it, I'll be happy to share.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8585) Create jump-tables for DocValues at index-time
[
https://issues.apache.org/jira/browse/LUCENE-8585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16746300#comment-16746300
]
Toke Eskildsen commented on LUCENE-8585:
Doing it now with the current version, but it takes 16 hours to run an index
upgrade with my fun-size shard, so it is a weekend project. My old benchmarks
took advantage of being able to turn the jump-tables on & off on the fly; there
is a bit more hand-holding with the always-on. Given plenty of time, I would
like to spend a week of performance-testing and possibly tweaking DENSE-rank
size before merging to {{master}}. As things are, I think it is better to push
to {{master}} first and then do adjustments (if needed) based on my own and
other benchmark data.
I have created a patch, applied it to {{master}} and am unit-testing it right
now. I plan to push directly to the GitHub-repo from my local check out and
then close the pull-issue, as that ensures a single commit and keeps the review
history in a usable format. If my logic is flawed, please advice on the correct
procedure.
> Create jump-tables for DocValues at index-time
> --
>
> Key: LUCENE-8585
> URL: https://issues.apache.org/jira/browse/LUCENE-8585
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
>Affects Versions: 8.0
> Reporter: Toke Eskildsen
>Priority: Minor
> Labels: performance
> Attachments: LUCENE-8585.patch, LUCENE-8585.patch,
> make_patch_lucene8585.sh
>
> Time Spent: 10h 10m
> Remaining Estimate: 0h
>
> As noted in LUCENE-7589, lookup of DocValues should use jump-tables to avoid
> long iterative walks. This is implemented in LUCENE-8374 at search-time
> (first request for DocValues from a field in a segment), with the benefit of
> working without changes to existing Lucene 7 indexes and the downside of
> introducing a startup time penalty and a memory overhead.
> As discussed in LUCENE-8374, the codec should be updated to create these
> jump-tables at index time. This eliminates the segment-open time & memory
> penalties, with the potential downside of increasing index-time for DocValues.
> The three elements of LUCENE-8374 should be transferable to index-time
> without much alteration of the core structures:
> * {{IndexedDISI}} block offset and index skips: A {{long}} (64 bits) for
> every 65536 documents, containing the offset of the block in 33 bits and the
> index (number of set bits) up to the block in 31 bits.
> It can be build sequentially and should be stored as a simple sequence of
> consecutive longs for caching of lookups.
> As it is fairly small, relative to document count, it might be better to
> simply memory cache it?
> * {{IndexedDISI}} DENSE (> 4095, < 65536 set bits) blocks: A {{short}} (16
> bits) for every 8 {{longs}} (512 bits) for a total of 256 bytes/DENSE_block.
> Each {{short}} represents the number of set bits up to right before the
> corresponding sub-block of 512 docIDs.
> The \{{shorts}} can be computed sequentially or when the DENSE block is
> flushed (probably the easiest). They should be stored as a simple sequence of
> consecutive shorts for caching of lookups, one logically independent sequence
> for each DENSE block. The logical position would be one sequence at the start
> of every DENSE block.
> Whether it is best to read all the 16 {{shorts}} up front when a DENSE block
> is accessed or whether it is best to only read any individual {{short}} when
> needed is not clear at this point.
> * Variable Bits Per Value: A {{long}} (64 bits) for every 16384 numeric
> values. Each {{long}} holds the offset to the corresponding block of values.
> The offsets can be computed sequentially and should be stored as a simple
> sequence of consecutive {{longs}} for caching of lookups.
> The vBPV-offsets has the largest space overhead og the 3 jump-tables and a
> lot of the 64 bits in each long are not used for most indexes. They could be
> represented as a simple {{PackedInts}} sequence or {{MonotonicLongValues}},
> with the downsides of a potential lookup-time overhead and the need for doing
> the compression after all offsets has been determined.
> I have no experience with the codec-parts responsible for creating
> index-structures. I'm quite willing to take a stab at this, although I
> probably won't do much about it before January 2019. Should anyone else wish
> to adopt this JIRA-issue or co-work on it, I'll be happy to share.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8585) Create jump-tables for DocValues at index-time
[
https://issues.apache.org/jira/browse/LUCENE-8585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16745859#comment-16745859
]
Toke Eskildsen commented on LUCENE-8585:
Ah yes, {{TestLucene80DocValuesFormat}}, thanks.
I'll await the ping from [~romseygeek] and if positive, merge {{master}} in, do
a last {{ant test}} and merge back to {{master}}.
Considering your involvement in this, I suggest I set the credits in the
{{CHANGES.txt}} to (Toke Eskildsen, Adrien Grand)?
> Create jump-tables for DocValues at index-time
> --
>
> Key: LUCENE-8585
> URL: https://issues.apache.org/jira/browse/LUCENE-8585
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
>Affects Versions: 8.0
> Reporter: Toke Eskildsen
>Priority: Minor
> Labels: performance
> Attachments: LUCENE-8585.patch, LUCENE-8585.patch,
> make_patch_lucene8585.sh
>
> Time Spent: 10h 10m
> Remaining Estimate: 0h
>
> As noted in LUCENE-7589, lookup of DocValues should use jump-tables to avoid
> long iterative walks. This is implemented in LUCENE-8374 at search-time
> (first request for DocValues from a field in a segment), with the benefit of
> working without changes to existing Lucene 7 indexes and the downside of
> introducing a startup time penalty and a memory overhead.
> As discussed in LUCENE-8374, the codec should be updated to create these
> jump-tables at index time. This eliminates the segment-open time & memory
> penalties, with the potential downside of increasing index-time for DocValues.
> The three elements of LUCENE-8374 should be transferable to index-time
> without much alteration of the core structures:
> * {{IndexedDISI}} block offset and index skips: A {{long}} (64 bits) for
> every 65536 documents, containing the offset of the block in 33 bits and the
> index (number of set bits) up to the block in 31 bits.
> It can be build sequentially and should be stored as a simple sequence of
> consecutive longs for caching of lookups.
> As it is fairly small, relative to document count, it might be better to
> simply memory cache it?
> * {{IndexedDISI}} DENSE (> 4095, < 65536 set bits) blocks: A {{short}} (16
> bits) for every 8 {{longs}} (512 bits) for a total of 256 bytes/DENSE_block.
> Each {{short}} represents the number of set bits up to right before the
> corresponding sub-block of 512 docIDs.
> The \{{shorts}} can be computed sequentially or when the DENSE block is
> flushed (probably the easiest). They should be stored as a simple sequence of
> consecutive shorts for caching of lookups, one logically independent sequence
> for each DENSE block. The logical position would be one sequence at the start
> of every DENSE block.
> Whether it is best to read all the 16 {{shorts}} up front when a DENSE block
> is accessed or whether it is best to only read any individual {{short}} when
> needed is not clear at this point.
> * Variable Bits Per Value: A {{long}} (64 bits) for every 16384 numeric
> values. Each {{long}} holds the offset to the corresponding block of values.
> The offsets can be computed sequentially and should be stored as a simple
> sequence of consecutive {{longs}} for caching of lookups.
> The vBPV-offsets has the largest space overhead og the 3 jump-tables and a
> lot of the 64 bits in each long are not used for most indexes. They could be
> represented as a simple {{PackedInts}} sequence or {{MonotonicLongValues}},
> with the downsides of a potential lookup-time overhead and the need for doing
> the compression after all offsets has been determined.
> I have no experience with the codec-parts responsible for creating
> index-structures. I'm quite willing to take a stab at this, although I
> probably won't do much about it before January 2019. Should anyone else wish
> to adopt this JIRA-issue or co-work on it, I'll be happy to share.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8585) Create jump-tables for DocValues at index-time
[
https://issues.apache.org/jira/browse/LUCENE-8585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16745060#comment-16745060
]
Toke Eskildsen commented on LUCENE-8585:
I am the one thanking for top-notch feedback. The current pull-request is way
better than the first take.
I have cleaned up the jump-tables relevant testing by pulling that code out of
the general {{BaseDocValuesFormatTestCase}} to {{TestLucene80DocValuesFormat}}
and expanded it a bit. {{TestIndexedDISI}} tests jump-tabls at
{{IndexedDIDI}}-level, so {{BaseDocValuesFormatTestCase}} focuses on Variable
Bits Per Value block jumps. This reduces the test time overhead significantly
and should address the rest of the issues you have raised on the pull request.
Our schedules seem to fit poorly, so there have been long delays in our
communication. I'll prioritize to react quickly during the next week, in the
hope that we can get this pushed to master.
I need a bit of guidance on how to fit this into the 8.0-release. Shall I just
merge the {{master}}-branch into the {{LUCENE-8585}}-branch to keep the
pull-request current?
> Create jump-tables for DocValues at index-time
> --
>
> Key: LUCENE-8585
> URL: https://issues.apache.org/jira/browse/LUCENE-8585
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
>Affects Versions: 8.0
> Reporter: Toke Eskildsen
>Priority: Minor
> Labels: performance
> Attachments: LUCENE-8585.patch, LUCENE-8585.patch,
> make_patch_lucene8585.sh
>
> Time Spent: 9h 40m
> Remaining Estimate: 0h
>
> As noted in LUCENE-7589, lookup of DocValues should use jump-tables to avoid
> long iterative walks. This is implemented in LUCENE-8374 at search-time
> (first request for DocValues from a field in a segment), with the benefit of
> working without changes to existing Lucene 7 indexes and the downside of
> introducing a startup time penalty and a memory overhead.
> As discussed in LUCENE-8374, the codec should be updated to create these
> jump-tables at index time. This eliminates the segment-open time & memory
> penalties, with the potential downside of increasing index-time for DocValues.
> The three elements of LUCENE-8374 should be transferable to index-time
> without much alteration of the core structures:
> * {{IndexedDISI}} block offset and index skips: A {{long}} (64 bits) for
> every 65536 documents, containing the offset of the block in 33 bits and the
> index (number of set bits) up to the block in 31 bits.
> It can be build sequentially and should be stored as a simple sequence of
> consecutive longs for caching of lookups.
> As it is fairly small, relative to document count, it might be better to
> simply memory cache it?
> * {{IndexedDISI}} DENSE (> 4095, < 65536 set bits) blocks: A {{short}} (16
> bits) for every 8 {{longs}} (512 bits) for a total of 256 bytes/DENSE_block.
> Each {{short}} represents the number of set bits up to right before the
> corresponding sub-block of 512 docIDs.
> The \{{shorts}} can be computed sequentially or when the DENSE block is
> flushed (probably the easiest). They should be stored as a simple sequence of
> consecutive shorts for caching of lookups, one logically independent sequence
> for each DENSE block. The logical position would be one sequence at the start
> of every DENSE block.
> Whether it is best to read all the 16 {{shorts}} up front when a DENSE block
> is accessed or whether it is best to only read any individual {{short}} when
> needed is not clear at this point.
> * Variable Bits Per Value: A {{long}} (64 bits) for every 16384 numeric
> values. Each {{long}} holds the offset to the corresponding block of values.
> The offsets can be computed sequentially and should be stored as a simple
> sequence of consecutive {{longs}} for caching of lookups.
> The vBPV-offsets has the largest space overhead og the 3 jump-tables and a
> lot of the 64 bits in each long are not used for most indexes. They could be
> represented as a simple {{PackedInts}} sequence or {{MonotonicLongValues}},
> with the downsides of a potential lookup-time overhead and the need for doing
> the compression after all offsets has been determined.
> I have no experience with the codec-parts responsible for creating
> index-structures. I'm quite willing to take a stab at this, although I
> probably won't do much about it before January 2019. Should anyone else wish
> to adopt this JIRA-issue or co-work on it, I'll be happy to share.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Solr Size Limitation upto 32 kb limitation
On Thu, 2019-01-17 at 07:12 +, Kranthi Kumar K wrote: > Can we have any updates on the below issue? We are awaiting your > reply. Erick Erickson posted a reply 5 hours after you asked your original question on the developer mailinglist. I suggest you write a follow-up to his answer if it did not solve your problem. Cc: to Kranthi as he might have mailinglist-related delivery problems. - Toke Eskildsen, royal Danish Library - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8585) Create jump-tables for DocValues at index-time
[
https://issues.apache.org/jira/browse/LUCENE-8585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737150#comment-16737150
]
Toke Eskildsen commented on LUCENE-8585:
[~jpountz] I have implemented all the code changes you suggested on the pull
request. Still pending/under discussion is the 200K test-docs in
{{BaseDocValuesFormatTestCase.doTestNumericsVsStoredFields}} - there are
already block-spanning tests in place for the lucene 80 codec, so this is
"just" about coverage.
I see that the 8.0-ball is beginning to roll. How do you see the status of
LUCENE-8585 and how does it fit into the 8.0-process?
> Create jump-tables for DocValues at index-time
> --
>
> Key: LUCENE-8585
> URL: https://issues.apache.org/jira/browse/LUCENE-8585
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
>Affects Versions: 8.0
>Reporter: Toke Eskildsen
>Priority: Minor
> Labels: performance
> Attachments: LUCENE-8585.patch, LUCENE-8585.patch,
> make_patch_lucene8585.sh
>
> Time Spent: 9h 20m
> Remaining Estimate: 0h
>
> As noted in LUCENE-7589, lookup of DocValues should use jump-tables to avoid
> long iterative walks. This is implemented in LUCENE-8374 at search-time
> (first request for DocValues from a field in a segment), with the benefit of
> working without changes to existing Lucene 7 indexes and the downside of
> introducing a startup time penalty and a memory overhead.
> As discussed in LUCENE-8374, the codec should be updated to create these
> jump-tables at index time. This eliminates the segment-open time & memory
> penalties, with the potential downside of increasing index-time for DocValues.
> The three elements of LUCENE-8374 should be transferable to index-time
> without much alteration of the core structures:
> * {{IndexedDISI}} block offset and index skips: A {{long}} (64 bits) for
> every 65536 documents, containing the offset of the block in 33 bits and the
> index (number of set bits) up to the block in 31 bits.
> It can be build sequentially and should be stored as a simple sequence of
> consecutive longs for caching of lookups.
> As it is fairly small, relative to document count, it might be better to
> simply memory cache it?
> * {{IndexedDISI}} DENSE (> 4095, < 65536 set bits) blocks: A {{short}} (16
> bits) for every 8 {{longs}} (512 bits) for a total of 256 bytes/DENSE_block.
> Each {{short}} represents the number of set bits up to right before the
> corresponding sub-block of 512 docIDs.
> The \{{shorts}} can be computed sequentially or when the DENSE block is
> flushed (probably the easiest). They should be stored as a simple sequence of
> consecutive shorts for caching of lookups, one logically independent sequence
> for each DENSE block. The logical position would be one sequence at the start
> of every DENSE block.
> Whether it is best to read all the 16 {{shorts}} up front when a DENSE block
> is accessed or whether it is best to only read any individual {{short}} when
> needed is not clear at this point.
> * Variable Bits Per Value: A {{long}} (64 bits) for every 16384 numeric
> values. Each {{long}} holds the offset to the corresponding block of values.
> The offsets can be computed sequentially and should be stored as a simple
> sequence of consecutive {{longs}} for caching of lookups.
> The vBPV-offsets has the largest space overhead og the 3 jump-tables and a
> lot of the 64 bits in each long are not used for most indexes. They could be
> represented as a simple {{PackedInts}} sequence or {{MonotonicLongValues}},
> with the downsides of a potential lookup-time overhead and the need for doing
> the compression after all offsets has been determined.
> I have no experience with the codec-parts responsible for creating
> index-structures. I'm quite willing to take a stab at this, although I
> probably won't do much about it before January 2019. Should anyone else wish
> to adopt this JIRA-issue or co-work on it, I'll be happy to share.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8628) ByteBuffersDirectory merge exception
[
https://issues.apache.org/jira/browse/LUCENE-8628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16732952#comment-16732952
]
Toke Eskildsen commented on LUCENE-8628:
Sigh. My JIRA-searching skills are clearly terrible.
> ByteBuffersDirectory merge exception
>
>
> Key: LUCENE-8628
> URL: https://issues.apache.org/jira/browse/LUCENE-8628
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/store
>Affects Versions: 7.5
> Reporter: Toke Eskildsen
>Priority: Major
>
> An exception during merge for {{ByteBufferDirectory}} was [reported on the
> lucene-dev
> mailinglist|https://mail-archives.apache.org/mod_mbox/lucene-dev/201812.mbox/raw/%3CCAL%3D08JvGioNvfy8xKbb7%2B999YnNnU4iQv6F6ktRkudrkr-6jGg%40mail.gmail.com%3E]:
>
> {{Exception in thread "Lucene Merge Thread #879"
> org.apache.lucene.index.MergePolicy$MergeException:
> org.apache.lucene.store.AlreadyClosedException: refusing to delete any files:
> this IndexWriter hit an unrecoverable exception}}{{ at
> org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:705)}}{{
> at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:685)}}{{Caused
> by: org.apache.lucene.store.AlreadyClosedException: refusing to delete any
> files: this IndexWriter hit an unrecoverable exception}}{{ at
> org.apache.lucene.index.IndexFileDeleter.ensureOpen(IndexFileDeleter.java:349)}}{{
> at
> org.apache.lucene.index.IndexFileDeleter.deleteFiles(IndexFileDeleter.java:669)}}{{
> at
> org.apache.lucene.index.IndexFileDeleter.deleteNewFiles(IndexFileDeleter.java:664)}}{{
> at
> org.apache.lucene.index.IndexWriter.deleteNewFiles(IndexWriter.java:5024)}}{{
> at
> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4539)}}{{
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4075)}}{{
> at
> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:626)}}{{
> at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:663)}}{{Caused
> by: java.lang.IllegalArgumentException: cannot write negative vLong (got:
> -4294878395)}}{{ at
> org.apache.lucene.store.DataOutput.writeVLong(DataOutput.java:225)}}{{
> at
> org.apache.lucene.codecs.lucene50.Lucene50SkipWriter.writeSkipData(Lucene50SkipWriter.java:180)}}{{
> at
> org.apache.lucene.codecs.MultiLevelSkipListWriter.bufferSkip(MultiLevelSkipListWriter.java:143)}}{{
> at
> org.apache.lucene.codecs.lucene50.Lucene50SkipWriter.bufferSkip(Lucene50SkipWriter.java:162)}}{{
> at
> org.apache.lucene.codecs.lucene50.Lucene50PostingsWriter.startDoc(Lucene50PostingsWriter.java:228)}}{{
> at
> org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm(PushPostingsWriterBase.java:148)}}{{
> at
> org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:865)}}{{
> at
> org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:344)}}{{
> at
> org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:105)}}{{
> at
> org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge(PerFieldPostingsFormat.java:169)}}{{
> at
> org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:244)}}{{
> at
> org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:139)}}{{
> at
> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4453)}}{{
> ... 3 more}}
> Following the trace shows no apparent problems with the
> {{Lucene50SkipWripter}}. However, it uses {{getFilePointer}} to resolve the
> skip lengths, which for {{ByteBuffersDirectory}} ends in
> {{ByteBuffersDataOutput.size()}} with the code
> {{ public long size() {}}
> {{ long size = 0;}}
> {{ int blockCount = blocks.size();}}
> {{ if (blockCount >= 1) {}}
> {{ int fullBlockSize = (blockCount - 1) * blockSize();}}
> {{ int lastBlockSize = blocks.getLast().position();}}
> {{ size = fullBlockSize + lastBlockSize;}}
> {{ }}}
> {{ return size;}}
> {{ }}}
> One problem here is that
> {{int fullBlockSize = (blockCount - 1) * blockSize();}}
> can overflow. This should be changed to something like
> {{long fullBlockSize = (blockCount - 1) * (long)blockSize();}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Resolved] (LUCENE-8628) ByteBuffersDirectory merge exception
[
https://issues.apache.org/jira/browse/LUCENE-8628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Toke Eskildsen resolved LUCENE-8628.
Resolution: Duplicate
> ByteBuffersDirectory merge exception
>
>
> Key: LUCENE-8628
> URL: https://issues.apache.org/jira/browse/LUCENE-8628
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/store
>Affects Versions: 7.5
> Reporter: Toke Eskildsen
>Priority: Major
>
> An exception during merge for {{ByteBufferDirectory}} was [reported on the
> lucene-dev
> mailinglist|https://mail-archives.apache.org/mod_mbox/lucene-dev/201812.mbox/raw/%3CCAL%3D08JvGioNvfy8xKbb7%2B999YnNnU4iQv6F6ktRkudrkr-6jGg%40mail.gmail.com%3E]:
>
> {{Exception in thread "Lucene Merge Thread #879"
> org.apache.lucene.index.MergePolicy$MergeException:
> org.apache.lucene.store.AlreadyClosedException: refusing to delete any files:
> this IndexWriter hit an unrecoverable exception}}{{ at
> org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:705)}}{{
> at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:685)}}{{Caused
> by: org.apache.lucene.store.AlreadyClosedException: refusing to delete any
> files: this IndexWriter hit an unrecoverable exception}}{{ at
> org.apache.lucene.index.IndexFileDeleter.ensureOpen(IndexFileDeleter.java:349)}}{{
> at
> org.apache.lucene.index.IndexFileDeleter.deleteFiles(IndexFileDeleter.java:669)}}{{
> at
> org.apache.lucene.index.IndexFileDeleter.deleteNewFiles(IndexFileDeleter.java:664)}}{{
> at
> org.apache.lucene.index.IndexWriter.deleteNewFiles(IndexWriter.java:5024)}}{{
> at
> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4539)}}{{
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4075)}}{{
> at
> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:626)}}{{
> at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:663)}}{{Caused
> by: java.lang.IllegalArgumentException: cannot write negative vLong (got:
> -4294878395)}}{{ at
> org.apache.lucene.store.DataOutput.writeVLong(DataOutput.java:225)}}{{
> at
> org.apache.lucene.codecs.lucene50.Lucene50SkipWriter.writeSkipData(Lucene50SkipWriter.java:180)}}{{
> at
> org.apache.lucene.codecs.MultiLevelSkipListWriter.bufferSkip(MultiLevelSkipListWriter.java:143)}}{{
> at
> org.apache.lucene.codecs.lucene50.Lucene50SkipWriter.bufferSkip(Lucene50SkipWriter.java:162)}}{{
> at
> org.apache.lucene.codecs.lucene50.Lucene50PostingsWriter.startDoc(Lucene50PostingsWriter.java:228)}}{{
> at
> org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm(PushPostingsWriterBase.java:148)}}{{
> at
> org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:865)}}{{
> at
> org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:344)}}{{
> at
> org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:105)}}{{
> at
> org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge(PerFieldPostingsFormat.java:169)}}{{
> at
> org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:244)}}{{
> at
> org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:139)}}{{
> at
> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4453)}}{{
> ... 3 more}}
> Following the trace shows no apparent problems with the
> {{Lucene50SkipWripter}}. However, it uses {{getFilePointer}} to resolve the
> skip lengths, which for {{ByteBuffersDirectory}} ends in
> {{ByteBuffersDataOutput.size()}} with the code
> {{ public long size() {}}
> {{ long size = 0;}}
> {{ int blockCount = blocks.size();}}
> {{ if (blockCount >= 1) {}}
> {{ int fullBlockSize = (blockCount - 1) * blockSize();}}
> {{ int lastBlockSize = blocks.getLast().position();}}
> {{ size = fullBlockSize + lastBlockSize;}}
> {{ }}}
> {{ return size;}}
> {{ }}}
> One problem here is that
> {{int fullBlockSize = (blockCount - 1) * blockSize();}}
> can overflow. This should be changed to something like
> {{long fullBlockSize = (blockCount - 1) * (long)blockSize();}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8628) ByteBuffersDirectory merge exception
[
https://issues.apache.org/jira/browse/LUCENE-8628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16732897#comment-16732897
]
Toke Eskildsen commented on LUCENE-8628:
The fix is easy, but a unit-test for this would mean creating a 2GB+
{{ByteBuffersDataOutput}}. That seems a bit extreme for a trivial overflow-bug,
so maybe it should just be changed? Ping to [~dweiss] as he seems to have made
the code originally.
> ByteBuffersDirectory merge exception
>
>
> Key: LUCENE-8628
> URL: https://issues.apache.org/jira/browse/LUCENE-8628
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/store
>Affects Versions: 7.5
> Reporter: Toke Eskildsen
>Priority: Major
>
> An exception during merge for {{ByteBufferDirectory}} was [reported on the
> lucene-dev
> mailinglist|https://mail-archives.apache.org/mod_mbox/lucene-dev/201812.mbox/raw/%3CCAL%3D08JvGioNvfy8xKbb7%2B999YnNnU4iQv6F6ktRkudrkr-6jGg%40mail.gmail.com%3E]:
>
> {{Exception in thread "Lucene Merge Thread #879"
> org.apache.lucene.index.MergePolicy$MergeException:
> org.apache.lucene.store.AlreadyClosedException: refusing to delete any files:
> this IndexWriter hit an unrecoverable exception}}{{ at
> org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:705)}}{{
> at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:685)}}{{Caused
> by: org.apache.lucene.store.AlreadyClosedException: refusing to delete any
> files: this IndexWriter hit an unrecoverable exception}}{{ at
> org.apache.lucene.index.IndexFileDeleter.ensureOpen(IndexFileDeleter.java:349)}}{{
> at
> org.apache.lucene.index.IndexFileDeleter.deleteFiles(IndexFileDeleter.java:669)}}{{
> at
> org.apache.lucene.index.IndexFileDeleter.deleteNewFiles(IndexFileDeleter.java:664)}}{{
> at
> org.apache.lucene.index.IndexWriter.deleteNewFiles(IndexWriter.java:5024)}}{{
> at
> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4539)}}{{
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4075)}}{{
> at
> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:626)}}{{
> at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:663)}}{{Caused
> by: java.lang.IllegalArgumentException: cannot write negative vLong (got:
> -4294878395)}}{{ at
> org.apache.lucene.store.DataOutput.writeVLong(DataOutput.java:225)}}{{
> at
> org.apache.lucene.codecs.lucene50.Lucene50SkipWriter.writeSkipData(Lucene50SkipWriter.java:180)}}{{
> at
> org.apache.lucene.codecs.MultiLevelSkipListWriter.bufferSkip(MultiLevelSkipListWriter.java:143)}}{{
> at
> org.apache.lucene.codecs.lucene50.Lucene50SkipWriter.bufferSkip(Lucene50SkipWriter.java:162)}}{{
> at
> org.apache.lucene.codecs.lucene50.Lucene50PostingsWriter.startDoc(Lucene50PostingsWriter.java:228)}}{{
> at
> org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm(PushPostingsWriterBase.java:148)}}{{
> at
> org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:865)}}{{
> at
> org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:344)}}{{
> at
> org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:105)}}{{
> at
> org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge(PerFieldPostingsFormat.java:169)}}{{
> at
> org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:244)}}{{
> at
> org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:139)}}{{
> at
> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4453)}}{{
> ... 3 more}}
> Following the trace shows no apparent problems with the
> {{Lucene50SkipWripter}}. However, it uses {{getFilePointer}} to resolve the
> skip lengths, which for {{ByteBuffersDirectory}} ends in
> {{ByteBuffersDataOutput.size()}} with the code
> {{ public long size() {}}
> {{ long size = 0;}}
> {{ int blockCount = blocks.size();}}
> {{ if (blockCount >= 1) {}}
> {{ int fullBlockSize = (blockCount - 1) * blockSize();}}
> {{ int lastBlockSize = blocks.getLast().position();}}
> {{ size = fullBlockSize + lastBlockSize;}}
> {{ }}}
> {{ return size;}}
> {{ }}}
> One problem here is that
> {{int fullBlockSize = (blockCount - 1) * blockSize();}}
> c
[jira] [Created] (LUCENE-8628) ByteBuffersDirectory merge exception
Toke Eskildsen created LUCENE-8628:
--
Summary: ByteBuffersDirectory merge exception
Key: LUCENE-8628
URL: https://issues.apache.org/jira/browse/LUCENE-8628
Project: Lucene - Core
Issue Type: Bug
Components: core/store
Affects Versions: 7.5
Reporter: Toke Eskildsen
An exception during merge for {{ByteBufferDirectory}} was [reported on the
lucene-dev
mailinglist|https://mail-archives.apache.org/mod_mbox/lucene-dev/201812.mbox/raw/%3CCAL%3D08JvGioNvfy8xKbb7%2B999YnNnU4iQv6F6ktRkudrkr-6jGg%40mail.gmail.com%3E]:
{{Exception in thread "Lucene Merge Thread #879"
org.apache.lucene.index.MergePolicy$MergeException:
org.apache.lucene.store.AlreadyClosedException: refusing to delete any files:
this IndexWriter hit an unrecoverable exception}}{{ at
org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:705)}}{{
at
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:685)}}{{Caused
by: org.apache.lucene.store.AlreadyClosedException: refusing to delete any
files: this IndexWriter hit an unrecoverable exception}}{{ at
org.apache.lucene.index.IndexFileDeleter.ensureOpen(IndexFileDeleter.java:349)}}{{
at
org.apache.lucene.index.IndexFileDeleter.deleteFiles(IndexFileDeleter.java:669)}}{{
at
org.apache.lucene.index.IndexFileDeleter.deleteNewFiles(IndexFileDeleter.java:664)}}{{
at
org.apache.lucene.index.IndexWriter.deleteNewFiles(IndexWriter.java:5024)}}{{
at
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4539)}}{{
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4075)}}{{
at
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:626)}}{{
at
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:663)}}{{Caused
by: java.lang.IllegalArgumentException: cannot write negative vLong (got:
-4294878395)}}{{ at
org.apache.lucene.store.DataOutput.writeVLong(DataOutput.java:225)}}{{
at
org.apache.lucene.codecs.lucene50.Lucene50SkipWriter.writeSkipData(Lucene50SkipWriter.java:180)}}{{
at
org.apache.lucene.codecs.MultiLevelSkipListWriter.bufferSkip(MultiLevelSkipListWriter.java:143)}}{{
at
org.apache.lucene.codecs.lucene50.Lucene50SkipWriter.bufferSkip(Lucene50SkipWriter.java:162)}}{{
at
org.apache.lucene.codecs.lucene50.Lucene50PostingsWriter.startDoc(Lucene50PostingsWriter.java:228)}}{{
at
org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm(PushPostingsWriterBase.java:148)}}{{
at
org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:865)}}{{
at
org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:344)}}{{
at
org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:105)}}{{
at
org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge(PerFieldPostingsFormat.java:169)}}{{
at
org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:244)}}{{
at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:139)}}{{
at
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4453)}}{{
... 3 more}}
Following the trace shows no apparent problems with the
{{Lucene50SkipWripter}}. However, it uses {{getFilePointer}} to resolve the
skip lengths, which for {{ByteBuffersDirectory}} ends in
{{ByteBuffersDataOutput.size()}} with the code
{{ public long size() {}}
{{ long size = 0;}}
{{ int blockCount = blocks.size();}}
{{ if (blockCount >= 1) {}}
{{ int fullBlockSize = (blockCount - 1) * blockSize();}}
{{ int lastBlockSize = blocks.getLast().position();}}
{{ size = fullBlockSize + lastBlockSize;}}
{{ }}}
{{ return size;}}
{{ }}}
One problem here is that
{{int fullBlockSize = (blockCount - 1) * blockSize();}}
can overflow. This should be changed to something like
{{long fullBlockSize = (blockCount - 1) * (long)blockSize();}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (LUCENE-8585) Create jump-tables for DocValues at index-time
[
https://issues.apache.org/jira/browse/LUCENE-8585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16726684#comment-16726684
]
Toke Eskildsen edited comment on LUCENE-8585 at 12/21/18 3:35 PM:
--
[~jpountz] A colleague suggested that we make the {{DENSE}} rank span
adjustable or at least plan for it. Right now it is fixed as an entry for every
8 longs and as we discussed earlier, that is just a qualified guess as to what
works best.
If we record the span in {{meta}} (only accepting powers of 2 for clean shifts
& masks) and hardcode it to 8 in the consumer for now, it is just a single
{{short}} spend to allow for future tweaking, without having to change the
underlying codec.
_Addition:_ Thinking about it, I seem to be falling into the usual trap of
adding features and not shipping. If the suggestion above will put the
8.0-target at risk, then let's just postpone it.
was (Author: toke):
[~jpountz] A colleague suggested that we make the {{DENSE}} rank span
adjustable or at least plan for it. Right now it is fixed as an entry for every
8 longs and as we discussed earlier, that is just a qualified guess as to what
works best.
If we record the span in {{meta}} (only accepting powers of 2 for clean shifts
& masks) and hardcode it to 8 in the consumer for now, it is just a single
{{short}} spend to allow for future tweaking, without having to change the
underlying codec.
> Create jump-tables for DocValues at index-time
> --
>
> Key: LUCENE-8585
> URL: https://issues.apache.org/jira/browse/LUCENE-8585
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
>Affects Versions: master (8.0)
>Reporter: Toke Eskildsen
>Priority: Minor
> Labels: performance
> Attachments: LUCENE-8585.patch, LUCENE-8585.patch,
> make_patch_lucene8585.sh
>
> Time Spent: 5h 10m
> Remaining Estimate: 0h
>
> As noted in LUCENE-7589, lookup of DocValues should use jump-tables to avoid
> long iterative walks. This is implemented in LUCENE-8374 at search-time
> (first request for DocValues from a field in a segment), with the benefit of
> working without changes to existing Lucene 7 indexes and the downside of
> introducing a startup time penalty and a memory overhead.
> As discussed in LUCENE-8374, the codec should be updated to create these
> jump-tables at index time. This eliminates the segment-open time & memory
> penalties, with the potential downside of increasing index-time for DocValues.
> The three elements of LUCENE-8374 should be transferable to index-time
> without much alteration of the core structures:
> * {{IndexedDISI}} block offset and index skips: A {{long}} (64 bits) for
> every 65536 documents, containing the offset of the block in 33 bits and the
> index (number of set bits) up to the block in 31 bits.
> It can be build sequentially and should be stored as a simple sequence of
> consecutive longs for caching of lookups.
> As it is fairly small, relative to document count, it might be better to
> simply memory cache it?
> * {{IndexedDISI}} DENSE (> 4095, < 65536 set bits) blocks: A {{short}} (16
> bits) for every 8 {{longs}} (512 bits) for a total of 256 bytes/DENSE_block.
> Each {{short}} represents the number of set bits up to right before the
> corresponding sub-block of 512 docIDs.
> The \{{shorts}} can be computed sequentially or when the DENSE block is
> flushed (probably the easiest). They should be stored as a simple sequence of
> consecutive shorts for caching of lookups, one logically independent sequence
> for each DENSE block. The logical position would be one sequence at the start
> of every DENSE block.
> Whether it is best to read all the 16 {{shorts}} up front when a DENSE block
> is accessed or whether it is best to only read any individual {{short}} when
> needed is not clear at this point.
> * Variable Bits Per Value: A {{long}} (64 bits) for every 16384 numeric
> values. Each {{long}} holds the offset to the corresponding block of values.
> The offsets can be computed sequentially and should be stored as a simple
> sequence of consecutive {{longs}} for caching of lookups.
> The vBPV-offsets has the largest space overhead og the 3 jump-tables and a
> lot of the 64 bits in each long are not used for most indexes. They could be
> represented as a simple {{PackedInts}} sequence or {{MonotonicLongValues}},
> with the downsides of a potential lookup-time overhead and the need for doing
> the compression after all offsets has been determined.
> I have no experience with the codec-parts r
[jira] [Commented] (LUCENE-8585) Create jump-tables for DocValues at index-time
[
https://issues.apache.org/jira/browse/LUCENE-8585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16726684#comment-16726684
]
Toke Eskildsen commented on LUCENE-8585:
[~jpountz] A colleague suggested that we make the {{DENSE}} rank span
adjustable or at least plan for it. Right now it is fixed as an entry for every
8 longs and as we discussed earlier, that is just a qualified guess as to what
works best.
If we record the span in {{meta}} (only accepting powers of 2 for clean shifts
& masks) and hardcode it to 8 in the consumer for now, it is just a single
{{short}} spend to allow for future tweaking, without having to change the
underlying codec.
> Create jump-tables for DocValues at index-time
> --
>
> Key: LUCENE-8585
> URL: https://issues.apache.org/jira/browse/LUCENE-8585
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
>Affects Versions: master (8.0)
> Reporter: Toke Eskildsen
>Priority: Minor
> Labels: performance
> Attachments: LUCENE-8585.patch, LUCENE-8585.patch,
> make_patch_lucene8585.sh
>
> Time Spent: 5h 10m
> Remaining Estimate: 0h
>
> As noted in LUCENE-7589, lookup of DocValues should use jump-tables to avoid
> long iterative walks. This is implemented in LUCENE-8374 at search-time
> (first request for DocValues from a field in a segment), with the benefit of
> working without changes to existing Lucene 7 indexes and the downside of
> introducing a startup time penalty and a memory overhead.
> As discussed in LUCENE-8374, the codec should be updated to create these
> jump-tables at index time. This eliminates the segment-open time & memory
> penalties, with the potential downside of increasing index-time for DocValues.
> The three elements of LUCENE-8374 should be transferable to index-time
> without much alteration of the core structures:
> * {{IndexedDISI}} block offset and index skips: A {{long}} (64 bits) for
> every 65536 documents, containing the offset of the block in 33 bits and the
> index (number of set bits) up to the block in 31 bits.
> It can be build sequentially and should be stored as a simple sequence of
> consecutive longs for caching of lookups.
> As it is fairly small, relative to document count, it might be better to
> simply memory cache it?
> * {{IndexedDISI}} DENSE (> 4095, < 65536 set bits) blocks: A {{short}} (16
> bits) for every 8 {{longs}} (512 bits) for a total of 256 bytes/DENSE_block.
> Each {{short}} represents the number of set bits up to right before the
> corresponding sub-block of 512 docIDs.
> The \{{shorts}} can be computed sequentially or when the DENSE block is
> flushed (probably the easiest). They should be stored as a simple sequence of
> consecutive shorts for caching of lookups, one logically independent sequence
> for each DENSE block. The logical position would be one sequence at the start
> of every DENSE block.
> Whether it is best to read all the 16 {{shorts}} up front when a DENSE block
> is accessed or whether it is best to only read any individual {{short}} when
> needed is not clear at this point.
> * Variable Bits Per Value: A {{long}} (64 bits) for every 16384 numeric
> values. Each {{long}} holds the offset to the corresponding block of values.
> The offsets can be computed sequentially and should be stored as a simple
> sequence of consecutive {{longs}} for caching of lookups.
> The vBPV-offsets has the largest space overhead og the 3 jump-tables and a
> lot of the 64 bits in each long are not used for most indexes. They could be
> represented as a simple {{PackedInts}} sequence or {{MonotonicLongValues}},
> with the downsides of a potential lookup-time overhead and the need for doing
> the compression after all offsets has been determined.
> I have no experience with the codec-parts responsible for creating
> index-structures. I'm quite willing to take a stab at this, although I
> probably won't do much about it before January 2019. Should anyone else wish
> to adopt this JIRA-issue or co-work on it, I'll be happy to share.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8585) Create jump-tables for DocValues at index-time
[
https://issues.apache.org/jira/browse/LUCENE-8585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16726304#comment-16726304
]
Toke Eskildsen commented on LUCENE-8585:
[~jpountz] The problem was indeed the initial position in the slice. Index
upgrade now works as expected and the fix is pushed to the GitHub pull request.
Besides the unit-test coverage consideration, I have no other things on the
to-do for LUCENE-8585.
> Create jump-tables for DocValues at index-time
> --
>
> Key: LUCENE-8585
> URL: https://issues.apache.org/jira/browse/LUCENE-8585
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
>Affects Versions: master (8.0)
> Reporter: Toke Eskildsen
>Priority: Minor
> Labels: performance
> Attachments: LUCENE-8585.patch, LUCENE-8585.patch,
> make_patch_lucene8585.sh
>
> Time Spent: 5h 10m
> Remaining Estimate: 0h
>
> As noted in LUCENE-7589, lookup of DocValues should use jump-tables to avoid
> long iterative walks. This is implemented in LUCENE-8374 at search-time
> (first request for DocValues from a field in a segment), with the benefit of
> working without changes to existing Lucene 7 indexes and the downside of
> introducing a startup time penalty and a memory overhead.
> As discussed in LUCENE-8374, the codec should be updated to create these
> jump-tables at index time. This eliminates the segment-open time & memory
> penalties, with the potential downside of increasing index-time for DocValues.
> The three elements of LUCENE-8374 should be transferable to index-time
> without much alteration of the core structures:
> * {{IndexedDISI}} block offset and index skips: A {{long}} (64 bits) for
> every 65536 documents, containing the offset of the block in 33 bits and the
> index (number of set bits) up to the block in 31 bits.
> It can be build sequentially and should be stored as a simple sequence of
> consecutive longs for caching of lookups.
> As it is fairly small, relative to document count, it might be better to
> simply memory cache it?
> * {{IndexedDISI}} DENSE (> 4095, < 65536 set bits) blocks: A {{short}} (16
> bits) for every 8 {{longs}} (512 bits) for a total of 256 bytes/DENSE_block.
> Each {{short}} represents the number of set bits up to right before the
> corresponding sub-block of 512 docIDs.
> The \{{shorts}} can be computed sequentially or when the DENSE block is
> flushed (probably the easiest). They should be stored as a simple sequence of
> consecutive shorts for caching of lookups, one logically independent sequence
> for each DENSE block. The logical position would be one sequence at the start
> of every DENSE block.
> Whether it is best to read all the 16 {{shorts}} up front when a DENSE block
> is accessed or whether it is best to only read any individual {{short}} when
> needed is not clear at this point.
> * Variable Bits Per Value: A {{long}} (64 bits) for every 16384 numeric
> values. Each {{long}} holds the offset to the corresponding block of values.
> The offsets can be computed sequentially and should be stored as a simple
> sequence of consecutive {{longs}} for caching of lookups.
> The vBPV-offsets has the largest space overhead og the 3 jump-tables and a
> lot of the 64 bits in each long are not used for most indexes. They could be
> represented as a simple {{PackedInts}} sequence or {{MonotonicLongValues}},
> with the downsides of a potential lookup-time overhead and the need for doing
> the compression after all offsets has been determined.
> I have no experience with the codec-parts responsible for creating
> index-structures. I'm quite willing to take a stab at this, although I
> probably won't do much about it before January 2019. Should anyone else wish
> to adopt this JIRA-issue or co-work on it, I'll be happy to share.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8585) Create jump-tables for DocValues at index-time
[
https://issues.apache.org/jira/browse/LUCENE-8585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16725907#comment-16725907
]
Toke Eskildsen commented on LUCENE-8585:
Quick note, [~jpountz]: I cannot run and index upgrade without getting an
exception. I have pin-pointed it to the re-use of previously created slices in
{{Lucene80NormsProducer}} {{getDataInput}} & {{getDisiInput}} that is activated
by {{merging}}. It works fine if I disable re-use there. Hopefully it is just a
question of ignoring the current position of the given slice in the
{{IndexedDISI}} constructor as the old code indicates that the
{{IndexedDISI}}-data always starts at offset 0 in the slice.
I hope to find the time to fix that soon, but things are a but busy the next
week.
> Create jump-tables for DocValues at index-time
> --
>
> Key: LUCENE-8585
> URL: https://issues.apache.org/jira/browse/LUCENE-8585
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
>Affects Versions: master (8.0)
> Reporter: Toke Eskildsen
>Priority: Minor
> Labels: performance
> Attachments: LUCENE-8585.patch, LUCENE-8585.patch,
> make_patch_lucene8585.sh
>
> Time Spent: 5h 10m
> Remaining Estimate: 0h
>
> As noted in LUCENE-7589, lookup of DocValues should use jump-tables to avoid
> long iterative walks. This is implemented in LUCENE-8374 at search-time
> (first request for DocValues from a field in a segment), with the benefit of
> working without changes to existing Lucene 7 indexes and the downside of
> introducing a startup time penalty and a memory overhead.
> As discussed in LUCENE-8374, the codec should be updated to create these
> jump-tables at index time. This eliminates the segment-open time & memory
> penalties, with the potential downside of increasing index-time for DocValues.
> The three elements of LUCENE-8374 should be transferable to index-time
> without much alteration of the core structures:
> * {{IndexedDISI}} block offset and index skips: A {{long}} (64 bits) for
> every 65536 documents, containing the offset of the block in 33 bits and the
> index (number of set bits) up to the block in 31 bits.
> It can be build sequentially and should be stored as a simple sequence of
> consecutive longs for caching of lookups.
> As it is fairly small, relative to document count, it might be better to
> simply memory cache it?
> * {{IndexedDISI}} DENSE (> 4095, < 65536 set bits) blocks: A {{short}} (16
> bits) for every 8 {{longs}} (512 bits) for a total of 256 bytes/DENSE_block.
> Each {{short}} represents the number of set bits up to right before the
> corresponding sub-block of 512 docIDs.
> The \{{shorts}} can be computed sequentially or when the DENSE block is
> flushed (probably the easiest). They should be stored as a simple sequence of
> consecutive shorts for caching of lookups, one logically independent sequence
> for each DENSE block. The logical position would be one sequence at the start
> of every DENSE block.
> Whether it is best to read all the 16 {{shorts}} up front when a DENSE block
> is accessed or whether it is best to only read any individual {{short}} when
> needed is not clear at this point.
> * Variable Bits Per Value: A {{long}} (64 bits) for every 16384 numeric
> values. Each {{long}} holds the offset to the corresponding block of values.
> The offsets can be computed sequentially and should be stored as a simple
> sequence of consecutive {{longs}} for caching of lookups.
> The vBPV-offsets has the largest space overhead og the 3 jump-tables and a
> lot of the 64 bits in each long are not used for most indexes. They could be
> represented as a simple {{PackedInts}} sequence or {{MonotonicLongValues}},
> with the downsides of a potential lookup-time overhead and the need for doing
> the compression after all offsets has been determined.
> I have no experience with the codec-parts responsible for creating
> index-structures. I'm quite willing to take a stab at this, although I
> probably won't do much about it before January 2019. Should anyone else wish
> to adopt this JIRA-issue or co-work on it, I'll be happy to share.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8585) Create jump-tables for DocValues at index-time
[
https://issues.apache.org/jira/browse/LUCENE-8585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16724857#comment-16724857
]
Toke Eskildsen commented on LUCENE-8585:
[~jpountz] I have implemented the rank pre-loading and pushed it to GitHub. It
took just an hour or two, followed by a day of tedious debugging to discover
that I shifted some bits the wrong way and that I had not checked that part
even though I thought I had. Sigh.
I'll experiment with index-upgrading. Still pending is how to handle the
{{doTestNumericsVsStoredFields}} unit-test: It did help me find and pinpoint
the shift-problem when I ran it with 20K documents and testing all jump-length
possibilities, but doing that bumps the full {{TestLucene80DocValuesFormat}} to
~15 minutes, up from ~1 minute.
I have noted the +1 and I have prioritized working on LUCENE-8585 for the next
weeks (interrupted by hollidays).
> Create jump-tables for DocValues at index-time
> --
>
> Key: LUCENE-8585
> URL: https://issues.apache.org/jira/browse/LUCENE-8585
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
>Affects Versions: master (8.0)
> Reporter: Toke Eskildsen
>Priority: Minor
> Labels: performance
> Attachments: LUCENE-8585.patch, LUCENE-8585.patch,
> make_patch_lucene8585.sh
>
> Time Spent: 4h 50m
> Remaining Estimate: 0h
>
> As noted in LUCENE-7589, lookup of DocValues should use jump-tables to avoid
> long iterative walks. This is implemented in LUCENE-8374 at search-time
> (first request for DocValues from a field in a segment), with the benefit of
> working without changes to existing Lucene 7 indexes and the downside of
> introducing a startup time penalty and a memory overhead.
> As discussed in LUCENE-8374, the codec should be updated to create these
> jump-tables at index time. This eliminates the segment-open time & memory
> penalties, with the potential downside of increasing index-time for DocValues.
> The three elements of LUCENE-8374 should be transferable to index-time
> without much alteration of the core structures:
> * {{IndexedDISI}} block offset and index skips: A {{long}} (64 bits) for
> every 65536 documents, containing the offset of the block in 33 bits and the
> index (number of set bits) up to the block in 31 bits.
> It can be build sequentially and should be stored as a simple sequence of
> consecutive longs for caching of lookups.
> As it is fairly small, relative to document count, it might be better to
> simply memory cache it?
> * {{IndexedDISI}} DENSE (> 4095, < 65536 set bits) blocks: A {{short}} (16
> bits) for every 8 {{longs}} (512 bits) for a total of 256 bytes/DENSE_block.
> Each {{short}} represents the number of set bits up to right before the
> corresponding sub-block of 512 docIDs.
> The \{{shorts}} can be computed sequentially or when the DENSE block is
> flushed (probably the easiest). They should be stored as a simple sequence of
> consecutive shorts for caching of lookups, one logically independent sequence
> for each DENSE block. The logical position would be one sequence at the start
> of every DENSE block.
> Whether it is best to read all the 16 {{shorts}} up front when a DENSE block
> is accessed or whether it is best to only read any individual {{short}} when
> needed is not clear at this point.
> * Variable Bits Per Value: A {{long}} (64 bits) for every 16384 numeric
> values. Each {{long}} holds the offset to the corresponding block of values.
> The offsets can be computed sequentially and should be stored as a simple
> sequence of consecutive {{longs}} for caching of lookups.
> The vBPV-offsets has the largest space overhead og the 3 jump-tables and a
> lot of the 64 bits in each long are not used for most indexes. They could be
> represented as a simple {{PackedInts}} sequence or {{MonotonicLongValues}},
> with the downsides of a potential lookup-time overhead and the need for doing
> the compression after all offsets has been determined.
> I have no experience with the codec-parts responsible for creating
> index-structures. I'm quite willing to take a stab at this, although I
> probably won't do much about it before January 2019. Should anyone else wish
> to adopt this JIRA-issue or co-work on it, I'll be happy to share.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8585) Create jump-tables for DocValues at index-time
[
https://issues.apache.org/jira/browse/LUCENE-8585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16723918#comment-16723918
]
Toke Eskildsen commented on LUCENE-8585:
[~jpountz] how do you see the chances of getting this in 8.0? Freezing seems
awfully close and a I would expect some baking time for a change such as
LUCENE-8585. On the other hand, it seems a bit backwards to roll a 8.0 with a
planned codec-update pending for 8.1. If you are in doubt (as I am), should we
discuss this in the relevant thread on the mailing list?
> Create jump-tables for DocValues at index-time
> --
>
> Key: LUCENE-8585
> URL: https://issues.apache.org/jira/browse/LUCENE-8585
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
>Affects Versions: master (8.0)
> Reporter: Toke Eskildsen
>Priority: Minor
> Labels: performance
> Attachments: LUCENE-8585.patch, LUCENE-8585.patch,
> make_patch_lucene8585.sh
>
> Time Spent: 4h 50m
> Remaining Estimate: 0h
>
> As noted in LUCENE-7589, lookup of DocValues should use jump-tables to avoid
> long iterative walks. This is implemented in LUCENE-8374 at search-time
> (first request for DocValues from a field in a segment), with the benefit of
> working without changes to existing Lucene 7 indexes and the downside of
> introducing a startup time penalty and a memory overhead.
> As discussed in LUCENE-8374, the codec should be updated to create these
> jump-tables at index time. This eliminates the segment-open time & memory
> penalties, with the potential downside of increasing index-time for DocValues.
> The three elements of LUCENE-8374 should be transferable to index-time
> without much alteration of the core structures:
> * {{IndexedDISI}} block offset and index skips: A {{long}} (64 bits) for
> every 65536 documents, containing the offset of the block in 33 bits and the
> index (number of set bits) up to the block in 31 bits.
> It can be build sequentially and should be stored as a simple sequence of
> consecutive longs for caching of lookups.
> As it is fairly small, relative to document count, it might be better to
> simply memory cache it?
> * {{IndexedDISI}} DENSE (> 4095, < 65536 set bits) blocks: A {{short}} (16
> bits) for every 8 {{longs}} (512 bits) for a total of 256 bytes/DENSE_block.
> Each {{short}} represents the number of set bits up to right before the
> corresponding sub-block of 512 docIDs.
> The \{{shorts}} can be computed sequentially or when the DENSE block is
> flushed (probably the easiest). They should be stored as a simple sequence of
> consecutive shorts for caching of lookups, one logically independent sequence
> for each DENSE block. The logical position would be one sequence at the start
> of every DENSE block.
> Whether it is best to read all the 16 {{shorts}} up front when a DENSE block
> is accessed or whether it is best to only read any individual {{short}} when
> needed is not clear at this point.
> * Variable Bits Per Value: A {{long}} (64 bits) for every 16384 numeric
> values. Each {{long}} holds the offset to the corresponding block of values.
> The offsets can be computed sequentially and should be stored as a simple
> sequence of consecutive {{longs}} for caching of lookups.
> The vBPV-offsets has the largest space overhead og the 3 jump-tables and a
> lot of the 64 bits in each long are not used for most indexes. They could be
> represented as a simple {{PackedInts}} sequence or {{MonotonicLongValues}},
> with the downsides of a potential lookup-time overhead and the need for doing
> the compression after all offsets has been determined.
> I have no experience with the codec-parts responsible for creating
> index-structures. I'm quite willing to take a stab at this, although I
> probably won't do much about it before January 2019. Should anyone else wish
> to adopt this JIRA-issue or co-work on it, I'll be happy to share.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8585) Create jump-tables for DocValues at index-time
[
https://issues.apache.org/jira/browse/LUCENE-8585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16719464#comment-16719464
]
Toke Eskildsen commented on LUCENE-8585:
Just a quick note: Running {{ant test -Dtestcase=TestLucene80DocValuesFormat}}
took 1:19 minutes with 300 documents and 7:30 minutes with 200K. There are
about 120 tests in {{BaseDocValuesFormatTestCase}} and 14 of them calls
{{doTestNumericsVsStoredFields}}. Back-of-the-envelope: Increasing to 200K
documents adds half a minute for each of the 14 tests. They all pass BTW.
> Create jump-tables for DocValues at index-time
> --
>
> Key: LUCENE-8585
> URL: https://issues.apache.org/jira/browse/LUCENE-8585
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
>Affects Versions: master (8.0)
> Reporter: Toke Eskildsen
>Priority: Minor
> Labels: performance
> Attachments: LUCENE-8585.patch, LUCENE-8585.patch,
> make_patch_lucene8585.sh
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> As noted in LUCENE-7589, lookup of DocValues should use jump-tables to avoid
> long iterative walks. This is implemented in LUCENE-8374 at search-time
> (first request for DocValues from a field in a segment), with the benefit of
> working without changes to existing Lucene 7 indexes and the downside of
> introducing a startup time penalty and a memory overhead.
> As discussed in LUCENE-8374, the codec should be updated to create these
> jump-tables at index time. This eliminates the segment-open time & memory
> penalties, with the potential downside of increasing index-time for DocValues.
> The three elements of LUCENE-8374 should be transferable to index-time
> without much alteration of the core structures:
> * {{IndexedDISI}} block offset and index skips: A {{long}} (64 bits) for
> every 65536 documents, containing the offset of the block in 33 bits and the
> index (number of set bits) up to the block in 31 bits.
> It can be build sequentially and should be stored as a simple sequence of
> consecutive longs for caching of lookups.
> As it is fairly small, relative to document count, it might be better to
> simply memory cache it?
> * {{IndexedDISI}} DENSE (> 4095, < 65536 set bits) blocks: A {{short}} (16
> bits) for every 8 {{longs}} (512 bits) for a total of 256 bytes/DENSE_block.
> Each {{short}} represents the number of set bits up to right before the
> corresponding sub-block of 512 docIDs.
> The \{{shorts}} can be computed sequentially or when the DENSE block is
> flushed (probably the easiest). They should be stored as a simple sequence of
> consecutive shorts for caching of lookups, one logically independent sequence
> for each DENSE block. The logical position would be one sequence at the start
> of every DENSE block.
> Whether it is best to read all the 16 {{shorts}} up front when a DENSE block
> is accessed or whether it is best to only read any individual {{short}} when
> needed is not clear at this point.
> * Variable Bits Per Value: A {{long}} (64 bits) for every 16384 numeric
> values. Each {{long}} holds the offset to the corresponding block of values.
> The offsets can be computed sequentially and should be stored as a simple
> sequence of consecutive {{longs}} for caching of lookups.
> The vBPV-offsets has the largest space overhead og the 3 jump-tables and a
> lot of the 64 bits in each long are not used for most indexes. They could be
> represented as a simple {{PackedInts}} sequence or {{MonotonicLongValues}},
> with the downsides of a potential lookup-time overhead and the need for doing
> the compression after all offsets has been determined.
> I have no experience with the codec-parts responsible for creating
> index-structures. I'm quite willing to take a stab at this, although I
> probably won't do much about it before January 2019. Should anyone else wish
> to adopt this JIRA-issue or co-work on it, I'll be happy to share.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8585) Create jump-tables for DocValues at index-time
[
https://issues.apache.org/jira/browse/LUCENE-8585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16719252#comment-16719252
]
Toke Eskildsen commented on LUCENE-8585:
Thank you for reviewing, [~jpountz].
I'll try and run an index-upgrade on one of our large shards and measure the
difference. I'll also take another look at {{doTestNumericsVsStoredFields}} to
see if anything can be done there.
> Create jump-tables for DocValues at index-time
> --
>
> Key: LUCENE-8585
> URL: https://issues.apache.org/jira/browse/LUCENE-8585
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
>Affects Versions: master (8.0)
> Reporter: Toke Eskildsen
>Priority: Minor
> Labels: performance
> Attachments: LUCENE-8585.patch, LUCENE-8585.patch,
> make_patch_lucene8585.sh
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> As noted in LUCENE-7589, lookup of DocValues should use jump-tables to avoid
> long iterative walks. This is implemented in LUCENE-8374 at search-time
> (first request for DocValues from a field in a segment), with the benefit of
> working without changes to existing Lucene 7 indexes and the downside of
> introducing a startup time penalty and a memory overhead.
> As discussed in LUCENE-8374, the codec should be updated to create these
> jump-tables at index time. This eliminates the segment-open time & memory
> penalties, with the potential downside of increasing index-time for DocValues.
> The three elements of LUCENE-8374 should be transferable to index-time
> without much alteration of the core structures:
> * {{IndexedDISI}} block offset and index skips: A {{long}} (64 bits) for
> every 65536 documents, containing the offset of the block in 33 bits and the
> index (number of set bits) up to the block in 31 bits.
> It can be build sequentially and should be stored as a simple sequence of
> consecutive longs for caching of lookups.
> As it is fairly small, relative to document count, it might be better to
> simply memory cache it?
> * {{IndexedDISI}} DENSE (> 4095, < 65536 set bits) blocks: A {{short}} (16
> bits) for every 8 {{longs}} (512 bits) for a total of 256 bytes/DENSE_block.
> Each {{short}} represents the number of set bits up to right before the
> corresponding sub-block of 512 docIDs.
> The \{{shorts}} can be computed sequentially or when the DENSE block is
> flushed (probably the easiest). They should be stored as a simple sequence of
> consecutive shorts for caching of lookups, one logically independent sequence
> for each DENSE block. The logical position would be one sequence at the start
> of every DENSE block.
> Whether it is best to read all the 16 {{shorts}} up front when a DENSE block
> is accessed or whether it is best to only read any individual {{short}} when
> needed is not clear at this point.
> * Variable Bits Per Value: A {{long}} (64 bits) for every 16384 numeric
> values. Each {{long}} holds the offset to the corresponding block of values.
> The offsets can be computed sequentially and should be stored as a simple
> sequence of consecutive {{longs}} for caching of lookups.
> The vBPV-offsets has the largest space overhead og the 3 jump-tables and a
> lot of the 64 bits in each long are not used for most indexes. They could be
> represented as a simple {{PackedInts}} sequence or {{MonotonicLongValues}},
> with the downsides of a potential lookup-time overhead and the need for doing
> the compression after all offsets has been determined.
> I have no experience with the codec-parts responsible for creating
> index-structures. I'm quite willing to take a stab at this, although I
> probably won't do much about it before January 2019. Should anyone else wish
> to adopt this JIRA-issue or co-work on it, I'll be happy to share.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8585) Create jump-tables for DocValues at index-time
[
https://issues.apache.org/jira/browse/LUCENE-8585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16719050#comment-16719050
]
Toke Eskildsen commented on LUCENE-8585:
I have cleaned up, moved old lucene70 codec classes and in general tried to
finish the job. For a change of pace and ease of review, I have created a
pull-request instead of a patch. I'll create a patch, should anyone want that
instead.
I have a single pending issue with unit-testing: The method
{{BaseDocValuesformatTestCase.doTestNumericsVsStoredFields}} is used a lot and
currently operates with 300 documents. This is far from enough when testing
jumps. Upping it to 200,000 means that jumping can be implicitly tested for all
the different test-cases in {{BaseDocValuesformatTestCase}}, but that increases
processing time a lot. I could make it switch from 300 to 200,000 when running
{{Nightly}} or I could hand-pick some of the tests and increase documents for
them, which would mean worse coverage but better speed?
> Create jump-tables for DocValues at index-time
> --
>
> Key: LUCENE-8585
> URL: https://issues.apache.org/jira/browse/LUCENE-8585
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
>Affects Versions: master (8.0)
> Reporter: Toke Eskildsen
>Priority: Minor
> Labels: performance
> Attachments: LUCENE-8585.patch, LUCENE-8585.patch,
> make_patch_lucene8585.sh
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> As noted in LUCENE-7589, lookup of DocValues should use jump-tables to avoid
> long iterative walks. This is implemented in LUCENE-8374 at search-time
> (first request for DocValues from a field in a segment), with the benefit of
> working without changes to existing Lucene 7 indexes and the downside of
> introducing a startup time penalty and a memory overhead.
> As discussed in LUCENE-8374, the codec should be updated to create these
> jump-tables at index time. This eliminates the segment-open time & memory
> penalties, with the potential downside of increasing index-time for DocValues.
> The three elements of LUCENE-8374 should be transferable to index-time
> without much alteration of the core structures:
> * {{IndexedDISI}} block offset and index skips: A {{long}} (64 bits) for
> every 65536 documents, containing the offset of the block in 33 bits and the
> index (number of set bits) up to the block in 31 bits.
> It can be build sequentially and should be stored as a simple sequence of
> consecutive longs for caching of lookups.
> As it is fairly small, relative to document count, it might be better to
> simply memory cache it?
> * {{IndexedDISI}} DENSE (> 4095, < 65536 set bits) blocks: A {{short}} (16
> bits) for every 8 {{longs}} (512 bits) for a total of 256 bytes/DENSE_block.
> Each {{short}} represents the number of set bits up to right before the
> corresponding sub-block of 512 docIDs.
> The \{{shorts}} can be computed sequentially or when the DENSE block is
> flushed (probably the easiest). They should be stored as a simple sequence of
> consecutive shorts for caching of lookups, one logically independent sequence
> for each DENSE block. The logical position would be one sequence at the start
> of every DENSE block.
> Whether it is best to read all the 16 {{shorts}} up front when a DENSE block
> is accessed or whether it is best to only read any individual {{short}} when
> needed is not clear at this point.
> * Variable Bits Per Value: A {{long}} (64 bits) for every 16384 numeric
> values. Each {{long}} holds the offset to the corresponding block of values.
> The offsets can be computed sequentially and should be stored as a simple
> sequence of consecutive {{longs}} for caching of lookups.
> The vBPV-offsets has the largest space overhead og the 3 jump-tables and a
> lot of the 64 bits in each long are not used for most indexes. They could be
> represented as a simple {{PackedInts}} sequence or {{MonotonicLongValues}},
> with the downsides of a potential lookup-time overhead and the need for doing
> the compression after all offsets has been determined.
> I have no experience with the codec-parts responsible for creating
> index-structures. I'm quite willing to take a stab at this, although I
> probably won't do much about it before January 2019. Should anyone else wish
> to adopt this JIRA-issue or co-work on it, I'll be happy to share.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Resolved] (LUCENE-8374) Reduce reads for sparse DocValues
[
https://issues.apache.org/jira/browse/LUCENE-8374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Toke Eskildsen resolved LUCENE-8374.
Resolution: Won't Fix
I am marking this issue as "Won't Fix": LUCENE-8585 is the logical successor
and implements the jump-tables at index-time.
> Reduce reads for sparse DocValues
> -
>
> Key: LUCENE-8374
> URL: https://issues.apache.org/jira/browse/LUCENE-8374
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
>Affects Versions: 7.5, master (8.0)
>Reporter: Toke Eskildsen
>Priority: Major
> Labels: performance
> Attachments: LUCENE-8374.patch, LUCENE-8374.patch, LUCENE-8374.patch,
> LUCENE-8374.patch, LUCENE-8374.patch, LUCENE-8374.patch, LUCENE-8374.patch,
> LUCENE-8374_branch_7_3.patch, LUCENE-8374_branch_7_3.patch.20181005,
> LUCENE-8374_branch_7_4.patch, LUCENE-8374_branch_7_5.patch,
> LUCENE-8374_part_1.patch, LUCENE-8374_part_2.patch, LUCENE-8374_part_3.patch,
> LUCENE-8374_part_4.patch, entire_index_logs.txt,
> image-2018-10-24-07-30-06-663.png, image-2018-10-24-07-30-56-962.png,
> single_vehicle_logs.txt,
> start-2018-10-24-1_snapshot___Users_tim_Snapshots__-_YourKit_Java_Profiler_2017_02-b75_-_64-bit.png,
>
> start-2018-10-24_snapshot___Users_tim_Snapshots__-_YourKit_Java_Profiler_2017_02-b75_-_64-bit.png
>
>
> The {{Lucene70DocValuesProducer}} has the internal classes
> {{SparseNumericDocValues}} and {{BaseSortedSetDocValues}} (sparse code path),
> which again uses {{IndexedDISI}} to handle the docID -> value-ordinal lookup.
> The value-ordinal is the index of the docID assuming an abstract tightly
> packed monotonically increasing list of docIDs: If the docIDs with
> corresponding values are {{[0, 4, 1432]}}, their value-ordinals will be {{[0,
> 1, 2]}}.
> h2. Outer blocks
> The lookup structure of {{IndexedDISI}} consists of blocks of 2^16 values
> (65536), where each block can be either {{ALL}}, {{DENSE}} (2^12 to 2^16
> values) or {{SPARSE}} (< 2^12 values ~= 6%). Consequently blocks vary quite a
> lot in size and ordinal resolving strategy.
> When a sparse Numeric DocValue is needed, the code first locates the block
> containing the wanted docID flag. It does so by iterating blocks one-by-one
> until it reaches the needed one, where each iteration requires a lookup in
> the underlying {{IndexSlice}}. For a common memory mapped index, this
> translates to either a cached request or a read operation. If a segment has
> 6M documents, worst-case is 91 lookups. In our web archive, our segments has
> ~300M values: A worst-case of 4577 lookups!
> One obvious solution is to use a lookup-table for blocks: A long[]-array with
> an entry for each block. For 6M documents, that is < 1KB and would allow for
> direct jumping (a single lookup) in all instances. Unfortunately this
> lookup-table cannot be generated upfront when the writing of values is purely
> streaming. It can be appended to the end of the stream before it is closed,
> but without knowing the position of the lookup-table the reader cannot seek
> to it.
> One strategy for creating such a lookup-table would be to generate it during
> reads and cache it for next lookup. This does not fit directly into how
> {{IndexedDISI}} currently works (it is created anew for each invocation), but
> could probably be added with a little work. An advantage to this is that this
> does not change the underlying format and thus could be used with existing
> indexes.
> h2. The lookup structure inside each block
> If {{ALL}} of the 2^16 values are defined, the structure is empty and the
> ordinal is simply the requested docID with some modulo and multiply math.
> Nothing to improve there.
> If the block is {{DENSE}} (2^12 to 2^16 values are defined), a bitmap is used
> and the number of set bits up to the wanted index (the docID modulo the block
> origo) are counted. That bitmap is a long[1024], meaning that worst case is
> to lookup and count all set bits for 1024 longs!
> One known solution to this is to use a [rank
> structure|[https://en.wikipedia.org/wiki/Succinct_data_structure]]. I
> [implemented
> it|[https://github.com/tokee/lucene-solr/blob/solr5894/solr/core/src/java/org/apache/solr/search/sparse/count/plane/RankCache.java]]
> for a related project and with that (), the rank-overhead for a {{DENSE}}
> block would be long[32] and would ensure a maximum of 9 lookups. It is not
> trivial to build the rank-structure and caching it (assuming all blocks are
> dense) for 6M documents would require 22
[jira] [Commented] (LUCENE-8374) Reduce reads for sparse DocValues
[
https://issues.apache.org/jira/browse/LUCENE-8374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16716896#comment-16716896
]
Toke Eskildsen commented on LUCENE-8374:
No reaction to that. Fair enough, I could have asked more directly. Looking
back through the git log for Lucene/Solr, I infer that doing the {{git
revert}}-thing is the correct course as it has been used multiple times before.
I'll start that process.
> Reduce reads for sparse DocValues
> -
>
> Key: LUCENE-8374
> URL: https://issues.apache.org/jira/browse/LUCENE-8374
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
>Affects Versions: 7.5, master (8.0)
> Reporter: Toke Eskildsen
>Priority: Major
> Labels: performance
> Attachments: LUCENE-8374.patch, LUCENE-8374.patch, LUCENE-8374.patch,
> LUCENE-8374.patch, LUCENE-8374.patch, LUCENE-8374.patch, LUCENE-8374.patch,
> LUCENE-8374_branch_7_3.patch, LUCENE-8374_branch_7_3.patch.20181005,
> LUCENE-8374_branch_7_4.patch, LUCENE-8374_branch_7_5.patch,
> LUCENE-8374_part_1.patch, LUCENE-8374_part_2.patch, LUCENE-8374_part_3.patch,
> LUCENE-8374_part_4.patch, entire_index_logs.txt,
> image-2018-10-24-07-30-06-663.png, image-2018-10-24-07-30-56-962.png,
> single_vehicle_logs.txt,
> start-2018-10-24-1_snapshot___Users_tim_Snapshots__-_YourKit_Java_Profiler_2017_02-b75_-_64-bit.png,
>
> start-2018-10-24_snapshot___Users_tim_Snapshots__-_YourKit_Java_Profiler_2017_02-b75_-_64-bit.png
>
>
> The {{Lucene70DocValuesProducer}} has the internal classes
> {{SparseNumericDocValues}} and {{BaseSortedSetDocValues}} (sparse code path),
> which again uses {{IndexedDISI}} to handle the docID -> value-ordinal lookup.
> The value-ordinal is the index of the docID assuming an abstract tightly
> packed monotonically increasing list of docIDs: If the docIDs with
> corresponding values are {{[0, 4, 1432]}}, their value-ordinals will be {{[0,
> 1, 2]}}.
> h2. Outer blocks
> The lookup structure of {{IndexedDISI}} consists of blocks of 2^16 values
> (65536), where each block can be either {{ALL}}, {{DENSE}} (2^12 to 2^16
> values) or {{SPARSE}} (< 2^12 values ~= 6%). Consequently blocks vary quite a
> lot in size and ordinal resolving strategy.
> When a sparse Numeric DocValue is needed, the code first locates the block
> containing the wanted docID flag. It does so by iterating blocks one-by-one
> until it reaches the needed one, where each iteration requires a lookup in
> the underlying {{IndexSlice}}. For a common memory mapped index, this
> translates to either a cached request or a read operation. If a segment has
> 6M documents, worst-case is 91 lookups. In our web archive, our segments has
> ~300M values: A worst-case of 4577 lookups!
> One obvious solution is to use a lookup-table for blocks: A long[]-array with
> an entry for each block. For 6M documents, that is < 1KB and would allow for
> direct jumping (a single lookup) in all instances. Unfortunately this
> lookup-table cannot be generated upfront when the writing of values is purely
> streaming. It can be appended to the end of the stream before it is closed,
> but without knowing the position of the lookup-table the reader cannot seek
> to it.
> One strategy for creating such a lookup-table would be to generate it during
> reads and cache it for next lookup. This does not fit directly into how
> {{IndexedDISI}} currently works (it is created anew for each invocation), but
> could probably be added with a little work. An advantage to this is that this
> does not change the underlying format and thus could be used with existing
> indexes.
> h2. The lookup structure inside each block
> If {{ALL}} of the 2^16 values are defined, the structure is empty and the
> ordinal is simply the requested docID with some modulo and multiply math.
> Nothing to improve there.
> If the block is {{DENSE}} (2^12 to 2^16 values are defined), a bitmap is used
> and the number of set bits up to the wanted index (the docID modulo the block
> origo) are counted. That bitmap is a long[1024], meaning that worst case is
> to lookup and count all set bits for 1024 longs!
> One known solution to this is to use a [rank
> structure|[https://en.wikipedia.org/wiki/Succinct_data_structure]]. I
> [implemented
> it|[https://github.com/tokee/lucene-solr/blob/solr5894/solr/core/src/java/org/apache/solr/search/sparse/count/plane/RankCache.java]]
> for a related project and with that (), the rank-overhead for a {{DENSE}}
> block would be long[32] and would ensure a maximum of 9 lookups. It
[jira] [Commented] (SOLR-13056) SortableTextField is trappy for faceting
[
https://issues.apache.org/jira/browse/SOLR-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16716593#comment-16716593
]
Toke Eskildsen commented on SOLR-13056:
---
As mentioned in LUCENE-11916 the documentation should be updated to note this
limitation.
An alternative to the current lookup would be to do the fine-counting on the
DocValues instead, but that works very poorly with many values.
I am not sure how to fix this properly: Using the {{StandardTokenizerFactory}}
means that the indexed terms in the example are split on whitespace, making it
problematic (i.e. I don't know how) to make an exact match on the full content
of the field. Maybe we need to have multiple analyzer-chains for the same field
(LUCENE-11917?).
> SortableTextField is trappy for faceting
>
>
> Key: SOLR-13056
> URL: https://issues.apache.org/jira/browse/SOLR-13056
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: search
>Affects Versions: 7.6
> Reporter: Toke Eskildsen
>Priority: Major
>
> Using {{SortableTextField}} for distributed faceting can lead to wrong
> results. This can be demonstrated by installing the cloud-version of the
> {{gettingstarted}} sample with
> {{./solr -e cloud}}
> using defaults all the way, except for {{shards}} which should be {{3}}.
> After that a corpus can be indexed with
> {{( echo '[' ; for J in $(seq 0 99); do ID=$((J)) ; echo
> "\{\"id\":\"$ID\",\"facet_t_sort\":\"a b $J\"},"; done ; echo
> '\{"id":"duplicate_1","facet_t_sort":"a
> b"},\{"id":"duplicate_2","facet_t_sort":"a b"}]' ) | curl -s -d @- -X POST -H
> 'Content-Type: application/json'
> 'http://localhost:8983/solr/gettingstarted/update?commit=true'}}
> This will index 100 documents with a single-valued field {{facet_t_sort:"a b
> X"}} where X is the document number + 2 documents with {{facet_t_sort:"a
> b"}}. The call
> {{curl
> 'http://localhost:8983/solr/gettingstarted/select?facet.field=facet_t_sort=5=on=**:**=0'}}
> should return "a b" as the top facet term with count 2, but returns
> {{ {}}
> {{ "responseHeader":{}}
> {{ "zkConnected":true,}}
> {{ "status":0,}}
> {{ "QTime":13,}}
> {{ "params":{}}
> {{ "facet.limit":"5",}}
> {{ "q":":",}}
> {{ "facet.field":"facet_t_sort",}}
> {{ "rows":"0",}}
> {{ "facet":"on"} },}}
> {{ "response":{"numFound":102,"start":0,"maxScore":1.0,"docs":[]}}
> {{ },}}
> {{ "facet_counts":{}}
> {{ "facet_queries":{},}}
> {{ "facet_fields":{}}
> {{ "facet_t_sort":[}}
> {{ "a b",36,}}
> {{ "a b 0",1,}}
> {{ "a b 1",1,}}
> {{ "a b 10",1,}}
> {{ "a b 11",1]},}}
> {{ "facet_ranges":{},}}
> {{ "facet_intervals":{},}}
> {{ "facet_heatmaps":{} } } }}
> The problem is the second phase of simple faceting, where the fine-counting
> happens. In the first phase, "a b" is returned from 1 or 2 of the 3 shards.
> It wins the popularity contest as there are 2 "a b"-terms and only 1 of all
> the other terms. The 1 or 2 shards that did not deliver "a b" in the first
> phase are then queried for the count for "a b", which happens in the form of
> a {{facet_t_sort:"a b"}}-lookup. It seems that this lookup uses the analyzer
> chain and thus matches _all_ the documents in that shard (approximately
> 102/3).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-13056) SortableTextField is trappy for faceting
[
https://issues.apache.org/jira/browse/SOLR-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Toke Eskildsen updated SOLR-13056:
--
Description:
Using {{SortableTextField}} for distributed faceting can lead to wrong results.
This can be demonstrated by installing the cloud-version of the
{{gettingstarted}} sample with
{{./solr -e cloud}}
using defaults all the way, except for {{shards}} which should be {{3}}. After
that a corpus can be indexed with
{{ ( echo '[' ; for J in $(seq 0 99); do ID=$((J)) ; echo
"\\{\"id\":\"$ID\",\"facet_t_sort\":\"a b $J\"},"; done ; echo
'\\{"id":"duplicate_1","facet_t_sort":"a
b"},\\{"id":"duplicate_2","facet_t_sort":"a b"}]' ) | curl -s -d @- -X POST -H
'Content-Type: application/json'
'http://localhost:8983/solr/gettingstarted/update?commit=true' }}
This will index 100 documents with a single-valued field {{facet_t_sort:"a b
X"}} where X is the document number + 2 documents with {{facet_t_sort:"a b"}}.
The call
{{curl
'http://localhost:8983/solr/gettingstarted/select?facet.field=facet_t_sort=5=on=**:**=0'}}
should return "a b" as the top facet term with count 2, but returns
{\{{}}
\{{ "responseHeader":{}}
\{{ "zkConnected":true,}}
\{{ "status":0,}}
\{{ "QTime":13,}}
\{{ "params":}}{{
{ "facet.limit":"5", "q":"*:*", "facet.field":"facet_t_sort", "rows":"0",
"facet":"on"}
}}{{},}}
\{{ "response":}}{{
{"numFound":102,"start":0,"maxScore":1.0,"docs":[] }
}}{{,}}
\{{ "facet_counts":{}}
\{{ "facet_queries":{},}}
\{{ "facet_fields":}}\{{{}}
\{{ "facet_t_sort":[ }}
\{{ "a b",36, }}
\{{ "a b 0",1, }}
\{{ "a b 1",1, }}
\{{ "a b 10",1, }}
{{ "a b 11",1]}}}{{,}}
\{{ "facet_ranges":{},}}
\{{ "facet_intervals":{},}}
\{{ "facet_heatmaps":{} } }}}
The problem is the second phase of simple faceting, where the fine-counting
happens. In the first phase, "a b" is returned from 1 or 2 of the 3 shards. It
wins the popularity contest as there are 2 "a b"-terms and only 1 of all the
other terms. The 1 or 2 shards that did not deliver "a b" in the first phase
are then queried for the count for "a b", which happens in the form of a
{{facet_t_sort:"a b"}}-lookup. It seems that this lookup uses the analyzer
chain and thus matches _all_ the documents in that shard (approximately 102/3).
was:
Using {{SortableTextField}} for distributed faceting can lead to wrong results.
This can be demonstrated by installing the cloud-version of the
{{gettingstarted}} sample with
{{./solr -e cloud}}
using defaults all the way, except for {{shards}} which should be {{3}}. After
that a corpus can be indexed with
{{( echo '[' ; for J in $(seq 0 99); do ID=$((J)) ; echo
"\\{\"id\":\"$ID\",\"facet_t_sort\":\"a b $J\"},"; done ; echo
'\\{"id":"duplicate_1","facet_t_sort":"a
b"},\\{"id":"duplicate_2","facet_t_sort":"a b"}]' ) | curl -s -d @- -X POST -H
'Content-Type: application/json'
'http://localhost:8983/solr/gettingstarted/update?commit=true'}}
This will index 100 documents with a single-valued field {{facet_t_sort:"a b
X"}} where X is the document number + 2 documents with {{facet_t_sort:"a b"}}.
The call
{{curl
'http://localhost:8983/solr/gettingstarted/select?facet.field=facet_t_sort=5=on=**:**=0'}}
should return "a b" as the top facet term with count 2, but returns
{\{{}}
\{{ "responseHeader":{}}
\{{ "zkConnected":true,}}
\{{ "status":0,}}
\{{ "QTime":13,}}
\{{ "params":}}{{
{ "facet.limit":"5", "q":"*:*", "facet.field":"facet_t_sort", "rows":"0",
"facet":"on"}
}}{{},}}
\{{ "response":}}{{
{"numFound":102,"start":0,"maxScore":1.0,"docs":[] }
}}{{,}}
\{{ "facet_counts":{}}
\{{ "facet_queries":{},}}
\{{ "facet_fields":}}\{{{}}
\{{ "facet_t_sort":[ }}
\{{ "a b",36, }}
\{{ "a b 0",1, }}
\{{ "a b 1",1, }}
\{{ "a b 10",1, }}
{{ "a b 11",1]}}}{{,}}
\{{ "facet_ranges":{},}}
\{{ "facet_intervals":{},}}
\{{ "facet_heatmaps":{
[jira] [Updated] (SOLR-13056) SortableTextField is trappy for faceting
[
https://issues.apache.org/jira/browse/SOLR-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Toke Eskildsen updated SOLR-13056:
--
Description:
Using {{SortableTextField}} for distributed faceting can lead to wrong results.
This can be demonstrated by installing the cloud-version of the
{{gettingstarted}} sample with
{{./solr -e cloud}}
using defaults all the way, except for {{shards}} which should be {{3}}. After
that a corpus can be indexed with
{{( echo '[' ; for J in $(seq 0 99); do ID=$((J)) ; echo
"\\{\"id\":\"$ID\",\"facet_t_sort\":\"a b $J\"},"; done ; echo
'\\{"id":"duplicate_1","facet_t_sort":"a
b"},\\{"id":"duplicate_2","facet_t_sort":"a b"}]' ) | curl -s -d @- -X POST -H
'Content-Type: application/json'
'http://localhost:8983/solr/gettingstarted/update?commit=true'}}
This will index 100 documents with a single-valued field {{facet_t_sort:"a b
X"}} where X is the document number + 2 documents with {{facet_t_sort:"a b"}}.
The call
{{curl
'http://localhost:8983/solr/gettingstarted/select?facet.field=facet_t_sort=5=on=**:**=0'}}
should return "a b" as the top facet term with count 2, but returns
{\{{}}
\{{ "responseHeader":{}}
\{{ "zkConnected":true,}}
\{{ "status":0,}}
\{{ "QTime":13,}}
\{{ "params":}}{{
{ "facet.limit":"5", "q":"*:*", "facet.field":"facet_t_sort", "rows":"0",
"facet":"on"}
}}{{},}}
\{{ "response":}}{{
{"numFound":102,"start":0,"maxScore":1.0,"docs":[] }
}}{{,}}
\{{ "facet_counts":{}}
\{{ "facet_queries":{},}}
\{{ "facet_fields":}}\{{{}}
\{{ "facet_t_sort":[ }}
\{{ "a b",36, }}
\{{ "a b 0",1, }}
\{{ "a b 1",1, }}
\{{ "a b 10",1, }}
{{ "a b 11",1]}}}{{,}}
\{{ "facet_ranges":{},}}
\{{ "facet_intervals":{},}}
\{{ "facet_heatmaps":{} } }}}
The problem is the second phase of simple faceting, where the fine-counting
happens. In the first phase, "a b" is returned from 1 or 2 of the 3 shards. It
wins the popularity contest as there are 2 "a b"-terms and only 1 of all the
other terms. The 1 or 2 shards that did not deliver "a b" in the first phase
are then queried for the count for "a b", which happens in the form of a
{{facet_t_sort:"a b"}}-lookup. It seems that this lookup uses the analyzer
chain and thus matches _all_ the documents in that shard (approximately 102/3).
was:
Using {{SortableTextField}} for distributed faceting can lead to wrong results.
This can be demonstrated by installing the cloud-version of the
{{gettingstarted}} sample with
{{./solr -e cloud}}
using defaults all the way, except for {{shards}} which should be {{3}}. After
that a corpus can be indexed with
{{( echo '[' ; for J in $(seq 0 99); do ID=$((J)) ; echo
"\\{\"id\":\"$ID\",\"facet_t_sort\":\"a b $J\"},"; done ; echo
'\\{"id":"duplicate_1","facet_t_sort":"a
b"},\\{"id":"duplicate_2","facet_t_sort":"a b"}]' ) | curl -s -d @- -X POST -H
'Content-Type: application/json'
'http://localhost:8983/solr/gettingstarted/update?commit=true'}}
This will index 100 documents with a single-valued field {{facet_t_sort:"a b
X"}} where X is the document number + 2 documents with {{facet_t_sort:"a b"}}.
The call
{{curl
'http://localhost:8983/solr/gettingstarted/select?facet.field=facet_t_sort=5=on=**:**=0'}}
should return "a b" as the top facet term with count 2, but returns
{{{}}
{{ "responseHeader":{}}
{{ "zkConnected":true,}}
{{ "status":0,}}
{{ "QTime":13,}}
{{ "params":}}{{{ "facet.limit":"5", "q":"*:*", "facet.field":"facet_t_sort",
"rows":"0", "facet":"on"}}}{{},}}
{{ "response":}}{{{"numFound":102,"start":0,"maxScore":1.0,"docs":[] }}}{{,}}
{{ "facet_counts":{}}
{{ "facet_queries":{},}}
{{ "facet_fields":}}{{{}}
{{ "facet_t_sort":[ }}
{{ "a b",36, }}
{{ "a b 0",1, }}
{{ "a b 1",1, }}
{{ "a b 10",1, }}
{{ "a b 11",1]}}}{{,}}
{{ "facet_ranges":{},}}
{{ "facet_intervals":{},}}
{{ "facet_heatmaps":{} } }}}
The problem is the second phase of simple faceting, wher
[jira] [Updated] (SOLR-13056) SortableTextField is trappy for faceting
[
https://issues.apache.org/jira/browse/SOLR-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Toke Eskildsen updated SOLR-13056:
--
Description:
Using {{SortableTextField}} for distributed faceting can lead to wrong results.
This can be demonstrated by installing the cloud-version of the
{{gettingstarted}} sample with
{{./solr -e cloud}}
using defaults all the way, except for {{shards}} which should be {{3}}. After
that a corpus can be indexed with
{{( echo '[' ; for J in $(seq 0 99); do ID=$((J)) ; echo
"\{\"id\":\"$ID\",\"facet_t_sort\":\"a b $J\"},"; done ; echo
'\{"id":"duplicate_1","facet_t_sort":"a
b"},\{"id":"duplicate_2","facet_t_sort":"a b"}]' ) | curl -s -d @- -X POST -H
'Content-Type: application/json'
'http://localhost:8983/solr/gettingstarted/update?commit=true'}}
This will index 100 documents with a single-valued field {{facet_t_sort:"a b
X"}} where X is the document number + 2 documents with {{facet_t_sort:"a b"}}.
The call
{{curl
'http://localhost:8983/solr/gettingstarted/select?facet.field=facet_t_sort=5=on=**:**=0'}}
should return "a b" as the top facet term with count 2, but returns
{{ {}}
{{ "responseHeader":{}}
{{ "zkConnected":true,}}
{{ "status":0,}}
{{ "QTime":13,}}
{{ "params":{}}
{{ "facet.limit":"5",}}
{{ "q":":",}}
{{ "facet.field":"facet_t_sort",}}
{{ "rows":"0",}}
{{ "facet":"on"} },}}
{{ "response":{"numFound":102,"start":0,"maxScore":1.0,"docs":[]}}
{{ },}}
{{ "facet_counts":{}}
{{ "facet_queries":{},}}
{{ "facet_fields":{}}
{{ "facet_t_sort":[}}
{{ "a b",36,}}
{{ "a b 0",1,}}
{{ "a b 1",1,}}
{{ "a b 10",1,}}
{{ "a b 11",1]},}}
{{ "facet_ranges":{},}}
{{ "facet_intervals":{},}}
{{ "facet_heatmaps":{} } } }}
The problem is the second phase of simple faceting, where the fine-counting
happens. In the first phase, "a b" is returned from 1 or 2 of the 3 shards. It
wins the popularity contest as there are 2 "a b"-terms and only 1 of all the
other terms. The 1 or 2 shards that did not deliver "a b" in the first phase
are then queried for the count for "a b", which happens in the form of a
{{facet_t_sort:"a b"}}-lookup. It seems that this lookup uses the analyzer
chain and thus matches _all_ the documents in that shard (approximately 102/3).
was:
Using {{SortableTextField}} for distributed faceting can lead to wrong results.
This can be demonstrated by installing the cloud-version of the
{{gettingstarted}} sample with
{{./solr -e cloud}}
using defaults all the way, except for {{shards}} which should be {{3}}. After
that a corpus can be indexed with
{{ ( echo '[' ; for J in $(seq 0 99); do ID=$((J)) ; echo
"\\{\"id\":\"$ID\",\"facet_t_sort\":\"a b $J\"},"; done ; echo
'\\{"id":"duplicate_1","facet_t_sort":"a
b"},\\{"id":"duplicate_2","facet_t_sort":"a b"}]' ) | curl -s -d @- -X POST -H
'Content-Type: application/json'
'http://localhost:8983/solr/gettingstarted/update?commit=true' }}
This will index 100 documents with a single-valued field {{facet_t_sort:"a b
X"}} where X is the document number + 2 documents with {{facet_t_sort:"a b"}}.
The call
{{curl
'http://localhost:8983/solr/gettingstarted/select?facet.field=facet_t_sort=5=on=**:**=0'}}
should return "a b" as the top facet term with count 2, but returns
{\{{}}
\{{ "responseHeader":{}}
\{{ "zkConnected":true,}}
\{{ "status":0,}}
\{{ "QTime":13,}}
\{{ "params":}}{{
{ "facet.limit":"5", "q":"*:*", "facet.field":"facet_t_sort", "rows":"0",
"facet":"on"}
}}{{},}}
\{{ "response":}}{{
{"numFound":102,"start":0,"maxScore":1.0,"docs":[] }
}}{{,}}
\{{ "facet_counts":{}}
\{{ "facet_queries":{},}}
\{{ "facet_fields":}}\{{{}}
\{{ "facet_t_sort":[ }}
\{{ "a b",36, }}
\{{ "a b 0",1, }}
\{{ "a b 1",1, }}
\{{ "a b 10",1, }}
{{ "a b 11",1]}}}{{,}}
\{{ "facet_ranges":{},}}
\{{ "facet_intervals":{},}}
\{{ "facet_heatmaps":{} } }}}
The problem is the second phase of simple faceting, where the fine-
[jira] [Created] (SOLR-13056) SortableTextField is trappy for faceting
Toke Eskildsen created SOLR-13056:
-
Summary: SortableTextField is trappy for faceting
Key: SOLR-13056
URL: https://issues.apache.org/jira/browse/SOLR-13056
Project: Solr
Issue Type: Bug
Security Level: Public (Default Security Level. Issues are Public)
Components: search
Affects Versions: 7.6
Reporter: Toke Eskildsen
Using {{SortableTextField}} for distributed faceting can lead to wrong results.
This can be demonstrated by installing the cloud-version of the
{{gettingstarted}} sample with
{{./solr -e cloud}}
using defaults all the way, except for {{shards}} which should be {{3}}. After
that a corpus can be indexed with
{{( echo '[' ; for J in $(seq 0 99); do ID=$((J)) ; echo
"\\{\"id\":\"$ID\",\"facet_t_sort\":\"a b $J\"},"; done ; echo
'\\{"id":"duplicate_1","facet_t_sort":"a
b"},\\{"id":"duplicate_2","facet_t_sort":"a b"}]' ) | curl -s -d @- -X POST -H
'Content-Type: application/json'
'http://localhost:8983/solr/gettingstarted/update?commit=true'}}
This will index 100 documents with a single-valued field {{facet_t_sort:"a b
X"}} where X is the document number + 2 documents with {{facet_t_sort:"a b"}}.
The call
{{curl
'http://localhost:8983/solr/gettingstarted/select?facet.field=facet_t_sort=5=on=**:**=0'}}
should return "a b" as the top facet term with count 2, but returns
{{{}}
{{ "responseHeader":{}}
{{ "zkConnected":true,}}
{{ "status":0,}}
{{ "QTime":13,}}
{{ "params":}}{{{ "facet.limit":"5", "q":"*:*", "facet.field":"facet_t_sort",
"rows":"0", "facet":"on"}}}{{},}}
{{ "response":}}{{{"numFound":102,"start":0,"maxScore":1.0,"docs":[] }}}{{,}}
{{ "facet_counts":{}}
{{ "facet_queries":{},}}
{{ "facet_fields":}}{{{}}
{{ "facet_t_sort":[ }}
{{ "a b",36, }}
{{ "a b 0",1, }}
{{ "a b 1",1, }}
{{ "a b 10",1, }}
{{ "a b 11",1]}}}{{,}}
{{ "facet_ranges":{},}}
{{ "facet_intervals":{},}}
{{ "facet_heatmaps":{} } }}}
The problem is the second phase of simple faceting, where the fine-counting
happens. In the first phase, "a b" is returned from 1 or 2 of the 3 shards. It
wins the popularity contest as there are 2 "a b"-terms and only 1 of all the
other terms. The 1 or 2 shards that did not deliver "a b" in the first phase
are then queried for the count for "a b", which happens in the form of a
{{facet_t_sort:"a b"}}-lookup. It seems that this lookup uses the analyzer
chain and thus matches _all_ the documents in that shard (approximately 102/3).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Issue Comment Deleted] (SOLR-8362) Add docValues support for TextField
[
https://issues.apache.org/jira/browse/SOLR-8362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Toke Eskildsen updated SOLR-8362:
-
Comment: was deleted
(was: [~hossman] using this field type for distributed faceting can lead to
wrong results. Maybe this should be noted in the JavaDoc or the Solr
documentation?
This can be demonstrated by installing the cloud-version of the
{{gettingstarted}} sample with
{{./solr -e cloud}}
using defaults all the way, except for {{shards}} which should be {{3}}. After
that a corpus can be indexed with
{{( echo '[' ; for J in $(seq 0 99); do ID=$((J)) ; echo
"\{\"id\":\"$ID\",\"facet_t_sort\":\"a b $J\"},"; done ; echo
'\{"id":"duplicate_1","facet_t_sort":"a
b"},\{"id":"duplicate_2","facet_t_sort":"a b"}]' ) | curl -s -d @- -X POST -H
'Content-Type: application/json'
'http://localhost:8983/solr/gettingstarted/update?commit=true'}}
This will index 100 documents with a single-valued field {{facet_t_sort:"a b
X"}} where X is the document number + 2 documents with {{facet_t_sort:"a b"}}.
The call
curl
'http://localhost:8983/solr/gettingstarted/select?facet.field=facet_t_sort=5=on=*:*=0'
should return "a b" as the top facet term with count 2, but returns
{{{}}
{{ "responseHeader":{}}
{{ "zkConnected":true,}}
{{ "status":0,}}
{{ "QTime":13,}}
{{ "params":{}}
{{ "facet.limit":"5",}}
{{ "q":"*:*",}}
{{ "facet.field":"facet_t_sort",}}
{{ "rows":"0",}}
{{ "facet":"on"}},}}
{{ "response":{"numFound":102,"start":0,"maxScore":1.0,"docs":[]}}
{{ },}}
{{ "facet_counts":{}}
{{ "facet_queries":{},}}
{{ "facet_fields":{}}
{{ "facet_t_sort":[}}
{{ "a b",36,}}
{{ "a b 0",1,}}
{{ "a b 1",1,}}
{{ "a b 10",1,}}
{{ "a b 11",1]},}}
{{ "facet_ranges":{},}}
{{ "facet_intervals":{},}}
{{ "facet_heatmaps":{}
The problem is the second phase of simple faceting, where the fine-counting
happens. In the first phase, "a b" is returned from 1 or 2 of the 3 shards. It
wins the popularity contest as there are 2 "a b"-terms and only 1 of all the
other terms. The 1 or 2 shards that did not deliver "a b" in the first phase
are then queried for the count for "a b", which happens in the form of a
{{facet_t_sort:"a b"}}-lookup. It seems that this lookup uses the analyzer
chain and thus matches _all_ the documents in that shard (approximately 102/3).
An alternative would be to do the fine-counting on the DocValues instead, but
that works very poorly with many values, so that seems more like a trap than a
solution.)
> Add docValues support for TextField
> ---
>
> Key: SOLR-8362
> URL: https://issues.apache.org/jira/browse/SOLR-8362
> Project: Solr
> Issue Type: Improvement
>Reporter: Hoss Man
>Priority: Major
>
> At the last lucene/solr revolution, Toke asked a question about why TextField
> doesn't support docValues. The short answer is because no one ever added it,
> but the longer answer was because we would have to think through carefully
> the _intent_ of supporting docValues for a "tokenized" field like TextField,
> and how to support various conflicting usecases where they could be handy.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11916) new SortableTextField using docValues built from the original string input
[
https://issues.apache.org/jira/browse/SOLR-11916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715516#comment-16715516
]
Toke Eskildsen commented on SOLR-11916:
---
[~hossman] using this field type for distributed faceting can lead to wrong
results. Maybe this should be noted in the JavaDoc or the Solr documentation?
This can be demonstrated by installing the cloud-version of the
{{gettingstarted}} sample with
{{./solr -e cloud}}
using defaults all the way, except for {{shards}} which should be {{3}}. After
that a corpus can be indexed with
{{( echo '[' ; for J in $(seq 0 99); do ID=$((J)) ; echo
"\{\"id\":\"$ID\",\"facet_t_sort\":\"a b $J\"},"; done ; echo
'\{"id":"duplicate_1","facet_t_sort":"a
b"},\{"id":"duplicate_2","facet_t_sort":"a b"}]' ) | curl -s -d @- -X POST -H
'Content-Type: application/json'
'http://localhost:8983/solr/gettingstarted/update?commit=true'}}
This will index 100 documents with a single-valued field {{facet_t_sort:"a b
X"}} where X is the document number + 2 documents with {{facet_t_sort:"a b"}}.
The call
curl
'http://localhost:8983/solr/gettingstarted/select?facet.field=facet_t_sort=5=on=*:*=0'
should return "a b" as the top facet term with count 2, but returns
{{{}}
{{ "responseHeader":{}}
{{ "zkConnected":true,}}
{{ "status":0,}}
{{ "QTime":13,}}
{{ "params":{}}
{{ "facet.limit":"5",}}
{{ "q":"*:*",}}
{{ "facet.field":"facet_t_sort",}}
{{ "rows":"0",}}
{{ "facet":"on"}},}}
{{ "response":{"numFound":102,"start":0,"maxScore":1.0,"docs":[]}}
{{ },}}
{{ "facet_counts":{}}
{{ "facet_queries":{},}}
{{ "facet_fields":{}}
{{ "facet_t_sort":[}}
{{ "a b",36,}}
{{ "a b 0",1,}}
{{ "a b 1",1,}}
{{ "a b 10",1,}}
{{ "a b 11",1]},}}
{{ "facet_ranges":{},}}
{{ "facet_intervals":{},}}
{{ "facet_heatmaps":{}
The problem is the second phase of simple faceting, where the fine-counting
happens. In the first phase, "a b" is returned from 1 or 2 of the 3 shards. It
wins the popularity contest as there are 2 "a b"-terms and only 1 of all the
other terms. The 1 or 2 shards that did not deliver "a b" in the first phase
are then queried for the count for "a b", which happens in the form of a
{{facet_t_sort:"a b"}}-lookup. It seems that this lookup uses the analyzer
chain and thus matches _all_ the documents in that shard (approximately 102/3).
An alternative would be to do the fine-counting on the DocValues instead, but
that works very poorly with many values, so that seems more like a trap than a
solution.
> new SortableTextField using docValues built from the original string input
> --
>
> Key: SOLR-11916
> URL: https://issues.apache.org/jira/browse/SOLR-11916
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: Schema and Analysis
>Reporter: Hoss Man
>Assignee: Hoss Man
>Priority: Major
> Fix For: 7.3, master (8.0)
>
> Attachments: SOLR-11916.patch, SOLR-11916.patch
>
>
> I propose adding a new SortableTextField subclass that would functionally
> work the same as TextField except:
> * {{docValues="true|false"}} could be configured, with the default being
> "true"
> * The docValues would contain the original input values (just like StrField)
> for sorting (or faceting)
> ** By default, to protect users from excessively large docValues, only the
> first 1024 of each field value would be used – but this could be overridden
> with configuration.
>
> Consider the following sample configuration:
> {code:java}
> indexed="true" docValues="true" stored="true" multiValued="false"/>
>
>
>...
>
>
>...
>
>
> {code}
> Given a document with a title of "Solr In Action"
> Users could:
> * Search for individual (indexed) terms in the "title" field:
> {{q=title:solr}}
> * Sort documents by title ( {{sort=title asc}} ) such that this document's
> sort value would be "Solr In Action"
> If another document had a "title" value that was longer then
[jira] [Comment Edited] (SOLR-8362) Add docValues support for TextField
[
https://issues.apache.org/jira/browse/SOLR-8362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715478#comment-16715478
]
Toke Eskildsen edited comment on SOLR-8362 at 12/10/18 7:53 PM:
[~hossman] using this field type for distributed faceting can lead to wrong
results. Maybe this should be noted in the JavaDoc or the Solr documentation?
This can be demonstrated by installing the cloud-version of the
{{gettingstarted}} sample with
{{./solr -e cloud}}
using defaults all the way, except for {{shards}} which should be {{3}}. After
that a corpus can be indexed with
{{( echo '[' ; for J in $(seq 0 99); do ID=$((J)) ; echo
"\{\"id\":\"$ID\",\"facet_t_sort\":\"a b $J\"},"; done ; echo
'\{"id":"duplicate_1","facet_t_sort":"a
b"},\{"id":"duplicate_2","facet_t_sort":"a b"}]' ) | curl -s -d @- -X POST -H
'Content-Type: application/json'
'http://localhost:8983/solr/gettingstarted/update?commit=true'}}
This will index 100 documents with a single-valued field {{facet_t_sort:"a b
X"}} where X is the document number + 2 documents with {{facet_t_sort:"a b"}}.
The call
curl
'http://localhost:8983/solr/gettingstarted/select?facet.field=facet_t_sort=5=on=*:*=0'
should return "a b" as the top facet term with count 2, but returns
{{{}}
{{ "responseHeader":{}}
{{ "zkConnected":true,}}
{{ "status":0,}}
{{ "QTime":13,}}
{{ "params":{}}
{{ "facet.limit":"5",}}
{{ "q":"*:*",}}
{{ "facet.field":"facet_t_sort",}}
{{ "rows":"0",}}
{{ "facet":"on"}},}}
{{ "response":{"numFound":102,"start":0,"maxScore":1.0,"docs":[]}}
{{ },}}
{{ "facet_counts":{}}
{{ "facet_queries":{},}}
{{ "facet_fields":{}}
{{ "facet_t_sort":[}}
{{ "a b",36,}}
{{ "a b 0",1,}}
{{ "a b 1",1,}}
{{ "a b 10",1,}}
{{ "a b 11",1]},}}
{{ "facet_ranges":{},}}
{{ "facet_intervals":{},}}
{{ "facet_heatmaps":{}
The problem is the second phase of simple faceting, where the fine-counting
happens. In the first phase, "a b" is returned from 1 or 2 of the 3 shards. It
wins the popularity contest as there are 2 "a b"-terms and only 1 of all the
other terms. The 1 or 2 shards that did not deliver "a b" in the first phase
are then queried for the count for "a b", which happens in the form of a
{{facet_t_sort:"a b"}}-lookup. It seems that this lookup uses the analyzer
chain and thus matches _all_ the documents in that shard (approximately 102/3).
An alternative would be to do the fine-counting on the DocValues instead, but
that works very poorly with many values, so that seems more like a trap than a
solution.
was (Author: toke):
[~hossman] using this field type for distributed faceting can lead to wrong
results. Maybe this should be noted in the JavaDoc or the Solr documentation?
This can be demonstrated by installing the cloud-version of the
{{gettingstarted}} sample with
{{./solr -e cloud}}
using defaults all the way, except for {{shards}} which should be {{3}}. After
that a corpus can be indexed with
{{( echo '[' ; for J in $(seq 0 99); do ID=$((J)) ; echo
"\\{\"id\":\"$ID\",\"facet_t_sort\":\"a b $J\"},"; done ; echo
'\\{"id":"duplicate_1","facet_t_sort":"a
b"},\\{"id":"duplicate_2","facet_t_sort":"a b"}]' ) | curl -s -d @- -X POST -H
'Content-Type: application/json'
'http://localhost:8983/solr/gettingstarted/update?commit=true'}}
This will index 100 documents with a single-valued field {{facet_t_sort:"a b
X"}} where X is the document number + 2 documents with {{facet_t_sort:"a b"}}.
The call
curl
'http://localhost:8983/solr/gettingstarted/select?facet.field=facet_t_sort=5=on=*:*=0'
should return "a b" as the top facet term with count 2, but returns
{{{}}
{{ "responseHeader":{}}
{{ "zkConnected":true,}}
{{ "status":0,}}
{{ "QTime":13,}}
{{ "params":{}}
{{ "facet.limit":"5",}}
{{ "q":"*:*",}}
{{ "facet.field":"facet_t_sort",}}
{{ "rows":"0",}}
{{ "facet":"on"}},}}
{{ "response":{"numFound":102,"start":0,"maxScore":1.0,"docs":[]}}
{{ },}}
{{ "facet_counts":{}}
{{ "facet_queries":{},}}
{{ "facet_fields":{}}
{{ "facet_
[jira] [Commented] (SOLR-8362) Add docValues support for TextField
[
https://issues.apache.org/jira/browse/SOLR-8362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715478#comment-16715478
]
Toke Eskildsen commented on SOLR-8362:
--
[~hossman] using this field type for distributed faceting can lead to wrong
results. Maybe this should be noted in the JavaDoc or the Solr documentation?
This can be demonstrated by installing the cloud-version of the
{{gettingstarted}} sample with
{{./solr -e cloud}}
using defaults all the way, except for {{shards}} which should be {{3}}. After
that a corpus can be indexed with
{{( echo '[' ; for J in $(seq 0 99); do ID=$((J)) ; echo
"\\{\"id\":\"$ID\",\"facet_t_sort\":\"a b $J\"},"; done ; echo
'\\{"id":"duplicate_1","facet_t_sort":"a
b"},\\{"id":"duplicate_2","facet_t_sort":"a b"}]' ) | curl -s -d @- -X POST -H
'Content-Type: application/json'
'http://localhost:8983/solr/gettingstarted/update?commit=true'}}
This will index 100 documents with a single-valued field {{facet_t_sort:"a b
X"}} where X is the document number + 2 documents with {{facet_t_sort:"a b"}}.
The call
curl
'http://localhost:8983/solr/gettingstarted/select?facet.field=facet_t_sort=5=on=*:*=0'
should return "a b" as the top facet term with count 2, but returns
{{{}}
{{ "responseHeader":{}}
{{ "zkConnected":true,}}
{{ "status":0,}}
{{ "QTime":13,}}
{{ "params":{}}
{{ "facet.limit":"5",}}
{{ "q":"*:*",}}
{{ "facet.field":"facet_t_sort",}}
{{ "rows":"0",}}
{{ "facet":"on"}},}}
{{ "response":{"numFound":102,"start":0,"maxScore":1.0,"docs":[]}}
{{ },}}
{{ "facet_counts":{}}
{{ "facet_queries":{},}}
{{ "facet_fields":{}}
{{ "facet_t_sort":[}}
{{ "a b",36,}}
{{ "a b 0",1,}}
{{ "a b 1",1,}}
{{ "a b 10",1,}}
{{ "a b 11",1]},}}
{{ "facet_ranges":{},}}
{{ "facet_intervals":{},}}
{{ "facet_heatmaps":{}
The problem is the second phase of simple faceting, where the fine-counting
happens. In the first phase, "a b" is returned from 1 or 2 of the 3 shards. It
wins the popularity contest as there are 2 "a b"-terms and only 1 of all the
other terms. The 1 or 2 shards that did not deliver "a b" in the first phase
are then queried for the count for "a b", which happens in the form of a
{{facet_t_sort:"a b"}}-lookup. It seems that this lookup uses the analyzer
chain and thus matches _all_ the documents in that shard (approximately 102/3).
An alternative would be to do the fine-counting on the DocValues instead, but
that works very poorly with many values, so that seems more like a trap than a
solution.
> Add docValues support for TextField
> ---
>
> Key: SOLR-8362
> URL: https://issues.apache.org/jira/browse/SOLR-8362
> Project: Solr
> Issue Type: Improvement
>Reporter: Hoss Man
>Priority: Major
>
> At the last lucene/solr revolution, Toke asked a question about why TextField
> doesn't support docValues. The short answer is because no one ever added it,
> but the longer answer was because we would have to think through carefully
> the _intent_ of supporting docValues for a "tokenized" field like TextField,
> and how to support various conflicting usecases where they could be handy.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8374) Reduce reads for sparse DocValues
[
https://issues.apache.org/jira/browse/LUCENE-8374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713665#comment-16713665
]
Toke Eskildsen commented on LUCENE-8374:
I kinda guessed you were, [~jpountz] ;). I'll revert the changes. I presume
that I should use the {{git revert}} for the 4 commits in reverse order? As far
as I understand, this leaves a proper git history.
> Reduce reads for sparse DocValues
> -
>
> Key: LUCENE-8374
> URL: https://issues.apache.org/jira/browse/LUCENE-8374
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
>Affects Versions: 7.5, master (8.0)
> Reporter: Toke Eskildsen
>Priority: Major
> Labels: performance
> Attachments: LUCENE-8374.patch, LUCENE-8374.patch, LUCENE-8374.patch,
> LUCENE-8374.patch, LUCENE-8374.patch, LUCENE-8374.patch, LUCENE-8374.patch,
> LUCENE-8374_branch_7_3.patch, LUCENE-8374_branch_7_3.patch.20181005,
> LUCENE-8374_branch_7_4.patch, LUCENE-8374_branch_7_5.patch,
> LUCENE-8374_part_1.patch, LUCENE-8374_part_2.patch, LUCENE-8374_part_3.patch,
> LUCENE-8374_part_4.patch, entire_index_logs.txt,
> image-2018-10-24-07-30-06-663.png, image-2018-10-24-07-30-56-962.png,
> single_vehicle_logs.txt,
> start-2018-10-24-1_snapshot___Users_tim_Snapshots__-_YourKit_Java_Profiler_2017_02-b75_-_64-bit.png,
>
> start-2018-10-24_snapshot___Users_tim_Snapshots__-_YourKit_Java_Profiler_2017_02-b75_-_64-bit.png
>
>
> The {{Lucene70DocValuesProducer}} has the internal classes
> {{SparseNumericDocValues}} and {{BaseSortedSetDocValues}} (sparse code path),
> which again uses {{IndexedDISI}} to handle the docID -> value-ordinal lookup.
> The value-ordinal is the index of the docID assuming an abstract tightly
> packed monotonically increasing list of docIDs: If the docIDs with
> corresponding values are {{[0, 4, 1432]}}, their value-ordinals will be {{[0,
> 1, 2]}}.
> h2. Outer blocks
> The lookup structure of {{IndexedDISI}} consists of blocks of 2^16 values
> (65536), where each block can be either {{ALL}}, {{DENSE}} (2^12 to 2^16
> values) or {{SPARSE}} (< 2^12 values ~= 6%). Consequently blocks vary quite a
> lot in size and ordinal resolving strategy.
> When a sparse Numeric DocValue is needed, the code first locates the block
> containing the wanted docID flag. It does so by iterating blocks one-by-one
> until it reaches the needed one, where each iteration requires a lookup in
> the underlying {{IndexSlice}}. For a common memory mapped index, this
> translates to either a cached request or a read operation. If a segment has
> 6M documents, worst-case is 91 lookups. In our web archive, our segments has
> ~300M values: A worst-case of 4577 lookups!
> One obvious solution is to use a lookup-table for blocks: A long[]-array with
> an entry for each block. For 6M documents, that is < 1KB and would allow for
> direct jumping (a single lookup) in all instances. Unfortunately this
> lookup-table cannot be generated upfront when the writing of values is purely
> streaming. It can be appended to the end of the stream before it is closed,
> but without knowing the position of the lookup-table the reader cannot seek
> to it.
> One strategy for creating such a lookup-table would be to generate it during
> reads and cache it for next lookup. This does not fit directly into how
> {{IndexedDISI}} currently works (it is created anew for each invocation), but
> could probably be added with a little work. An advantage to this is that this
> does not change the underlying format and thus could be used with existing
> indexes.
> h2. The lookup structure inside each block
> If {{ALL}} of the 2^16 values are defined, the structure is empty and the
> ordinal is simply the requested docID with some modulo and multiply math.
> Nothing to improve there.
> If the block is {{DENSE}} (2^12 to 2^16 values are defined), a bitmap is used
> and the number of set bits up to the wanted index (the docID modulo the block
> origo) are counted. That bitmap is a long[1024], meaning that worst case is
> to lookup and count all set bits for 1024 longs!
> One known solution to this is to use a [rank
> structure|[https://en.wikipedia.org/wiki/Succinct_data_structure]]. I
> [implemented
> it|[https://github.com/tokee/lucene-solr/blob/solr5894/solr/core/src/java/org/apache/solr/search/sparse/count/plane/RankCache.java]]
> for a related project and with that (), the rank-overhead for a {{DENSE}}
> block would be long[32] and would ensure a maximum of 9 lookups. It is not
> trivial to build the rank-struct
[jira] [Commented] (LUCENE-8585) Create jump-tables for DocValues at index-time
[
https://issues.apache.org/jira/browse/LUCENE-8585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16712878#comment-16712878
]
Toke Eskildsen commented on LUCENE-8585:
Thank you for the clarifications, [~jpountz].
Regarding where to put the jump-data:
{quote}If the access pattern is sequential, which I assume would be the case in
both cases, then it's fine to keep them on storage.
{quote}
Well, that really depends on the access pattern from the outside ;). But as the
jump-entries are stored sequentially then a request hitting a smaller subset of
the documents in a manner that will benefit from jumps means that the
jump-entries will be accessed in increasing order. They won't be used if the
jumps are within the current block or to the block immediately following the
current one.
{quote}We can also move the 7.0 format to lucene/backward-codecs since
lucene/core only keeps formats that are used for the current codec.
{quote}
Before I began there was a single file {{Lucene80Codec.java}} in the
{{lucene80}} package, picking codec-parts from both 50, 60 and 70. After having
implemented the jumps, I have not touched the {{Lucene70Norms*}}-part. I
_guess_ I should move the {{Lucene70DocValues*}}-files from {{lucene70}} to
{{backward-codecs}}, leaving the norms-classes?
Since the norms-classes also uses {{IndexedDISI}}, I expect it would be best to
upgrade them too. This would leave the core {{lucene70}} folder empty of active
code.
{quote}If you move the 7.0 format to lucene/backward-codecs, then you'll need
to move it to
lucene/backward-codecs/src/resources/META-INF/services/org.apache.lucene.codecs.DocValuesFormat.
{quote}
That makes sense, thanks!
> Create jump-tables for DocValues at index-time
> --
>
> Key: LUCENE-8585
> URL: https://issues.apache.org/jira/browse/LUCENE-8585
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
>Affects Versions: master (8.0)
> Reporter: Toke Eskildsen
>Priority: Minor
> Labels: performance
> Attachments: LUCENE-8585.patch, make_patch_lucene8585.sh
>
>
> As noted in LUCENE-7589, lookup of DocValues should use jump-tables to avoid
> long iterative walks. This is implemented in LUCENE-8374 at search-time
> (first request for DocValues from a field in a segment), with the benefit of
> working without changes to existing Lucene 7 indexes and the downside of
> introducing a startup time penalty and a memory overhead.
> As discussed in LUCENE-8374, the codec should be updated to create these
> jump-tables at index time. This eliminates the segment-open time & memory
> penalties, with the potential downside of increasing index-time for DocValues.
> The three elements of LUCENE-8374 should be transferable to index-time
> without much alteration of the core structures:
> * {{IndexedDISI}} block offset and index skips: A {{long}} (64 bits) for
> every 65536 documents, containing the offset of the block in 33 bits and the
> index (number of set bits) up to the block in 31 bits.
> It can be build sequentially and should be stored as a simple sequence of
> consecutive longs for caching of lookups.
> As it is fairly small, relative to document count, it might be better to
> simply memory cache it?
> * {{IndexedDISI}} DENSE (> 4095, < 65536 set bits) blocks: A {{short}} (16
> bits) for every 8 {{longs}} (512 bits) for a total of 256 bytes/DENSE_block.
> Each {{short}} represents the number of set bits up to right before the
> corresponding sub-block of 512 docIDs.
> The \{{shorts}} can be computed sequentially or when the DENSE block is
> flushed (probably the easiest). They should be stored as a simple sequence of
> consecutive shorts for caching of lookups, one logically independent sequence
> for each DENSE block. The logical position would be one sequence at the start
> of every DENSE block.
> Whether it is best to read all the 16 {{shorts}} up front when a DENSE block
> is accessed or whether it is best to only read any individual {{short}} when
> needed is not clear at this point.
> * Variable Bits Per Value: A {{long}} (64 bits) for every 16384 numeric
> values. Each {{long}} holds the offset to the corresponding block of values.
> The offsets can be computed sequentially and should be stored as a simple
> sequence of consecutive {{longs}} for caching of lookups.
> The vBPV-offsets has the largest space overhead og the 3 jump-tables and a
> lot of the 64 bits in each long are not used for most indexes. They could be
> represented as a simple {{PackedInts}} sequence or {{MonotonicLongValues}},
> with the downsides of a p
[jira] [Commented] (LUCENE-8374) Reduce reads for sparse DocValues
[
https://issues.apache.org/jira/browse/LUCENE-8374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16712431#comment-16712431
]
Toke Eskildsen commented on LUCENE-8374:
I have my hands full with LUCENE-8585 (index-time DV jump-tables), which seems
to be coming along nicely. With the expectation that it will be in a
reviewable/testable state in the near future and (hope hope hope) be in Lucene
7.7+, some decision should be made on LUCENE-8374.
# I can let it stay and let it be further tested & bugfixed. I would like this
for the reasons stated earlier, but I accept that there is no consensus for
this.
# I can make it an optional expert/experimental feature, with the default
being off (and add opt-in in Solr config so it is easily usable there). As the
jump-table using parts of the codecs are extremely localized (more or less
{{jumpIfPossible(...)}} followed by the old code for fallback), I believe this
can be done in a manner that is both very low-risk and easy to review. I like
this solution.
# I can do a full revert when we have LUCENE-8585. Obviously I don't like
this, but I accept the validity of the argument that the value of a plug'n'play
performance upgrade diminishes greatly when a technically better
plug'n'index-upgrade'n'play performance solution exists.
# I can do a full revert ASAP. I am against this as I see DV-scaling as very
poor without jump-tables and want to be sure we have jump-tables from here on
out: Keep it until a better alternative is in place.
Under the assumption of a soon-to-be working LUCENE-8585, it is my impression
that #3 or #4 are the most favored actions. Until I learn otherwise, I'll go
with #3 and pour my focus into LUCENE-8585.
> Reduce reads for sparse DocValues
> -
>
> Key: LUCENE-8374
> URL: https://issues.apache.org/jira/browse/LUCENE-8374
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
>Affects Versions: 7.5, master (8.0)
> Reporter: Toke Eskildsen
>Priority: Major
> Labels: performance
> Attachments: LUCENE-8374.patch, LUCENE-8374.patch, LUCENE-8374.patch,
> LUCENE-8374.patch, LUCENE-8374.patch, LUCENE-8374.patch, LUCENE-8374.patch,
> LUCENE-8374_branch_7_3.patch, LUCENE-8374_branch_7_3.patch.20181005,
> LUCENE-8374_branch_7_4.patch, LUCENE-8374_branch_7_5.patch,
> LUCENE-8374_part_1.patch, LUCENE-8374_part_2.patch, LUCENE-8374_part_3.patch,
> LUCENE-8374_part_4.patch, entire_index_logs.txt,
> image-2018-10-24-07-30-06-663.png, image-2018-10-24-07-30-56-962.png,
> single_vehicle_logs.txt,
> start-2018-10-24-1_snapshot___Users_tim_Snapshots__-_YourKit_Java_Profiler_2017_02-b75_-_64-bit.png,
>
> start-2018-10-24_snapshot___Users_tim_Snapshots__-_YourKit_Java_Profiler_2017_02-b75_-_64-bit.png
>
>
> The {{Lucene70DocValuesProducer}} has the internal classes
> {{SparseNumericDocValues}} and {{BaseSortedSetDocValues}} (sparse code path),
> which again uses {{IndexedDISI}} to handle the docID -> value-ordinal lookup.
> The value-ordinal is the index of the docID assuming an abstract tightly
> packed monotonically increasing list of docIDs: If the docIDs with
> corresponding values are {{[0, 4, 1432]}}, their value-ordinals will be {{[0,
> 1, 2]}}.
> h2. Outer blocks
> The lookup structure of {{IndexedDISI}} consists of blocks of 2^16 values
> (65536), where each block can be either {{ALL}}, {{DENSE}} (2^12 to 2^16
> values) or {{SPARSE}} (< 2^12 values ~= 6%). Consequently blocks vary quite a
> lot in size and ordinal resolving strategy.
> When a sparse Numeric DocValue is needed, the code first locates the block
> containing the wanted docID flag. It does so by iterating blocks one-by-one
> until it reaches the needed one, where each iteration requires a lookup in
> the underlying {{IndexSlice}}. For a common memory mapped index, this
> translates to either a cached request or a read operation. If a segment has
> 6M documents, worst-case is 91 lookups. In our web archive, our segments has
> ~300M values: A worst-case of 4577 lookups!
> One obvious solution is to use a lookup-table for blocks: A long[]-array with
> an entry for each block. For 6M documents, that is < 1KB and would allow for
> direct jumping (a single lookup) in all instances. Unfortunately this
> lookup-table cannot be generated upfront when the writing of values is purely
> streaming. It can be appended to the end of the stream before it is closed,
> but without knowing the position of the lookup-table the reader cannot seek
> to it.
> One strategy for creating such a lookup-table would be to generate it during
> reads and cache it for nex
[jira] [Commented] (LUCENE-8585) Create jump-tables for DocValues at index-time
[
https://issues.apache.org/jira/browse/LUCENE-8585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16711062#comment-16711062
]
Toke Eskildsen commented on LUCENE-8585:
[~jpountz] the confusion probably stems from the fact that I was (and a little
bit still am) unsure if it is best to load the the {{IndexedDISI}} and {{vBPV}}
jump-tables to heap or keep them on storage, from a pure performance
perspective. Since I am leaning to keeping them on storage and you green light
my first take (append to data), everything's great.
I would appreciate a sanity check on how to change the codec:
I have copied {{LuceneDocValuesFormat}}, {{LuceneDocValuesConsumer}},
{{LuceneDocValuesProducer}} and {{IndexedDISI}} from {{lucene70}} to
{{lucene80}} and made the changes there. The test-class {{TestIndexedDISI}} is
specifically for the 7.0 version due to package privacy, so I copied that one
too. Was that correct? It seems a bit clunky to have a copy of a class with
only a few methods added, but on the other hand it does work well for
separation of codec versions.
I added {{org.apache.lucene.codecs.lucene80.Lucene80DocValuesFormat}} to the
{{org.apache.lucene.codecs.DocValuesFormat}} in {{META_INF/services}}, to get
the Lucene 80 codec to discover the new DocValues. I kept the old
{{org.apache.lucene.codecs.lucene70.Lucene70DocValuesFormat}} in the file. Was
that the right way to do it?
> Create jump-tables for DocValues at index-time
> --
>
> Key: LUCENE-8585
> URL: https://issues.apache.org/jira/browse/LUCENE-8585
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
>Affects Versions: master (8.0)
> Reporter: Toke Eskildsen
>Priority: Minor
>
> As noted in LUCENE-7589, lookup of DocValues should use jump-tables to avoid
> long iterative walks. This is implemented in LUCENE-8374 at search-time
> (first request for DocValues from a field in a segment), with the benefit of
> working without changes to existing Lucene 7 indexes and the downside of
> introducing a startup time penalty and a memory overhead.
> As discussed in LUCENE-8374, the codec should be updated to create these
> jump-tables at index time. This eliminates the segment-open time & memory
> penalties, with the potential downside of increasing index-time for DocValues.
> The three elements of LUCENE-8374 should be transferable to index-time
> without much alteration of the core structures:
> * {{IndexedDISI}} block offset and index skips: A {{long}} (64 bits) for
> every 65536 documents, containing the offset of the block in 33 bits and the
> index (number of set bits) up to the block in 31 bits.
> It can be build sequentially and should be stored as a simple sequence of
> consecutive longs for caching of lookups.
> As it is fairly small, relative to document count, it might be better to
> simply memory cache it?
> * {{IndexedDISI}} DENSE (> 4095, < 65536 set bits) blocks: A {{short}} (16
> bits) for every 8 {{longs}} (512 bits) for a total of 256 bytes/DENSE_block.
> Each {{short}} represents the number of set bits up to right before the
> corresponding sub-block of 512 docIDs.
> The \{{shorts}} can be computed sequentially or when the DENSE block is
> flushed (probably the easiest). They should be stored as a simple sequence of
> consecutive shorts for caching of lookups, one logically independent sequence
> for each DENSE block. The logical position would be one sequence at the start
> of every DENSE block.
> Whether it is best to read all the 16 {{shorts}} up front when a DENSE block
> is accessed or whether it is best to only read any individual {{short}} when
> needed is not clear at this point.
> * Variable Bits Per Value: A {{long}} (64 bits) for every 16384 numeric
> values. Each {{long}} holds the offset to the corresponding block of values.
> The offsets can be computed sequentially and should be stored as a simple
> sequence of consecutive {{longs}} for caching of lookups.
> The vBPV-offsets has the largest space overhead og the 3 jump-tables and a
> lot of the 64 bits in each long are not used for most indexes. They could be
> represented as a simple {{PackedInts}} sequence or {{MonotonicLongValues}},
> with the downsides of a potential lookup-time overhead and the need for doing
> the compression after all offsets has been determined.
> I have no experience with the codec-parts responsible for creating
> index-structures. I'm quite willing to take a stab at this, although I
> probably won't do much about it before January 2019. Should anyone else wish
> to adopt this JIRA-issue or co-work on it, I
[jira] [Commented] (LUCENE-8585) Create jump-tables for DocValues at index-time
[
https://issues.apache.org/jira/browse/LUCENE-8585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710601#comment-16710601
]
Toke Eskildsen commented on LUCENE-8585:
[~jim.ferenczi] you are off the hook :P. I did not plan on working on this, but
it seems that it must be done in order to get DV-scaling in some form into
Lucene for the release after 7.6. I have currently hacked {{IndexedDISI}} block
jump-tables as well as DENSE block ranking into the codec, but as can be seen
above, I might have stored it in the wrong slices. I hope to find the
time/energy to look at variable bits per value blocks tomorrow, but I kind of
marathoned the first parts today, so no promises.
As I did not know how to approach this logistically, I just used my own fork.
Should anyone be curious, it is on
https://github.com/tokee/lucene-solr/tree/lucene8585 - work in progress and all
that. Do you suggest that I make a pull-request from that one to the Lucene
GitHub repository in order to benefit from its fine tools for reviews etc.?
Apart from the ever-present need for reviewing at some point, some kind of
performance testing would be great, should anyone feel like it. Unfortunately
this is not a light task as an index of non-trivial size needs to be build (or
index-upgraded?).
> Create jump-tables for DocValues at index-time
> --
>
> Key: LUCENE-8585
> URL: https://issues.apache.org/jira/browse/LUCENE-8585
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
>Affects Versions: master (8.0)
> Reporter: Toke Eskildsen
>Priority: Minor
>
> As noted in LUCENE-7589, lookup of DocValues should use jump-tables to avoid
> long iterative walks. This is implemented in LUCENE-8374 at search-time
> (first request for DocValues from a field in a segment), with the benefit of
> working without changes to existing Lucene 7 indexes and the downside of
> introducing a startup time penalty and a memory overhead.
> As discussed in LUCENE-8374, the codec should be updated to create these
> jump-tables at index time. This eliminates the segment-open time & memory
> penalties, with the potential downside of increasing index-time for DocValues.
> The three elements of LUCENE-8374 should be transferable to index-time
> without much alteration of the core structures:
> * {{IndexedDISI}} block offset and index skips: A {{long}} (64 bits) for
> every 65536 documents, containing the offset of the block in 33 bits and the
> index (number of set bits) up to the block in 31 bits.
> It can be build sequentially and should be stored as a simple sequence of
> consecutive longs for caching of lookups.
> As it is fairly small, relative to document count, it might be better to
> simply memory cache it?
> * {{IndexedDISI}} DENSE (> 4095, < 65536 set bits) blocks: A {{short}} (16
> bits) for every 8 {{longs}} (512 bits) for a total of 256 bytes/DENSE_block.
> Each {{short}} represents the number of set bits up to right before the
> corresponding sub-block of 512 docIDs.
> The \{{shorts}} can be computed sequentially or when the DENSE block is
> flushed (probably the easiest). They should be stored as a simple sequence of
> consecutive shorts for caching of lookups, one logically independent sequence
> for each DENSE block. The logical position would be one sequence at the start
> of every DENSE block.
> Whether it is best to read all the 16 {{shorts}} up front when a DENSE block
> is accessed or whether it is best to only read any individual {{short}} when
> needed is not clear at this point.
> * Variable Bits Per Value: A {{long}} (64 bits) for every 16384 numeric
> values. Each {{long}} holds the offset to the corresponding block of values.
> The offsets can be computed sequentially and should be stored as a simple
> sequence of consecutive {{longs}} for caching of lookups.
> The vBPV-offsets has the largest space overhead og the 3 jump-tables and a
> lot of the 64 bits in each long are not used for most indexes. They could be
> represented as a simple {{PackedInts}} sequence or {{MonotonicLongValues}},
> with the downsides of a potential lookup-time overhead and the need for doing
> the compression after all offsets has been determined.
> I have no experience with the codec-parts responsible for creating
> index-structures. I'm quite willing to take a stab at this, although I
> probably won't do much about it before January 2019. Should anyone else wish
> to adopt this JIRA-issue or co-work on it, I'll be happy to share.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8585) Create jump-tables for DocValues at index-time
[
https://issues.apache.org/jira/browse/LUCENE-8585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710238#comment-16710238
]
Toke Eskildsen commented on LUCENE-8585:
Thank you for your suggestions, [~jpountz]. I am a bit surprised that you
advocate for having the jump-tables for {{IndexedDISI}} and {{vBPV}} on heap?
They are not that big, but it still means heap + extra startup-time to fetch
them!? It also means that it will be done for all fields, regardless of whether
they are used or not, if I understand Entry-loading correctly.
I have a hard time discerning what the principal difference is between this and
search-time build of the jump-tables. The only difference I see is that the
index-time version loads the data contiguously instead of having to visit every
block, which boils down to segment-open performance. I am not opposed to
pre-computing and having them on heap - if nothing else it will make lookups
slightly faster - I just can't follow the logic.
I tinkered a bit and got the {{IndexedDISI}} block jump-tables to work when
stored as I described in the first comment (after the regular blocks, accessed
as needed). The scary codec beast is not _that_ scary once you get to know it.
It should be easy enough to flush the data to meta instead.
As this has the potential to be more of a collaborative project, what is the
easiest way (ping to [~jim.ferenczi])? Sending patches back and forth seems a
bit heavy to me, so perhaps a git branch? Should I create one in the Apache git
repository or on GitHub?
> Create jump-tables for DocValues at index-time
> --
>
> Key: LUCENE-8585
> URL: https://issues.apache.org/jira/browse/LUCENE-8585
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
>Affects Versions: master (8.0)
> Reporter: Toke Eskildsen
>Priority: Minor
>
> As noted in LUCENE-7589, lookup of DocValues should use jump-tables to avoid
> long iterative walks. This is implemented in LUCENE-8374 at search-time
> (first request for DocValues from a field in a segment), with the benefit of
> working without changes to existing Lucene 7 indexes and the downside of
> introducing a startup time penalty and a memory overhead.
> As discussed in LUCENE-8374, the codec should be updated to create these
> jump-tables at index time. This eliminates the segment-open time & memory
> penalties, with the potential downside of increasing index-time for DocValues.
> The three elements of LUCENE-8374 should be transferable to index-time
> without much alteration of the core structures:
> * {{IndexedDISI}} block offset and index skips: A {{long}} (64 bits) for
> every 65536 documents, containing the offset of the block in 33 bits and the
> index (number of set bits) up to the block in 31 bits.
> It can be build sequentially and should be stored as a simple sequence of
> consecutive longs for caching of lookups.
> As it is fairly small, relative to document count, it might be better to
> simply memory cache it?
> * {{IndexedDISI}} DENSE (> 4095, < 65536 set bits) blocks: A {{short}} (16
> bits) for every 8 {{longs}} (512 bits) for a total of 256 bytes/DENSE_block.
> Each {{short}} represents the number of set bits up to right before the
> corresponding sub-block of 512 docIDs.
> The \{{shorts}} can be computed sequentially or when the DENSE block is
> flushed (probably the easiest). They should be stored as a simple sequence of
> consecutive shorts for caching of lookups, one logically independent sequence
> for each DENSE block. The logical position would be one sequence at the start
> of every DENSE block.
> Whether it is best to read all the 16 {{shorts}} up front when a DENSE block
> is accessed or whether it is best to only read any individual {{short}} when
> needed is not clear at this point.
> * Variable Bits Per Value: A {{long}} (64 bits) for every 16384 numeric
> values. Each {{long}} holds the offset to the corresponding block of values.
> The offsets can be computed sequentially and should be stored as a simple
> sequence of consecutive {{longs}} for caching of lookups.
> The vBPV-offsets has the largest space overhead og the 3 jump-tables and a
> lot of the 64 bits in each long are not used for most indexes. They could be
> represented as a simple {{PackedInts}} sequence or {{MonotonicLongValues}},
> with the downsides of a potential lookup-time overhead and the need for doing
> the compression after all offsets has been determined.
> I have no experience with the codec-parts responsible for creating
> index-structures. I'm quite willing to take a stab at
[jira] [Commented] (LUCENE-8585) Create jump-tables for DocValues at index-time
[
https://issues.apache.org/jira/browse/LUCENE-8585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16709234#comment-16709234
]
Toke Eskildsen commented on LUCENE-8585:
Preliminary notes after inspecting the Lucene7DocValues-codec: It seems that
the {{meta}} data for the DocValue entries are loaded one-off and that any
subsequently needed data is fetched from the {{data}}-slices.
In line with {{TermsDictEntry}}, the jump-tables for IndexedDISI-blocks and
vBPV can be stored at the end of their respective {{data}}-slices, with only
the offsets to the jump-tables being stored in {{meta}}. The downside to this
solution is that the full jump-tables needs to be kept in memory until the data
has been written, for a worst-case temporary overhead of 2MB and 8MB when
flushing 2 billion documents with values.
As mentioned in the description, DENSE ranks belongs naturally in their blocks
in {{data}}. This makes their representation extremely simple and makes the
rank & data be disk cached together. {{meta}} is not involved here.
> Create jump-tables for DocValues at index-time
> --
>
> Key: LUCENE-8585
> URL: https://issues.apache.org/jira/browse/LUCENE-8585
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
>Affects Versions: master (8.0)
> Reporter: Toke Eskildsen
>Priority: Minor
>
> As noted in LUCENE-7589, lookup of DocValues should use jump-tables to avoid
> long iterative walks. This is implemented in LUCENE-8374 at search-time
> (first request for DocValues from a field in a segment), with the benefit of
> working without changes to existing Lucene 7 indexes and the downside of
> introducing a startup time penalty and a memory overhead.
> As discussed in LUCENE-8374, the codec should be updated to create these
> jump-tables at index time. This eliminates the segment-open time & memory
> penalties, with the potential downside of increasing index-time for DocValues.
> The three elements of LUCENE-8374 should be transferable to index-time
> without much alteration of the core structures:
> * {{IndexedDISI}} block offset and index skips: A {{long}} (64 bits) for
> every 65536 documents, containing the offset of the block in 33 bits and the
> index (number of set bits) up to the block in 31 bits.
> It can be build sequentially and should be stored as a simple sequence of
> consecutive longs for caching of lookups.
> As it is fairly small, relative to document count, it might be better to
> simply memory cache it?
> * {{IndexedDISI}} DENSE (> 4095, < 65536 set bits) blocks: A {{short}} (16
> bits) for every 8 {{longs}} (512 bits) for a total of 256 bytes/DENSE_block.
> Each {{short}} represents the number of set bits up to right before the
> corresponding sub-block of 512 docIDs.
> The \{{shorts}} can be computed sequentially or when the DENSE block is
> flushed (probably the easiest). They should be stored as a simple sequence of
> consecutive shorts for caching of lookups, one logically independent sequence
> for each DENSE block. The logical position would be one sequence at the start
> of every DENSE block.
> Whether it is best to read all the 16 {{shorts}} up front when a DENSE block
> is accessed or whether it is best to only read any individual {{short}} when
> needed is not clear at this point.
> * Variable Bits Per Value: A {{long}} (64 bits) for every 16384 numeric
> values. Each {{long}} holds the offset to the corresponding block of values.
> The offsets can be computed sequentially and should be stored as a simple
> sequence of consecutive {{longs}} for caching of lookups.
> The vBPV-offsets has the largest space overhead og the 3 jump-tables and a
> lot of the 64 bits in each long are not used for most indexes. They could be
> represented as a simple {{PackedInts}} sequence or {{MonotonicLongValues}},
> with the downsides of a potential lookup-time overhead and the need for doing
> the compression after all offsets has been determined.
> I have no experience with the codec-parts responsible for creating
> index-structures. I'm quite willing to take a stab at this, although I
> probably won't do much about it before January 2019. Should anyone else wish
> to adopt this JIRA-issue or co-work on it, I'll be happy to share.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8374) Reduce reads for sparse DocValues
[
https://issues.apache.org/jira/browse/LUCENE-8374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16708966#comment-16708966
]
Toke Eskildsen commented on LUCENE-8374:
[~joel.bernstein] as for turning on optionally, then it was part of my first
patch as a static global switch (for debugging). Easy to put back in that form
and if JIT does what it's supposed to, a {{if (cache != null)}} check should
not be noticeable.
Making a switch non-static is a question of how it fits into the whole
codec-instantiation, specifically {{Lucene70DocValuesProducer}}. I don't know
how that is done without kludging it in.
> Reduce reads for sparse DocValues
> -
>
> Key: LUCENE-8374
> URL: https://issues.apache.org/jira/browse/LUCENE-8374
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
>Affects Versions: 7.5, master (8.0)
> Reporter: Toke Eskildsen
>Priority: Major
> Labels: performance
> Attachments: LUCENE-8374.patch, LUCENE-8374.patch, LUCENE-8374.patch,
> LUCENE-8374.patch, LUCENE-8374.patch, LUCENE-8374.patch, LUCENE-8374.patch,
> LUCENE-8374_branch_7_3.patch, LUCENE-8374_branch_7_3.patch.20181005,
> LUCENE-8374_branch_7_4.patch, LUCENE-8374_branch_7_5.patch,
> LUCENE-8374_part_1.patch, LUCENE-8374_part_2.patch, LUCENE-8374_part_3.patch,
> LUCENE-8374_part_4.patch, entire_index_logs.txt,
> image-2018-10-24-07-30-06-663.png, image-2018-10-24-07-30-56-962.png,
> single_vehicle_logs.txt,
> start-2018-10-24-1_snapshot___Users_tim_Snapshots__-_YourKit_Java_Profiler_2017_02-b75_-_64-bit.png,
>
> start-2018-10-24_snapshot___Users_tim_Snapshots__-_YourKit_Java_Profiler_2017_02-b75_-_64-bit.png
>
>
> The {{Lucene70DocValuesProducer}} has the internal classes
> {{SparseNumericDocValues}} and {{BaseSortedSetDocValues}} (sparse code path),
> which again uses {{IndexedDISI}} to handle the docID -> value-ordinal lookup.
> The value-ordinal is the index of the docID assuming an abstract tightly
> packed monotonically increasing list of docIDs: If the docIDs with
> corresponding values are {{[0, 4, 1432]}}, their value-ordinals will be {{[0,
> 1, 2]}}.
> h2. Outer blocks
> The lookup structure of {{IndexedDISI}} consists of blocks of 2^16 values
> (65536), where each block can be either {{ALL}}, {{DENSE}} (2^12 to 2^16
> values) or {{SPARSE}} (< 2^12 values ~= 6%). Consequently blocks vary quite a
> lot in size and ordinal resolving strategy.
> When a sparse Numeric DocValue is needed, the code first locates the block
> containing the wanted docID flag. It does so by iterating blocks one-by-one
> until it reaches the needed one, where each iteration requires a lookup in
> the underlying {{IndexSlice}}. For a common memory mapped index, this
> translates to either a cached request or a read operation. If a segment has
> 6M documents, worst-case is 91 lookups. In our web archive, our segments has
> ~300M values: A worst-case of 4577 lookups!
> One obvious solution is to use a lookup-table for blocks: A long[]-array with
> an entry for each block. For 6M documents, that is < 1KB and would allow for
> direct jumping (a single lookup) in all instances. Unfortunately this
> lookup-table cannot be generated upfront when the writing of values is purely
> streaming. It can be appended to the end of the stream before it is closed,
> but without knowing the position of the lookup-table the reader cannot seek
> to it.
> One strategy for creating such a lookup-table would be to generate it during
> reads and cache it for next lookup. This does not fit directly into how
> {{IndexedDISI}} currently works (it is created anew for each invocation), but
> could probably be added with a little work. An advantage to this is that this
> does not change the underlying format and thus could be used with existing
> indexes.
> h2. The lookup structure inside each block
> If {{ALL}} of the 2^16 values are defined, the structure is empty and the
> ordinal is simply the requested docID with some modulo and multiply math.
> Nothing to improve there.
> If the block is {{DENSE}} (2^12 to 2^16 values are defined), a bitmap is used
> and the number of set bits up to the wanted index (the docID modulo the block
> origo) are counted. That bitmap is a long[1024], meaning that worst case is
> to lookup and count all set bits for 1024 longs!
> One known solution to this is to use a [rank
> structure|[https://en.wikipedia.org/wiki/Succinct_data_structure]]. I
> [implemented
> it|[https://github.com/tokee/lucene-solr/blob/solr5894/solr/core/src/java/org/apache/solr/s
[jira] [Commented] (LUCENE-8374) Reduce reads for sparse DocValues
[
https://issues.apache.org/jira/browse/LUCENE-8374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16708907#comment-16708907
]
Toke Eskildsen commented on LUCENE-8374:
[~jpountz] thank you for the well-defined arguments. I see your point much more
clearly now.
{quote}The fact that the default codec happily computes data at search-time
rather than index-time may set a precedent...
{quote}
I think you place way too little faith in codec developers. Also, the codec is
a scary beast and I seriously doubt that changes here will fly under the radar
of you or other developers. Concretely the injection of caches is in a few key
places and one could well add explicit comments of them being a band aid, to
ensure no inspiration follows.
{quote}The introduced complexity of search-time-only support for jump tables
makes it harder to evolve this doc-value format so that it computes jump tables
at index time.
{quote}
I don't see how that follows. If anything the patch points out the places that
should use the index-time structures, which (unless LUCENE-8585 changes that)
works the same way as search-time. Where possible, the skipping code has been
extracted to helper methods, making it even easier to build on.
{quote}We are reintroducing a feature that potentially allows the memory usage
of readers to grow over time rather than being set in stone after the reader is
open...
{quote}
True. I was not aware of the problems with this. One "solution" would be to
cache all the DocValues up front. That would mean wasted time & space for
non-used fields, so I don't like it.
{quote}Search-time-only support introduces contention and/or complexity due to
the need to access a cache concurrently.
{quote}
True (and I think I have already found a problem there). With the addendum that
we have the same problem if we decide that some structures should be kept in
memory for LUCENE-8585.
>From my point of view you are prioritizing architecture over functionality to
>a very high degree.
It might well be more in line with the Lucene code base than my prioritization
and on that ground I concede that the search-time code should be removed (and
thereby that plug-in functionality for existing indexes goes away) when an
index-time solution has been made. Removing a working band aid solution before
a working clean one has been committed is not something I support at this point.
On the more agreeable side, thanks to [~jim.ferenczi] for volunteering with
index-time. With a little luck we can wrap that up fairly quick. I do have time
before January, just not much. I guess we should talk details in the
LUCENE-8585 issue?
> Reduce reads for sparse DocValues
> -
>
> Key: LUCENE-8374
> URL: https://issues.apache.org/jira/browse/LUCENE-8374
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
>Affects Versions: 7.5, master (8.0)
>Reporter: Toke Eskildsen
>Priority: Major
> Labels: performance
> Attachments: LUCENE-8374.patch, LUCENE-8374.patch, LUCENE-8374.patch,
> LUCENE-8374.patch, LUCENE-8374.patch, LUCENE-8374.patch, LUCENE-8374.patch,
> LUCENE-8374_branch_7_3.patch, LUCENE-8374_branch_7_3.patch.20181005,
> LUCENE-8374_branch_7_4.patch, LUCENE-8374_branch_7_5.patch,
> LUCENE-8374_part_1.patch, LUCENE-8374_part_2.patch, LUCENE-8374_part_3.patch,
> LUCENE-8374_part_4.patch, entire_index_logs.txt,
> image-2018-10-24-07-30-06-663.png, image-2018-10-24-07-30-56-962.png,
> single_vehicle_logs.txt,
> start-2018-10-24-1_snapshot___Users_tim_Snapshots__-_YourKit_Java_Profiler_2017_02-b75_-_64-bit.png,
>
> start-2018-10-24_snapshot___Users_tim_Snapshots__-_YourKit_Java_Profiler_2017_02-b75_-_64-bit.png
>
>
> The {{Lucene70DocValuesProducer}} has the internal classes
> {{SparseNumericDocValues}} and {{BaseSortedSetDocValues}} (sparse code path),
> which again uses {{IndexedDISI}} to handle the docID -> value-ordinal lookup.
> The value-ordinal is the index of the docID assuming an abstract tightly
> packed monotonically increasing list of docIDs: If the docIDs with
> corresponding values are {{[0, 4, 1432]}}, their value-ordinals will be {{[0,
> 1, 2]}}.
> h2. Outer blocks
> The lookup structure of {{IndexedDISI}} consists of blocks of 2^16 values
> (65536), where each block can be either {{ALL}}, {{DENSE}} (2^12 to 2^16
> values) or {{SPARSE}} (< 2^12 values ~= 6%). Consequently blocks vary quite a
> lot in size and ordinal resolving strategy.
> When a sparse Numeric DocValue is needed, the code first locates the block
> containing the wanted docID flag. It does so by iterating blocks one-by-one
> until it reaches the need
[jira] [Commented] (LUCENE-8374) Reduce reads for sparse DocValues
[
https://issues.apache.org/jira/browse/LUCENE-8374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16708799#comment-16708799
]
Toke Eskildsen commented on LUCENE-8374:
[~jim.ferenczi] this all started with us (Royal Danish Library) having large
segments in our webarchive index. Our own experience, as well as that for other
institutions we are in contact with, is that large indexes are unwieldy. One
does not simply run an index upgrade on the full collection. In that context
the search-time implementation means that a (relatively) simple upgrade of
Solr, especially if ported to the 7.x-series, improves performance considerably.
So your "why two versions?" is akin to "why backport anything?". We do it
because we acknowledge that circumstances differ greatly across organizations &
installations and that there is a need for balancing legacy maintenance
overhead and making people play the upgrade game.
> Reduce reads for sparse DocValues
> -
>
> Key: LUCENE-8374
> URL: https://issues.apache.org/jira/browse/LUCENE-8374
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
>Affects Versions: 7.5, master (8.0)
>Reporter: Toke Eskildsen
>Priority: Major
> Labels: performance
> Attachments: LUCENE-8374.patch, LUCENE-8374.patch, LUCENE-8374.patch,
> LUCENE-8374.patch, LUCENE-8374.patch, LUCENE-8374.patch, LUCENE-8374.patch,
> LUCENE-8374_branch_7_3.patch, LUCENE-8374_branch_7_3.patch.20181005,
> LUCENE-8374_branch_7_4.patch, LUCENE-8374_branch_7_5.patch,
> LUCENE-8374_part_1.patch, LUCENE-8374_part_2.patch, LUCENE-8374_part_3.patch,
> LUCENE-8374_part_4.patch, entire_index_logs.txt,
> image-2018-10-24-07-30-06-663.png, image-2018-10-24-07-30-56-962.png,
> single_vehicle_logs.txt,
> start-2018-10-24-1_snapshot___Users_tim_Snapshots__-_YourKit_Java_Profiler_2017_02-b75_-_64-bit.png,
>
> start-2018-10-24_snapshot___Users_tim_Snapshots__-_YourKit_Java_Profiler_2017_02-b75_-_64-bit.png
>
>
> The {{Lucene70DocValuesProducer}} has the internal classes
> {{SparseNumericDocValues}} and {{BaseSortedSetDocValues}} (sparse code path),
> which again uses {{IndexedDISI}} to handle the docID -> value-ordinal lookup.
> The value-ordinal is the index of the docID assuming an abstract tightly
> packed monotonically increasing list of docIDs: If the docIDs with
> corresponding values are {{[0, 4, 1432]}}, their value-ordinals will be {{[0,
> 1, 2]}}.
> h2. Outer blocks
> The lookup structure of {{IndexedDISI}} consists of blocks of 2^16 values
> (65536), where each block can be either {{ALL}}, {{DENSE}} (2^12 to 2^16
> values) or {{SPARSE}} (< 2^12 values ~= 6%). Consequently blocks vary quite a
> lot in size and ordinal resolving strategy.
> When a sparse Numeric DocValue is needed, the code first locates the block
> containing the wanted docID flag. It does so by iterating blocks one-by-one
> until it reaches the needed one, where each iteration requires a lookup in
> the underlying {{IndexSlice}}. For a common memory mapped index, this
> translates to either a cached request or a read operation. If a segment has
> 6M documents, worst-case is 91 lookups. In our web archive, our segments has
> ~300M values: A worst-case of 4577 lookups!
> One obvious solution is to use a lookup-table for blocks: A long[]-array with
> an entry for each block. For 6M documents, that is < 1KB and would allow for
> direct jumping (a single lookup) in all instances. Unfortunately this
> lookup-table cannot be generated upfront when the writing of values is purely
> streaming. It can be appended to the end of the stream before it is closed,
> but without knowing the position of the lookup-table the reader cannot seek
> to it.
> One strategy for creating such a lookup-table would be to generate it during
> reads and cache it for next lookup. This does not fit directly into how
> {{IndexedDISI}} currently works (it is created anew for each invocation), but
> could probably be added with a little work. An advantage to this is that this
> does not change the underlying format and thus could be used with existing
> indexes.
> h2. The lookup structure inside each block
> If {{ALL}} of the 2^16 values are defined, the structure is empty and the
> ordinal is simply the requested docID with some modulo and multiply math.
> Nothing to improve there.
> If the block is {{DENSE}} (2^12 to 2^16 values are defined), a bitmap is used
> and the number of set bits up to the wanted index (the docID modulo the block
> origo) are counted. That bitmap is a long[1024], me
[jira] [Commented] (LUCENE-8374) Reduce reads for sparse DocValues
[
https://issues.apache.org/jira/browse/LUCENE-8374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16708764#comment-16708764
]
Toke Eskildsen commented on LUCENE-8374:
[~jpountz] something might have been lost in translation or culture here. For
me, "preference" is a neutral term, used in this context to signal that I
understand that you do not see having a search-time cache as the right choice.
We have
# A DV-implementation (pre LUCENE-8374 commit) that is O(n), with poor scaling
up
# An instantly working band aid
# A plan to implement a better index-time solution for future segments
I don't claim that search-time is a superior solution to index-time, when
compared neutrally. You are absolutely right that it does lazy-load of data and
it is technologically better to do index-time. But it is a red herring. The
issue is whether the band-aid makes sense for existing setups.
Now, this is all on master, so no need for panic. But at some point we will
roll a 7.7 or an 8.0. Hopefully we will have the index-time implementation
ready then. But on the off chance that we don't, let me ask you: Do you think
it would be better to have no DV jump-tables at all, rather than have the
search-time implementation only in a release?
> Reduce reads for sparse DocValues
> -
>
> Key: LUCENE-8374
> URL: https://issues.apache.org/jira/browse/LUCENE-8374
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
>Affects Versions: 7.5, master (8.0)
>Reporter: Toke Eskildsen
>Priority: Major
> Labels: performance
> Attachments: LUCENE-8374.patch, LUCENE-8374.patch, LUCENE-8374.patch,
> LUCENE-8374.patch, LUCENE-8374.patch, LUCENE-8374.patch, LUCENE-8374.patch,
> LUCENE-8374_branch_7_3.patch, LUCENE-8374_branch_7_3.patch.20181005,
> LUCENE-8374_branch_7_4.patch, LUCENE-8374_branch_7_5.patch,
> LUCENE-8374_part_1.patch, LUCENE-8374_part_2.patch, LUCENE-8374_part_3.patch,
> LUCENE-8374_part_4.patch, entire_index_logs.txt,
> image-2018-10-24-07-30-06-663.png, image-2018-10-24-07-30-56-962.png,
> single_vehicle_logs.txt,
> start-2018-10-24-1_snapshot___Users_tim_Snapshots__-_YourKit_Java_Profiler_2017_02-b75_-_64-bit.png,
>
> start-2018-10-24_snapshot___Users_tim_Snapshots__-_YourKit_Java_Profiler_2017_02-b75_-_64-bit.png
>
>
> The {{Lucene70DocValuesProducer}} has the internal classes
> {{SparseNumericDocValues}} and {{BaseSortedSetDocValues}} (sparse code path),
> which again uses {{IndexedDISI}} to handle the docID -> value-ordinal lookup.
> The value-ordinal is the index of the docID assuming an abstract tightly
> packed monotonically increasing list of docIDs: If the docIDs with
> corresponding values are {{[0, 4, 1432]}}, their value-ordinals will be {{[0,
> 1, 2]}}.
> h2. Outer blocks
> The lookup structure of {{IndexedDISI}} consists of blocks of 2^16 values
> (65536), where each block can be either {{ALL}}, {{DENSE}} (2^12 to 2^16
> values) or {{SPARSE}} (< 2^12 values ~= 6%). Consequently blocks vary quite a
> lot in size and ordinal resolving strategy.
> When a sparse Numeric DocValue is needed, the code first locates the block
> containing the wanted docID flag. It does so by iterating blocks one-by-one
> until it reaches the needed one, where each iteration requires a lookup in
> the underlying {{IndexSlice}}. For a common memory mapped index, this
> translates to either a cached request or a read operation. If a segment has
> 6M documents, worst-case is 91 lookups. In our web archive, our segments has
> ~300M values: A worst-case of 4577 lookups!
> One obvious solution is to use a lookup-table for blocks: A long[]-array with
> an entry for each block. For 6M documents, that is < 1KB and would allow for
> direct jumping (a single lookup) in all instances. Unfortunately this
> lookup-table cannot be generated upfront when the writing of values is purely
> streaming. It can be appended to the end of the stream before it is closed,
> but without knowing the position of the lookup-table the reader cannot seek
> to it.
> One strategy for creating such a lookup-table would be to generate it during
> reads and cache it for next lookup. This does not fit directly into how
> {{IndexedDISI}} currently works (it is created anew for each invocation), but
> could probably be added with a little work. An advantage to this is that this
> does not change the underlying format and thus could be used with existing
> indexes.
> h2. The lookup structure inside each block
> If {{ALL}} of the 2^16 values are defined, the structure is empty and the
> ordinal is si
[jira] [Comment Edited] (LUCENE-8374) Reduce reads for sparse DocValues
[
https://issues.apache.org/jira/browse/LUCENE-8374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16708764#comment-16708764
]
Toke Eskildsen edited comment on LUCENE-8374 at 12/4/18 2:29 PM:
-
[~jpountz] something might have been lost in translation or culture here. For
me, "preference" is a neutral term, used in this context to signal that I
understand that you do not see having a search-time cache as the right choice.
We have
# A DV-implementation (pre LUCENE-8374 commit) that is O\(n), with poor
scaling up
# An instantly working band aid
# A plan to implement a better index-time solution for future segments
I don't claim that search-time is a superior solution to index-time, when
compared neutrally. You are absolutely right that it does lazy-load of data and
it is technologically better to do index-time. But it is a red herring. The
issue is whether the band-aid makes sense for existing setups.
Now, this is all on master, so no need for panic. But at some point we will
roll a 7.7 or an 8.0. Hopefully we will have the index-time implementation
ready then. But on the off chance that we don't, let me ask you: Do you think
it would be better to have no DV jump-tables at all, rather than have the
search-time implementation only in a release?
was (Author: toke):
[~jpountz] something might have been lost in translation or culture here. For
me, "preference" is a neutral term, used in this context to signal that I
understand that you do not see having a search-time cache as the right choice.
We have
# A DV-implementation (pre LUCENE-8374 commit) that is O(n), with poor scaling
up
# An instantly working band aid
# A plan to implement a better index-time solution for future segments
I don't claim that search-time is a superior solution to index-time, when
compared neutrally. You are absolutely right that it does lazy-load of data and
it is technologically better to do index-time. But it is a red herring. The
issue is whether the band-aid makes sense for existing setups.
Now, this is all on master, so no need for panic. But at some point we will
roll a 7.7 or an 8.0. Hopefully we will have the index-time implementation
ready then. But on the off chance that we don't, let me ask you: Do you think
it would be better to have no DV jump-tables at all, rather than have the
search-time implementation only in a release?
> Reduce reads for sparse DocValues
> -
>
> Key: LUCENE-8374
> URL: https://issues.apache.org/jira/browse/LUCENE-8374
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
>Affects Versions: 7.5, master (8.0)
>Reporter: Toke Eskildsen
>Priority: Major
> Labels: performance
> Attachments: LUCENE-8374.patch, LUCENE-8374.patch, LUCENE-8374.patch,
> LUCENE-8374.patch, LUCENE-8374.patch, LUCENE-8374.patch, LUCENE-8374.patch,
> LUCENE-8374_branch_7_3.patch, LUCENE-8374_branch_7_3.patch.20181005,
> LUCENE-8374_branch_7_4.patch, LUCENE-8374_branch_7_5.patch,
> LUCENE-8374_part_1.patch, LUCENE-8374_part_2.patch, LUCENE-8374_part_3.patch,
> LUCENE-8374_part_4.patch, entire_index_logs.txt,
> image-2018-10-24-07-30-06-663.png, image-2018-10-24-07-30-56-962.png,
> single_vehicle_logs.txt,
> start-2018-10-24-1_snapshot___Users_tim_Snapshots__-_YourKit_Java_Profiler_2017_02-b75_-_64-bit.png,
>
> start-2018-10-24_snapshot___Users_tim_Snapshots__-_YourKit_Java_Profiler_2017_02-b75_-_64-bit.png
>
>
> The {{Lucene70DocValuesProducer}} has the internal classes
> {{SparseNumericDocValues}} and {{BaseSortedSetDocValues}} (sparse code path),
> which again uses {{IndexedDISI}} to handle the docID -> value-ordinal lookup.
> The value-ordinal is the index of the docID assuming an abstract tightly
> packed monotonically increasing list of docIDs: If the docIDs with
> corresponding values are {{[0, 4, 1432]}}, their value-ordinals will be {{[0,
> 1, 2]}}.
> h2. Outer blocks
> The lookup structure of {{IndexedDISI}} consists of blocks of 2^16 values
> (65536), where each block can be either {{ALL}}, {{DENSE}} (2^12 to 2^16
> values) or {{SPARSE}} (< 2^12 values ~= 6%). Consequently blocks vary quite a
> lot in size and ordinal resolving strategy.
> When a sparse Numeric DocValue is needed, the code first locates the block
> containing the wanted docID flag. It does so by iterating blocks one-by-one
> until it reaches the needed one, where each iteration requires a lookup in
> the underlying {{IndexSlice}}. For a common memory mapped index, this
> translates to either a cached request or a read operation. If a segment has
> 6M documents, worst-case is 91 look
[jira] [Comment Edited] (LUCENE-8374) Reduce reads for sparse DocValues
[
https://issues.apache.org/jira/browse/LUCENE-8374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16708764#comment-16708764
]
Toke Eskildsen edited comment on LUCENE-8374 at 12/4/18 2:29 PM:
-
[~jpountz] something might have been lost in translation or culture here. For
me, "preference" is a neutral term, used in this context to signal that I
understand that you do not see having a search-time cache as the right choice.
We have
# A DV-implementation (pre LUCENE-8374 commit) that is O(n), with poor scaling
up
# An instantly working band aid
# A plan to implement a better index-time solution for future segments
I don't claim that search-time is a superior solution to index-time, when
compared neutrally. You are absolutely right that it does lazy-load of data and
it is technologically better to do index-time. But it is a red herring. The
issue is whether the band-aid makes sense for existing setups.
Now, this is all on master, so no need for panic. But at some point we will
roll a 7.7 or an 8.0. Hopefully we will have the index-time implementation
ready then. But on the off chance that we don't, let me ask you: Do you think
it would be better to have no DV jump-tables at all, rather than have the
search-time implementation only in a release?
was (Author: toke):
[~jpountz] something might have been lost in translation or culture here. For
me, "preference" is a neutral term, used in this context to signal that I
understand that you do not see having a search-time cache as the right choice.
We have
# A DV-implementation (pre LUCENE-8374 commit) that is O(n), with poor scaling
up
# An instantly working band aid
# A plan to implement a better index-time solution for future segments
I don't claim that search-time is a superior solution to index-time, when
compared neutrally. You are absolutely right that it does lazy-load of data and
it is technologically better to do index-time. But it is a red herring. The
issue is whether the band-aid makes sense for existing setups.
Now, this is all on master, so no need for panic. But at some point we will
roll a 7.7 or an 8.0. Hopefully we will have the index-time implementation
ready then. But on the off chance that we don't, let me ask you: Do you think
it would be better to have no DV jump-tables at all, rather than have the
search-time implementation only in a release?
> Reduce reads for sparse DocValues
> -
>
> Key: LUCENE-8374
> URL: https://issues.apache.org/jira/browse/LUCENE-8374
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
>Affects Versions: 7.5, master (8.0)
>Reporter: Toke Eskildsen
>Priority: Major
> Labels: performance
> Attachments: LUCENE-8374.patch, LUCENE-8374.patch, LUCENE-8374.patch,
> LUCENE-8374.patch, LUCENE-8374.patch, LUCENE-8374.patch, LUCENE-8374.patch,
> LUCENE-8374_branch_7_3.patch, LUCENE-8374_branch_7_3.patch.20181005,
> LUCENE-8374_branch_7_4.patch, LUCENE-8374_branch_7_5.patch,
> LUCENE-8374_part_1.patch, LUCENE-8374_part_2.patch, LUCENE-8374_part_3.patch,
> LUCENE-8374_part_4.patch, entire_index_logs.txt,
> image-2018-10-24-07-30-06-663.png, image-2018-10-24-07-30-56-962.png,
> single_vehicle_logs.txt,
> start-2018-10-24-1_snapshot___Users_tim_Snapshots__-_YourKit_Java_Profiler_2017_02-b75_-_64-bit.png,
>
> start-2018-10-24_snapshot___Users_tim_Snapshots__-_YourKit_Java_Profiler_2017_02-b75_-_64-bit.png
>
>
> The {{Lucene70DocValuesProducer}} has the internal classes
> {{SparseNumericDocValues}} and {{BaseSortedSetDocValues}} (sparse code path),
> which again uses {{IndexedDISI}} to handle the docID -> value-ordinal lookup.
> The value-ordinal is the index of the docID assuming an abstract tightly
> packed monotonically increasing list of docIDs: If the docIDs with
> corresponding values are {{[0, 4, 1432]}}, their value-ordinals will be {{[0,
> 1, 2]}}.
> h2. Outer blocks
> The lookup structure of {{IndexedDISI}} consists of blocks of 2^16 values
> (65536), where each block can be either {{ALL}}, {{DENSE}} (2^12 to 2^16
> values) or {{SPARSE}} (< 2^12 values ~= 6%). Consequently blocks vary quite a
> lot in size and ordinal resolving strategy.
> When a sparse Numeric DocValue is needed, the code first locates the block
> containing the wanted docID flag. It does so by iterating blocks one-by-one
> until it reaches the needed one, where each iteration requires a lookup in
> the underlying {{IndexSlice}}. For a common memory mapped index, this
> translates to either a cached request or a read operation. If a segment has
> 6M documents, worst-case is 91 look
Re: [1/4] lucene-solr:master: LUCENE-8374 part 1/4: Reduce reads for sparse DocValues
Toke Eskildsen wrote: > Gus Heck wrote: >> Precommit appears to be failing related to this series of commits > I apologize and will correct it right away. Fixed. ant precommit now passes for me on master. Thanks for the note Gus, Toke Eskildsen - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: [1/4] lucene-solr:master: LUCENE-8374 part 1/4: Reduce reads for sparse DocValues
From: Gus Heck wrote: > Precommit appears to be failing related to this series of commits Thank you. I clearly did not perform this step, even if I thought I did. I apologize and will correct it right away. Toke Eskildsen - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (LUCENE-8585) Create jump-tables for DocValues at index-time
Toke Eskildsen created LUCENE-8585:
--
Summary: Create jump-tables for DocValues at index-time
Key: LUCENE-8585
URL: https://issues.apache.org/jira/browse/LUCENE-8585
Project: Lucene - Core
Issue Type: Improvement
Components: core/codecs
Affects Versions: master (8.0)
Reporter: Toke Eskildsen
As noted in LUCENE-7589, lookup of DocValues should use jump-tables to avoid
long iterative walks. This is implemented in LUCENE-8374 at search-time (first
request for DocValues from a field in a segment), with the benefit of working
without changes to existing Lucene 7 indexes and the downside of introducing a
startup time penalty and a memory overhead.
As discussed in LUCENE-8374, the codec should be updated to create these
jump-tables at index time. This eliminates the segment-open time & memory
penalties, with the potential downside of increasing index-time for DocValues.
The three elements of LUCENE-8374 should be transferable to index-time without
much alteration of the core structures:
* {{IndexedDISI}} block offset and index skips: A {{long}} (64 bits) for every
65536 documents, containing the offset of the block in 33 bits and the index
(number of set bits) up to the block in 31 bits.
It can be build sequentially and should be stored as a simple sequence of
consecutive longs for caching of lookups.
As it is fairly small, relative to document count, it might be better to
simply memory cache it?
* {{IndexedDISI}} DENSE (> 4095, < 65536 set bits) blocks: A {{short}} (16
bits) for every 8 {{longs}} (512 bits) for a total of 256 bytes/DENSE_block.
Each {{short}} represents the number of set bits up to right before the
corresponding sub-block of 512 docIDs.
The \{{shorts}} can be computed sequentially or when the DENSE block is
flushed (probably the easiest). They should be stored as a simple sequence of
consecutive shorts for caching of lookups, one logically independent sequence
for each DENSE block. The logical position would be one sequence at the start
of every DENSE block.
Whether it is best to read all the 16 {{shorts}} up front when a DENSE block
is accessed or whether it is best to only read any individual {{short}} when
needed is not clear at this point.
* Variable Bits Per Value: A {{long}} (64 bits) for every 16384 numeric
values. Each {{long}} holds the offset to the corresponding block of values.
The offsets can be computed sequentially and should be stored as a simple
sequence of consecutive {{longs}} for caching of lookups.
The vBPV-offsets has the largest space overhead og the 3 jump-tables and a lot
of the 64 bits in each long are not used for most indexes. They could be
represented as a simple {{PackedInts}} sequence or {{MonotonicLongValues}},
with the downsides of a potential lookup-time overhead and the need for doing
the compression after all offsets has been determined.
I have no experience with the codec-parts responsible for creating
index-structures. I'm quite willing to take a stab at this, although I probably
won't do much about it before January 2019. Should anyone else wish to adopt
this JIRA-issue or co-work on it, I'll be happy to share.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8374) Reduce reads for sparse DocValues
[
https://issues.apache.org/jira/browse/LUCENE-8374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16707247#comment-16707247
]
Toke Eskildsen commented on LUCENE-8374:
[~jpountz] Your preference is noted. I have presented my arguments for having a
search-time version, you have presented yours. With no mediating party and as I
saw no progress in our discussion, I decided to follow [~dsmiley]s suggestion
and push to see what Jenkins and other testers says.
The patch is mostly new files and thus fairly easy to revert, even if there are
subsequent changes to {{Lucene70DocValuesProducer}} & {{IndexedDISI}}. I hope
this will only be considered *after* we as a collective has examined its impact
on setups of different sizes and request types.
> Reduce reads for sparse DocValues
> -
>
> Key: LUCENE-8374
> URL: https://issues.apache.org/jira/browse/LUCENE-8374
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
>Affects Versions: 7.5, master (8.0)
> Reporter: Toke Eskildsen
>Priority: Major
> Labels: performance
> Attachments: LUCENE-8374.patch, LUCENE-8374.patch, LUCENE-8374.patch,
> LUCENE-8374.patch, LUCENE-8374.patch, LUCENE-8374.patch, LUCENE-8374.patch,
> LUCENE-8374_branch_7_3.patch, LUCENE-8374_branch_7_3.patch.20181005,
> LUCENE-8374_branch_7_4.patch, LUCENE-8374_branch_7_5.patch,
> LUCENE-8374_part_1.patch, LUCENE-8374_part_2.patch, LUCENE-8374_part_3.patch,
> LUCENE-8374_part_4.patch, entire_index_logs.txt,
> image-2018-10-24-07-30-06-663.png, image-2018-10-24-07-30-56-962.png,
> single_vehicle_logs.txt,
> start-2018-10-24-1_snapshot___Users_tim_Snapshots__-_YourKit_Java_Profiler_2017_02-b75_-_64-bit.png,
>
> start-2018-10-24_snapshot___Users_tim_Snapshots__-_YourKit_Java_Profiler_2017_02-b75_-_64-bit.png
>
>
> The {{Lucene70DocValuesProducer}} has the internal classes
> {{SparseNumericDocValues}} and {{BaseSortedSetDocValues}} (sparse code path),
> which again uses {{IndexedDISI}} to handle the docID -> value-ordinal lookup.
> The value-ordinal is the index of the docID assuming an abstract tightly
> packed monotonically increasing list of docIDs: If the docIDs with
> corresponding values are {{[0, 4, 1432]}}, their value-ordinals will be {{[0,
> 1, 2]}}.
> h2. Outer blocks
> The lookup structure of {{IndexedDISI}} consists of blocks of 2^16 values
> (65536), where each block can be either {{ALL}}, {{DENSE}} (2^12 to 2^16
> values) or {{SPARSE}} (< 2^12 values ~= 6%). Consequently blocks vary quite a
> lot in size and ordinal resolving strategy.
> When a sparse Numeric DocValue is needed, the code first locates the block
> containing the wanted docID flag. It does so by iterating blocks one-by-one
> until it reaches the needed one, where each iteration requires a lookup in
> the underlying {{IndexSlice}}. For a common memory mapped index, this
> translates to either a cached request or a read operation. If a segment has
> 6M documents, worst-case is 91 lookups. In our web archive, our segments has
> ~300M values: A worst-case of 4577 lookups!
> One obvious solution is to use a lookup-table for blocks: A long[]-array with
> an entry for each block. For 6M documents, that is < 1KB and would allow for
> direct jumping (a single lookup) in all instances. Unfortunately this
> lookup-table cannot be generated upfront when the writing of values is purely
> streaming. It can be appended to the end of the stream before it is closed,
> but without knowing the position of the lookup-table the reader cannot seek
> to it.
> One strategy for creating such a lookup-table would be to generate it during
> reads and cache it for next lookup. This does not fit directly into how
> {{IndexedDISI}} currently works (it is created anew for each invocation), but
> could probably be added with a little work. An advantage to this is that this
> does not change the underlying format and thus could be used with existing
> indexes.
> h2. The lookup structure inside each block
> If {{ALL}} of the 2^16 values are defined, the structure is empty and the
> ordinal is simply the requested docID with some modulo and multiply math.
> Nothing to improve there.
> If the block is {{DENSE}} (2^12 to 2^16 values are defined), a bitmap is used
> and the number of set bits up to the wanted index (the docID modulo the block
> origo) are counted. That bitmap is a long[1024], meaning that worst case is
> to lookup and count all set bits for 1024 longs!
> One known solution to this is to use a [rank
> structure|[https://en.wikipedia.
[jira] [Commented] (LUCENE-8374) Reduce reads for sparse DocValues
[
https://issues.apache.org/jira/browse/LUCENE-8374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16707216#comment-16707216
]
Toke Eskildsen commented on LUCENE-8374:
Pushed to master. Let's see what happens.
Besides following up on issues stemming from the search-time jump-tables, it's
time to look at making an index-time version. I think this calls for a new
JIRA-issue?
> Reduce reads for sparse DocValues
> -
>
> Key: LUCENE-8374
> URL: https://issues.apache.org/jira/browse/LUCENE-8374
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
>Affects Versions: 7.5, master (8.0)
> Reporter: Toke Eskildsen
>Priority: Major
> Labels: performance
> Attachments: LUCENE-8374.patch, LUCENE-8374.patch, LUCENE-8374.patch,
> LUCENE-8374.patch, LUCENE-8374.patch, LUCENE-8374.patch, LUCENE-8374.patch,
> LUCENE-8374_branch_7_3.patch, LUCENE-8374_branch_7_3.patch.20181005,
> LUCENE-8374_branch_7_4.patch, LUCENE-8374_branch_7_5.patch,
> LUCENE-8374_part_1.patch, LUCENE-8374_part_2.patch, LUCENE-8374_part_3.patch,
> LUCENE-8374_part_4.patch, entire_index_logs.txt,
> image-2018-10-24-07-30-06-663.png, image-2018-10-24-07-30-56-962.png,
> single_vehicle_logs.txt,
> start-2018-10-24-1_snapshot___Users_tim_Snapshots__-_YourKit_Java_Profiler_2017_02-b75_-_64-bit.png,
>
> start-2018-10-24_snapshot___Users_tim_Snapshots__-_YourKit_Java_Profiler_2017_02-b75_-_64-bit.png
>
>
> The {{Lucene70DocValuesProducer}} has the internal classes
> {{SparseNumericDocValues}} and {{BaseSortedSetDocValues}} (sparse code path),
> which again uses {{IndexedDISI}} to handle the docID -> value-ordinal lookup.
> The value-ordinal is the index of the docID assuming an abstract tightly
> packed monotonically increasing list of docIDs: If the docIDs with
> corresponding values are {{[0, 4, 1432]}}, their value-ordinals will be {{[0,
> 1, 2]}}.
> h2. Outer blocks
> The lookup structure of {{IndexedDISI}} consists of blocks of 2^16 values
> (65536), where each block can be either {{ALL}}, {{DENSE}} (2^12 to 2^16
> values) or {{SPARSE}} (< 2^12 values ~= 6%). Consequently blocks vary quite a
> lot in size and ordinal resolving strategy.
> When a sparse Numeric DocValue is needed, the code first locates the block
> containing the wanted docID flag. It does so by iterating blocks one-by-one
> until it reaches the needed one, where each iteration requires a lookup in
> the underlying {{IndexSlice}}. For a common memory mapped index, this
> translates to either a cached request or a read operation. If a segment has
> 6M documents, worst-case is 91 lookups. In our web archive, our segments has
> ~300M values: A worst-case of 4577 lookups!
> One obvious solution is to use a lookup-table for blocks: A long[]-array with
> an entry for each block. For 6M documents, that is < 1KB and would allow for
> direct jumping (a single lookup) in all instances. Unfortunately this
> lookup-table cannot be generated upfront when the writing of values is purely
> streaming. It can be appended to the end of the stream before it is closed,
> but without knowing the position of the lookup-table the reader cannot seek
> to it.
> One strategy for creating such a lookup-table would be to generate it during
> reads and cache it for next lookup. This does not fit directly into how
> {{IndexedDISI}} currently works (it is created anew for each invocation), but
> could probably be added with a little work. An advantage to this is that this
> does not change the underlying format and thus could be used with existing
> indexes.
> h2. The lookup structure inside each block
> If {{ALL}} of the 2^16 values are defined, the structure is empty and the
> ordinal is simply the requested docID with some modulo and multiply math.
> Nothing to improve there.
> If the block is {{DENSE}} (2^12 to 2^16 values are defined), a bitmap is used
> and the number of set bits up to the wanted index (the docID modulo the block
> origo) are counted. That bitmap is a long[1024], meaning that worst case is
> to lookup and count all set bits for 1024 longs!
> One known solution to this is to use a [rank
> structure|[https://en.wikipedia.org/wiki/Succinct_data_structure]]. I
> [implemented
> it|[https://github.com/tokee/lucene-solr/blob/solr5894/solr/core/src/java/org/apache/solr/search/sparse/count/plane/RankCache.java]]
> for a related project and with that (), the rank-overhead for a {{DENSE}}
> block would be long[32] and would ensure a maximum of 9 lookups. It is not
> trivial to build the rank-struct
[jira] [Commented] (SOLR-13013) Change export to extract DocValues in docID order
[ https://issues.apache.org/jira/browse/SOLR-13013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16702126#comment-16702126 ] Toke Eskildsen commented on SOLR-13013: --- Ah! I understand now, thanks. Guess I got a bit too focused on index data IO. > Change export to extract DocValues in docID order > - > > Key: SOLR-13013 > URL: https://issues.apache.org/jira/browse/SOLR-13013 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: Export Writer >Affects Versions: 7.5, master (8.0) > Reporter: Toke Eskildsen >Priority: Major > Fix For: master (8.0) > > Attachments: SOLR-13013_proof_of_concept.patch, > SOLR-13013_proof_of_concept.patch > > > The streaming export writer uses a sliding window of 30,000 documents for > paging through the result set in a given sort order. Each time a window has > been calculated, the values for the export fields are retrieved from the > underlying DocValues structures in document sort order and delivered. > The iterative DocValues API introduced in Lucene/Solr 7 does not support > random access. The current export implementation bypasses this by creating a > new DocValues-iterator for each individual value to retrieve. This slows down > export as the iterator has to seek to the given docID from start for each > value. The slowdown scales with shard size (see LUCENE-8374 for details). An > alternative is to extract the DocValues in docID-order, with re-use of > DocValues-iterators. The idea is as follows: > # Change the FieldWriters for export to re-use the DocValues-iterators if > subsequent requests are for docIDs higher than the previous ones > # Calculate the sliding window of SortDocs as usual > # Take a note of the order of the SortDocs in the sliding window > # Re-sort the SortDocs in docID-order > # Extract the DocValues to a temporary on-heap structure > # Re-sort the extracted values to the original sliding window order > Deliver the values > One big difference from the current export code is of course the need to hold > the whole sliding window scaled result set in memory. This might well be a > showstopper as there is no real limit to how large this partial result set > can be. Maybe such an optimization could be requested explicitly if the user > knows that there is enough memory? -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13013) Change export to extract DocValues in docID order
[ https://issues.apache.org/jira/browse/SOLR-13013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701946#comment-16701946 ] Toke Eskildsen commented on SOLR-13013: --- [~joel.bernstein] Unfortunately I don't have proper hardware at hand to test with our large shards in a multi-shard setup. I _could_ put them on a spinning drive, now that I think about it, but I am also afraid that my test-box does not have adequate memory to fully cache the DocValues structures when using multiple shards, so that would complicate testing somewhat. I'll see what else we have lying around and if nothing else, I could just delete 3/4th of the data in 4 of the shards and run with those instead (takes some days to do though). Up until now we have used export exclusively to do simple query-bases data-dumps, so that was my go-to case. It is probably due to my limited understanding of Streaming Expressions that I do not understand the methodological problem in my test: I get that multi-sharding, replicas and hashing (bit unsure about the hashing part) can distribute and parallelize the load to make processing faster, but only the "more and smaller shards" of those 3 would reduce the total amount of work, as I understand it? So with regard to that, any optimization to the export should work equally well for a single-shard simple export and a more complex distributed setup, measured as total work to be done? I am on (even) more shaky grounds with the local reduce operation. Isn't that step after the export part and therefore extremely dependent on raw export speed? Or is there some shortcut mechanism I haven't understood? > Change export to extract DocValues in docID order > - > > Key: SOLR-13013 > URL: https://issues.apache.org/jira/browse/SOLR-13013 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: Export Writer >Affects Versions: 7.5, master (8.0) >Reporter: Toke Eskildsen >Priority: Major > Fix For: master (8.0) > > Attachments: SOLR-13013_proof_of_concept.patch, > SOLR-13013_proof_of_concept.patch > > > The streaming export writer uses a sliding window of 30,000 documents for > paging through the result set in a given sort order. Each time a window has > been calculated, the values for the export fields are retrieved from the > underlying DocValues structures in document sort order and delivered. > The iterative DocValues API introduced in Lucene/Solr 7 does not support > random access. The current export implementation bypasses this by creating a > new DocValues-iterator for each individual value to retrieve. This slows down > export as the iterator has to seek to the given docID from start for each > value. The slowdown scales with shard size (see LUCENE-8374 for details). An > alternative is to extract the DocValues in docID-order, with re-use of > DocValues-iterators. The idea is as follows: > # Change the FieldWriters for export to re-use the DocValues-iterators if > subsequent requests are for docIDs higher than the previous ones > # Calculate the sliding window of SortDocs as usual > # Take a note of the order of the SortDocs in the sliding window > # Re-sort the SortDocs in docID-order > # Extract the DocValues to a temporary on-heap structure > # Re-sort the extracted values to the original sliding window order > Deliver the values > One big difference from the current export code is of course the need to hold > the whole sliding window scaled result set in memory. This might well be a > showstopper as there is no real limit to how large this partial result set > can be. Maybe such an optimization could be requested explicitly if the user > knows that there is enough memory? -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13013) Change export to extract DocValues in docID order
[
https://issues.apache.org/jira/browse/SOLR-13013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701727#comment-16701727
]
Toke Eskildsen commented on SOLR-13013:
---
I cherry-picked some DENSE fields from our netarchive index and tried exporting
them from a single shard, to demonstrate the problem with large indexes in
Lucene/Solr 7+ and to performance test the current patch.
I made sure everything was warmed (practically zero IO on the index-SSD
according to iostat) and tested with combinations of SOLR-13013 and LUCENE-8374
turned on and off:
{code}
> curl -s "http://localhost:9090/solr/ns80/select?q=*:*; | jq .response.numFound
307171504
> curl -s "http://localhost:9090/solr/ns80/select?q=text:hestevogn; | jq
> .response.numFound'
52654
> curl -s -w "%\{time_total} seconds"$\'\n\'
> "http://localhost:9090/solr/ns80/export?q=text:hestevogn=id+asc=content_type_ext,content_type_served,crawl_date,content_length=true=true;
> -o t_export_true_true
0.433661 seconds
> curl -s -w "%\{time_total} seconds"$\'\n\'
> "http://localhost:9090/solr/ns80/export?q=text:hestevogn=id+asc=content_type_ext,content_type_served,crawl_date,content_length=true=false;
> -o t_export_true_false
0.555844 seconds
> curl -s -w "%\{time_total} seconds"$\'\n\'
> "http://localhost:9090/solr/ns80/export?q=text:hestevogn=id+asc=content_type_ext,content_type_served,crawl_date,content_length=false=true;
> -o t_export_false_true
1.037004 seconds
> curl -s -w "%\{time_total} seconds"$\'\n\'
> "http://localhost:9090/solr/ns80/export?q=text:hestevogn=id+asc=content_type_ext,content_type_served,crawl_date,content_length=false=false;
> -o t_export_false_false
843.477925 seconds
> diff -s t_export_true_true t_export_true_false ; diff -s t_export_true_true
> t_export_false_true ; diff -s t_export_true_true t_export_false_false
Files t_export_true_true and t_export_true_false are identical
Files t_export_true_true and t_export_false_true are identical
Files t_export_true_true and t_export_false_false are identical
{code}
Observations from this ad-hoc test (which of course should be independently
verified): Exporting from a large index with vanilla Solr master is not ideal.
It does not make much sense to talk about what performance-factors the patches
provides as they are mostly about changing time complexity: Our factor 1500
speed-up with SOLR-13013 with this shard with this request will be something
quite else for other setups.
* The explicit sort in SOLR-13013 seems the superior solution and the addition
of the O(n) → O(1) lookup-improvement in LUCENE-8374 only makes it slightly
faster.
* On the other hand, LUCENE-8374 works quite well for export and does not
require any changes to export. This might influence whether or not energy
should be spend on a "best as possible" fallback in case of memory problems or
if simpler "full fallback to sliding window sort order" is preferable.
* On the gripping hand, testing with a smaller index is likely to result in
SOLR-13013 being (relative to LUCENE-8374) even faster, as SOLR-13013 avoids
re-opening DV-readers all the time. More testing needed (no surprise there).
> Change export to extract DocValues in docID order
> -
>
> Key: SOLR-13013
> URL: https://issues.apache.org/jira/browse/SOLR-13013
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: Export Writer
>Affects Versions: 7.5, master (8.0)
>Reporter: Toke Eskildsen
>Priority: Major
> Fix For: master (8.0)
>
> Attachments: SOLR-13013_proof_of_concept.patch,
> SOLR-13013_proof_of_concept.patch
>
>
> The streaming export writer uses a sliding window of 30,000 documents for
> paging through the result set in a given sort order. Each time a window has
> been calculated, the values for the export fields are retrieved from the
> underlying DocValues structures in document sort order and delivered.
> The iterative DocValues API introduced in Lucene/Solr 7 does not support
> random access. The current export implementation bypasses this by creating a
> new DocValues-iterator for each individual value to retrieve. This slows down
> export as the iterator has to seek to the given docID from start for each
> value. The slowdown scales with shard size (see LUCENE-8374 for details). An
> alternative is to extract the DocValues in docID-order, with re-use of
> DocValues-iterators. The idea is as follows:
> # Change the FieldWriters for export to re-use the DocValues-iterators if
[jira] [Comment Edited] (SOLR-13013) Change export to extract DocValues in docID order
[
https://issues.apache.org/jira/browse/SOLR-13013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701727#comment-16701727
]
Toke Eskildsen edited comment on SOLR-13013 at 11/28/18 11:15 AM:
--
I cherry-picked some DENSE fields from our netarchive index and tried exporting
them from a single shard, to demonstrate the problem with large indexes in
Lucene/Solr 7+ and to performance test the current patch.
I made sure everything was warmed (practically zero IO on the index-SSD
according to iostat) and tested with combinations of SOLR-13013 and LUCENE-8374
turned on and off:
{code:java}
> curl -s "http://localhost:9090/solr/ns80/select?q=*:*; | jq .response.numFound
307171504
> curl -s "http://localhost:9090/solr/ns80/select?q=text:hestevogn; | jq
> .response.numFound'
52654
> curl -s -w "%\{time_total} seconds"$\'\n\'
> "http://localhost:9090/solr/ns80/export?q=text:hestevogn=id+asc=content_type_ext,content_type_served,crawl_date,content_length=true=true;
> -o t_export_true_true
0.433661 seconds
> curl -s -w "%\{time_total} seconds"$\'\n\'
> "http://localhost:9090/solr/ns80/export?q=text:hestevogn=id+asc=content_type_ext,content_type_served,crawl_date,content_length=true=false;
> -o t_export_true_false
0.555844 seconds
> curl -s -w "%\{time_total} seconds"$\'\n\'
> "http://localhost:9090/solr/ns80/export?q=text:hestevogn=id+asc=content_type_ext,content_type_served,crawl_date,content_length=false=true;
> -o t_export_false_true
1.037004 seconds
> curl -s -w "%\{time_total} seconds"$\'\n\'
> "http://localhost:9090/solr/ns80/export?q=text:hestevogn=id+asc=content_type_ext,content_type_served,crawl_date,content_length=false=false;
> -o t_export_false_false
843.477925 seconds
> diff -s t_export_true_true t_export_true_false ; diff -s t_export_true_true
> t_export_false_true ; diff -s t_export_true_true t_export_false_false
Files t_export_true_true and t_export_true_false are identical
Files t_export_true_true and t_export_false_true are identical
Files t_export_true_true and t_export_false_false are identical
{code}
Observations from this ad-hoc test (which of course should be independently
verified): Exporting from a large index with vanilla Solr master is not ideal.
It does not make much sense to talk about what performance-factors the patches
provides as they are mostly about changing time complexity: Our factor 1500
speed-up with SOLR-13013 with this shard with this request will be something
quite else for other setups.
* The explicit sort in SOLR-13013 seems the superior solution and the addition
of the O\(n) → O(1) lookup-improvement in LUCENE-8374 only makes it slightly
faster.
* On the other hand, LUCENE-8374 works quite well for export and does not
require any changes to export. This might influence whether or not energy
should be spend on a "best as possible" fallback in case of memory problems or
if simpler "full fallback to sliding window sort order" is preferable.
* On the gripping hand, testing with a smaller index is likely to result in
SOLR-13013 being (relative to LUCENE-8374) even faster, as SOLR-13013 avoids
re-opening DV-readers all the time. More testing needed (no surprise there).
was (Author: toke):
I cherry-picked some DENSE fields from our netarchive index and tried exporting
them from a single shard, to demonstrate the problem with large indexes in
Lucene/Solr 7+ and to performance test the current patch.
I made sure everything was warmed (practically zero IO on the index-SSD
according to iostat) and tested with combinations of SOLR-13013 and LUCENE-8374
turned on and off:
{code}
> curl -s "http://localhost:9090/solr/ns80/select?q=*:*; | jq .response.numFound
307171504
> curl -s "http://localhost:9090/solr/ns80/select?q=text:hestevogn; | jq
> .response.numFound'
52654
> curl -s -w "%\{time_total} seconds"$\'\n\'
> "http://localhost:9090/solr/ns80/export?q=text:hestevogn=id+asc=content_type_ext,content_type_served,crawl_date,content_length=true=true;
> -o t_export_true_true
0.433661 seconds
> curl -s -w "%\{time_total} seconds"$\'\n\'
> "http://localhost:9090/solr/ns80/export?q=text:hestevogn=id+asc=content_type_ext,content_type_served,crawl_date,content_length=true=false;
> -o t_export_true_false
0.555844 seconds
> curl -s -w "%\{time_total} seconds"$\'\n\'
> "http://localhost:9090/solr/ns80/export?q=text:hestevogn=id+asc=content_type_ext,content_type_served,crawl_date,content_length=false=true;
> -o t_export_false_true
1.037004 seconds
> curl -s -w "%\{time_total} seconds"$\'\n\'
> "http://localhost:9090/solr/ns80/export?q=text:hestevogn=id+asc=content_type_ext,content_t
[jira] [Commented] (SOLR-13013) Change export to extract DocValues in docID order
[
https://issues.apache.org/jira/browse/SOLR-13013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699385#comment-16699385
]
Toke Eskildsen commented on SOLR-13013:
---
[~joel.bernstein] I am glad that it looks useful. I expect that it needs at
least a full re-implementation of the {{MapWriter}}-parts. I am unfamiliar with
that part of the code, and it would be great if you took over. I won't take any
offense if you rewrite everything.
I'd be happy to try and review or at least do some testing on our 300M
docs/segment shards as they are very affected by the DV API change.
> Change export to extract DocValues in docID order
> -
>
> Key: SOLR-13013
> URL: https://issues.apache.org/jira/browse/SOLR-13013
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: Export Writer
>Affects Versions: 7.5, master (8.0)
> Reporter: Toke Eskildsen
>Priority: Major
> Fix For: master (8.0)
>
> Attachments: SOLR-13013_proof_of_concept.patch,
> SOLR-13013_proof_of_concept.patch
>
>
> The streaming export writer uses a sliding window of 30,000 documents for
> paging through the result set in a given sort order. Each time a window has
> been calculated, the values for the export fields are retrieved from the
> underlying DocValues structures in document sort order and delivered.
> The iterative DocValues API introduced in Lucene/Solr 7 does not support
> random access. The current export implementation bypasses this by creating a
> new DocValues-iterator for each individual value to retrieve. This slows down
> export as the iterator has to seek to the given docID from start for each
> value. The slowdown scales with shard size (see LUCENE-8374 for details). An
> alternative is to extract the DocValues in docID-order, with re-use of
> DocValues-iterators. The idea is as follows:
> # Change the FieldWriters for export to re-use the DocValues-iterators if
> subsequent requests are for docIDs higher than the previous ones
> # Calculate the sliding window of SortDocs as usual
> # Take a note of the order of the SortDocs in the sliding window
> # Re-sort the SortDocs in docID-order
> # Extract the DocValues to a temporary on-heap structure
> # Re-sort the extracted values to the original sliding window order
> Deliver the values
> One big difference from the current export code is of course the need to hold
> the whole sliding window scaled result set in memory. This might well be a
> showstopper as there is no real limit to how large this partial result set
> can be. Maybe such an optimization could be requested explicitly if the user
> knows that there is enough memory?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13013) Change export to extract DocValues in docID order
[ https://issues.apache.org/jira/browse/SOLR-13013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16698207#comment-16698207 ] Toke Eskildsen commented on SOLR-13013: --- [~janhoy] Nice idea! The values are collected as Objects, so it would involve some instanceof-checks to estimate memory overhead. I am unsure how that would affect performance, but it would be great if we could both avoid the risk of OOM and make it simpler for the user. Technically a fallback would be easy to do. It would even be possible to do a partial fallback: If the memory limit for the buffered values is reached before all 30K SortDocs has been processed, switch back to standard "sort order DV-resolving with immediate delivery", but use the already collected values whenever possible. Adjusting windows size for subsequent windows is tricky, as is requires weighing query + sort cost vs. DV-retrieval cost. It would be possible to collect a run time profile of the two parts and use that for qualified guessing, but then it's sounding like quite a large project. If we determine that the base idea of this JIRA-issue has merit, the first version could just use a simple fallback to sort order DV-resolving, without any re-use of already collected values, and stay in that mode for the rest of the current export. Re-use and/or window shrinking could be later enhancements. > Change export to extract DocValues in docID order > - > > Key: SOLR-13013 > URL: https://issues.apache.org/jira/browse/SOLR-13013 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: Export Writer >Affects Versions: 7.5, master (8.0) >Reporter: Toke Eskildsen >Priority: Major > Fix For: master (8.0) > > Attachments: SOLR-13013_proof_of_concept.patch > > > The streaming export writer uses a sliding window of 30,000 documents for > paging through the result set in a given sort order. Each time a window has > been calculated, the values for the export fields are retrieved from the > underlying DocValues structures in document sort order and delivered. > The iterative DocValues API introduced in Lucene/Solr 7 does not support > random access. The current export implementation bypasses this by creating a > new DocValues-iterator for each individual value to retrieve. This slows down > export as the iterator has to seek to the given docID from start for each > value. The slowdown scales with shard size (see LUCENE-8374 for details). An > alternative is to extract the DocValues in docID-order, with re-use of > DocValues-iterators. The idea is as follows: > # Change the FieldWriters for export to re-use the DocValues-iterators if > subsequent requests are for docIDs higher than the previous ones > # Calculate the sliding window of SortDocs as usual > # Take a note of the order of the SortDocs in the sliding window > # Re-sort the SortDocs in docID-order > # Extract the DocValues to a temporary on-heap structure > # Re-sort the extracted values to the original sliding window order > Deliver the values > One big difference from the current export code is of course the need to hold > the whole sliding window scaled result set in memory. This might well be a > showstopper as there is no real limit to how large this partial result set > can be. Maybe such an optimization could be requested explicitly if the user > knows that there is enough memory? -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13010) max rollup for dates
[
https://issues.apache.org/jira/browse/SOLR-13010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16697909#comment-16697909
]
Toke Eskildsen commented on SOLR-13010:
---
For the years 1-, the ISO-formatted {{String}} representations are
comparable in alphanumeric order. Instead of parsing to {{Instant}} all the
time, maybe do a quick check to see if the {{String}} representation of the
timestamp is within simple comparison-range and if so, use {{String}}
comparison and {{String}} for {{dateMax}}?
Caveat lector: I am not familiar with the streaming code at all, so the above
is very much guesswork.
> max rollup for dates
>
>
> Key: SOLR-13010
> URL: https://issues.apache.org/jira/browse/SOLR-13010
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: streaming expressions
>Affects Versions: master (8.0)
>Reporter: Gus Heck
>Priority: Minor
> Attachments: SOLR-13010.patch
>
>
> Sometimes it's useful to know the maximum or minimum value of a date field
> that's passing through a rollup() command. For example in the following
> expression, the effect would be to portray the freshness of the data for the
> bucket on which the counts have been generated.
> {code:java}
> rollup(search(activate2018-tra, q="racecar_id:*",
> fl="id,presented_date,racecar_id", sort="racecar_id
> asc",rows=9),over=racecar_id, max(presented_date),count(*)){code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-13013) Change export to extract DocValues in docID order
[
https://issues.apache.org/jira/browse/SOLR-13013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16697846#comment-16697846
]
Toke Eskildsen edited comment on SOLR-13013 at 11/24/18 3:12 PM:
-
I have uploaded a proof of concept for the idea in the issue description. The
structure that collects and holds the temporary values is made by mashing the
keyboard until it worked and the performance test is Frankensteined from
existing unit-test code in {{TestExportWriter}}. Nevertheless unit-tests in
{{TestExportWriter}} passes and a performance test can be executed with
{code}
TES_SIZES="1000,1,10,20,30" ant -Dtests.heapsize=5g
-Dtests.codec=Lucene80 -Dtestmethod=testExportSpeed -Dtestcase=TestExportWriter
test | grep "TES:"
{code}
It takes a 10+ minutes and writes a summary at the end, For a quicker test, use
{{TES_SIZES="1000,1"}} or something like that. For my desktop the result was
{code}
Test 1/5: 1000 docs, trie: 11098 / 7525 docs/sec ( 147%), points: 7639 /
11552 docs/sec ( 66%)
Test 2/5: 1000 docs, trie: 15135 / 9269 docs/sec ( 163%), points: 27769 /
15986 docs/sec ( 174%)
Test 3/5: 1000 docs, trie: 11505 / 9593 docs/sec ( 120%), points: 37643 /
13584 docs/sec ( 277%)
Test 4/5: 1000 docs, trie: 17495 / 9730 docs/sec ( 180%), points: 39103 /
18222 docs/sec ( 215%)
Test 5/5: 1000 docs, trie: 17657 / 10331 docs/sec ( 171%), points: 37633 /
19104 docs/sec ( 197%)
--
Test 1/5: 1 docs, trie: 17018 / 7901 docs/sec ( 215%), points: 38606 /
12381 docs/sec ( 312%)
Test 2/5: 1 docs, trie: 17191 / 7879 docs/sec ( 218%), points: 39920 /
12404 docs/sec ( 322%)
Test 3/5: 1 docs, trie: 17218 / 7881 docs/sec ( 218%), points: 41696 /
12410 docs/sec ( 336%)
Test 4/5: 1 docs, trie: 17451 / 7884 docs/sec ( 221%), points: 41719 /
12360 docs/sec ( 338%)
Test 5/5: 1 docs, trie: 17227 / 7855 docs/sec ( 219%), points: 41879 /
12436 docs/sec ( 337%)
--
Test 1/5: 10 docs, trie: 15849 / 3718 docs/sec ( 426%), points: 36037 /
4841 docs/sec ( 744%)
Test 2/5: 10 docs, trie: 16348 / 3717 docs/sec ( 440%), points: 37994 /
4858 docs/sec ( 782%)
Test 3/5: 10 docs, trie: 15378 / 3718 docs/sec ( 414%), points: 38831 /
4872 docs/sec ( 797%)
Test 4/5: 10 docs, trie: 16042 / 3710 docs/sec ( 432%), points: 39084 /
4876 docs/sec ( 802%)
Test 5/5: 10 docs, trie: 16009 / 3713 docs/sec ( 431%), points: 39503 /
4865 docs/sec ( 812%)
--
Test 1/5: 20 docs, trie: 15403 / 3031 docs/sec ( 508%), points: 37349 /
3531 docs/sec (1058%)
Test 2/5: 20 docs, trie: 15853 / 3018 docs/sec ( 525%), points: 37509 /
3544 docs/sec (1058%)
Test 3/5: 20 docs, trie: 14993 / 3018 docs/sec ( 497%), points: 38468 /
3547 docs/sec (1084%)
Test 4/5: 20 docs, trie: 15191 / 3023 docs/sec ( 502%), points: 38684 /
3538 docs/sec (1093%)
Test 5/5: 20 docs, trie: 15678 / 3035 docs/sec ( 517%), points: 38729 /
3542 docs/sec (1093%)
--
Test 1/5: 30 docs, trie: 15529 / 2834 docs/sec ( 548%), points: 36911 /
3652 docs/sec (1011%)
Test 2/5: 30 docs, trie: 15455 / 2846 docs/sec ( 543%), points: 37705 /
3630 docs/sec (1039%)
Test 3/5: 30 docs, trie: 15805 / 2866 docs/sec ( 551%), points: 37583 /
3660 docs/sec (1027%)
Test 4/5: 30 docs, trie: 15653 / 2883 docs/sec ( 543%), points: 39365 /
3591 docs/sec (1096%)
Test 5/5: 30 docs, trie: 15736 / 2895 docs/sec ( 543%), points: 38606 /
3667 docs/sec (1053%)
{code}
The two numbers for trie and points are sorted followed by non_sorted. The
numbers in the parentheses are sorted/non_sorted. As can be seen, non_sorted
export performance degrades as index size (measured in number of documents)
goes up. Also, as can be seen from the percentages, reusing the
DocValues-iterators and ensuring docID order improved the speed significantly,
The patch is not at alll production-ready. See it as a "is this idea worth
exploring?". Ping to [~joel.bernstein], as I expect he will be interested in
this.
was (Author: toke):
I have uploaded a proof of concept for the idea in the issue description. The
structure that collects and holds the temporary values is made by mashing the
keyboard until it worked and the performance test is Frankensteined from
existing unit-test code in TestExportWriter. Nevertheless unit-tests in
TestExportWriter passes and a performance test can be executed with
{code}
TES_SIZES="1000,1,10,20,30" ant -Dtests.heapsize=5g
-Dtests.codec=Lucene80 -Dtestmethod=testExportSpeed -Dtestcase=TestExportWriter
test | grep "TES:"
{code}
It takes a 10+ minutes and writes a summary at the end, For a quicker test, use
TES_SIZES="1000,1" or something like that. For my desktop the result was
{code}
Concatena
[jira] [Comment Edited] (SOLR-13013) Change export to extract DocValues in docID order
[
https://issues.apache.org/jira/browse/SOLR-13013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16697846#comment-16697846
]
Toke Eskildsen edited comment on SOLR-13013 at 11/24/18 3:08 PM:
-
I have uploaded a proof of concept for the idea in the issue description. The
structure that collects and holds the temporary values is made by mashing the
keyboard until it worked and the performance test is Frankensteined from
existing unit-test code in TestExportWriter. Nevertheless unit-tests in
TestExportWriter passes and a performance test can be executed with
{code}
TES_SIZES="1000,1,10,20,30" ant -Dtests.heapsize=5g
-Dtests.codec=Lucene80 -Dtestmethod=testExportSpeed -Dtestcase=TestExportWriter
test | grep "TES:"
{code}
It takes a 10+ minutes and writes a summary at the end, For a quicker test, use
TES_SIZES="1000,1" or something like that. For my desktop the result was
{code}
Concatenated output:
Test 1/5: 1000 documents, trie: 11098 / 7525 docs/sec ( 147%), points:
7639 / 11552 docs/sec ( 66%)
Test 2/5: 1000 documents, trie: 15135 / 9269 docs/sec ( 163%), points:
27769 / 15986 docs/sec ( 174%)
Test 3/5: 1000 documents, trie: 11505 / 9593 docs/sec ( 120%), points:
37643 / 13584 docs/sec ( 277%)
Test 4/5: 1000 documents, trie: 17495 / 9730 docs/sec ( 180%), points:
39103 / 18222 docs/sec ( 215%)
Test 5/5: 1000 documents, trie: 17657 / 10331 docs/sec ( 171%), points:
37633 / 19104 docs/sec ( 197%)
--
Test 1/5: 1 documents, trie: 17018 / 7901 docs/sec ( 215%), points:
38606 / 12381 docs/sec ( 312%)
Test 2/5: 1 documents, trie: 17191 / 7879 docs/sec ( 218%), points:
39920 / 12404 docs/sec ( 322%)
Test 3/5: 1 documents, trie: 17218 / 7881 docs/sec ( 218%), points:
41696 / 12410 docs/sec ( 336%)
Test 4/5: 1 documents, trie: 17451 / 7884 docs/sec ( 221%), points:
41719 / 12360 docs/sec ( 338%)
Test 5/5: 1 documents, trie: 17227 / 7855 docs/sec ( 219%), points:
41879 / 12436 docs/sec ( 337%)
--
Test 1/5: 10 documents, trie: 15849 / 3718 docs/sec ( 426%), points:
36037 / 4841 docs/sec ( 744%)
Test 2/5: 10 documents, trie: 16348 / 3717 docs/sec ( 440%), points:
37994 / 4858 docs/sec ( 782%)
Test 3/5: 10 documents, trie: 15378 / 3718 docs/sec ( 414%), points:
38831 / 4872 docs/sec ( 797%)
Test 4/5: 10 documents, trie: 16042 / 3710 docs/sec ( 432%), points:
39084 / 4876 docs/sec ( 802%)
Test 5/5: 10 documents, trie: 16009 / 3713 docs/sec ( 431%), points:
39503 / 4865 docs/sec ( 812%)
--
Test 1/5: 20 documents, trie: 15403 / 3031 docs/sec ( 508%), points:
37349 / 3531 docs/sec (1058%)
Test 2/5: 20 documents, trie: 15853 / 3018 docs/sec ( 525%), points:
37509 / 3544 docs/sec (1058%)
Test 3/5: 20 documents, trie: 14993 / 3018 docs/sec ( 497%), points:
38468 / 3547 docs/sec (1084%)
Test 4/5: 20 documents, trie: 15191 / 3023 docs/sec ( 502%), points:
38684 / 3538 docs/sec (1093%)
Test 5/5: 20 documents, trie: 15678 / 3035 docs/sec ( 517%), points:
38729 / 3542 docs/sec (1093%)
--
Test 1/5: 30 documents, trie: 15529 / 2834 docs/sec ( 548%), points:
36911 / 3652 docs/sec (1011%)
Test 2/5: 30 documents, trie: 15455 / 2846 docs/sec ( 543%), points:
37705 / 3630 docs/sec (1039%)
Test 3/5: 30 documents, trie: 15805 / 2866 docs/sec ( 551%), points:
37583 / 3660 docs/sec (1027%)
Test 4/5: 30 documents, trie: 15653 / 2883 docs/sec ( 543%), points:
39365 / 3591 docs/sec (1096%)
Test 5/5: 30 documents, trie: 15736 / 2895 docs/sec ( 543%), points:
38606 / 3667 docs/sec (1053%)
{code}
The two numbers for trie and points are sorted followed by non_sorted. The
numbers in the parentheses are sorted/non_sorted. As can be seen, non_sorted
export performance degrades as index size (measured in number of documents)
goes up. Also, as can be seen from the percentages, reusing the
DocValues-iterators and ensuring docID order improved the speed significantly,
The patch is not at alll production-ready. See it as a "is this idea worth
exploring?". Ping to [~joel.bernstein], as I expect he will be interested in
this.
was (Author: toke):
I have uploaded a proof of concept for the idea in the issue description. The
structure that collects and holds the temporary values is made by mashing the
keyboard until it worked and the performance test is Frankensteined from
existing unit-test code in TestExportWriter. Nevertheless unit-tests in
TestExportWriter passes and a performance test can be executed with
{{TES_SIZES="1000,1,10,20,30" ant -Dtests.heapsize=5g
-Dtests.codec=Lucene80 -Dtestmethod=testExportSpeed -Dtestcase=TestExportWr
[jira] [Commented] (SOLR-13013) Change export to extract DocValues in docID order
[
https://issues.apache.org/jira/browse/SOLR-13013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16697846#comment-16697846
]
Toke Eskildsen commented on SOLR-13013:
---
I have uploaded a proof of concept for the idea in the issue description. The
structure that collects and holds the temporary values is made by mashing the
keyboard until it worked and the performance test is Frankensteined from
existing unit-test code in TestExportWriter. Nevertheless unit-tests in
TestExportWriter passes and a performance test can be executed with
{{TES_SIZES="1000,1,10,20,30" ant -Dtests.heapsize=5g
-Dtests.codec=Lucene80 -Dtestmethod=testExportSpeed -Dtestcase=TestExportWriter
test | grep "TES:"}}
It takes a 10+ minutes and writes a summary at the end, For a quicker test, use
TES_SIZES="1000,1" or something like that. For my desktop the result was
{{[junit4] 1> TES: Concatenated output:}}
{{ [junit4] 1> TES: Test 1/5: 1000 documents, trie: 11098 / 7525 docs/sec (
147%), points: 7639 / 11552 docs/sec ( 66%)}}
{{ [junit4] 1> TES: Test 2/5: 1000 documents, trie: 15135 / 9269 docs/sec (
163%), points: 27769 / 15986 docs/sec ( 174%)}}
{{ [junit4] 1> TES: Test 3/5: 1000 documents, trie: 11505 / 9593 docs/sec (
120%), points: 37643 / 13584 docs/sec ( 277%)}}
{{ [junit4] 1> TES: Test 4/5: 1000 documents, trie: 17495 / 9730 docs/sec (
180%), points: 39103 / 18222 docs/sec ( 215%)}}
{{ [junit4] 1> TES: Test 5/5: 1000 documents, trie: 17657 / 10331 docs/sec (
171%), points: 37633 / 19104 docs/sec ( 197%)}}
{{ [junit4] 1> TES: --}}
{{ [junit4] 1> TES: Test 1/5: 1 documents, trie: 17018 / 7901 docs/sec (
215%), points: 38606 / 12381 docs/sec ( 312%)}}
{{ [junit4] 1> TES: Test 2/5: 1 documents, trie: 17191 / 7879 docs/sec (
218%), points: 39920 / 12404 docs/sec ( 322%)}}
{{ [junit4] 1> TES: Test 3/5: 1 documents, trie: 17218 / 7881 docs/sec (
218%), points: 41696 / 12410 docs/sec ( 336%)}}
{{ [junit4] 1> TES: Test 4/5: 1 documents, trie: 17451 / 7884 docs/sec (
221%), points: 41719 / 12360 docs/sec ( 338%)}}
{{ [junit4] 1> TES: Test 5/5: 1 documents, trie: 17227 / 7855 docs/sec (
219%), points: 41879 / 12436 docs/sec ( 337%)}}
{{ [junit4] 1> TES: --}}
{{ [junit4] 1> TES: Test 1/5: 10 documents, trie: 15849 / 3718 docs/sec (
426%), points: 36037 / 4841 docs/sec ( 744%)}}
{{ [junit4] 1> TES: Test 2/5: 10 documents, trie: 16348 / 3717 docs/sec (
440%), points: 37994 / 4858 docs/sec ( 782%)}}
{{ [junit4] 1> TES: Test 3/5: 10 documents, trie: 15378 / 3718 docs/sec (
414%), points: 38831 / 4872 docs/sec ( 797%)}}
{{ [junit4] 1> TES: Test 4/5: 10 documents, trie: 16042 / 3710 docs/sec (
432%), points: 39084 / 4876 docs/sec ( 802%)}}
{{ [junit4] 1> TES: Test 5/5: 10 documents, trie: 16009 / 3713 docs/sec (
431%), points: 39503 / 4865 docs/sec ( 812%)}}
{{ [junit4] 1> TES: --}}
{{ [junit4] 1> TES: Test 1/5: 20 documents, trie: 15403 / 3031 docs/sec (
508%), points: 37349 / 3531 docs/sec (1058%)}}
{{ [junit4] 1> TES: Test 2/5: 20 documents, trie: 15853 / 3018 docs/sec (
525%), points: 37509 / 3544 docs/sec (1058%)}}
{{ [junit4] 1> TES: Test 3/5: 20 documents, trie: 14993 / 3018 docs/sec (
497%), points: 38468 / 3547 docs/sec (1084%)}}
{{ [junit4] 1> TES: Test 4/5: 20 documents, trie: 15191 / 3023 docs/sec (
502%), points: 38684 / 3538 docs/sec (1093%)}}
{{ [junit4] 1> TES: Test 5/5: 20 documents, trie: 15678 / 3035 docs/sec (
517%), points: 38729 / 3542 docs/sec (1093%)}}
{{ [junit4] 1> TES: --}}
{{ [junit4] 1> TES: Test 1/5: 30 documents, trie: 15529 / 2834 docs/sec (
548%), points: 36911 / 3652 docs/sec (1011%)}}
{{ [junit4] 1> TES: Test 2/5: 30 documents, trie: 15455 / 2846 docs/sec (
543%), points: 37705 / 3630 docs/sec (1039%)}}
{{ [junit4] 1> TES: Test 3/5: 30 documents, trie: 15805 / 2866 docs/sec (
551%), points: 37583 / 3660 docs/sec (1027%)}}
{{ [junit4] 1> TES: Test 4/5: 30 documents, trie: 15653 / 2883 docs/sec (
543%), points: 39365 / 3591 docs/sec (1096%)}}
{{ [junit4] 1> TES: Test 5/5: 30 documents, trie: 15736 / 2895 docs/sec (
543%), points: 38606 / 3667 docs/sec (1053%)}}
The two numbers for trie and points are sorted followed by non_sorted. The
numbers in the parentheses are sorted/non_sorted. As can be seen, non_sorted
export performance degrades as index size (measured in number of documents)
goes up. Also, as can be seen from the percentages, reusing the
DocValues-iterators and ensuring docID order improved the speed significantly,
The patch is not at alll production-ready. See it as a "is this idea worth
exploring?". Ping to [~joel.bernstein], as I expect he will be interested in
this.
> Change export to extract D
[jira] [Updated] (SOLR-13013) Change export to extract DocValues in docID order
[ https://issues.apache.org/jira/browse/SOLR-13013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Toke Eskildsen updated SOLR-13013: -- Attachment: SOLR-13013_proof_of_concept.patch > Change export to extract DocValues in docID order > - > > Key: SOLR-13013 > URL: https://issues.apache.org/jira/browse/SOLR-13013 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: Export Writer >Affects Versions: 7.5, master (8.0) > Reporter: Toke Eskildsen >Priority: Major > Fix For: master (8.0) > > Attachments: SOLR-13013_proof_of_concept.patch > > > The streaming export writer uses a sliding window of 30,000 documents for > paging through the result set in a given sort order. Each time a window has > been calculated, the values for the export fields are retrieved from the > underlying DocValues structures in document sort order and delivered. > The iterative DocValues API introduced in Lucene/Solr 7 does not support > random access. The current export implementation bypasses this by creating a > new DocValues-iterator for each individual value to retrieve. This slows down > export as the iterator has to seek to the given docID from start for each > value. The slowdown scales with shard size (see LUCENE-8374 for details). An > alternative is to extract the DocValues in docID-order, with re-use of > DocValues-iterators. The idea is as follows: > # Change the FieldWriters for export to re-use the DocValues-iterators if > subsequent requests are for docIDs higher than the previous ones > # Calculate the sliding window of SortDocs as usual > # Take a note of the order of the SortDocs in the sliding window > # Re-sort the SortDocs in docID-order > # Extract the DocValues to a temporary on-heap structure > # Re-sort the extracted values to the original sliding window order > Deliver the values > One big difference from the current export code is of course the need to hold > the whole sliding window scaled result set in memory. This might well be a > showstopper as there is no real limit to how large this partial result set > can be. Maybe such an optimization could be requested explicitly if the user > knows that there is enough memory? -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-13013) Change export to extract DocValues in docID order
Toke Eskildsen created SOLR-13013: - Summary: Change export to extract DocValues in docID order Key: SOLR-13013 URL: https://issues.apache.org/jira/browse/SOLR-13013 Project: Solr Issue Type: Improvement Security Level: Public (Default Security Level. Issues are Public) Components: Export Writer Affects Versions: 7.5, master (8.0) Reporter: Toke Eskildsen Fix For: master (8.0) The streaming export writer uses a sliding window of 30,000 documents for paging through the result set in a given sort order. Each time a window has been calculated, the values for the export fields are retrieved from the underlying DocValues structures in document sort order and delivered. The iterative DocValues API introduced in Lucene/Solr 7 does not support random access. The current export implementation bypasses this by creating a new DocValues-iterator for each individual value to retrieve. This slows down export as the iterator has to seek to the given docID from start for each value. The slowdown scales with shard size (see LUCENE-8374 for details). An alternative is to extract the DocValues in docID-order, with re-use of DocValues-iterators. The idea is as follows: # Change the FieldWriters for export to re-use the DocValues-iterators if subsequent requests are for docIDs higher than the previous ones # Calculate the sliding window of SortDocs as usual # Take a note of the order of the SortDocs in the sliding window # Re-sort the SortDocs in docID-order # Extract the DocValues to a temporary on-heap structure # Re-sort the extracted values to the original sliding window order Deliver the values One big difference from the current export code is of course the need to hold the whole sliding window scaled result set in memory. This might well be a showstopper as there is no real limit to how large this partial result set can be. Maybe such an optimization could be requested explicitly if the user knows that there is enough memory? -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Resolved] (LUCENE-2369) Locale-based sort by field with low memory overhead
[ https://issues.apache.org/jira/browse/LUCENE-2369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Toke Eskildsen resolved LUCENE-2369. Resolution: Won't Fix I am closing this as I have not touched it for years and since it was probably a bad idea to start with. > Locale-based sort by field with low memory overhead > --- > > Key: LUCENE-2369 > URL: https://issues.apache.org/jira/browse/LUCENE-2369 > Project: Lucene - Core > Issue Type: New Feature > Components: core/search >Affects Versions: 4.0 > Reporter: Toke Eskildsen >Priority: Minor > Labels: memory, search, sort, time > Attachments: LUCENE-2369.patch, LUCENE-2369.patch, LUCENE-2369.patch, > LUCENE-2369.patch, LUCENE-2369.patch, lucene-2369-20101011.patch > > > The current implementation of locale-based sort in Lucene uses the FieldCache > which keeps all sort terms in memory. Beside the huge memory overhead, > searching requires comparison of terms with collator.compare every time, > making searches with millions of hits fairly expensive. > This proposed alternative implementation is to create a packed list of > pre-sorted ordinals for the sort terms and a map from document-IDs to entries > in the sorted ordinals list. This results in very low memory overhead and > faster sorted searches, at the cost of increased startup-time. As the > ordinals can be resolved to terms after the sorting has been performed, this > approach supports fillFields=true. > This issue is related to https://issues.apache.org/jira/browse/LUCENE-2335 > which contain previous discussions on the subject. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8374) Reduce reads for sparse DocValues
[
https://issues.apache.org/jira/browse/LUCENE-8374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16693349#comment-16693349
]
Toke Eskildsen commented on LUCENE-8374:
{quote}this patch spends significant CPU and I/O when opening the reader
{quote}
This patch spends the IO equivalent of making a single worst-case search
matching 1/16384th of the documents in the newly opened segment, sorting on a
numeric DocValues field, for each used DocValued field. I am hard pressed to
imagine a setup where that IO-penalty wouldn't pay off very quickly.
As for CPU it does perform a compression on the DENSE data structure, to lower
the memory overhead. In Tim's setup, I could not get the timing for his smaller
segments as they took less than 1ms. For his largest segment (19M documents) it
took a total of 23ms across 41 DocValues fields. For Tim his startup overhead
(a total of 412ms across all segments & fields) was paid off in less than a
second of operation.
Based on this, I disagree on your assertion of "significant" performance
overhead for search-time. It might turn out to be significant when applied
broadly or tested in depth, but at this point neither theory nor observations
suggests so.
{quote}For instance it will be easier to track which change a slowdown/speedup
relates to in Mike's nightly benchmarks.
{quote}
As [discussed|https://github.com/mikemccand/luceneutil/issues/23] on Mike's
GitHub space, his nightly benchmarks are not really tracking this part
(large-jump access of DocValues) of Lucene. If they were, an alarm would have
been raised on the switch to the iterative API.
It is quite understandable that such tests are missing, as large indexes
requires a lot of computing resources and/or time. How to quickly catch such
regressions in the future, without spending a lump sum on hardware, is a
challenge that I have no ready answers for.
In its current form, Mike's benchmarks seems fine for tracking LUCENE-8374
regressions in the areas where there is no expectation of a speed-up, which is
an important part of testing the value of an attempted optimization.
{quote}I'll take care of reviews.
{quote}
Thank you, I appreciate that.
As for the wait-for-7.6, I just based it on the note in
[https://mail-archives.apache.org/mod_mbox/lucene-dev/201811.mbox/ajax/%3CCABM%2Bu9JU%3DQFshN_juicOohU8dZQBtZ0Nc7Zs79_Wt9wZRjTVqA%40mail.gmail.com%3E]
about big changes. I am happy to classify this patch as a non-big change.
> Reduce reads for sparse DocValues
> -
>
> Key: LUCENE-8374
> URL: https://issues.apache.org/jira/browse/LUCENE-8374
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
>Affects Versions: 7.5, master (8.0)
>Reporter: Toke Eskildsen
>Priority: Major
> Labels: performance
> Fix For: 7.6
>
> Attachments: LUCENE-8374.patch, LUCENE-8374.patch, LUCENE-8374.patch,
> LUCENE-8374.patch, LUCENE-8374.patch, LUCENE-8374.patch, LUCENE-8374.patch,
> LUCENE-8374_branch_7_3.patch, LUCENE-8374_branch_7_3.patch.20181005,
> LUCENE-8374_branch_7_4.patch, LUCENE-8374_branch_7_5.patch,
> entire_index_logs.txt, image-2018-10-24-07-30-06-663.png,
> image-2018-10-24-07-30-56-962.png, single_vehicle_logs.txt,
> start-2018-10-24-1_snapshot___Users_tim_Snapshots__-_YourKit_Java_Profiler_2017_02-b75_-_64-bit.png,
>
> start-2018-10-24_snapshot___Users_tim_Snapshots__-_YourKit_Java_Profiler_2017_02-b75_-_64-bit.png
>
>
> The {{Lucene70DocValuesProducer}} has the internal classes
> {{SparseNumericDocValues}} and {{BaseSortedSetDocValues}} (sparse code path),
> which again uses {{IndexedDISI}} to handle the docID -> value-ordinal lookup.
> The value-ordinal is the index of the docID assuming an abstract tightly
> packed monotonically increasing list of docIDs: If the docIDs with
> corresponding values are {{[0, 4, 1432]}}, their value-ordinals will be {{[0,
> 1, 2]}}.
> h2. Outer blocks
> The lookup structure of {{IndexedDISI}} consists of blocks of 2^16 values
> (65536), where each block can be either {{ALL}}, {{DENSE}} (2^12 to 2^16
> values) or {{SPARSE}} (< 2^12 values ~= 6%). Consequently blocks vary quite a
> lot in size and ordinal resolving strategy.
> When a sparse Numeric DocValue is needed, the code first locates the block
> containing the wanted docID flag. It does so by iterating blocks one-by-one
> until it reaches the needed one, where each iteration requires a lookup in
> the underlying {{IndexSlice}}. For a common memory mapped index, this
> translates to either a cached request or a read operation. If a segment has
> 6M documents, worst-case is 91 lookups. In our we
[jira] [Commented] (SOLR-12074) Add numeric typed equivalents to StrField
[
https://issues.apache.org/jira/browse/SOLR-12074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16691027#comment-16691027
]
Toke Eskildsen commented on SOLR-12074:
---
How would this work with facet-sorting on index and the stats component? Would
it behave as a numeric or a String field?
> Add numeric typed equivalents to StrField
> -
>
> Key: SOLR-12074
> URL: https://issues.apache.org/jira/browse/SOLR-12074
> Project: Solr
> Issue Type: New Feature
> Security Level: Public(Default Security Level. Issues are Public)
> Components: Schema and Analysis
>Reporter: David Smiley
>Priority: Major
> Labels: newdev, numeric-tries-to-points
>
> There ought to be numeric typed equivalents to StrField in the schema. The
> TrieField types can be configured to do this with precisionStep=0, but the
> TrieFields are deprecated and slated for removal in 8.0. PointFields may be
> adequate for some use cases but, unlike TrieField, it's not as efficient for
> simple field:value lookup queries. They probably should use the same
> internal sortable full-precision term format that TrieField uses (details
> currently in {{LegacyNumericUtils}} (which are used by the deprecated Trie
> fields).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8374) Reduce reads for sparse DocValues
[
https://issues.apache.org/jira/browse/LUCENE-8374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16691026#comment-16691026
]
Toke Eskildsen commented on LUCENE-8374:
[~jpountz] I do not understand your resistance against providing an
out-of-the-box performance improvement to Lucene-based search engines. I do not
see index-upgrades as trivial exercises and I have previously encountered quite
a lot of resistance against trying patched versions of Lucene/Solr. As a
developer it is probably second nature to you. For a lot of users, not so much.
Your patch-splitting suggestion is interesting. The three parts [DISI-blocks,
dense and variable bits per value] could reasonably easy be split into
[DISI-blocks, dense] and [variable bits per value]. Or they could not-so-easily
be split into all 3 parts. They would be additive though (as opposed to
independent), as they share a lot of structure. If splitting helps acceptance,
I'll try and commit that way. I'm also game to make a split patch, if someone
commits to do a review.
Note: I am aware that Lucene/Solr 7.6 is in pre-release and that larger patches
should not be committed to master until 7.6 has been released.
> Reduce reads for sparse DocValues
> -
>
> Key: LUCENE-8374
> URL: https://issues.apache.org/jira/browse/LUCENE-8374
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
>Affects Versions: 7.5, master (8.0)
> Reporter: Toke Eskildsen
>Priority: Major
> Labels: performance
> Fix For: 7.6
>
> Attachments: LUCENE-8374.patch, LUCENE-8374.patch, LUCENE-8374.patch,
> LUCENE-8374.patch, LUCENE-8374.patch, LUCENE-8374.patch, LUCENE-8374.patch,
> LUCENE-8374_branch_7_3.patch, LUCENE-8374_branch_7_3.patch.20181005,
> LUCENE-8374_branch_7_4.patch, LUCENE-8374_branch_7_5.patch,
> entire_index_logs.txt, image-2018-10-24-07-30-06-663.png,
> image-2018-10-24-07-30-56-962.png, single_vehicle_logs.txt,
> start-2018-10-24-1_snapshot___Users_tim_Snapshots__-_YourKit_Java_Profiler_2017_02-b75_-_64-bit.png,
>
> start-2018-10-24_snapshot___Users_tim_Snapshots__-_YourKit_Java_Profiler_2017_02-b75_-_64-bit.png
>
>
> The {{Lucene70DocValuesProducer}} has the internal classes
> {{SparseNumericDocValues}} and {{BaseSortedSetDocValues}} (sparse code path),
> which again uses {{IndexedDISI}} to handle the docID -> value-ordinal lookup.
> The value-ordinal is the index of the docID assuming an abstract tightly
> packed monotonically increasing list of docIDs: If the docIDs with
> corresponding values are {{[0, 4, 1432]}}, their value-ordinals will be {{[0,
> 1, 2]}}.
> h2. Outer blocks
> The lookup structure of {{IndexedDISI}} consists of blocks of 2^16 values
> (65536), where each block can be either {{ALL}}, {{DENSE}} (2^12 to 2^16
> values) or {{SPARSE}} (< 2^12 values ~= 6%). Consequently blocks vary quite a
> lot in size and ordinal resolving strategy.
> When a sparse Numeric DocValue is needed, the code first locates the block
> containing the wanted docID flag. It does so by iterating blocks one-by-one
> until it reaches the needed one, where each iteration requires a lookup in
> the underlying {{IndexSlice}}. For a common memory mapped index, this
> translates to either a cached request or a read operation. If a segment has
> 6M documents, worst-case is 91 lookups. In our web archive, our segments has
> ~300M values: A worst-case of 4577 lookups!
> One obvious solution is to use a lookup-table for blocks: A long[]-array with
> an entry for each block. For 6M documents, that is < 1KB and would allow for
> direct jumping (a single lookup) in all instances. Unfortunately this
> lookup-table cannot be generated upfront when the writing of values is purely
> streaming. It can be appended to the end of the stream before it is closed,
> but without knowing the position of the lookup-table the reader cannot seek
> to it.
> One strategy for creating such a lookup-table would be to generate it during
> reads and cache it for next lookup. This does not fit directly into how
> {{IndexedDISI}} currently works (it is created anew for each invocation), but
> could probably be added with a little work. An advantage to this is that this
> does not change the underlying format and thus could be used with existing
> indexes.
> h2. The lookup structure inside each block
> If {{ALL}} of the 2^16 values are defined, the structure is empty and the
> ordinal is simply the requested docID with some modulo and multiply math.
> Nothing to improve there.
> If the block is {{DENSE}} (2^12 to 2^16 values are defi
Re: Lucene/Solr 7.6
jim ferenczi wrote: > That's a big patch and as you said it needs a proper review. Reviewing large patches is a problem in itself. I guess some baking time on master is a secondary alternative, which unfortunately does not fit into the idea of getting this in 7.6. > Considering that the release should be in two weeks I think we > shouldn't rush this in 7.6. That's my 2 cents but it also depends > if we do a 7.7 or not. That is one reason I am pushing this a bit. It would be quite a missed opportunity to not get this into 7.x at some point, as I know the reluctance to upgrade to new major versions. Thanks, Toke Eskildsen - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Lucene/Solr 7.6
On Tue, 2018-11-06 at 15:41 -0600, Nicholas Knize wrote: > Let me know if there are any other blockers that need to be resolved > prior to cutting the branch. If not, I will plan to cut the branch on > Friday or (provided they are close to resolution) whenever these > issues are resolved. Not a blocker, but I have LUCENE-8374 (DocValues performance optimization) which I expect would be very beneficial for 7.6. It needs back porting from master (30 minutes by me as the relevant code paths are virtually the same between 7.6 & master), but the main problem is that it is a 2000+ line patch that has not been reviewed. It seems a bit forced to add it to 7.6, but on the other hand it will be tested thoroughly as part of the release process. What is the best action here? - Toke Eskildsen, Royal Danish Library - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12964) Use advanceExact instead of advance in a few remaining json facet use cases
[ https://issues.apache.org/jira/browse/SOLR-12964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16679479#comment-16679479 ] Toke Eskildsen commented on SOLR-12964: --- Looks like a clean change that also makes the code easier to read. > Use advanceExact instead of advance in a few remaining json facet use cases > --- > > Key: SOLR-12964 > URL: https://issues.apache.org/jira/browse/SOLR-12964 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: Facet Module >Affects Versions: 7.5 >Reporter: Tim Underwood >Assignee: David Smiley >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > This updates 2 places in the JSON Facets code that uses the advance()/docID() > pattern instead of the simpler advanceExact(). Most other usages in the > faceting code already make use of advanceExact(). > The only remaining usage of advance() in org.apache.solr.search.facet is in: > * UniqueAgg.BaseNumericAcc.collect > * HLLAgg..BaseNumericAcc.collect > The code for those of those looks very similar and probably makes sense to > update but it would require changing the return type of the protected > docIdSetIterator() method to return a DocValuesIterator in order to be able > to call the advanceExact() method. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8374) Reduce reads for sparse DocValues
[
https://issues.apache.org/jira/browse/LUCENE-8374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676295#comment-16676295
]
Toke Eskildsen commented on LUCENE-8374:
Thank you, [~dsmiley] & [~thetaphi]. The jump-tables are field-oriented, so the
amount of output is currently {{#segments * #DocValue_fields * #reopens =
verbose}}. Much too fine-grained from what Uwe describes. I'll remove it all.
Same goes for the options for enabling & disabling the caches. Should it be
relevant at a later point, that part is quite easy to re-introduce.
> Reduce reads for sparse DocValues
> -
>
> Key: LUCENE-8374
> URL: https://issues.apache.org/jira/browse/LUCENE-8374
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
>Affects Versions: 7.5, master (8.0)
>Reporter: Toke Eskildsen
>Priority: Major
> Labels: performance
> Fix For: 7.6
>
> Attachments: LUCENE-8374.patch, LUCENE-8374.patch, LUCENE-8374.patch,
> LUCENE-8374.patch, LUCENE-8374.patch, LUCENE-8374_branch_7_3.patch,
> LUCENE-8374_branch_7_3.patch.20181005, LUCENE-8374_branch_7_4.patch,
> LUCENE-8374_branch_7_5.patch, entire_index_logs.txt,
> image-2018-10-24-07-30-06-663.png, image-2018-10-24-07-30-56-962.png,
> single_vehicle_logs.txt,
> start-2018-10-24-1_snapshot___Users_tim_Snapshots__-_YourKit_Java_Profiler_2017_02-b75_-_64-bit.png,
>
> start-2018-10-24_snapshot___Users_tim_Snapshots__-_YourKit_Java_Profiler_2017_02-b75_-_64-bit.png
>
>
> The {{Lucene70DocValuesProducer}} has the internal classes
> {{SparseNumericDocValues}} and {{BaseSortedSetDocValues}} (sparse code path),
> which again uses {{IndexedDISI}} to handle the docID -> value-ordinal lookup.
> The value-ordinal is the index of the docID assuming an abstract tightly
> packed monotonically increasing list of docIDs: If the docIDs with
> corresponding values are {{[0, 4, 1432]}}, their value-ordinals will be {{[0,
> 1, 2]}}.
> h2. Outer blocks
> The lookup structure of {{IndexedDISI}} consists of blocks of 2^16 values
> (65536), where each block can be either {{ALL}}, {{DENSE}} (2^12 to 2^16
> values) or {{SPARSE}} (< 2^12 values ~= 6%). Consequently blocks vary quite a
> lot in size and ordinal resolving strategy.
> When a sparse Numeric DocValue is needed, the code first locates the block
> containing the wanted docID flag. It does so by iterating blocks one-by-one
> until it reaches the needed one, where each iteration requires a lookup in
> the underlying {{IndexSlice}}. For a common memory mapped index, this
> translates to either a cached request or a read operation. If a segment has
> 6M documents, worst-case is 91 lookups. In our web archive, our segments has
> ~300M values: A worst-case of 4577 lookups!
> One obvious solution is to use a lookup-table for blocks: A long[]-array with
> an entry for each block. For 6M documents, that is < 1KB and would allow for
> direct jumping (a single lookup) in all instances. Unfortunately this
> lookup-table cannot be generated upfront when the writing of values is purely
> streaming. It can be appended to the end of the stream before it is closed,
> but without knowing the position of the lookup-table the reader cannot seek
> to it.
> One strategy for creating such a lookup-table would be to generate it during
> reads and cache it for next lookup. This does not fit directly into how
> {{IndexedDISI}} currently works (it is created anew for each invocation), but
> could probably be added with a little work. An advantage to this is that this
> does not change the underlying format and thus could be used with existing
> indexes.
> h2. The lookup structure inside each block
> If {{ALL}} of the 2^16 values are defined, the structure is empty and the
> ordinal is simply the requested docID with some modulo and multiply math.
> Nothing to improve there.
> If the block is {{DENSE}} (2^12 to 2^16 values are defined), a bitmap is used
> and the number of set bits up to the wanted index (the docID modulo the block
> origo) are counted. That bitmap is a long[1024], meaning that worst case is
> to lookup and count all set bits for 1024 longs!
> One known solution to this is to use a [rank
> structure|[https://en.wikipedia.org/wiki/Succinct_data_structure]]. I
> [implemented
> it|[https://github.com/tokee/lucene-solr/blob/solr5894/solr/core/src/java/org/apache/solr/search/sparse/count/plane/RankCache.java]]
> for a related project and with that (), the rank-overhead for a {{DENSE}}
> block would be long[32] and would ensure a maximum of
[jira] [Commented] (LUCENE-8374) Reduce reads for sparse DocValues
[
https://issues.apache.org/jira/browse/LUCENE-8374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16674489#comment-16674489
]
Toke Eskildsen commented on LUCENE-8374:
We've used the patch in production for a few months and found zero issues.
[~tpunder] was kind enough to provide independent verification of its impact,
for a case where the jump-tables are activated as well as one where they aren't
(to check for performance regression). To me this seems solid enough to move
forward.
I understand the recommendation from [~jpountz] to bake this into the DocValues
codec. From my perspective, it makes sense to _also_ make a search-time version
for the current codec, which is what the current patch does. I have a few
reasons for that
# We (the Royal Danish Library) really needs it. Well, we don't as we already
use the patch, but figuratively we do: We have 70TB+ of indexes which are
logistically hard to run through an index upgrader. I imagine other people with
large indexes faces similar challenges. By supporting search-time jump-tables
the upgrade becomes plug'n'play for those.
# It is practically free. The patch needs a clean-up, but unless I really
messed up, this should be a minor thing.
# My previous “let's improve Solr performance”-project died the legacy-code
death by becoming unwieldy; hard to review and hard to port. I would very much
like to avoid a repeat and so I push for the minimum viable product approach.
Unless there is direct opposal to adding the search-time patch to Lucene/Solr
master (with a back-port to 7), I will go ahead with this process. For that I
need some help.
# What to do about messaging? The current patch spews statistics on stdout
whenever a new segment is accessed. The Lucene codebase seems very silent, so
should I just remove all that? Or can I tie into some sort of “enable debugging
messages”-system?
# Currently the different jump-tables can be turned off and on on a global
basis. This is extremely useful to evaluate the effect, but not something
people would normally have a use for. I could just leave it as it does no
functional harm, but my suspicion is that it is seen ad clutter in the code
base?
# Review after I have fixed 1+2? It would be nice, but I know that it is also
a heavy thing. I guess I'll leave it as a wish for some time and then go ahead
with a merge if no one bites.
> Reduce reads for sparse DocValues
> -
>
> Key: LUCENE-8374
> URL: https://issues.apache.org/jira/browse/LUCENE-8374
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
>Affects Versions: 7.5, master (8.0)
> Reporter: Toke Eskildsen
>Priority: Major
> Labels: performance
> Fix For: 7.6
>
> Attachments: LUCENE-8374.patch, LUCENE-8374.patch, LUCENE-8374.patch,
> LUCENE-8374.patch, LUCENE-8374.patch, LUCENE-8374_branch_7_3.patch,
> LUCENE-8374_branch_7_3.patch.20181005, LUCENE-8374_branch_7_4.patch,
> LUCENE-8374_branch_7_5.patch, entire_index_logs.txt,
> image-2018-10-24-07-30-06-663.png, image-2018-10-24-07-30-56-962.png,
> single_vehicle_logs.txt,
> start-2018-10-24-1_snapshot___Users_tim_Snapshots__-_YourKit_Java_Profiler_2017_02-b75_-_64-bit.png,
>
> start-2018-10-24_snapshot___Users_tim_Snapshots__-_YourKit_Java_Profiler_2017_02-b75_-_64-bit.png
>
>
> The {{Lucene70DocValuesProducer}} has the internal classes
> {{SparseNumericDocValues}} and {{BaseSortedSetDocValues}} (sparse code path),
> which again uses {{IndexedDISI}} to handle the docID -> value-ordinal lookup.
> The value-ordinal is the index of the docID assuming an abstract tightly
> packed monotonically increasing list of docIDs: If the docIDs with
> corresponding values are {{[0, 4, 1432]}}, their value-ordinals will be {{[0,
> 1, 2]}}.
> h2. Outer blocks
> The lookup structure of {{IndexedDISI}} consists of blocks of 2^16 values
> (65536), where each block can be either {{ALL}}, {{DENSE}} (2^12 to 2^16
> values) or {{SPARSE}} (< 2^12 values ~= 6%). Consequently blocks vary quite a
> lot in size and ordinal resolving strategy.
> When a sparse Numeric DocValue is needed, the code first locates the block
> containing the wanted docID flag. It does so by iterating blocks one-by-one
> until it reaches the needed one, where each iteration requires a lookup in
> the underlying {{IndexSlice}}. For a common memory mapped index, this
> translates to either a cached request or a read operation. If a segment has
> 6M documents, worst-case is 91 lookups. In our web archive, our segments has
> ~300M values: A worst-case of 4577 lookups!
> One obvious solution is to use a lookup-table for blocks: A
[jira] [Commented] (LUCENE-8374) Reduce reads for sparse DocValues
[
https://issues.apache.org/jira/browse/LUCENE-8374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16668773#comment-16668773
]
Toke Eskildsen commented on LUCENE-8374:
[~tpunder] it might be that it is because of sampling that the
{{advanceExactWithinBlock}} does not show up. Don't fret about it - I'll try
some profiling experiments myself to see what's up with that.
Your logs are very informative, thanks:
{{
Segments: 41
Fields: Total=41, vBPV(long)=30 (blocks: 128095)
DISI-blocks: ALL=41, DENSE=76793, SPARSE=123, EMPTY=0
Overhead: 20.98 MB, 412 ms startup
}}
The patch introduces jumps for DISI-blocks, DENSE and vBPV. As most of your
fields has this exact combination, your index is very receptive to the
optimizations. Good for you and a caveat that this level of speed-up is not to
be expected generally.
There is a lot of numerics to a front-end that uses string-facets. I guess you
represent the facet entries as IDs and look up the string representations from
another source? Is this because you found string faceting to be too slow? I
tried looking at DocValues string resolving but there did not seem to be any
easy gains like the one for numerics.
> Reduce reads for sparse DocValues
> -
>
> Key: LUCENE-8374
> URL: https://issues.apache.org/jira/browse/LUCENE-8374
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
>Affects Versions: 7.5, master (8.0)
> Reporter: Toke Eskildsen
>Priority: Major
> Labels: performance
> Fix For: 7.6
>
> Attachments: LUCENE-8374.patch, LUCENE-8374.patch, LUCENE-8374.patch,
> LUCENE-8374.patch, LUCENE-8374.patch, LUCENE-8374_branch_7_3.patch,
> LUCENE-8374_branch_7_3.patch.20181005, LUCENE-8374_branch_7_4.patch,
> LUCENE-8374_branch_7_5.patch, entire_index_logs.txt,
> image-2018-10-24-07-30-06-663.png, image-2018-10-24-07-30-56-962.png,
> single_vehicle_logs.txt,
> start-2018-10-24-1_snapshot___Users_tim_Snapshots__-_YourKit_Java_Profiler_2017_02-b75_-_64-bit.png,
>
> start-2018-10-24_snapshot___Users_tim_Snapshots__-_YourKit_Java_Profiler_2017_02-b75_-_64-bit.png
>
>
> The {{Lucene70DocValuesProducer}} has the internal classes
> {{SparseNumericDocValues}} and {{BaseSortedSetDocValues}} (sparse code path),
> which again uses {{IndexedDISI}} to handle the docID -> value-ordinal lookup.
> The value-ordinal is the index of the docID assuming an abstract tightly
> packed monotonically increasing list of docIDs: If the docIDs with
> corresponding values are {{[0, 4, 1432]}}, their value-ordinals will be {{[0,
> 1, 2]}}.
> h2. Outer blocks
> The lookup structure of {{IndexedDISI}} consists of blocks of 2^16 values
> (65536), where each block can be either {{ALL}}, {{DENSE}} (2^12 to 2^16
> values) or {{SPARSE}} (< 2^12 values ~= 6%). Consequently blocks vary quite a
> lot in size and ordinal resolving strategy.
> When a sparse Numeric DocValue is needed, the code first locates the block
> containing the wanted docID flag. It does so by iterating blocks one-by-one
> until it reaches the needed one, where each iteration requires a lookup in
> the underlying {{IndexSlice}}. For a common memory mapped index, this
> translates to either a cached request or a read operation. If a segment has
> 6M documents, worst-case is 91 lookups. In our web archive, our segments has
> ~300M values: A worst-case of 4577 lookups!
> One obvious solution is to use a lookup-table for blocks: A long[]-array with
> an entry for each block. For 6M documents, that is < 1KB and would allow for
> direct jumping (a single lookup) in all instances. Unfortunately this
> lookup-table cannot be generated upfront when the writing of values is purely
> streaming. It can be appended to the end of the stream before it is closed,
> but without knowing the position of the lookup-table the reader cannot seek
> to it.
> One strategy for creating such a lookup-table would be to generate it during
> reads and cache it for next lookup. This does not fit directly into how
> {{IndexedDISI}} currently works (it is created anew for each invocation), but
> could probably be added with a little work. An advantage to this is that this
> does not change the underlying format and thus could be used with existing
> indexes.
> h2. The lookup structure inside each block
> If {{ALL}} of the 2^16 values are defined, the structure is empty and the
> ordinal is simply the requested docID with some modulo and multiply math.
> Nothing to improve there.
> If the block is {{DENSE}} (2^12 to 2^16 values are defined), a bitmap is used
> and the number of set
[jira] [Comment Edited] (LUCENE-8374) Reduce reads for sparse DocValues
[
https://issues.apache.org/jira/browse/LUCENE-8374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16663475#comment-16663475
]
Toke Eskildsen edited comment on LUCENE-8374 at 10/25/18 2:30 PM:
--
[~tpunder] Profiling, nice! Must. Resist. Urge. To. Look. At.
{{MonotonicDocValues}} :)
The differences for the profiles for "Faceting on most of the index" seems like
jitter to me. Good to verify that.
"Faceting on data for a single vehicle" has multiple interesting measurements:
* I am surprised that {{FieldCacheImpl#Cache.get}} takes 68,485ms for cached
and only 4,792ms for non-cached. The LUCENE-8374 caches are lazy-created so I
would expect the start-up overhead for them to be paid under
{{SparseNumericDocValues.advanceExact}} or similar, but not as part of
requesting the cache. I wonder why there is such a difference?
* {{SparseNumericDocValues.advanceExact}} is reduced by a factor 6 with
caching. This factor should scale to the number of documents in each segment:
If you try the 300M parent/child structure and the number of segments stays
about the same, I would expect the factor to increase to 15-20.
* {{IndexedDISI.advanceBlock}} + {{IndexedDISI.rankSkip}} for cached is
46,540ms and {{IndexedDISI.advanceBlock}} is 79,185ms for non-cached, telling
me that the BLOCK-cache is a noticeable win in itself. Glad to see that as my
artificial test showed a lesser win.
* I wonder why {{IndexedDISI.advanceExactWithinBlock}} is not visible in any
of the profiles. Could it be that practically all your documents has values
defined for the fields used for faceting?
In the current patch, statistics for LUCENE-8374 are written to stdout, which
Solr collects in the logfile {{solr--console.log}}. If possible, I would
like to see the content of that file as it contains a break-down of the
DocValues-fields, thanks.
was (Author: toke):
[~tpunder] Profiling, nice! Must. Resist. Urge. To. Look. At.
{{MonotonicDocValues}} :)
The differences for the profiles for "Faceting on most of the index" seems like
jitter to me. Good to verify that.
"Faceting on data for a single vehicle" has multiple interesting measurements:
* I am surprised that {FieldCacheImpl#Cache.get} takes 68,485ms for cached and
only 4,792ms for non-cached. The LUCENE-8374 caches are lazy-created so I would
expect the start-up overhead for them to be paid under
{SparseNumericDocValues.advanceExact} or similar, but not as part of requesting
the cache. I wonder why there is such a difference?
* {SparseNumericDocValues.advanceExact} is reduced by a factor 6 with caching.
This factor should scale to the number of documents in each segment: If you try
the 300M parent/child structure and the number of segments stays about the
same, I would expect the factor to increase to 15-20.
* {IndexedDISI.advanceBlock} + {IndexedDISI.rankSkip} for cached is 46,540ms
and {IndexedDISI.advanceBlock} is 79,185ms for non-cached, telling me that the
BLOCK-cache is a noticeable win in itself. Glad to see that as my artificial
test showed a lesser win.
* I wonder why {{IndexedDISI.advanceExactWithinBlock} is not visible in any of
the profiles. Could it be that practically all your documents has values
defined for the fields used for faceting?
In the current patch, statistics for LUCENE-8374 are written to stdout, which
Solr collects in the logfile {solr--console.log}. If possible, I would like
to see the content of that file as it contains a break-down of the
DocValues-fields, thanks.
> Reduce reads for sparse DocValues
> -
>
> Key: LUCENE-8374
> URL: https://issues.apache.org/jira/browse/LUCENE-8374
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
> Affects Versions: 7.5, master (8.0)
>Reporter: Toke Eskildsen
>Priority: Major
> Labels: performance
> Fix For: 7.6
>
> Attachments: LUCENE-8374.patch, LUCENE-8374.patch, LUCENE-8374.patch,
> LUCENE-8374.patch, LUCENE-8374.patch, LUCENE-8374_branch_7_3.patch,
> LUCENE-8374_branch_7_3.patch.20181005, LUCENE-8374_branch_7_4.patch,
> LUCENE-8374_branch_7_5.patch, image-2018-10-24-07-30-06-663.png,
> image-2018-10-24-07-30-56-962.png,
> start-2018-10-24-1_snapshot___Users_tim_Snapshots__-_YourKit_Java_Profiler_2017_02-b75_-_64-bit.png,
>
> start-2018-10-24_snapshot___Users_tim_Snapshots__-_YourKit_Java_Profiler_2017_02-b75_-_64-bit.png
>
>
> The {{Lucene70DocValuesProducer}} has the internal classes
> {{SparseNumericDocValues}} and {{BaseSortedSetDocValues}} (sparse code path),
> which again uses {{IndexedDISI}} to handle the docID -> value-ordinal lookup.
> The value-
[jira] [Commented] (LUCENE-8374) Reduce reads for sparse DocValues
[
https://issues.apache.org/jira/browse/LUCENE-8374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16663475#comment-16663475
]
Toke Eskildsen commented on LUCENE-8374:
[~tpunder] Profiling, nice! Must. Resist. Urge. To. Look. At.
{{MonotonicDocValues}} :)
The differences for the profiles for "Faceting on most of the index" seems like
jitter to me. Good to verify that.
"Faceting on data for a single vehicle" has multiple interesting measurements:
* I am surprised that {FieldCacheImpl#Cache.get} takes 68,485ms for cached and
only 4,792ms for non-cached. The LUCENE-8374 caches are lazy-created so I would
expect the start-up overhead for them to be paid under
{SparseNumericDocValues.advanceExact} or similar, but not as part of requesting
the cache. I wonder why there is such a difference?
* {SparseNumericDocValues.advanceExact} is reduced by a factor 6 with caching.
This factor should scale to the number of documents in each segment: If you try
the 300M parent/child structure and the number of segments stays about the
same, I would expect the factor to increase to 15-20.
* {IndexedDISI.advanceBlock} + {IndexedDISI.rankSkip} for cached is 46,540ms
and {IndexedDISI.advanceBlock} is 79,185ms for non-cached, telling me that the
BLOCK-cache is a noticeable win in itself. Glad to see that as my artificial
test showed a lesser win.
* I wonder why {{IndexedDISI.advanceExactWithinBlock} is not visible in any of
the profiles. Could it be that practically all your documents has values
defined for the fields used for faceting?
In the current patch, statistics for LUCENE-8374 are written to stdout, which
Solr collects in the logfile {solr--console.log}. If possible, I would like
to see the content of that file as it contains a break-down of the
DocValues-fields, thanks.
> Reduce reads for sparse DocValues
> -
>
> Key: LUCENE-8374
> URL: https://issues.apache.org/jira/browse/LUCENE-8374
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
>Affects Versions: 7.5, master (8.0)
>Reporter: Toke Eskildsen
>Priority: Major
> Labels: performance
> Fix For: 7.6
>
> Attachments: LUCENE-8374.patch, LUCENE-8374.patch, LUCENE-8374.patch,
> LUCENE-8374.patch, LUCENE-8374.patch, LUCENE-8374_branch_7_3.patch,
> LUCENE-8374_branch_7_3.patch.20181005, LUCENE-8374_branch_7_4.patch,
> LUCENE-8374_branch_7_5.patch, image-2018-10-24-07-30-06-663.png,
> image-2018-10-24-07-30-56-962.png,
> start-2018-10-24-1_snapshot___Users_tim_Snapshots__-_YourKit_Java_Profiler_2017_02-b75_-_64-bit.png,
>
> start-2018-10-24_snapshot___Users_tim_Snapshots__-_YourKit_Java_Profiler_2017_02-b75_-_64-bit.png
>
>
> The {{Lucene70DocValuesProducer}} has the internal classes
> {{SparseNumericDocValues}} and {{BaseSortedSetDocValues}} (sparse code path),
> which again uses {{IndexedDISI}} to handle the docID -> value-ordinal lookup.
> The value-ordinal is the index of the docID assuming an abstract tightly
> packed monotonically increasing list of docIDs: If the docIDs with
> corresponding values are {{[0, 4, 1432]}}, their value-ordinals will be {{[0,
> 1, 2]}}.
> h2. Outer blocks
> The lookup structure of {{IndexedDISI}} consists of blocks of 2^16 values
> (65536), where each block can be either {{ALL}}, {{DENSE}} (2^12 to 2^16
> values) or {{SPARSE}} (< 2^12 values ~= 6%). Consequently blocks vary quite a
> lot in size and ordinal resolving strategy.
> When a sparse Numeric DocValue is needed, the code first locates the block
> containing the wanted docID flag. It does so by iterating blocks one-by-one
> until it reaches the needed one, where each iteration requires a lookup in
> the underlying {{IndexSlice}}. For a common memory mapped index, this
> translates to either a cached request or a read operation. If a segment has
> 6M documents, worst-case is 91 lookups. In our web archive, our segments has
> ~300M values: A worst-case of 4577 lookups!
> One obvious solution is to use a lookup-table for blocks: A long[]-array with
> an entry for each block. For 6M documents, that is < 1KB and would allow for
> direct jumping (a single lookup) in all instances. Unfortunately this
> lookup-table cannot be generated upfront when the writing of values is purely
> streaming. It can be appended to the end of the stream before it is closed,
> but without knowing the position of the lookup-table the reader cannot seek
> to it.
> One strategy for creating such a lookup-table would be to generate it during
> reads and cache it for next lookup. This does not fit directly into how
> {{In
[jira] [Updated] (LUCENE-8374) Reduce reads for sparse DocValues
[
https://issues.apache.org/jira/browse/LUCENE-8374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Toke Eskildsen updated LUCENE-8374:
---
Fix Version/s: 7.6
> Reduce reads for sparse DocValues
> -
>
> Key: LUCENE-8374
> URL: https://issues.apache.org/jira/browse/LUCENE-8374
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
>Affects Versions: 7.5, master (8.0)
> Reporter: Toke Eskildsen
>Priority: Major
> Labels: performance
> Fix For: 7.6
>
> Attachments: LUCENE-8374.patch, LUCENE-8374.patch, LUCENE-8374.patch,
> LUCENE-8374.patch, LUCENE-8374.patch, LUCENE-8374_branch_7_3.patch,
> LUCENE-8374_branch_7_3.patch.20181005, LUCENE-8374_branch_7_4.patch,
> LUCENE-8374_branch_7_5.patch
>
>
> The {{Lucene70DocValuesProducer}} has the internal classes
> {{SparseNumericDocValues}} and {{BaseSortedSetDocValues}} (sparse code path),
> which again uses {{IndexedDISI}} to handle the docID -> value-ordinal lookup.
> The value-ordinal is the index of the docID assuming an abstract tightly
> packed monotonically increasing list of docIDs: If the docIDs with
> corresponding values are {{[0, 4, 1432]}}, their value-ordinals will be {{[0,
> 1, 2]}}.
> h2. Outer blocks
> The lookup structure of {{IndexedDISI}} consists of blocks of 2^16 values
> (65536), where each block can be either {{ALL}}, {{DENSE}} (2^12 to 2^16
> values) or {{SPARSE}} (< 2^12 values ~= 6%). Consequently blocks vary quite a
> lot in size and ordinal resolving strategy.
> When a sparse Numeric DocValue is needed, the code first locates the block
> containing the wanted docID flag. It does so by iterating blocks one-by-one
> until it reaches the needed one, where each iteration requires a lookup in
> the underlying {{IndexSlice}}. For a common memory mapped index, this
> translates to either a cached request or a read operation. If a segment has
> 6M documents, worst-case is 91 lookups. In our web archive, our segments has
> ~300M values: A worst-case of 4577 lookups!
> One obvious solution is to use a lookup-table for blocks: A long[]-array with
> an entry for each block. For 6M documents, that is < 1KB and would allow for
> direct jumping (a single lookup) in all instances. Unfortunately this
> lookup-table cannot be generated upfront when the writing of values is purely
> streaming. It can be appended to the end of the stream before it is closed,
> but without knowing the position of the lookup-table the reader cannot seek
> to it.
> One strategy for creating such a lookup-table would be to generate it during
> reads and cache it for next lookup. This does not fit directly into how
> {{IndexedDISI}} currently works (it is created anew for each invocation), but
> could probably be added with a little work. An advantage to this is that this
> does not change the underlying format and thus could be used with existing
> indexes.
> h2. The lookup structure inside each block
> If {{ALL}} of the 2^16 values are defined, the structure is empty and the
> ordinal is simply the requested docID with some modulo and multiply math.
> Nothing to improve there.
> If the block is {{DENSE}} (2^12 to 2^16 values are defined), a bitmap is used
> and the number of set bits up to the wanted index (the docID modulo the block
> origo) are counted. That bitmap is a long[1024], meaning that worst case is
> to lookup and count all set bits for 1024 longs!
> One known solution to this is to use a [rank
> structure|[https://en.wikipedia.org/wiki/Succinct_data_structure]]. I
> [implemented
> it|[https://github.com/tokee/lucene-solr/blob/solr5894/solr/core/src/java/org/apache/solr/search/sparse/count/plane/RankCache.java]]
> for a related project and with that (), the rank-overhead for a {{DENSE}}
> block would be long[32] and would ensure a maximum of 9 lookups. It is not
> trivial to build the rank-structure and caching it (assuming all blocks are
> dense) for 6M documents would require 22 KB (3.17% overhead). It would be far
> better to generate the rank-structure at index time and store it immediately
> before the bitset (this is possible with streaming as each block is fully
> resolved before flushing), but of course that would require a change to the
> codec.
> If {{SPARSE}} (< 2^12 values ~= 6%) are defined, the docIDs are simply in the
> form of a list. As a comment in the code suggests, a binary search through
> these would be faster, although that would mean seeking backwards. If that is
> not acceptable, I don't have any immediate idea for
[jira] [Commented] (LUCENE-8374) Reduce reads for sparse DocValues
[
https://issues.apache.org/jira/browse/LUCENE-8374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16662036#comment-16662036
]
Toke Eskildsen commented on LUCENE-8374:
[~dsmiley] I think I got it now, thanks. I was confused by the upload-dialog,
where the Fix version is available: I thought I should set it to reflect the
concrete upload (e.g. a 7.3 patch for testing) rather than the overall issue. I
will adjust it to 7.6 with master implied. I will also consider making a
feature branch: If this it to be part of the codec instead of search-time only,
I'll probably need assistance to make it happen. Thank you for your help.
[~tpunder] Thank you for the follow-up. I would expect a parent/child structure
with 300M documents to fit extremely well with the patch. Your new information
supports that there are non-trivial gains even when everything is disk cached
(yay!) and that small result sets benefits the most (expected, but nice to
verify).
As for the "Faceting on most of the index", I guess all the time is spend on
calculating the facets and that any speed-up for the retrieval of the facet
values themselves are insignificant compared to the overall time. While the
patch is also relevant for the calculation phase (to determine the ordinals for
the facet values for the documents), a match for nearly all documents means a
sequential iteration with extremely small gaps where the jump tables are not
used.
It is nice to see that there is no penalty for having the optimizations
enabled, though strictly speaking one needs to compare unpatched vs. patched
rather than patch disabled/enabled to be sure that the {{if (patch_enabled)
...}} -check does not in itself make a difference (and to be sure I did not
mess up elsewhere). But measuring tiny differences between individual runs is
of course quite hard.
> Reduce reads for sparse DocValues
> -
>
> Key: LUCENE-8374
> URL: https://issues.apache.org/jira/browse/LUCENE-8374
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
>Affects Versions: 7.5, master (8.0)
>Reporter: Toke Eskildsen
>Priority: Major
> Labels: performance
> Attachments: LUCENE-8374.patch, LUCENE-8374.patch, LUCENE-8374.patch,
> LUCENE-8374.patch, LUCENE-8374.patch, LUCENE-8374_branch_7_3.patch,
> LUCENE-8374_branch_7_3.patch.20181005, LUCENE-8374_branch_7_4.patch,
> LUCENE-8374_branch_7_5.patch
>
>
> The {{Lucene70DocValuesProducer}} has the internal classes
> {{SparseNumericDocValues}} and {{BaseSortedSetDocValues}} (sparse code path),
> which again uses {{IndexedDISI}} to handle the docID -> value-ordinal lookup.
> The value-ordinal is the index of the docID assuming an abstract tightly
> packed monotonically increasing list of docIDs: If the docIDs with
> corresponding values are {{[0, 4, 1432]}}, their value-ordinals will be {{[0,
> 1, 2]}}.
> h2. Outer blocks
> The lookup structure of {{IndexedDISI}} consists of blocks of 2^16 values
> (65536), where each block can be either {{ALL}}, {{DENSE}} (2^12 to 2^16
> values) or {{SPARSE}} (< 2^12 values ~= 6%). Consequently blocks vary quite a
> lot in size and ordinal resolving strategy.
> When a sparse Numeric DocValue is needed, the code first locates the block
> containing the wanted docID flag. It does so by iterating blocks one-by-one
> until it reaches the needed one, where each iteration requires a lookup in
> the underlying {{IndexSlice}}. For a common memory mapped index, this
> translates to either a cached request or a read operation. If a segment has
> 6M documents, worst-case is 91 lookups. In our web archive, our segments has
> ~300M values: A worst-case of 4577 lookups!
> One obvious solution is to use a lookup-table for blocks: A long[]-array with
> an entry for each block. For 6M documents, that is < 1KB and would allow for
> direct jumping (a single lookup) in all instances. Unfortunately this
> lookup-table cannot be generated upfront when the writing of values is purely
> streaming. It can be appended to the end of the stream before it is closed,
> but without knowing the position of the lookup-table the reader cannot seek
> to it.
> One strategy for creating such a lookup-table would be to generate it during
> reads and cache it for next lookup. This does not fit directly into how
> {{IndexedDISI}} currently works (it is created anew for each invocation), but
> could probably be added with a little work. An advantage to this is that this
> does not change the underlying format and thus could be used with existing
> indexes.
> h2. The lookup structure inside each block
&g
[jira] [Commented] (LUCENE-8374) Reduce reads for sparse DocValues
[
https://issues.apache.org/jira/browse/LUCENE-8374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16659044#comment-16659044
]
Toke Eskildsen commented on LUCENE-8374:
[~dsmiley] the idea for making patches for older versions was to make it easier
to measure its effect on existing setups: Testing an existing Solr 7.3 vs. a
patched Solr 7.3 is much cleaner than testing a Solr 7.3 vs. patched master.
If that collides with the established workflow here, can you suggest how I can
support easy testing?
> Reduce reads for sparse DocValues
> -
>
> Key: LUCENE-8374
> URL: https://issues.apache.org/jira/browse/LUCENE-8374
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
>Affects Versions: 7.5, master (8.0)
> Reporter: Toke Eskildsen
>Priority: Major
> Labels: performance
> Attachments: LUCENE-8374.patch, LUCENE-8374.patch, LUCENE-8374.patch,
> LUCENE-8374.patch, LUCENE-8374.patch, LUCENE-8374_branch_7_3.patch,
> LUCENE-8374_branch_7_3.patch.20181005, LUCENE-8374_branch_7_4.patch,
> LUCENE-8374_branch_7_5.patch
>
>
> The {{Lucene70DocValuesProducer}} has the internal classes
> {{SparseNumericDocValues}} and {{BaseSortedSetDocValues}} (sparse code path),
> which again uses {{IndexedDISI}} to handle the docID -> value-ordinal lookup.
> The value-ordinal is the index of the docID assuming an abstract tightly
> packed monotonically increasing list of docIDs: If the docIDs with
> corresponding values are {{[0, 4, 1432]}}, their value-ordinals will be {{[0,
> 1, 2]}}.
> h2. Outer blocks
> The lookup structure of {{IndexedDISI}} consists of blocks of 2^16 values
> (65536), where each block can be either {{ALL}}, {{DENSE}} (2^12 to 2^16
> values) or {{SPARSE}} (< 2^12 values ~= 6%). Consequently blocks vary quite a
> lot in size and ordinal resolving strategy.
> When a sparse Numeric DocValue is needed, the code first locates the block
> containing the wanted docID flag. It does so by iterating blocks one-by-one
> until it reaches the needed one, where each iteration requires a lookup in
> the underlying {{IndexSlice}}. For a common memory mapped index, this
> translates to either a cached request or a read operation. If a segment has
> 6M documents, worst-case is 91 lookups. In our web archive, our segments has
> ~300M values: A worst-case of 4577 lookups!
> One obvious solution is to use a lookup-table for blocks: A long[]-array with
> an entry for each block. For 6M documents, that is < 1KB and would allow for
> direct jumping (a single lookup) in all instances. Unfortunately this
> lookup-table cannot be generated upfront when the writing of values is purely
> streaming. It can be appended to the end of the stream before it is closed,
> but without knowing the position of the lookup-table the reader cannot seek
> to it.
> One strategy for creating such a lookup-table would be to generate it during
> reads and cache it for next lookup. This does not fit directly into how
> {{IndexedDISI}} currently works (it is created anew for each invocation), but
> could probably be added with a little work. An advantage to this is that this
> does not change the underlying format and thus could be used with existing
> indexes.
> h2. The lookup structure inside each block
> If {{ALL}} of the 2^16 values are defined, the structure is empty and the
> ordinal is simply the requested docID with some modulo and multiply math.
> Nothing to improve there.
> If the block is {{DENSE}} (2^12 to 2^16 values are defined), a bitmap is used
> and the number of set bits up to the wanted index (the docID modulo the block
> origo) are counted. That bitmap is a long[1024], meaning that worst case is
> to lookup and count all set bits for 1024 longs!
> One known solution to this is to use a [rank
> structure|[https://en.wikipedia.org/wiki/Succinct_data_structure]]. I
> [implemented
> it|[https://github.com/tokee/lucene-solr/blob/solr5894/solr/core/src/java/org/apache/solr/search/sparse/count/plane/RankCache.java]]
> for a related project and with that (), the rank-overhead for a {{DENSE}}
> block would be long[32] and would ensure a maximum of 9 lookups. It is not
> trivial to build the rank-structure and caching it (assuming all blocks are
> dense) for 6M documents would require 22 KB (3.17% overhead). It would be far
> better to generate the rank-structure at index time and store it immediately
> before the bitset (this is possible with streaming as each block is fully
> resolved before flushing), but of course that would require a change to the
&
[jira] [Updated] (LUCENE-8374) Reduce reads for sparse DocValues
[
https://issues.apache.org/jira/browse/LUCENE-8374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Toke Eskildsen updated LUCENE-8374:
---
Fix Version/s: 7.3.1
7.5
> Reduce reads for sparse DocValues
> -
>
> Key: LUCENE-8374
> URL: https://issues.apache.org/jira/browse/LUCENE-8374
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
>Affects Versions: 7.5, master (8.0)
> Reporter: Toke Eskildsen
>Priority: Major
> Labels: performance
> Fix For: trunk, 7.3.1, 7.5
>
> Attachments: LUCENE-8374.patch, LUCENE-8374.patch, LUCENE-8374.patch,
> LUCENE-8374.patch, LUCENE-8374.patch, LUCENE-8374_branch_7_3.patch,
> LUCENE-8374_branch_7_3.patch.20181005, LUCENE-8374_branch_7_4.patch,
> LUCENE-8374_branch_7_5.patch
>
>
> The {{Lucene70DocValuesProducer}} has the internal classes
> {{SparseNumericDocValues}} and {{BaseSortedSetDocValues}} (sparse code path),
> which again uses {{IndexedDISI}} to handle the docID -> value-ordinal lookup.
> The value-ordinal is the index of the docID assuming an abstract tightly
> packed monotonically increasing list of docIDs: If the docIDs with
> corresponding values are {{[0, 4, 1432]}}, their value-ordinals will be {{[0,
> 1, 2]}}.
> h2. Outer blocks
> The lookup structure of {{IndexedDISI}} consists of blocks of 2^16 values
> (65536), where each block can be either {{ALL}}, {{DENSE}} (2^12 to 2^16
> values) or {{SPARSE}} (< 2^12 values ~= 6%). Consequently blocks vary quite a
> lot in size and ordinal resolving strategy.
> When a sparse Numeric DocValue is needed, the code first locates the block
> containing the wanted docID flag. It does so by iterating blocks one-by-one
> until it reaches the needed one, where each iteration requires a lookup in
> the underlying {{IndexSlice}}. For a common memory mapped index, this
> translates to either a cached request or a read operation. If a segment has
> 6M documents, worst-case is 91 lookups. In our web archive, our segments has
> ~300M values: A worst-case of 4577 lookups!
> One obvious solution is to use a lookup-table for blocks: A long[]-array with
> an entry for each block. For 6M documents, that is < 1KB and would allow for
> direct jumping (a single lookup) in all instances. Unfortunately this
> lookup-table cannot be generated upfront when the writing of values is purely
> streaming. It can be appended to the end of the stream before it is closed,
> but without knowing the position of the lookup-table the reader cannot seek
> to it.
> One strategy for creating such a lookup-table would be to generate it during
> reads and cache it for next lookup. This does not fit directly into how
> {{IndexedDISI}} currently works (it is created anew for each invocation), but
> could probably be added with a little work. An advantage to this is that this
> does not change the underlying format and thus could be used with existing
> indexes.
> h2. The lookup structure inside each block
> If {{ALL}} of the 2^16 values are defined, the structure is empty and the
> ordinal is simply the requested docID with some modulo and multiply math.
> Nothing to improve there.
> If the block is {{DENSE}} (2^12 to 2^16 values are defined), a bitmap is used
> and the number of set bits up to the wanted index (the docID modulo the block
> origo) are counted. That bitmap is a long[1024], meaning that worst case is
> to lookup and count all set bits for 1024 longs!
> One known solution to this is to use a [rank
> structure|[https://en.wikipedia.org/wiki/Succinct_data_structure]]. I
> [implemented
> it|[https://github.com/tokee/lucene-solr/blob/solr5894/solr/core/src/java/org/apache/solr/search/sparse/count/plane/RankCache.java]]
> for a related project and with that (), the rank-overhead for a {{DENSE}}
> block would be long[32] and would ensure a maximum of 9 lookups. It is not
> trivial to build the rank-structure and caching it (assuming all blocks are
> dense) for 6M documents would require 22 KB (3.17% overhead). It would be far
> better to generate the rank-structure at index time and store it immediately
> before the bitset (this is possible with streaming as each block is fully
> resolved before flushing), but of course that would require a change to the
> codec.
> If {{SPARSE}} (< 2^12 values ~= 6%) are defined, the docIDs are simply in the
> form of a list. As a comment in the code suggests, a binary search through
> these would be faster, although that would mean seeking backwards. If that is
> not accepta
[jira] [Updated] (LUCENE-8374) Reduce reads for sparse DocValues
[
https://issues.apache.org/jira/browse/LUCENE-8374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Toke Eskildsen updated LUCENE-8374:
---
Fix Version/s: trunk
> Reduce reads for sparse DocValues
> -
>
> Key: LUCENE-8374
> URL: https://issues.apache.org/jira/browse/LUCENE-8374
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
>Affects Versions: 7.5, master (8.0)
> Reporter: Toke Eskildsen
>Priority: Major
> Labels: performance
> Fix For: trunk
>
> Attachments: LUCENE-8374.patch, LUCENE-8374.patch, LUCENE-8374.patch,
> LUCENE-8374.patch, LUCENE-8374.patch, LUCENE-8374_branch_7_3.patch,
> LUCENE-8374_branch_7_3.patch.20181005, LUCENE-8374_branch_7_4.patch,
> LUCENE-8374_branch_7_5.patch
>
>
> The {{Lucene70DocValuesProducer}} has the internal classes
> {{SparseNumericDocValues}} and {{BaseSortedSetDocValues}} (sparse code path),
> which again uses {{IndexedDISI}} to handle the docID -> value-ordinal lookup.
> The value-ordinal is the index of the docID assuming an abstract tightly
> packed monotonically increasing list of docIDs: If the docIDs with
> corresponding values are {{[0, 4, 1432]}}, their value-ordinals will be {{[0,
> 1, 2]}}.
> h2. Outer blocks
> The lookup structure of {{IndexedDISI}} consists of blocks of 2^16 values
> (65536), where each block can be either {{ALL}}, {{DENSE}} (2^12 to 2^16
> values) or {{SPARSE}} (< 2^12 values ~= 6%). Consequently blocks vary quite a
> lot in size and ordinal resolving strategy.
> When a sparse Numeric DocValue is needed, the code first locates the block
> containing the wanted docID flag. It does so by iterating blocks one-by-one
> until it reaches the needed one, where each iteration requires a lookup in
> the underlying {{IndexSlice}}. For a common memory mapped index, this
> translates to either a cached request or a read operation. If a segment has
> 6M documents, worst-case is 91 lookups. In our web archive, our segments has
> ~300M values: A worst-case of 4577 lookups!
> One obvious solution is to use a lookup-table for blocks: A long[]-array with
> an entry for each block. For 6M documents, that is < 1KB and would allow for
> direct jumping (a single lookup) in all instances. Unfortunately this
> lookup-table cannot be generated upfront when the writing of values is purely
> streaming. It can be appended to the end of the stream before it is closed,
> but without knowing the position of the lookup-table the reader cannot seek
> to it.
> One strategy for creating such a lookup-table would be to generate it during
> reads and cache it for next lookup. This does not fit directly into how
> {{IndexedDISI}} currently works (it is created anew for each invocation), but
> could probably be added with a little work. An advantage to this is that this
> does not change the underlying format and thus could be used with existing
> indexes.
> h2. The lookup structure inside each block
> If {{ALL}} of the 2^16 values are defined, the structure is empty and the
> ordinal is simply the requested docID with some modulo and multiply math.
> Nothing to improve there.
> If the block is {{DENSE}} (2^12 to 2^16 values are defined), a bitmap is used
> and the number of set bits up to the wanted index (the docID modulo the block
> origo) are counted. That bitmap is a long[1024], meaning that worst case is
> to lookup and count all set bits for 1024 longs!
> One known solution to this is to use a [rank
> structure|[https://en.wikipedia.org/wiki/Succinct_data_structure]]. I
> [implemented
> it|[https://github.com/tokee/lucene-solr/blob/solr5894/solr/core/src/java/org/apache/solr/search/sparse/count/plane/RankCache.java]]
> for a related project and with that (), the rank-overhead for a {{DENSE}}
> block would be long[32] and would ensure a maximum of 9 lookups. It is not
> trivial to build the rank-structure and caching it (assuming all blocks are
> dense) for 6M documents would require 22 KB (3.17% overhead). It would be far
> better to generate the rank-structure at index time and store it immediately
> before the bitset (this is possible with streaming as each block is fully
> resolved before flushing), but of course that would require a change to the
> codec.
> If {{SPARSE}} (< 2^12 values ~= 6%) are defined, the docIDs are simply in the
> form of a list. As a comment in the code suggests, a binary search through
> these would be faster, although that would mean seeking backwards. If that is
> not acceptable, I don't have any immediate idea for
[jira] [Updated] (LUCENE-8374) Reduce reads for sparse DocValues
[
https://issues.apache.org/jira/browse/LUCENE-8374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Toke Eskildsen updated LUCENE-8374:
---
Fix Version/s: (was: 7.3.1)
> Reduce reads for sparse DocValues
> -
>
> Key: LUCENE-8374
> URL: https://issues.apache.org/jira/browse/LUCENE-8374
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
>Affects Versions: 7.5, master (8.0)
> Reporter: Toke Eskildsen
>Priority: Major
> Labels: performance
> Fix For: trunk
>
> Attachments: LUCENE-8374.patch, LUCENE-8374.patch, LUCENE-8374.patch,
> LUCENE-8374.patch, LUCENE-8374.patch, LUCENE-8374_branch_7_3.patch,
> LUCENE-8374_branch_7_3.patch.20181005, LUCENE-8374_branch_7_4.patch,
> LUCENE-8374_branch_7_5.patch
>
>
> The {{Lucene70DocValuesProducer}} has the internal classes
> {{SparseNumericDocValues}} and {{BaseSortedSetDocValues}} (sparse code path),
> which again uses {{IndexedDISI}} to handle the docID -> value-ordinal lookup.
> The value-ordinal is the index of the docID assuming an abstract tightly
> packed monotonically increasing list of docIDs: If the docIDs with
> corresponding values are {{[0, 4, 1432]}}, their value-ordinals will be {{[0,
> 1, 2]}}.
> h2. Outer blocks
> The lookup structure of {{IndexedDISI}} consists of blocks of 2^16 values
> (65536), where each block can be either {{ALL}}, {{DENSE}} (2^12 to 2^16
> values) or {{SPARSE}} (< 2^12 values ~= 6%). Consequently blocks vary quite a
> lot in size and ordinal resolving strategy.
> When a sparse Numeric DocValue is needed, the code first locates the block
> containing the wanted docID flag. It does so by iterating blocks one-by-one
> until it reaches the needed one, where each iteration requires a lookup in
> the underlying {{IndexSlice}}. For a common memory mapped index, this
> translates to either a cached request or a read operation. If a segment has
> 6M documents, worst-case is 91 lookups. In our web archive, our segments has
> ~300M values: A worst-case of 4577 lookups!
> One obvious solution is to use a lookup-table for blocks: A long[]-array with
> an entry for each block. For 6M documents, that is < 1KB and would allow for
> direct jumping (a single lookup) in all instances. Unfortunately this
> lookup-table cannot be generated upfront when the writing of values is purely
> streaming. It can be appended to the end of the stream before it is closed,
> but without knowing the position of the lookup-table the reader cannot seek
> to it.
> One strategy for creating such a lookup-table would be to generate it during
> reads and cache it for next lookup. This does not fit directly into how
> {{IndexedDISI}} currently works (it is created anew for each invocation), but
> could probably be added with a little work. An advantage to this is that this
> does not change the underlying format and thus could be used with existing
> indexes.
> h2. The lookup structure inside each block
> If {{ALL}} of the 2^16 values are defined, the structure is empty and the
> ordinal is simply the requested docID with some modulo and multiply math.
> Nothing to improve there.
> If the block is {{DENSE}} (2^12 to 2^16 values are defined), a bitmap is used
> and the number of set bits up to the wanted index (the docID modulo the block
> origo) are counted. That bitmap is a long[1024], meaning that worst case is
> to lookup and count all set bits for 1024 longs!
> One known solution to this is to use a [rank
> structure|[https://en.wikipedia.org/wiki/Succinct_data_structure]]. I
> [implemented
> it|[https://github.com/tokee/lucene-solr/blob/solr5894/solr/core/src/java/org/apache/solr/search/sparse/count/plane/RankCache.java]]
> for a related project and with that (), the rank-overhead for a {{DENSE}}
> block would be long[32] and would ensure a maximum of 9 lookups. It is not
> trivial to build the rank-structure and caching it (assuming all blocks are
> dense) for 6M documents would require 22 KB (3.17% overhead). It would be far
> better to generate the rank-structure at index time and store it immediately
> before the bitset (this is possible with streaming as each block is fully
> resolved before flushing), but of course that would require a change to the
> codec.
> If {{SPARSE}} (< 2^12 values ~= 6%) are defined, the docIDs are simply in the
> form of a list. As a comment in the code suggests, a binary search through
> these would be faster, although that would mean seeking backwards. If that is
> not acceptable, I don't have any immedi
[jira] [Commented] (LUCENE-8374) Reduce reads for sparse DocValues
[
https://issues.apache.org/jira/browse/LUCENE-8374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16658705#comment-16658705
]
Toke Eskildsen commented on LUCENE-8374:
That is great news, [~tpunder]. Thank you very much for testing!
I was pleasantly surprised about your relatively large performance increase as
[my own
tests|https://sbdevel.wordpress.com/2018/10/02/faster-docvalues-in-lucene-solr-7/]
had a more modest increase for faceting. Visiting your site, I see that you
faceting is much richer than my simple facets, which would explain it. I guess
that my base response time (query match + logistics) is relatively large - I
should try and make a test with more facets (and one with less) to probe that.
I doubt that you will see much difference with your 8M docs index, but I would
love to be proven wrong on that.
Could you tell me how much of your index data are memory-cached?
Anyway, your test verifies that the patch is relevant for other real-world
setups, so I'll try and move forward with it. Thanks again.
> Reduce reads for sparse DocValues
> -
>
> Key: LUCENE-8374
> URL: https://issues.apache.org/jira/browse/LUCENE-8374
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
>Affects Versions: 7.4, master (8.0)
> Reporter: Toke Eskildsen
>Priority: Major
> Labels: performance
> Fix For: master (8.0)
>
> Attachments: LUCENE-8374.patch, LUCENE-8374.patch, LUCENE-8374.patch,
> LUCENE-8374.patch, LUCENE-8374.patch, LUCENE-8374_branch_7_3.patch,
> LUCENE-8374_branch_7_4.patch
>
>
> The {{Lucene70DocValuesProducer}} has the internal classes
> {{SparseNumericDocValues}} and {{BaseSortedSetDocValues}} (sparse code path),
> which again uses {{IndexedDISI}} to handle the docID -> value-ordinal lookup.
> The value-ordinal is the index of the docID assuming an abstract tightly
> packed monotonically increasing list of docIDs: If the docIDs with
> corresponding values are {{[0, 4, 1432]}}, their value-ordinals will be {{[0,
> 1, 2]}}.
> h2. Outer blocks
> The lookup structure of {{IndexedDISI}} consists of blocks of 2^16 values
> (65536), where each block can be either {{ALL}}, {{DENSE}} (2^12 to 2^16
> values) or {{SPARSE}} (< 2^12 values ~= 6%). Consequently blocks vary quite a
> lot in size and ordinal resolving strategy.
> When a sparse Numeric DocValue is needed, the code first locates the block
> containing the wanted docID flag. It does so by iterating blocks one-by-one
> until it reaches the needed one, where each iteration requires a lookup in
> the underlying {{IndexSlice}}. For a common memory mapped index, this
> translates to either a cached request or a read operation. If a segment has
> 6M documents, worst-case is 91 lookups. In our web archive, our segments has
> ~300M values: A worst-case of 4577 lookups!
> One obvious solution is to use a lookup-table for blocks: A long[]-array with
> an entry for each block. For 6M documents, that is < 1KB and would allow for
> direct jumping (a single lookup) in all instances. Unfortunately this
> lookup-table cannot be generated upfront when the writing of values is purely
> streaming. It can be appended to the end of the stream before it is closed,
> but without knowing the position of the lookup-table the reader cannot seek
> to it.
> One strategy for creating such a lookup-table would be to generate it during
> reads and cache it for next lookup. This does not fit directly into how
> {{IndexedDISI}} currently works (it is created anew for each invocation), but
> could probably be added with a little work. An advantage to this is that this
> does not change the underlying format and thus could be used with existing
> indexes.
> h2. The lookup structure inside each block
> If {{ALL}} of the 2^16 values are defined, the structure is empty and the
> ordinal is simply the requested docID with some modulo and multiply math.
> Nothing to improve there.
> If the block is {{DENSE}} (2^12 to 2^16 values are defined), a bitmap is used
> and the number of set bits up to the wanted index (the docID modulo the block
> origo) are counted. That bitmap is a long[1024], meaning that worst case is
> to lookup and count all set bits for 1024 longs!
> One known solution to this is to use a [rank
> structure|[https://en.wikipedia.org/wiki/Succinct_data_structure]]. I
> [implemented
> it|[https://github.com/tokee/lucene-solr/blob/solr5894/solr/core/src/java/org/apache/solr/search/sparse/count/plane/RankCache.java]]
> for a related project and with that (), the rank-overhead for a {{DENS
Re: DocValues, retrieval performance and policy
Erick Erickson wrote: > I think part of it is locality. By that I mean two docValues fields in > the same document have no relation to each other in terms of their > location on disk. So _assuming_ all your DocValues can't be contained > in memory, you may be doing a bunch of disk seeks. Fair enough: Doc Values overhead scales linear with the number of fields, whereas stored is more constant-ish. As you note with export, Doc Values can be faster than stored with a few fields but using them for hundreds would probably be quite a lot slower. > And maybe part of it is the notion of stuffing large text fields into > a DocValues field just to return it seems like abusing DV. That seems like a reasonable explanation to me. If that is what the talk of misuse is about, I can understand it. It is not a case I have any current interest in optimizing and I agree that "real" compression (as opposed to the light prefix-reuse from Doc Values) is the best choice. > That said, the Streaming code uses DV fields exclusively and I got > 200K rows/second returned without tuning a single thing which I doubt > you're going to get with stored fields! > So I think as usual, "it depends". I would like to think so, as that implies that it does make sense to consider if changes to Doc Values codec representation causes a performance regression, when using them to populate documents. - Toke Eskildsen - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: DocValues, retrieval performance and policy
David Smiley wrote: > I don't think it makes a difference if some people think docValues should > never be used for value-retrieval. When that performance drop occurred > due to those changes, I'm sure it would have affected sorting & faceting > as well as value-retrieval. Some more than others perhaps. Yes. The iterative API is fine for relatively small jumps, so it works perfectly for sorting on medium- to large result sets. Depending on the type of faceting it's the same. Grouping and faceting on small result sets is (probably) relatively affected, but as the amount of needed data is small in those cases, the (assumed) impact is not that high. Retrieving documents is different as there are typically more fields involved and the amount of documents itself is nearly always small, which means large jumps repeated for all the fields. > I don't see any disagreement about improving docValues in the ways > you suggest. You are right about that. I apologize if I was being unclear: It is not the concrete patch I am asking about, that's just how this started. I am asking for background on why it is considered misuse to use Doc Values for document retrieval. - Toke Eskildsen - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
DocValues, retrieval performance and policy
The Solr 7 switch to iterative API for Doc Values https://issues.apache.org/jira/browse/LUCENE-7407 meant a severe performance regression for Solr export and document retrieval with our web archive index, which is distinguished by having quite large segments (300M docs / 900GB) and using primarily doc values to hold field content. Technically there is a working patch https://issues.apache.org/jira/browse/LUCENE-8374 but during discussion of performance measurements elsewhere https://github.com/mikemccand/luceneutil/issues/23 it came up that doc values are not intended for document retrieval and as such that Lucene should not be optimized towards that. >From my point of view, using doc values to build retrieval documents is quite natural: The data are there, so making a double representation by also making them stored seems a waste of space. If this is somehow a misuse of Doc Values, maybe I could be explained what the problem is or directed towards more information? - Toke Eskildsen, Royal Danish Library - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8374) Reduce reads for sparse DocValues
[
https://issues.apache.org/jira/browse/LUCENE-8374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594728#comment-16594728
]
Toke Eskildsen commented on LUCENE-8374:
Regarding overhead:
The {{advance}}/{{advanceExact}} for blocks in {{IndexedDISI}} still has the
old check for whether the current block is the right one:
https://github.com/tokee/lucene-solr/blob/lucene8374/lucene/core/src/java/org/apache/lucene/codecs/lucene70/IndexedDISI.java#L176-L219
The difference between skipping to the next block using the old {{blockEnd}}
and skipping by looking up the offset in the {{long[]}} is tiny, but I guess
the case "the wanted block is the block immediately after the current one"
could be micro-optimized.
{{advanceWithinBlock}} and {{advanceExactWithinBlock}} are only modified for
{{DENSE}}. It always involves a call to
[rankSkip|https://github.com/tokee/lucene-solr/blob/lucene8374/lucene/core/src/java/org/apache/lucene/codecs/lucene70/IndexedDISI.java#L410].
Looking at the code, I see that it stupidly does not take advantage of the
current IndexedDISI-state and always does the somewhat costly rank-calculations
and seek, even when it would be faster to just go ahead the old way. I will fix
that. Thanks for bringing it to my attention.
The variable bits per value (vBPV) structures used by numerics are equivalent
to the DISI-blocks: If the current vBPV-block is the right one, there is no
overhead. Getting a subsequent vBPV-block offset is a simple lookup in a
{{long[]}}. As with DISI-blocks it could be optimized for the vBPB-block
following immediately and it might be worth it if the offset-cache is read from
index data and not form memory. It is a very simple optimization, so I'll just
do it.
I'll think about search-time vs. index-time a bit and come back to you on that.
> Reduce reads for sparse DocValues
> -
>
> Key: LUCENE-8374
> URL: https://issues.apache.org/jira/browse/LUCENE-8374
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
>Affects Versions: 7.4, master (8.0)
>Reporter: Toke Eskildsen
>Priority: Major
> Labels: performance
> Fix For: 7.4, master (8.0)
>
> Attachments: LUCENE-8374.patch, LUCENE-8374.patch, LUCENE-8374.patch,
> LUCENE-8374.patch, LUCENE-8374_branch_7_3.patch, LUCENE-8374_branch_7_4.patch
>
>
> The {{Lucene70DocValuesProducer}} has the internal classes
> {{SparseNumericDocValues}} and {{BaseSortedSetDocValues}} (sparse code path),
> which again uses {{IndexedDISI}} to handle the docID -> value-ordinal lookup.
> The value-ordinal is the index of the docID assuming an abstract tightly
> packed monotonically increasing list of docIDs: If the docIDs with
> corresponding values are {{[0, 4, 1432]}}, their value-ordinals will be {{[0,
> 1, 2]}}.
> h2. Outer blocks
> The lookup structure of {{IndexedDISI}} consists of blocks of 2^16 values
> (65536), where each block can be either {{ALL}}, {{DENSE}} (2^12 to 2^16
> values) or {{SPARSE}} (< 2^12 values ~= 6%). Consequently blocks vary quite a
> lot in size and ordinal resolving strategy.
> When a sparse Numeric DocValue is needed, the code first locates the block
> containing the wanted docID flag. It does so by iterating blocks one-by-one
> until it reaches the needed one, where each iteration requires a lookup in
> the underlying {{IndexSlice}}. For a common memory mapped index, this
> translates to either a cached request or a read operation. If a segment has
> 6M documents, worst-case is 91 lookups. In our web archive, our segments has
> ~300M values: A worst-case of 4577 lookups!
> One obvious solution is to use a lookup-table for blocks: A long[]-array with
> an entry for each block. For 6M documents, that is < 1KB and would allow for
> direct jumping (a single lookup) in all instances. Unfortunately this
> lookup-table cannot be generated upfront when the writing of values is purely
> streaming. It can be appended to the end of the stream before it is closed,
> but without knowing the position of the lookup-table the reader cannot seek
> to it.
> One strategy for creating such a lookup-table would be to generate it during
> reads and cache it for next lookup. This does not fit directly into how
> {{IndexedDISI}} currently works (it is created anew for each invocation), but
> could probably be added with a little work. An advantage to this is that this
> does not change the underlying format and thus could be used with existing
> indexes.
> h2. The lookup structure inside each block
> If {{ALL}} of the 2^16 values are defined, the structure is empty and the
&
[jira] [Comment Edited] (LUCENE-8374) Reduce reads for sparse DocValues
[
https://issues.apache.org/jira/browse/LUCENE-8374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593536#comment-16593536
]
Toke Eskildsen edited comment on LUCENE-8374 at 8/27/18 11:47 AM:
--
My previous stats script was slightly off. I isolated the block /
DENSE-structures and got
{{Index: 890GB / 51 fields / 307M docs}}
{{ Blocks: Total 238971 (blocks: ALL=102693, SPARSE=28739, DENSE=102421,
EMPTY=5118) / 1866KB}}
{{ DENSE-Rank: 28118KB (raw DENSE data: 102421*8KB = 819368KB)}}
{{ BPV(long): 7 / 654KB}}
{{ Total: 30653KB, 13 seconds start up}}
DENSE is by far the biggest consumer of cache-space in our setup. Interestingly
enough, the vBPV-caching was the ones that gave us by far the biggest benefit,
for the few long fields that we have.
I looked at skip lists, both in the
[MultiLevelSkipListWriter|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/codecs/MultiLevelSkipListWriter.java]
and [on Wikipedia|[https://en.wikipedia.org/wiki/Skip_list].] As I understand
it they are essentially a {{rank+select}} implementation that allows for
varying-size skips and works well with a linked list that can be modified. The
varying-size and mutability does not seem to be used/relevant for Lucene.
What I don't really understand is the benefit of skip list's multi-level
approach in this case. How would a skip list be better than the current
direct-lookup in the array of longs representing offset+bitcount? If the point
is to save further memory, the block-offsets could be stated for every 4th
block or so, just as the skip lists does. But the current overhead of 2MB for a
rather large segment does not seem problematic to me and it does mean that 0
superfluous blocks needs to be seeked.
New point: I would very much like to make this issue two-step.
As the performance regression gets worse linear with segment size, it seems
plausible that the people that will benefit the most from the patch are also
people where a full re-index is not trivial. From my point of view, search-time
caching should be present for present segments, independently of codec-changes
to future segments. The current patch needs polishing, but is functionwise
ready and does exactly this.
As the search-time cache is non-destructive, rolling this as step 1 would be a
conservative update with easy rollback. Step 2 is of course to change the codec
to embed the caching structures, if they prove their worth.
was (Author: toke):
My previous stats script was slightly off. I isolated the block /
DENSE-structures and got
{{Index: 890GB / 51 fields / 307M docs}}
Blocks: Total 238971 (blocks: ALL=102693, SPARSE=28739, DENSE=102421,
EMPTY=5118) / 1866KB
DENSE-Rank: 28118KB (raw DENSE data: 102421*8KB = 819368KB)
BPV(long): 7 / 654KB
Total: 30653KB, 13 seconds start up
DENSE is by far the biggest consumer of cache-space in our setup. Interestingly
enough, the vBPV-caching was the ones that gave us by far the biggest benefit,
for the few long fields that we have.
I looked at skip lists, both in the
[MultiLevelSkipListWriter|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/codecs/MultiLevelSkipListWriter.java]
and [on Wikipedia|[https://en.wikipedia.org/wiki/Skip_list].] As I understand
it they are essentially a {{rank+select}} implementation that allows for
varying-size skips and works well with a linked list that can be modified. The
varying-size and mutability does not seem to be used/relevant for Lucene.
What I don't really understand is the benefit of skip list's multi-level
approach in this case. How would a skip list be better than the current
direct-lookup in the array of longs representing offset+bitcount? If the point
is to save further memory, the block-offsets could be stated for every 4th
block or so, just as the skip lists does. But the current overhead of 2MB for a
rather large segment does not seem problematic to me and it does mean that 0
superfluous blocks needs to be seeked.
New point: I would very much like to make this issue two-step.
As the performance regression gets worse linear with segment size, it seems
plausible that the people that will benefit the most from the patch are also
people where a full re-index is not trivial. From my point of view, search-time
caching should be present for present segments, independently of codec-changes
to future segments. The current patch needs polishing, but is functionwise
ready and does exactly this.
As the search-time cache is non-destructive, rolling this as step 1 would be a
conservative update with easy rollback. Step 2 is of course to change the codec
to embed the caching structures, if they prove their worth.
> Reduce reads for sparse DocValues
> -
>
> Key: LUCENE-8374
>
[jira] [Comment Edited] (LUCENE-8374) Reduce reads for sparse DocValues
[
https://issues.apache.org/jira/browse/LUCENE-8374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593536#comment-16593536
]
Toke Eskildsen edited comment on LUCENE-8374 at 8/27/18 11:46 AM:
--
My previous stats script was slightly off. I isolated the block /
DENSE-structures and got
{{Index: 890GB / 51 fields / 307M docs}}
Blocks: Total 238971 (blocks: ALL=102693, SPARSE=28739, DENSE=102421,
EMPTY=5118) / 1866KB
DENSE-Rank: 28118KB (raw DENSE data: 102421*8KB = 819368KB)
BPV(long): 7 / 654KB
Total: 30653KB, 13 seconds start up
DENSE is by far the biggest consumer of cache-space in our setup. Interestingly
enough, the vBPV-caching was the ones that gave us by far the biggest benefit,
for the few long fields that we have.
I looked at skip lists, both in the
[MultiLevelSkipListWriter|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/codecs/MultiLevelSkipListWriter.java]
and [on Wikipedia|[https://en.wikipedia.org/wiki/Skip_list].] As I understand
it they are essentially a {{rank+select}} implementation that allows for
varying-size skips and works well with a linked list that can be modified. The
varying-size and mutability does not seem to be used/relevant for Lucene.
What I don't really understand is the benefit of skip list's multi-level
approach in this case. How would a skip list be better than the current
direct-lookup in the array of longs representing offset+bitcount? If the point
is to save further memory, the block-offsets could be stated for every 4th
block or so, just as the skip lists does. But the current overhead of 2MB for a
rather large segment does not seem problematic to me and it does mean that 0
superfluous blocks needs to be seeked.
New point: I would very much like to make this issue two-step.
As the performance regression gets worse linear with segment size, it seems
plausible that the people that will benefit the most from the patch are also
people where a full re-index is not trivial. From my point of view, search-time
caching should be present for present segments, independently of codec-changes
to future segments. The current patch needs polishing, but is functionwise
ready and does exactly this.
As the search-time cache is non-destructive, rolling this as step 1 would be a
conservative update with easy rollback. Step 2 is of course to change the codec
to embed the caching structures, if they prove their worth.
was (Author: toke):
My previous stats script was slightly off. I isolated the block /
DENSE-structures and got
{{Index: 890GB / 51 fields / 307M docs}}
{{ Blocks: Total 238971 (blocks: ALL=102693, SPARSE=28739, DENSE=102421,
EMPTY=5118) / 1866KB}}
{{ DENSE-Rank: 28118KB (raw DENSE data: 102421*8KB = 819368KB)}}
{{ vBPV(long): 7 / 654KB}}
{{ Total: 30653KB, 13 seconds start up}}
DENSE is by far the biggest consumer of cache-space in our setup. Interestingly
enough, the vBPV-caching was the ones that gave us by far the biggest benefit,
for the few long fields that we have.
I looked at skip lists, both in the
[MultiLevelSkipListWriter|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/codecs/MultiLevelSkipListWriter.java]
and [on Wikipedia|[https://en.wikipedia.org/wiki/Skip_list].] As I understand
it they are essentially a {{rank+select}} implementation that allows for
varying-size skips and works well with a linked list that can be modified. The
varying-size and mutability does not seem to be used/relevant for Lucene.
What I don't really understand is the benefit of skip list's multi-level
approach in this case. How would a skip list be better than the current
direct-lookup in the array of longs representing offset+bitcount? If the point
is to save further memory, the block-offsets could be stated for every 4th
block or so, just as the skip lists does. But the current overhead of 2MB for a
rather large segment does not seem problematic to me and it does mean that 0
superfluous blocks needs to be seeked.
New point: I would very much like to make this issue two-step.
As the performance regression gets worse linear with segment size, it seems
plausible that the people that will benefit the most from the patch are also
people where a full re-index is not trivial. From my point of view, search-time
caching should be present for present segments, independently of codec-changes
to future segments. The current patch needs polishing, but is functionwise
ready and does exactly this.
As the search-time cache is non-destructive, rolling this as step 1 would be a
conservative update with easy rollback. Step 2 is of course to change the codec
to embed the caching structures, if they prove their worth.
> Reduce reads for sparse DocValues
> -
>
> Key: LUCENE-8374
>
[jira] [Commented] (LUCENE-8374) Reduce reads for sparse DocValues
[
https://issues.apache.org/jira/browse/LUCENE-8374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593547#comment-16593547
]
Toke Eskildsen commented on LUCENE-8374:
Some independent verification/debunking of this performance issue would be
highly appreciated.
The patch speeds up {{DocValues}}-retrieval and the larger the segments, the
more the benefit. An index with 50M+ documents, using {{DocValues}} when
returning documents (preferably with one or more int/long/date-fields), would
be ideal. Checking that it does not hurt performance for smaller indexes also
seems important.
I'll port the patch to any Lucene/Solr 7.x-version if it helps.
> Reduce reads for sparse DocValues
> -
>
> Key: LUCENE-8374
> URL: https://issues.apache.org/jira/browse/LUCENE-8374
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
>Affects Versions: 7.4, master (8.0)
> Reporter: Toke Eskildsen
>Priority: Major
> Labels: performance
> Fix For: 7.4, master (8.0)
>
> Attachments: LUCENE-8374.patch, LUCENE-8374.patch, LUCENE-8374.patch,
> LUCENE-8374.patch, LUCENE-8374_branch_7_3.patch, LUCENE-8374_branch_7_4.patch
>
>
> The {{Lucene70DocValuesProducer}} has the internal classes
> {{SparseNumericDocValues}} and {{BaseSortedSetDocValues}} (sparse code path),
> which again uses {{IndexedDISI}} to handle the docID -> value-ordinal lookup.
> The value-ordinal is the index of the docID assuming an abstract tightly
> packed monotonically increasing list of docIDs: If the docIDs with
> corresponding values are {{[0, 4, 1432]}}, their value-ordinals will be {{[0,
> 1, 2]}}.
> h2. Outer blocks
> The lookup structure of {{IndexedDISI}} consists of blocks of 2^16 values
> (65536), where each block can be either {{ALL}}, {{DENSE}} (2^12 to 2^16
> values) or {{SPARSE}} (< 2^12 values ~= 6%). Consequently blocks vary quite a
> lot in size and ordinal resolving strategy.
> When a sparse Numeric DocValue is needed, the code first locates the block
> containing the wanted docID flag. It does so by iterating blocks one-by-one
> until it reaches the needed one, where each iteration requires a lookup in
> the underlying {{IndexSlice}}. For a common memory mapped index, this
> translates to either a cached request or a read operation. If a segment has
> 6M documents, worst-case is 91 lookups. In our web archive, our segments has
> ~300M values: A worst-case of 4577 lookups!
> One obvious solution is to use a lookup-table for blocks: A long[]-array with
> an entry for each block. For 6M documents, that is < 1KB and would allow for
> direct jumping (a single lookup) in all instances. Unfortunately this
> lookup-table cannot be generated upfront when the writing of values is purely
> streaming. It can be appended to the end of the stream before it is closed,
> but without knowing the position of the lookup-table the reader cannot seek
> to it.
> One strategy for creating such a lookup-table would be to generate it during
> reads and cache it for next lookup. This does not fit directly into how
> {{IndexedDISI}} currently works (it is created anew for each invocation), but
> could probably be added with a little work. An advantage to this is that this
> does not change the underlying format and thus could be used with existing
> indexes.
> h2. The lookup structure inside each block
> If {{ALL}} of the 2^16 values are defined, the structure is empty and the
> ordinal is simply the requested docID with some modulo and multiply math.
> Nothing to improve there.
> If the block is {{DENSE}} (2^12 to 2^16 values are defined), a bitmap is used
> and the number of set bits up to the wanted index (the docID modulo the block
> origo) are counted. That bitmap is a long[1024], meaning that worst case is
> to lookup and count all set bits for 1024 longs!
> One known solution to this is to use a [rank
> structure|[https://en.wikipedia.org/wiki/Succinct_data_structure]]. I
> [implemented
> it|[https://github.com/tokee/lucene-solr/blob/solr5894/solr/core/src/java/org/apache/solr/search/sparse/count/plane/RankCache.java]]
> for a related project and with that (), the rank-overhead for a {{DENSE}}
> block would be long[32] and would ensure a maximum of 9 lookups. It is not
> trivial to build the rank-structure and caching it (assuming all blocks are
> dense) for 6M documents would require 22 KB (3.17% overhead). It would be far
> better to generate the rank-structure at index time and store it immediately
> before the bitset (this is possible with streaming as each bl
[jira] [Comment Edited] (LUCENE-8374) Reduce reads for sparse DocValues
[
https://issues.apache.org/jira/browse/LUCENE-8374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593536#comment-16593536
]
Toke Eskildsen edited comment on LUCENE-8374 at 8/27/18 11:45 AM:
--
My previous stats script was slightly off. I isolated the block /
DENSE-structures and got
{{Index: 890GB / 51 fields / 307M docs}}
{{ Blocks: Total 238971 (blocks: ALL=102693, SPARSE=28739, DENSE=102421,
EMPTY=5118) / 1866KB}}
{{ DENSE-Rank: 28118KB (raw DENSE data: 102421*8KB = 819368KB)}}
{{ vBPV(long): 7 / 654KB}}
{{ Total: 30653KB, 13 seconds start up}}
DENSE is by far the biggest consumer of cache-space in our setup. Interestingly
enough, the vBPV-caching was the ones that gave us by far the biggest benefit,
for the few long fields that we have.
I looked at skip lists, both in the
[MultiLevelSkipListWriter|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/codecs/MultiLevelSkipListWriter.java]
and [on Wikipedia|[https://en.wikipedia.org/wiki/Skip_list].] As I understand
it they are essentially a {{rank+select}} implementation that allows for
varying-size skips and works well with a linked list that can be modified. The
varying-size and mutability does not seem to be used/relevant for Lucene.
What I don't really understand is the benefit of skip list's multi-level
approach in this case. How would a skip list be better than the current
direct-lookup in the array of longs representing offset+bitcount? If the point
is to save further memory, the block-offsets could be stated for every 4th
block or so, just as the skip lists does. But the current overhead of 2MB for a
rather large segment does not seem problematic to me and it does mean that 0
superfluous blocks needs to be seeked.
New point: I would very much like to make this issue two-step.
As the performance regression gets worse linear with segment size, it seems
plausible that the people that will benefit the most from the patch are also
people where a full re-index is not trivial. From my point of view, search-time
caching should be present for present segments, independently of codec-changes
to future segments. The current patch needs polishing, but is functionwise
ready and does exactly this.
As the search-time cache is non-destructive, rolling this as step 1 would be a
conservative update with easy rollback. Step 2 is of course to change the codec
to embed the caching structures, if they prove their worth.
was (Author: toke):
My previous stats script was slightly off. I isolated the block /
DENSE-structures and got
```
Index: 890GB / 51 fields / 307M docs
Blocks: Total 238971 (blocks: ALL=102693, SPARSE=28739, DENSE=102421,
EMPTY=5118) / 1866KB
DENSE-Rank: 28118KB (raw DENSE data: 102421*8KB = 819368KB)
vBPV(long): 7 / 654KB
Total: 30653KB, 13 seconds start up
```
DENSE is by far the biggest consumer of cache-space in our setup. Interestingly
enough, the vBPV-caching was the ones that gave us by far the biggest benefit,
for the few long fields that we have.
I looked at skip lists, both in the
[MultiLevelSkipListWriter|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/codecs/MultiLevelSkipListWriter.java]
and [on Wikipedia|[https://en.wikipedia.org/wiki/Skip_list].] As I understand
it they are essentially a {{rank+select}} implementation that allows for
varying-size skips and works well with a linked list that can be modified. The
varying-size and mutability does not seem to be used/relevant for Lucene.
What I don't really understand is the benefit of skip list's multi-level
approach in this case. How would a skip list be better than the current
direct-lookup in the array of longs representing offset+bitcount? If the point
is to save further memory, the block-offsets could be stated for every 4th
block or so, just as the skip lists does. But the current overhead of 2MB for a
rather large segment does not seem problematic to me and it does mean that 0
superfluous blocks needs to be seeked.
New point: I would very much like to make this issue two-step.
As the performance regression gets worse linear with segment size, it seems
plausible that the people that will benefit the most from the patch are also
people where a full re-index is not trivial. From my point of view, search-time
caching should be present for present segments, independently of codec-changes
to future segments. The current patch needs polishing, but is functionwise
ready and does exactly this.
As the search-time cache is non-destructive, rolling this as step 1 would be a
conservative update with easy rollback. Step 2 is of course to change the codec
to embed the caching structures, if they prove their worth.
> Reduce reads for sparse DocValues
> -
>
> Key: LUCENE-8374
>
[jira] [Commented] (LUCENE-8374) Reduce reads for sparse DocValues
[
https://issues.apache.org/jira/browse/LUCENE-8374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593536#comment-16593536
]
Toke Eskildsen commented on LUCENE-8374:
My previous stats script was slightly off. I isolated the block /
DENSE-structures and got
```
Index: 890GB / 51 fields / 307M docs
Blocks: Total 238971 (blocks: ALL=102693, SPARSE=28739, DENSE=102421,
EMPTY=5118) / 1866KB
DENSE-Rank: 28118KB (raw DENSE data: 102421*8KB = 819368KB)
vBPV(long): 7 / 654KB
Total: 30653KB, 13 seconds start up
```
DENSE is by far the biggest consumer of cache-space in our setup. Interestingly
enough, the vBPV-caching was the ones that gave us by far the biggest benefit,
for the few long fields that we have.
I looked at skip lists, both in the
[MultiLevelSkipListWriter|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/codecs/MultiLevelSkipListWriter.java]
and [on Wikipedia|[https://en.wikipedia.org/wiki/Skip_list].] As I understand
it they are essentially a {{rank+select}} implementation that allows for
varying-size skips and works well with a linked list that can be modified. The
varying-size and mutability does not seem to be used/relevant for Lucene.
What I don't really understand is the benefit of skip list's multi-level
approach in this case. How would a skip list be better than the current
direct-lookup in the array of longs representing offset+bitcount? If the point
is to save further memory, the block-offsets could be stated for every 4th
block or so, just as the skip lists does. But the current overhead of 2MB for a
rather large segment does not seem problematic to me and it does mean that 0
superfluous blocks needs to be seeked.
New point: I would very much like to make this issue two-step.
As the performance regression gets worse linear with segment size, it seems
plausible that the people that will benefit the most from the patch are also
people where a full re-index is not trivial. From my point of view, search-time
caching should be present for present segments, independently of codec-changes
to future segments. The current patch needs polishing, but is functionwise
ready and does exactly this.
As the search-time cache is non-destructive, rolling this as step 1 would be a
conservative update with easy rollback. Step 2 is of course to change the codec
to embed the caching structures, if they prove their worth.
> Reduce reads for sparse DocValues
> -
>
> Key: LUCENE-8374
> URL: https://issues.apache.org/jira/browse/LUCENE-8374
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
>Affects Versions: 7.4, master (8.0)
> Reporter: Toke Eskildsen
>Priority: Major
> Labels: performance
> Fix For: 7.4, master (8.0), 7.3.1
>
> Attachments: LUCENE-8374.patch, LUCENE-8374.patch, LUCENE-8374.patch,
> LUCENE-8374_branch_7_3.patch, LUCENE-8374_branch_7_4.patch
>
>
> The {{Lucene70DocValuesProducer}} has the internal classes
> {{SparseNumericDocValues}} and {{BaseSortedSetDocValues}} (sparse code path),
> which again uses {{IndexedDISI}} to handle the docID -> value-ordinal lookup.
> The value-ordinal is the index of the docID assuming an abstract tightly
> packed monotonically increasing list of docIDs: If the docIDs with
> corresponding values are {{[0, 4, 1432]}}, their value-ordinals will be {{[0,
> 1, 2]}}.
> h2. Outer blocks
> The lookup structure of {{IndexedDISI}} consists of blocks of 2^16 values
> (65536), where each block can be either {{ALL}}, {{DENSE}} (2^12 to 2^16
> values) or {{SPARSE}} (< 2^12 values ~= 6%). Consequently blocks vary quite a
> lot in size and ordinal resolving strategy.
> When a sparse Numeric DocValue is needed, the code first locates the block
> containing the wanted docID flag. It does so by iterating blocks one-by-one
> until it reaches the needed one, where each iteration requires a lookup in
> the underlying {{IndexSlice}}. For a common memory mapped index, this
> translates to either a cached request or a read operation. If a segment has
> 6M documents, worst-case is 91 lookups. In our web archive, our segments has
> ~300M values: A worst-case of 4577 lookups!
> One obvious solution is to use a lookup-table for blocks: A long[]-array with
> an entry for each block. For 6M documents, that is < 1KB and would allow for
> direct jumping (a single lookup) in all instances. Unfortunately this
> lookup-table cannot be generated upfront when the writing of values is purely
> streaming. It can be appended to the end of the stream before it is closed,
> but without knowing the position of the lookup-table the
[jira] [Commented] (LUCENE-8374) Reduce reads for sparse DocValues
[
https://issues.apache.org/jira/browse/LUCENE-8374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16579877#comment-16579877
]
Toke Eskildsen commented on LUCENE-8374:
The patch implements the jump list suggested in LUCENE-7589.
> Reduce reads for sparse DocValues
> -
>
> Key: LUCENE-8374
> URL: https://issues.apache.org/jira/browse/LUCENE-8374
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
>Affects Versions: 7.4, master (8.0)
> Reporter: Toke Eskildsen
>Priority: Major
> Labels: performance
> Fix For: 7.4, master (8.0), 7.3.1
>
> Attachments: LUCENE-8374.patch, LUCENE-8374.patch,
> LUCENE-8374_branch_7_3.patch, LUCENE-8374_branch_7_4.patch
>
>
> The {{Lucene70DocValuesProducer}} has the internal classes
> {{SparseNumericDocValues}} and {{BaseSortedSetDocValues}} (sparse code path),
> which again uses {{IndexedDISI}} to handle the docID -> value-ordinal lookup.
> The value-ordinal is the index of the docID assuming an abstract tightly
> packed monotonically increasing list of docIDs: If the docIDs with
> corresponding values are {{[0, 4, 1432]}}, their value-ordinals will be {{[0,
> 1, 2]}}.
> h2. Outer blocks
> The lookup structure of {{IndexedDISI}} consists of blocks of 2^16 values
> (65536), where each block can be either {{ALL}}, {{DENSE}} (2^12 to 2^16
> values) or {{SPARSE}} (< 2^12 values ~= 6%). Consequently blocks vary quite a
> lot in size and ordinal resolving strategy.
> When a sparse Numeric DocValue is needed, the code first locates the block
> containing the wanted docID flag. It does so by iterating blocks one-by-one
> until it reaches the needed one, where each iteration requires a lookup in
> the underlying {{IndexSlice}}. For a common memory mapped index, this
> translates to either a cached request or a read operation. If a segment has
> 6M documents, worst-case is 91 lookups. In our web archive, our segments has
> ~300M values: A worst-case of 4577 lookups!
> One obvious solution is to use a lookup-table for blocks: A long[]-array with
> an entry for each block. For 6M documents, that is < 1KB and would allow for
> direct jumping (a single lookup) in all instances. Unfortunately this
> lookup-table cannot be generated upfront when the writing of values is purely
> streaming. It can be appended to the end of the stream before it is closed,
> but without knowing the position of the lookup-table the reader cannot seek
> to it.
> One strategy for creating such a lookup-table would be to generate it during
> reads and cache it for next lookup. This does not fit directly into how
> {{IndexedDISI}} currently works (it is created anew for each invocation), but
> could probably be added with a little work. An advantage to this is that this
> does not change the underlying format and thus could be used with existing
> indexes.
> h2. The lookup structure inside each block
> If {{ALL}} of the 2^16 values are defined, the structure is empty and the
> ordinal is simply the requested docID with some modulo and multiply math.
> Nothing to improve there.
> If the block is {{DENSE}} (2^12 to 2^16 values are defined), a bitmap is used
> and the number of set bits up to the wanted index (the docID modulo the block
> origo) are counted. That bitmap is a long[1024], meaning that worst case is
> to lookup and count all set bits for 1024 longs!
> One known solution to this is to use a [rank
> structure|[https://en.wikipedia.org/wiki/Succinct_data_structure]]. I
> [implemented
> it|[https://github.com/tokee/lucene-solr/blob/solr5894/solr/core/src/java/org/apache/solr/search/sparse/count/plane/RankCache.java]]
> for a related project and with that (), the rank-overhead for a {{DENSE}}
> block would be long[32] and would ensure a maximum of 9 lookups. It is not
> trivial to build the rank-structure and caching it (assuming all blocks are
> dense) for 6M documents would require 22 KB (3.17% overhead). It would be far
> better to generate the rank-structure at index time and store it immediately
> before the bitset (this is possible with streaming as each block is fully
> resolved before flushing), but of course that would require a change to the
> codec.
> If {{SPARSE}} (< 2^12 values ~= 6%) are defined, the docIDs are simply in the
> form of a list. As a comment in the code suggests, a binary search through
> these would be faster, although that would mean seeking backwards. If that is
> not acceptable, I don't have any immediate idea for avoiding the full
> iteration.
> I
[jira] [Updated] (LUCENE-8374) Reduce reads for sparse DocValues
[
https://issues.apache.org/jira/browse/LUCENE-8374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Toke Eskildsen updated LUCENE-8374:
---
Affects Version/s: (was: 7.3.1)
7.4
> Reduce reads for sparse DocValues
> -
>
> Key: LUCENE-8374
> URL: https://issues.apache.org/jira/browse/LUCENE-8374
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
>Affects Versions: 7.4, master (8.0)
> Reporter: Toke Eskildsen
>Priority: Major
> Labels: performance
> Fix For: 7.4, master (8.0), 7.3.1
>
> Attachments: LUCENE-8374.patch, LUCENE-8374.patch,
> LUCENE-8374_branch_7_3.patch, LUCENE-8374_branch_7_4.patch
>
>
> The {{Lucene70DocValuesProducer}} has the internal classes
> {{SparseNumericDocValues}} and {{BaseSortedSetDocValues}} (sparse code path),
> which again uses {{IndexedDISI}} to handle the docID -> value-ordinal lookup.
> The value-ordinal is the index of the docID assuming an abstract tightly
> packed monotonically increasing list of docIDs: If the docIDs with
> corresponding values are {{[0, 4, 1432]}}, their value-ordinals will be {{[0,
> 1, 2]}}.
> h2. Outer blocks
> The lookup structure of {{IndexedDISI}} consists of blocks of 2^16 values
> (65536), where each block can be either {{ALL}}, {{DENSE}} (2^12 to 2^16
> values) or {{SPARSE}} (< 2^12 values ~= 6%). Consequently blocks vary quite a
> lot in size and ordinal resolving strategy.
> When a sparse Numeric DocValue is needed, the code first locates the block
> containing the wanted docID flag. It does so by iterating blocks one-by-one
> until it reaches the needed one, where each iteration requires a lookup in
> the underlying {{IndexSlice}}. For a common memory mapped index, this
> translates to either a cached request or a read operation. If a segment has
> 6M documents, worst-case is 91 lookups. In our web archive, our segments has
> ~300M values: A worst-case of 4577 lookups!
> One obvious solution is to use a lookup-table for blocks: A long[]-array with
> an entry for each block. For 6M documents, that is < 1KB and would allow for
> direct jumping (a single lookup) in all instances. Unfortunately this
> lookup-table cannot be generated upfront when the writing of values is purely
> streaming. It can be appended to the end of the stream before it is closed,
> but without knowing the position of the lookup-table the reader cannot seek
> to it.
> One strategy for creating such a lookup-table would be to generate it during
> reads and cache it for next lookup. This does not fit directly into how
> {{IndexedDISI}} currently works (it is created anew for each invocation), but
> could probably be added with a little work. An advantage to this is that this
> does not change the underlying format and thus could be used with existing
> indexes.
> h2. The lookup structure inside each block
> If {{ALL}} of the 2^16 values are defined, the structure is empty and the
> ordinal is simply the requested docID with some modulo and multiply math.
> Nothing to improve there.
> If the block is {{DENSE}} (2^12 to 2^16 values are defined), a bitmap is used
> and the number of set bits up to the wanted index (the docID modulo the block
> origo) are counted. That bitmap is a long[1024], meaning that worst case is
> to lookup and count all set bits for 1024 longs!
> One known solution to this is to use a [rank
> structure|[https://en.wikipedia.org/wiki/Succinct_data_structure]]. I
> [implemented
> it|[https://github.com/tokee/lucene-solr/blob/solr5894/solr/core/src/java/org/apache/solr/search/sparse/count/plane/RankCache.java]]
> for a related project and with that (), the rank-overhead for a {{DENSE}}
> block would be long[32] and would ensure a maximum of 9 lookups. It is not
> trivial to build the rank-structure and caching it (assuming all blocks are
> dense) for 6M documents would require 22 KB (3.17% overhead). It would be far
> better to generate the rank-structure at index time and store it immediately
> before the bitset (this is possible with streaming as each block is fully
> resolved before flushing), but of course that would require a change to the
> codec.
> If {{SPARSE}} (< 2^12 values ~= 6%) are defined, the docIDs are simply in the
> form of a list. As a comment in the code suggests, a binary search through
> these would be faster, although that would mean seeking backwards. If that is
> not acceptable, I don't have any immediate idea for avoiding the full
> iteration.
> I propose implementing query-t
[jira] [Updated] (LUCENE-8374) Reduce reads for sparse DocValues
[
https://issues.apache.org/jira/browse/LUCENE-8374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Toke Eskildsen updated LUCENE-8374:
---
Fix Version/s: 7.4
master (8.0)
> Reduce reads for sparse DocValues
> -
>
> Key: LUCENE-8374
> URL: https://issues.apache.org/jira/browse/LUCENE-8374
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
>Affects Versions: master (8.0), 7.3.1
> Reporter: Toke Eskildsen
>Priority: Major
> Labels: performance
> Fix For: 7.4, master (8.0), 7.3.1
>
> Attachments: LUCENE-8374.patch, LUCENE-8374.patch,
> LUCENE-8374_branch_7_3.patch, LUCENE-8374_branch_7_4.patch
>
>
> The {{Lucene70DocValuesProducer}} has the internal classes
> {{SparseNumericDocValues}} and {{BaseSortedSetDocValues}} (sparse code path),
> which again uses {{IndexedDISI}} to handle the docID -> value-ordinal lookup.
> The value-ordinal is the index of the docID assuming an abstract tightly
> packed monotonically increasing list of docIDs: If the docIDs with
> corresponding values are {{[0, 4, 1432]}}, their value-ordinals will be {{[0,
> 1, 2]}}.
> h2. Outer blocks
> The lookup structure of {{IndexedDISI}} consists of blocks of 2^16 values
> (65536), where each block can be either {{ALL}}, {{DENSE}} (2^12 to 2^16
> values) or {{SPARSE}} (< 2^12 values ~= 6%). Consequently blocks vary quite a
> lot in size and ordinal resolving strategy.
> When a sparse Numeric DocValue is needed, the code first locates the block
> containing the wanted docID flag. It does so by iterating blocks one-by-one
> until it reaches the needed one, where each iteration requires a lookup in
> the underlying {{IndexSlice}}. For a common memory mapped index, this
> translates to either a cached request or a read operation. If a segment has
> 6M documents, worst-case is 91 lookups. In our web archive, our segments has
> ~300M values: A worst-case of 4577 lookups!
> One obvious solution is to use a lookup-table for blocks: A long[]-array with
> an entry for each block. For 6M documents, that is < 1KB and would allow for
> direct jumping (a single lookup) in all instances. Unfortunately this
> lookup-table cannot be generated upfront when the writing of values is purely
> streaming. It can be appended to the end of the stream before it is closed,
> but without knowing the position of the lookup-table the reader cannot seek
> to it.
> One strategy for creating such a lookup-table would be to generate it during
> reads and cache it for next lookup. This does not fit directly into how
> {{IndexedDISI}} currently works (it is created anew for each invocation), but
> could probably be added with a little work. An advantage to this is that this
> does not change the underlying format and thus could be used with existing
> indexes.
> h2. The lookup structure inside each block
> If {{ALL}} of the 2^16 values are defined, the structure is empty and the
> ordinal is simply the requested docID with some modulo and multiply math.
> Nothing to improve there.
> If the block is {{DENSE}} (2^12 to 2^16 values are defined), a bitmap is used
> and the number of set bits up to the wanted index (the docID modulo the block
> origo) are counted. That bitmap is a long[1024], meaning that worst case is
> to lookup and count all set bits for 1024 longs!
> One known solution to this is to use a [rank
> structure|[https://en.wikipedia.org/wiki/Succinct_data_structure]]. I
> [implemented
> it|[https://github.com/tokee/lucene-solr/blob/solr5894/solr/core/src/java/org/apache/solr/search/sparse/count/plane/RankCache.java]]
> for a related project and with that (), the rank-overhead for a {{DENSE}}
> block would be long[32] and would ensure a maximum of 9 lookups. It is not
> trivial to build the rank-structure and caching it (assuming all blocks are
> dense) for 6M documents would require 22 KB (3.17% overhead). It would be far
> better to generate the rank-structure at index time and store it immediately
> before the bitset (this is possible with streaming as each block is fully
> resolved before flushing), but of course that would require a change to the
> codec.
> If {{SPARSE}} (< 2^12 values ~= 6%) are defined, the docIDs are simply in the
> form of a list. As a comment in the code suggests, a binary search through
> these would be faster, although that would mean seeking backwards. If that is
> not acceptable, I don't have any immediate idea for avoiding the full
> iteration.
> I propose implementing query-time caching of bo
[jira] [Commented] (LUCENE-8374) Reduce reads for sparse DocValues
[
https://issues.apache.org/jira/browse/LUCENE-8374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16527685#comment-16527685
]
Toke Eskildsen commented on LUCENE-8374:
[~dsmiley] thank you. Let's hope my activeness sticks this time. Why I have
been inactive is probably not suitable for Jira :P
> Reduce reads for sparse DocValues
> -
>
> Key: LUCENE-8374
> URL: https://issues.apache.org/jira/browse/LUCENE-8374
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
>Affects Versions: master (8.0), 7.3.1
> Reporter: Toke Eskildsen
>Priority: Major
>
> The {{Lucene70DocValuesProducer}} has the internal classes
> {{SparseNumericDocValues}} and {{BaseSortedSetDocValues}} (sparse code path),
> which again uses {{IndexedDISI}} to handle the docID -> value-ordinal lookup.
> The value-ordinal is the index of the docID assuming an abstract tightly
> packed monotonically increasing list of docIDs: If the docIDs with
> corresponding values are {{[0, 4, 1432]}}, their value-ordinals will be {{[0,
> 1, 2]}}.
> h2. Outer blocks
> The lookup structure of {{IndexedDISI}} consists of blocks of 2^16 values
> (65536), where each block can be either {{ALL}}, {{DENSE}} (2^12 to 2^16
> values) or {{SPARSE}} (< 2^12 values ~= 6%). Consequently blocks vary quite a
> lot in size and ordinal resolving strategy.
> When a sparse Numeric DocValue is needed, the code first locates the block
> containing the wanted docID flag. It does so by iterating blocks one-by-one
> until it reaches the needed one, where each iteration requires a lookup in
> the underlying {{IndexSlice}}. For a common memory mapped index, this
> translates to either a cached request or a read operation. If a segment has
> 6M documents, worst-case is 91 lookups. In our web archive, our segments has
> ~300M values: A worst-case of 4577 lookups!
> One obvious solution is to use a lookup-table for blocks: A long[]-array with
> an entry for each block. For 6M documents, that is < 1KB and would allow for
> direct jumping (a single lookup) in all instances. Unfortunately this
> lookup-table cannot be generated upfront when the writing of values is purely
> streaming. It can be appended to the end of the stream before it is closed,
> but without knowing the position of the lookup-table the reader cannot seek
> to it.
> One strategy for creating such a lookup-table would be to generate it during
> reads and cache it for next lookup. This does not fit directly into how
> {{IndexedDISI}} currently works (it is created anew for each invocation), but
> could probably be added with a little work. An advantage to this is that this
> does not change the underlying format and thus could be used with existing
> indexes.
> h2. The lookup structure inside each block
> If {{ALL}} of the 2^16 values are defined, the structure is empty and the
> ordinal is simply the requested docID with some modulo and multiply math.
> Nothing to improve there.
> If the block is {{DENSE}} (2^12 to 2^16 values are defined), a bitmap is used
> and the number of set bits up to the wanted index (the docID modulo the block
> origo) are counted. That bitmap is a long[1024], meaning that worst case is
> to lookup and count all set bits for 1024 longs!
> One known solution to this is to use a [rank
> structure|[https://en.wikipedia.org/wiki/Succinct_data_structure]]. I
> [implemented
> it|[https://github.com/tokee/lucene-solr/blob/solr5894/solr/core/src/java/org/apache/solr/search/sparse/count/plane/RankCache.java]]
> for a related project and with that (), the rank-overhead for a {{DENSE}}
> block would be long[32] and would ensure a maximum of 9 lookups. It is not
> trivial to build the rank-structure and caching it (assuming all blocks are
> dense) for 6M documents would require 22 KB (3.17% overhead). It would be far
> better to generate the rank-structure at index time and store it immediately
> before the bitset (this is possible with streaming as each block is fully
> resolved before flushing), but of course that would require a change to the
> codec.
> If {{SPARSE}} (< 2^12 values ~= 6%) are defined, the docIDs are simply in the
> form of a list. As a comment in the code suggests, a binary search through
> these would be faster, although that would mean seeking backwards. If that is
> not acceptable, I don't have any immediate idea for avoiding the full
> iteration.
> I propose implementing query-time caching of both block-jumps and inner-block
> lookups for {{DENSE}} (using rank) as first improvement and an index-ti
[jira] [Commented] (LUCENE-8374) Reduce reads for sparse DocValues
[
https://issues.apache.org/jira/browse/LUCENE-8374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16527681#comment-16527681
]
Toke Eskildsen commented on LUCENE-8374:
Putting the rank-structure with the data would have the benefit that they would
only be stored for the DENSE blocks: Putting it in meta would mean either
unused entries for non-DENSE blocks or some sort of sparse representation,
adding yet another complicated layer. And yes, if the rank is always used
(which would be my guess), there would probably be very little performance
difference between having it in-memory or together with the data that needs to
be read anyway.
As for the rank structure itself, it is trade-off between space and lookups.
* A common structure is to have 1 rank-entry (a long) for each 2048 bits (1/32
overhead): The first 32 bits in the rank is the offset, followed by 3*10 bits
for the bit-counts for the first 3*512 bits in the 2048 bits. With 8 longs in a
512 bit chunk, this means a worst-case of 8 count-bits-in-longs.
* The 2048 bit structure is quite nice from a binary exponential point of
view, but for this case it is not optimal as we know that there is a maximum of
65535 set bits, so only 16 bits are needed for the offset-part, not 32. Another
choice might be 16 + 5*9 bits every 1576 bits (1/24 overhead) with a worst-case
of 4 count-bits-in-longs.
* Or it could be 16 + 4*10 bits every 2560 bits (1/40 overhead) with a
worst-case of 8 count-bits-in-longs (same as the first, just with slightly less
overhead).
* Or maybe 16 + 4*11 bits every 5120 bits (1/80) overhead with a worst-case of
16 count-bits-in-longs.
>From an overhead/performance perspective, the 2560 bit version looks good to
>me. Unfortunately it does not align well with 65536 bits, so it's a bit messy
>compared to the very clean 2048 bit one. ...I am probably over-thinking it
>here. The difference between iterative and rank-based lookups is hopefully be
>apparent either way.
{quote}For what it's worth, the codec API makes it very easy to deal with
backward compatibility, so there would be no problem with completely changing
the default doc-value format in a minor release. It doesn't have to wait for
8.0.
{quote}
This surprises me. A bit of a dangerous thing, is is not? No temporarily
switching back to a known stable sub-version of Solr if a new release within
the same major version turns out to have severe problems.
> Reduce reads for sparse DocValues
> -
>
> Key: LUCENE-8374
> URL: https://issues.apache.org/jira/browse/LUCENE-8374
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
>Affects Versions: master (8.0), 7.3.1
>Reporter: Toke Eskildsen
>Priority: Major
>
> The {{Lucene70DocValuesProducer}} has the internal classes
> {{SparseNumericDocValues}} and {{BaseSortedSetDocValues}} (sparse code path),
> which again uses {{IndexedDISI}} to handle the docID -> value-ordinal lookup.
> The value-ordinal is the index of the docID assuming an abstract tightly
> packed monotonically increasing list of docIDs: If the docIDs with
> corresponding values are {{[0, 4, 1432]}}, their value-ordinals will be {{[0,
> 1, 2]}}.
> h2. Outer blocks
> The lookup structure of {{IndexedDISI}} consists of blocks of 2^16 values
> (65536), where each block can be either {{ALL}}, {{DENSE}} (2^12 to 2^16
> values) or {{SPARSE}} (< 2^12 values ~= 6%). Consequently blocks vary quite a
> lot in size and ordinal resolving strategy.
> When a sparse Numeric DocValue is needed, the code first locates the block
> containing the wanted docID flag. It does so by iterating blocks one-by-one
> until it reaches the needed one, where each iteration requires a lookup in
> the underlying {{IndexSlice}}. For a common memory mapped index, this
> translates to either a cached request or a read operation. If a segment has
> 6M documents, worst-case is 91 lookups. In our web archive, our segments has
> ~300M values: A worst-case of 4577 lookups!
> One obvious solution is to use a lookup-table for blocks: A long[]-array with
> an entry for each block. For 6M documents, that is < 1KB and would allow for
> direct jumping (a single lookup) in all instances. Unfortunately this
> lookup-table cannot be generated upfront when the writing of values is purely
> streaming. It can be appended to the end of the stream before it is closed,
> but without knowing the position of the lookup-table the reader cannot seek
> to it.
> One strategy for creating such a lookup-table would be to generate it during
> reads and cache it for next lookup. This does not fit directly into how
> {{IndexedDISI}} cur
[jira] [Commented] (LUCENE-8374) Reduce reads for sparse DocValues
[
https://issues.apache.org/jira/browse/LUCENE-8374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525300#comment-16525300
]
Toke Eskildsen commented on LUCENE-8374:
Thank you for the suggestions, [~jpountz]. I am not at home in this part of the
code base, so guidance is much appreciated. I will look into how postings work.
Putting the block-index in the separate meta file sounds like the right
solution. What about also putting the rank structure in there too? That way the
existing data file would keep its format and if a setup chooses not to use the
lookup structures (time will show if that makes sense for any setups), its
inactive existence will not affect disk caching of the data file. One downside
I see is that it would require all the rank structures to be kept in-memory
instead of being temporarily loaded as part of accessing a DENSE block.
On the other hand, offloading the support structures to the meta file ties very
well into a two-stage approach where the first stage is query-time improvement
for version 7 and the second is a codec-change for version 8. I must admit to
having a selfish reason for trying to get version 7 to perform better: Our
index takes 50 CPU-years / 3 realtime months to regenerate, so we would like
not to do so.
What is the contract for the slicer? Is seeking strictly forward, so that a
binary search for {{SPARSE}} can only be done by loading the full block
temporarily onto the heap? It would be nice to also have an improvement for
{{SPARSE}}, if possible.
> Reduce reads for sparse DocValues
> -
>
> Key: LUCENE-8374
> URL: https://issues.apache.org/jira/browse/LUCENE-8374
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
>Affects Versions: master (8.0), 7.3.1
> Reporter: Toke Eskildsen
>Priority: Major
>
> The {{Lucene70DocValuesProducer}} has the internal classes
> {{SparseNumericDocValues}} and {{BaseSortedSetDocValues}} (sparse code path),
> which again uses {{IndexedDISI}} to handle the docID -> value-ordinal lookup.
> The value-ordinal is the index of the docID assuming an abstract tightly
> packed monotonically increasing list of docIDs: If the docIDs with
> corresponding values are {{[0, 4, 1432]}}, their value-ordinals will be {{[0,
> 1, 2]}}.
> h2. Outer blocks
> The lookup structure of {{IndexedDISI}} consists of blocks of 2^16 values
> (65536), where each block can be either {{ALL}}, {{DENSE}} (2^12 to 2^16
> values) or {{SPARSE}} (< 2^12 values ~= 6%). Consequently blocks vary quite a
> lot in size and ordinal resolving strategy.
> When a sparse Numeric DocValue is needed, the code first locates the block
> containing the wanted docID flag. It does so by iterating blocks one-by-one
> until it reaches the needed one, where each iteration requires a lookup in
> the underlying {{IndexSlice}}. For a common memory mapped index, this
> translates to either a cached request or a read operation. If a segment has
> 6M documents, worst-case is 91 lookups. In our web archive, our segments has
> ~300M values: A worst-case of 4577 lookups!
> One obvious solution is to use a lookup-table for blocks: A long[]-array with
> an entry for each block. For 6M documents, that is < 1KB and would allow for
> direct jumping (a single lookup) in all instances. Unfortunately this
> lookup-table cannot be generated upfront when the writing of values is purely
> streaming. It can be appended to the end of the stream before it is closed,
> but without knowing the position of the lookup-table the reader cannot seek
> to it.
> One strategy for creating such a lookup-table would be to generate it during
> reads and cache it for next lookup. This does not fit directly into how
> {{IndexedDISI}} currently works (it is created anew for each invocation), but
> could probably be added with a little work. An advantage to this is that this
> does not change the underlying format and thus could be used with existing
> indexes.
> h2. The lookup structure inside each block
> If {{ALL}} of the 2^16 values are defined, the structure is empty and the
> ordinal is simply the requested docID with some modulo and multiply math.
> Nothing to improve there.
> If the block is {{DENSE}} (2^12 to 2^16 values are defined), a bitmap is used
> and the number of set bits up to the wanted index (the docID modulo the block
> origo) are counted. That bitmap is a long[1024], meaning that worst case is
> to lookup and count all set bits for 1024 longs!
> One known solution to this is to use a [rank
> structure|[https://en.wikipedia.org/wiki/Succinct_data_structure]]. I
>
[jira] [Created] (LUCENE-8374) Reduce reads for sparse DocValues
Toke Eskildsen created LUCENE-8374:
--
Summary: Reduce reads for sparse DocValues
Key: LUCENE-8374
URL: https://issues.apache.org/jira/browse/LUCENE-8374
Project: Lucene - Core
Issue Type: Improvement
Components: core/codecs
Affects Versions: 7.3.1, master (8.0)
Reporter: Toke Eskildsen
The {{Lucene70DocValuesProducer}} has the internal classes
{{SparseNumericDocValues}} and {{BaseSortedSetDocValues}} (sparse code path),
which again uses {{IndexedDISI}} to handle the docID -> value-ordinal lookup.
The value-ordinal is the index of the docID assuming an abstract tightly packed
monotonically increasing list of docIDs: If the docIDs with corresponding
values are {{[0, 4, 1432]}}, their value-ordinals will be {{[0, 1, 2]}}.
h2. Outer blocks
The lookup structure of {{IndexedDISI}} consists of blocks of 2^16 values
(65536), where each block can be either {{ALL}}, {{DENSE}} (2^12 to 2^16
values) or {{SPARSE}} (< 2^12 values ~= 6%). Consequently blocks vary quite a
lot in size and ordinal resolving strategy.
When a sparse Numeric DocValue is needed, the code first locates the block
containing the wanted docID flag. It does so by iterating blocks one-by-one
until it reaches the needed one, where each iteration requires a lookup in the
underlying {{IndexSlice}}. For a common memory mapped index, this translates to
either a cached request or a read operation. If a segment has 6M documents,
worst-case is 91 lookups. In our web archive, our segments has ~300M values: A
worst-case of 4577 lookups!
One obvious solution is to use a lookup-table for blocks: A long[]-array with
an entry for each block. For 6M documents, that is < 1KB and would allow for
direct jumping (a single lookup) in all instances. Unfortunately this
lookup-table cannot be generated upfront when the writing of values is purely
streaming. It can be appended to the end of the stream before it is closed, but
without knowing the position of the lookup-table the reader cannot seek to it.
One strategy for creating such a lookup-table would be to generate it during
reads and cache it for next lookup. This does not fit directly into how
{{IndexedDISI}} currently works (it is created anew for each invocation), but
could probably be added with a little work. An advantage to this is that this
does not change the underlying format and thus could be used with existing
indexes.
h2. The lookup structure inside each block
If {{ALL}} of the 2^16 values are defined, the structure is empty and the
ordinal is simply the requested docID with some modulo and multiply math.
Nothing to improve there.
If the block is {{DENSE}} (2^12 to 2^16 values are defined), a bitmap is used
and the number of set bits up to the wanted index (the docID modulo the block
origo) are counted. That bitmap is a long[1024], meaning that worst case is to
lookup and count all set bits for 1024 longs!
One known solution to this is to use a [rank
structure|[https://en.wikipedia.org/wiki/Succinct_data_structure]]. I
[implemented
it|[https://github.com/tokee/lucene-solr/blob/solr5894/solr/core/src/java/org/apache/solr/search/sparse/count/plane/RankCache.java]]
for a related project and with that (), the rank-overhead for a {{DENSE}}
block would be long[32] and would ensure a maximum of 9 lookups. It is not
trivial to build the rank-structure and caching it (assuming all blocks are
dense) for 6M documents would require 22 KB (3.17% overhead). It would be far
better to generate the rank-structure at index time and store it immediately
before the bitset (this is possible with streaming as each block is fully
resolved before flushing), but of course that would require a change to the
codec.
If {{SPARSE}} (< 2^12 values ~= 6%) are defined, the docIDs are simply in the
form of a list. As a comment in the code suggests, a binary search through
these would be faster, although that would mean seeking backwards. If that is
not acceptable, I don't have any immediate idea for avoiding the full iteration.
I propose implementing query-time caching of both block-jumps and inner-block
lookups for {{DENSE}} (using rank) as first improvement and an index-time
{{DENSE}}-rank structure for future improvement. As query-time caching is
likely to be too costly for rapidly-changing indexes, it should probably be an
opt-in in solrconfig.xml.
h2. Some real-world observations
This analysis was triggered by massive (10x) slowdown problems with both simple
querying and large exports from our webarchive index after upgrading from Solr
4.10 to 7.3.1. The query-matching itself takes ½-2 seconds, but returning the
top-10 documents takes 5-20 seconds (~50 non-stored DocValues fields), up from
½-2 seconds in total from Solr 4.10 (more of a mix of stored vs. DocValues, so
might not be directly comparable).
Measuring with Vis
