[jira] [Commented] (SOLR-6581) Efficient DocValues support and numeric collapse field implementations for Collapse and Expand
[
https://issues.apache.org/jira/browse/SOLR-6581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15392504#comment-15392504
]
Jeff Wartes commented on SOLR-6581:
---
For what it's worth, I recall having a bad experience with that hint in a Solr
5.4 cluster late last year. I never did dig into why though.
I had a similar case where I was collapsing on a highly distinct field, and as
Joel indicates, the memory allocation rate was bad enough I had to give up on
the whole thing. Joel and I discussed this a little in SOLR-9125 if you're
curious.
> Efficient DocValues support and numeric collapse field implementations for
> Collapse and Expand
> --
>
> Key: SOLR-6581
> URL: https://issues.apache.org/jira/browse/SOLR-6581
> Project: Solr
> Issue Type: Bug
>Reporter: Joel Bernstein
>Assignee: Joel Bernstein
>Priority: Minor
> Fix For: 5.0, 6.0
>
> Attachments: SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch,
> SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch,
> SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch,
> SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch,
> renames.diff
>
>
> The 4x implementation of the CollapsingQParserPlugin and the ExpandComponent
> are optimized to work with a top level FieldCache. Top level FieldCaches have
> a very fast docID to top-level ordinal lookup. Fast access to the top-level
> ordinals allows for very high performance field collapsing on high
> cardinality fields.
> LUCENE-5666 unified the DocValues and FieldCache api's so that the top level
> FieldCache is no longer in regular use. Instead all top level caches are
> accessed through MultiDocValues.
> This ticket does the following:
> 1) Optimizes Collapse and Expand to use MultiDocValues and makes this the
> default approach when collapsing on String fields
> 2) Provides an option to use a top level FieldCache if the performance of
> MultiDocValues is a blocker. The mechanism for switching to the FieldCache is
> a new "hint" parameter. If the hint parameter is set to "top_fc" then the
> top-level FieldCache would be used for both Collapse and Expand.
> Example syntax:
> {code}
> fq={!collapse field=x hint=TOP_FC}
> {code}
> 3) Adds numeric collapse field implementations.
> 4) Resolves issue SOLR-6066
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6581) Efficient DocValues support and numeric collapse field implementations for Collapse and Expand
[
https://issues.apache.org/jira/browse/SOLR-6581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15392478#comment-15392478
]
David Smiley commented on SOLR-6581:
This is just one shard; the actual total collection is sharded beyond the
levels most people shard.
In any case, maybe some time I'll peek into profiling what's going on to see if
I can find any insights. But that's low priority as our Solr 6 upgrade without
the hint=top_fc is already a huge improvement overall.
> Efficient DocValues support and numeric collapse field implementations for
> Collapse and Expand
> --
>
> Key: SOLR-6581
> URL: https://issues.apache.org/jira/browse/SOLR-6581
> Project: Solr
> Issue Type: Bug
>Reporter: Joel Bernstein
>Assignee: Joel Bernstein
>Priority: Minor
> Fix For: 5.0, 6.0
>
> Attachments: SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch,
> SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch,
> SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch,
> SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch,
> renames.diff
>
>
> The 4x implementation of the CollapsingQParserPlugin and the ExpandComponent
> are optimized to work with a top level FieldCache. Top level FieldCaches have
> a very fast docID to top-level ordinal lookup. Fast access to the top-level
> ordinals allows for very high performance field collapsing on high
> cardinality fields.
> LUCENE-5666 unified the DocValues and FieldCache api's so that the top level
> FieldCache is no longer in regular use. Instead all top level caches are
> accessed through MultiDocValues.
> This ticket does the following:
> 1) Optimizes Collapse and Expand to use MultiDocValues and makes this the
> default approach when collapsing on String fields
> 2) Provides an option to use a top level FieldCache if the performance of
> MultiDocValues is a blocker. The mechanism for switching to the FieldCache is
> a new "hint" parameter. If the hint parameter is set to "top_fc" then the
> top-level FieldCache would be used for both Collapse and Expand.
> Example syntax:
> {code}
> fq={!collapse field=x hint=TOP_FC}
> {code}
> 3) Adds numeric collapse field implementations.
> 4) Resolves issue SOLR-6066
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6581) Efficient DocValues support and numeric collapse field implementations for Collapse and Expand
[
https://issues.apache.org/jira/browse/SOLR-6581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15392469#comment-15392469
]
Joel Bernstein commented on SOLR-6581:
--
Wow that's a hard case.
The string collapse is done into a fixed array the size of the unique values in
the field. Similar to faceting on a string field. So we're talking about a huge
amount of memory for a single query. Still not sure why MultiDocValues would
outperform the top level field cache in this scenario.
In this scenario sharding would be very useful, but you would have to shard on
the collapse field.
> Efficient DocValues support and numeric collapse field implementations for
> Collapse and Expand
> --
>
> Key: SOLR-6581
> URL: https://issues.apache.org/jira/browse/SOLR-6581
> Project: Solr
> Issue Type: Bug
>Reporter: Joel Bernstein
>Assignee: Joel Bernstein
>Priority: Minor
> Fix For: 5.0, 6.0
>
> Attachments: SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch,
> SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch,
> SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch,
> SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch,
> renames.diff
>
>
> The 4x implementation of the CollapsingQParserPlugin and the ExpandComponent
> are optimized to work with a top level FieldCache. Top level FieldCaches have
> a very fast docID to top-level ordinal lookup. Fast access to the top-level
> ordinals allows for very high performance field collapsing on high
> cardinality fields.
> LUCENE-5666 unified the DocValues and FieldCache api's so that the top level
> FieldCache is no longer in regular use. Instead all top level caches are
> accessed through MultiDocValues.
> This ticket does the following:
> 1) Optimizes Collapse and Expand to use MultiDocValues and makes this the
> default approach when collapsing on String fields
> 2) Provides an option to use a top level FieldCache if the performance of
> MultiDocValues is a blocker. The mechanism for switching to the FieldCache is
> a new "hint" parameter. If the hint parameter is set to "top_fc" then the
> top-level FieldCache would be used for both Collapse and Expand.
> Example syntax:
> {code}
> fq={!collapse field=x hint=TOP_FC}
> {code}
> 3) Adds numeric collapse field implementations.
> 4) Resolves issue SOLR-6066
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6581) Efficient DocValues support and numeric collapse field implementations for Collapse and Expand
[
https://issues.apache.org/jira/browse/SOLR-6581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15392451#comment-15392451
]
Joel Bernstein commented on SOLR-6581:
--
Ok, false alarm. my initial tests were faulty. I thought a had loaded 5,000,000
docs and actually a had set the job to load 50,000,000 docs. So the test was
running with the indexing running.
After running a proper test I found that things are as expected. I'm seeing
to_fc queries running almost 3 times faster with the top_fc hint.
I was running with these simple queries:
{code}
{!collapse field=test_s hint=top_fc}
and
{!collapse field=test_s}
{code}
I had an index of 5 million docs and the test_s field had 1.8 million unique
values.
With the top_fc hint the query was taking around 160 millis.
Without the top_fc hint the query was taking around 440 millis.
> Efficient DocValues support and numeric collapse field implementations for
> Collapse and Expand
> --
>
> Key: SOLR-6581
> URL: https://issues.apache.org/jira/browse/SOLR-6581
> Project: Solr
> Issue Type: Bug
>Reporter: Joel Bernstein
>Assignee: Joel Bernstein
>Priority: Minor
> Fix For: 5.0, 6.0
>
> Attachments: SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch,
> SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch,
> SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch,
> SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch,
> renames.diff
>
>
> The 4x implementation of the CollapsingQParserPlugin and the ExpandComponent
> are optimized to work with a top level FieldCache. Top level FieldCaches have
> a very fast docID to top-level ordinal lookup. Fast access to the top-level
> ordinals allows for very high performance field collapsing on high
> cardinality fields.
> LUCENE-5666 unified the DocValues and FieldCache api's so that the top level
> FieldCache is no longer in regular use. Instead all top level caches are
> accessed through MultiDocValues.
> This ticket does the following:
> 1) Optimizes Collapse and Expand to use MultiDocValues and makes this the
> default approach when collapsing on String fields
> 2) Provides an option to use a top level FieldCache if the performance of
> MultiDocValues is a blocker. The mechanism for switching to the FieldCache is
> a new "hint" parameter. If the hint parameter is set to "top_fc" then the
> top-level FieldCache would be used for both Collapse and Expand.
> Example syntax:
> {code}
> fq={!collapse field=x hint=TOP_FC}
> {code}
> 3) Adds numeric collapse field implementations.
> 4) Resolves issue SOLR-6066
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6581) Efficient DocValues support and numeric collapse field implementations for Collapse and Expand
[
https://issues.apache.org/jira/browse/SOLR-6581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15392402#comment-15392402
]
David Smiley commented on SOLR-6581:
[~joel.bernstein] It's a single index (shard) of 20.8M docs spanning 180
segments (kinda a lot I know) containing a string parent-document-id field with
docValues. The field is always populated and has almost as many distinct
values as there are documents -- 18.9M. The collapse usage has
{{max=(aBooleanField)}} and no sort. I do see a field cache (UninvertedReader)
entry on this field via looking at the admin screen (says it takes up ~403MB).
> Efficient DocValues support and numeric collapse field implementations for
> Collapse and Expand
> --
>
> Key: SOLR-6581
> URL: https://issues.apache.org/jira/browse/SOLR-6581
> Project: Solr
> Issue Type: Bug
>Reporter: Joel Bernstein
>Assignee: Joel Bernstein
>Priority: Minor
> Fix For: 5.0, 6.0
>
> Attachments: SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch,
> SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch,
> SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch,
> SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch,
> renames.diff
>
>
> The 4x implementation of the CollapsingQParserPlugin and the ExpandComponent
> are optimized to work with a top level FieldCache. Top level FieldCaches have
> a very fast docID to top-level ordinal lookup. Fast access to the top-level
> ordinals allows for very high performance field collapsing on high
> cardinality fields.
> LUCENE-5666 unified the DocValues and FieldCache api's so that the top level
> FieldCache is no longer in regular use. Instead all top level caches are
> accessed through MultiDocValues.
> This ticket does the following:
> 1) Optimizes Collapse and Expand to use MultiDocValues and makes this the
> default approach when collapsing on String fields
> 2) Provides an option to use a top level FieldCache if the performance of
> MultiDocValues is a blocker. The mechanism for switching to the FieldCache is
> a new "hint" parameter. If the hint parameter is set to "top_fc" then the
> top-level FieldCache would be used for both Collapse and Expand.
> Example syntax:
> {code}
> fq={!collapse field=x hint=TOP_FC}
> {code}
> 3) Adds numeric collapse field implementations.
> 4) Resolves issue SOLR-6066
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6581) Efficient DocValues support and numeric collapse field implementations for Collapse and Expand
[
https://issues.apache.org/jira/browse/SOLR-6581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15392392#comment-15392392
]
Joel Bernstein commented on SOLR-6581:
--
I just ran a test with collapse on trunk and validated [~dsmiley] findings.
Collapse at this point is so slow both with and without top_fc that it's really
not fit for purpose.
I'll dig a little to see if I can see what's happening.
> Efficient DocValues support and numeric collapse field implementations for
> Collapse and Expand
> --
>
> Key: SOLR-6581
> URL: https://issues.apache.org/jira/browse/SOLR-6581
> Project: Solr
> Issue Type: Bug
>Reporter: Joel Bernstein
>Assignee: Joel Bernstein
>Priority: Minor
> Fix For: 5.0, 6.0
>
> Attachments: SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch,
> SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch,
> SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch,
> SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch,
> renames.diff
>
>
> The 4x implementation of the CollapsingQParserPlugin and the ExpandComponent
> are optimized to work with a top level FieldCache. Top level FieldCaches have
> a very fast docID to top-level ordinal lookup. Fast access to the top-level
> ordinals allows for very high performance field collapsing on high
> cardinality fields.
> LUCENE-5666 unified the DocValues and FieldCache api's so that the top level
> FieldCache is no longer in regular use. Instead all top level caches are
> accessed through MultiDocValues.
> This ticket does the following:
> 1) Optimizes Collapse and Expand to use MultiDocValues and makes this the
> default approach when collapsing on String fields
> 2) Provides an option to use a top level FieldCache if the performance of
> MultiDocValues is a blocker. The mechanism for switching to the FieldCache is
> a new "hint" parameter. If the hint parameter is set to "top_fc" then the
> top-level FieldCache would be used for both Collapse and Expand.
> Example syntax:
> {code}
> fq={!collapse field=x hint=TOP_FC}
> {code}
> 3) Adds numeric collapse field implementations.
> 4) Resolves issue SOLR-6066
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6581) Efficient DocValues support and numeric collapse field implementations for Collapse and Expand
[
https://issues.apache.org/jira/browse/SOLR-6581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15392310#comment-15392310
]
Joel Bernstein commented on SOLR-6581:
--
[~dsmiley], can you post the query you were using. And also some specs on the
data set and result set. Things like how many unique values in the collapse
field, how documents in the result set pre-collapse. Thanks!
> Efficient DocValues support and numeric collapse field implementations for
> Collapse and Expand
> --
>
> Key: SOLR-6581
> URL: https://issues.apache.org/jira/browse/SOLR-6581
> Project: Solr
> Issue Type: Bug
>Reporter: Joel Bernstein
>Assignee: Joel Bernstein
>Priority: Minor
> Fix For: 5.0, 6.0
>
> Attachments: SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch,
> SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch,
> SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch,
> SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch,
> renames.diff
>
>
> The 4x implementation of the CollapsingQParserPlugin and the ExpandComponent
> are optimized to work with a top level FieldCache. Top level FieldCaches have
> a very fast docID to top-level ordinal lookup. Fast access to the top-level
> ordinals allows for very high performance field collapsing on high
> cardinality fields.
> LUCENE-5666 unified the DocValues and FieldCache api's so that the top level
> FieldCache is no longer in regular use. Instead all top level caches are
> accessed through MultiDocValues.
> This ticket does the following:
> 1) Optimizes Collapse and Expand to use MultiDocValues and makes this the
> default approach when collapsing on String fields
> 2) Provides an option to use a top level FieldCache if the performance of
> MultiDocValues is a blocker. The mechanism for switching to the FieldCache is
> a new "hint" parameter. If the hint parameter is set to "top_fc" then the
> top-level FieldCache would be used for both Collapse and Expand.
> Example syntax:
> {code}
> fq={!collapse field=x hint=TOP_FC}
> {code}
> 3) Adds numeric collapse field implementations.
> 4) Resolves issue SOLR-6066
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6581) Efficient DocValues support and numeric collapse field implementations for Collapse and Expand
[
https://issues.apache.org/jira/browse/SOLR-6581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15392287#comment-15392287
]
Joel Bernstein commented on SOLR-6581:
--
Were you using the sort param or max/min param? All the top_fc performance
testing was done with max/min param. The sort param came later and I'm not sure
how it performs with top_fc.
The top_fc param should always be the fastest approach when using min/max. If
it's not then something odd is going on.
> Efficient DocValues support and numeric collapse field implementations for
> Collapse and Expand
> --
>
> Key: SOLR-6581
> URL: https://issues.apache.org/jira/browse/SOLR-6581
> Project: Solr
> Issue Type: Bug
>Reporter: Joel Bernstein
>Assignee: Joel Bernstein
>Priority: Minor
> Fix For: 5.0, 6.0
>
> Attachments: SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch,
> SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch,
> SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch,
> SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch,
> renames.diff
>
>
> The 4x implementation of the CollapsingQParserPlugin and the ExpandComponent
> are optimized to work with a top level FieldCache. Top level FieldCaches have
> a very fast docID to top-level ordinal lookup. Fast access to the top-level
> ordinals allows for very high performance field collapsing on high
> cardinality fields.
> LUCENE-5666 unified the DocValues and FieldCache api's so that the top level
> FieldCache is no longer in regular use. Instead all top level caches are
> accessed through MultiDocValues.
> This ticket does the following:
> 1) Optimizes Collapse and Expand to use MultiDocValues and makes this the
> default approach when collapsing on String fields
> 2) Provides an option to use a top level FieldCache if the performance of
> MultiDocValues is a blocker. The mechanism for switching to the FieldCache is
> a new "hint" parameter. If the hint parameter is set to "top_fc" then the
> top-level FieldCache would be used for both Collapse and Expand.
> Example syntax:
> {code}
> fq={!collapse field=x hint=TOP_FC}
> {code}
> 3) Adds numeric collapse field implementations.
> 4) Resolves issue SOLR-6066
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6581) Efficient DocValues support and numeric collapse field implementations for Collapse and Expand
[
https://issues.apache.org/jira/browse/SOLR-6581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15392175#comment-15392175
]
David Smiley commented on SOLR-6581:
Dallan:
bq. Any ideas how much slower numeric collapse/expand implementation is than
string collapse/expand with the top_fc hint?
I would guess a numeric impl would beat a string impl every time.
Hey FYI everyone... this is weird but while I just finished doing a major Solr
4.10 -> Solr 6.1.0 upgrade, I found that the top_fc for the collapse had quite
the opposite effect in a test environment _with no indexing/commits_. top_fc
took twice as long as without. Shrug; no clue. So folks don't go setting this
blindly without actually measuirng before & after in your own environment.
> Efficient DocValues support and numeric collapse field implementations for
> Collapse and Expand
> --
>
> Key: SOLR-6581
> URL: https://issues.apache.org/jira/browse/SOLR-6581
> Project: Solr
> Issue Type: Bug
>Reporter: Joel Bernstein
>Assignee: Joel Bernstein
>Priority: Minor
> Fix For: 5.0, 6.0
>
> Attachments: SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch,
> SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch,
> SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch,
> SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch,
> renames.diff
>
>
> The 4x implementation of the CollapsingQParserPlugin and the ExpandComponent
> are optimized to work with a top level FieldCache. Top level FieldCaches have
> a very fast docID to top-level ordinal lookup. Fast access to the top-level
> ordinals allows for very high performance field collapsing on high
> cardinality fields.
> LUCENE-5666 unified the DocValues and FieldCache api's so that the top level
> FieldCache is no longer in regular use. Instead all top level caches are
> accessed through MultiDocValues.
> This ticket does the following:
> 1) Optimizes Collapse and Expand to use MultiDocValues and makes this the
> default approach when collapsing on String fields
> 2) Provides an option to use a top level FieldCache if the performance of
> MultiDocValues is a blocker. The mechanism for switching to the FieldCache is
> a new "hint" parameter. If the hint parameter is set to "top_fc" then the
> top-level FieldCache would be used for both Collapse and Expand.
> Example syntax:
> {code}
> fq={!collapse field=x hint=TOP_FC}
> {code}
> 3) Adds numeric collapse field implementations.
> 4) Resolves issue SOLR-6066
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6581) Efficient DocValues support and numeric collapse field implementations for Collapse and Expand
[
https://issues.apache.org/jira/browse/SOLR-6581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14335405#comment-14335405
]
Dallan Quass commented on SOLR-6581:
Any ideas how much slower numeric collapse/expand implementation is than string
collapse/expand with the top_fc hint? I'm trying to decide if I should re-index
my int collapse field as a string. (I don't care about real-time performance.)
Efficient DocValues support and numeric collapse field implementations for
Collapse and Expand
--
Key: SOLR-6581
URL: https://issues.apache.org/jira/browse/SOLR-6581
Project: Solr
Issue Type: Bug
Reporter: Joel Bernstein
Assignee: Joel Bernstein
Priority: Minor
Fix For: 5.0, Trunk
Attachments: SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch,
SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch,
SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch,
SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch,
renames.diff
The 4x implementation of the CollapsingQParserPlugin and the ExpandComponent
are optimized to work with a top level FieldCache. Top level FieldCaches have
a very fast docID to top-level ordinal lookup. Fast access to the top-level
ordinals allows for very high performance field collapsing on high
cardinality fields.
LUCENE-5666 unified the DocValues and FieldCache api's so that the top level
FieldCache is no longer in regular use. Instead all top level caches are
accessed through MultiDocValues.
This ticket does the following:
1) Optimizes Collapse and Expand to use MultiDocValues and makes this the
default approach when collapsing on String fields
2) Provides an option to use a top level FieldCache if the performance of
MultiDocValues is a blocker. The mechanism for switching to the FieldCache is
a new hint parameter. If the hint parameter is set to top_fc then the
top-level FieldCache would be used for both Collapse and Expand.
Example syntax:
{code}
fq={!collapse field=x hint=TOP_FC}
{code}
3) Adds numeric collapse field implementations.
4) Resolves issue SOLR-6066
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6581) Efficient DocValues support and numeric collapse field implementations for Collapse and Expand
[
https://issues.apache.org/jira/browse/SOLR-6581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14286240#comment-14286240
]
Joel Bernstein commented on SOLR-6581:
--
The hint in the code is still upper case TOP_FC. This was meant to be lower
case. I'll open another issue for this and have it accept both cases. 5.0 will
go out with the upper case syntax though so I'll update the documentation.
Efficient DocValues support and numeric collapse field implementations for
Collapse and Expand
--
Key: SOLR-6581
URL: https://issues.apache.org/jira/browse/SOLR-6581
Project: Solr
Issue Type: Bug
Reporter: Joel Bernstein
Assignee: Joel Bernstein
Priority: Minor
Fix For: 5.0, Trunk
Attachments: SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch,
SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch,
SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch,
SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch,
renames.diff
The 4x implementation of the CollapsingQParserPlugin and the ExpandComponent
are optimized to work with a top level FieldCache. Top level FieldCaches have
a very fast docID to top-level ordinal lookup. Fast access to the top-level
ordinals allows for very high performance field collapsing on high
cardinality fields.
LUCENE-5666 unified the DocValues and FieldCache api's so that the top level
FieldCache is no longer in regular use. Instead all top level caches are
accessed through MultiDocValues.
This ticket does the following:
1) Optimizes Collapse and Expand to use MultiDocValues and makes this the
default approach when collapsing on String fields
2) Provides an option to use a top level FieldCache if the performance of
MultiDocValues is a blocker. The mechanism for switching to the FieldCache is
a new hint parameter. If the hint parameter is set to top_fc then the
top-level FieldCache would be used for both Collapse and Expand.
Example syntax:
{code}
fq={!collapse field=x hint=TOP_FC}
{code}
3) Adds numeric collapse field implementations.
4) Resolves issue SOLR-6066
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6581) Efficient DocValues support and numeric collapse field implementations for Collapse and Expand
[
https://issues.apache.org/jira/browse/SOLR-6581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14276958#comment-14276958
]
ASF subversion and git services commented on SOLR-6581:
---
Commit 1651685 from [~joel.bernstein] in branch 'dev/trunk'
[ https://svn.apache.org/r1651685 ]
SOLR-6581: Additional test for Collapse and fixed problem with Expand float
tests
Efficient DocValues support and numeric collapse field implementations for
Collapse and Expand
--
Key: SOLR-6581
URL: https://issues.apache.org/jira/browse/SOLR-6581
Project: Solr
Issue Type: Bug
Reporter: Joel Bernstein
Assignee: Joel Bernstein
Priority: Minor
Fix For: 5.0, Trunk
Attachments: SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch,
SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch,
SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch,
SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch,
renames.diff
The 4x implementation of the CollapsingQParserPlugin and the ExpandComponent
are optimized to work with a top level FieldCache. Top level FieldCaches have
a very fast docID to top-level ordinal lookup. Fast access to the top-level
ordinals allows for very high performance field collapsing on high
cardinality fields.
LUCENE-5666 unified the DocValues and FieldCache api's so that the top level
FieldCache is no longer in regular use. Instead all top level caches are
accessed through MultiDocValues.
This ticket does the following:
1) Optimizes Collapse and Expand to use MultiDocValues and makes this the
default approach when collapsing on String fields
2) Provides an option to use a top level FieldCache if the performance of
MultiDocValues is a blocker. The mechanism for switching to the FieldCache is
a new hint parameter. If the hint parameter is set to top_fc then the
top-level FieldCache would be used for both Collapse and Expand.
Example syntax:
{code}
fq={!collapse field=x hint=top_fc}
{code}
3) Adds numeric collapse field implementations.
4) Resolves issue SOLR-6066
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6581) Efficient DocValues support and numeric collapse field implementations for Collapse and Expand
[
https://issues.apache.org/jira/browse/SOLR-6581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14277018#comment-14277018
]
ASF subversion and git services commented on SOLR-6581:
---
Commit 1651693 from [~joel.bernstein] in branch 'dev/branches/branch_5x'
[ https://svn.apache.org/r1651693 ]
SOLR-6581: Additional test for Collapse and fixed problem with Expand float
tests
Efficient DocValues support and numeric collapse field implementations for
Collapse and Expand
--
Key: SOLR-6581
URL: https://issues.apache.org/jira/browse/SOLR-6581
Project: Solr
Issue Type: Bug
Reporter: Joel Bernstein
Assignee: Joel Bernstein
Priority: Minor
Fix For: 5.0, Trunk
Attachments: SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch,
SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch,
SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch,
SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch,
renames.diff
The 4x implementation of the CollapsingQParserPlugin and the ExpandComponent
are optimized to work with a top level FieldCache. Top level FieldCaches have
a very fast docID to top-level ordinal lookup. Fast access to the top-level
ordinals allows for very high performance field collapsing on high
cardinality fields.
LUCENE-5666 unified the DocValues and FieldCache api's so that the top level
FieldCache is no longer in regular use. Instead all top level caches are
accessed through MultiDocValues.
This ticket does the following:
1) Optimizes Collapse and Expand to use MultiDocValues and makes this the
default approach when collapsing on String fields
2) Provides an option to use a top level FieldCache if the performance of
MultiDocValues is a blocker. The mechanism for switching to the FieldCache is
a new hint parameter. If the hint parameter is set to top_fc then the
top-level FieldCache would be used for both Collapse and Expand.
Example syntax:
{code}
fq={!collapse field=x hint=top_fc}
{code}
3) Adds numeric collapse field implementations.
4) Resolves issue SOLR-6066
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6581) Efficient DocValues support and numeric collapse field implementations for Collapse and Expand
[
https://issues.apache.org/jira/browse/SOLR-6581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14277259#comment-14277259
]
ASF subversion and git services commented on SOLR-6581:
---
Commit 1651736 from [~joel.bernstein] in branch 'dev/branches/lucene_solr_5_0'
[ https://svn.apache.org/r1651736 ]
SOLR-6581: Additional test for Collapse and fixed problem with Expand float
tests
Efficient DocValues support and numeric collapse field implementations for
Collapse and Expand
--
Key: SOLR-6581
URL: https://issues.apache.org/jira/browse/SOLR-6581
Project: Solr
Issue Type: Bug
Reporter: Joel Bernstein
Assignee: Joel Bernstein
Priority: Minor
Fix For: 5.0, Trunk
Attachments: SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch,
SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch,
SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch,
SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch,
renames.diff
The 4x implementation of the CollapsingQParserPlugin and the ExpandComponent
are optimized to work with a top level FieldCache. Top level FieldCaches have
a very fast docID to top-level ordinal lookup. Fast access to the top-level
ordinals allows for very high performance field collapsing on high
cardinality fields.
LUCENE-5666 unified the DocValues and FieldCache api's so that the top level
FieldCache is no longer in regular use. Instead all top level caches are
accessed through MultiDocValues.
This ticket does the following:
1) Optimizes Collapse and Expand to use MultiDocValues and makes this the
default approach when collapsing on String fields
2) Provides an option to use a top level FieldCache if the performance of
MultiDocValues is a blocker. The mechanism for switching to the FieldCache is
a new hint parameter. If the hint parameter is set to top_fc then the
top-level FieldCache would be used for both Collapse and Expand.
Example syntax:
{code}
fq={!collapse field=x hint=top_fc}
{code}
3) Adds numeric collapse field implementations.
4) Resolves issue SOLR-6066
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6581) Efficient DocValues support and numeric collapse field implementations for Collapse and Expand
[
https://issues.apache.org/jira/browse/SOLR-6581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14273671#comment-14273671
]
ASF subversion and git services commented on SOLR-6581:
---
Commit 1651109 from [~joel.bernstein] in branch 'dev/branches/branch_5x'
[ https://svn.apache.org/r1651109 ]
SOLR-6581: Efficient DocValues support and numeric collapse field
implementations for Collapse and Expand
Efficient DocValues support and numeric collapse field implementations for
Collapse and Expand
--
Key: SOLR-6581
URL: https://issues.apache.org/jira/browse/SOLR-6581
Project: Solr
Issue Type: Bug
Reporter: Joel Bernstein
Assignee: Joel Bernstein
Priority: Minor
Fix For: 5.0
Attachments: SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch,
SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch,
SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch,
SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch,
renames.diff
The 4x implementation of the CollapsingQParserPlugin and the ExpandComponent
are optimized to work with a top level FieldCache. Top level FieldCaches have
a very fast docID to top-level ordinal lookup. Fast access to the top-level
ordinals allows for very high performance field collapsing on high
cardinality fields.
LUCENE-5666 unified the DocValues and FieldCache api's so that the top level
FieldCache is no longer in regular use. Instead all top level caches are
accessed through MultiDocValues.
This ticket does the following:
1) Optimizes Collapse and Expand to use MultiDocValues and makes this the
default approach when collapsing on String fields
2) Provides an option to use a top level FieldCache if the performance of
MultiDocValues is a blocker. The mechanism for switching to the FieldCache is
a new hint parameter. If the hint parameter is set to FAST_QUERY then the
top-level FieldCache would be used for both Collapse and Expand.
Example syntax:
{code}
fq={!collapse field=x hint=FAST_QUERY}
{code}
3) Adds numeric collapse field implementations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6581) Efficient DocValues support and numeric collapse field implementations for Collapse and Expand
[
https://issues.apache.org/jira/browse/SOLR-6581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14273597#comment-14273597
]
ASF subversion and git services commented on SOLR-6581:
---
Commit 1651087 from [~joel.bernstein] in branch 'dev/trunk'
[ https://svn.apache.org/r1651087 ]
SOLR-6581: Efficient DocValues support and numeric collapse field
implementations for Collapse and Expand
Efficient DocValues support and numeric collapse field implementations for
Collapse and Expand
--
Key: SOLR-6581
URL: https://issues.apache.org/jira/browse/SOLR-6581
Project: Solr
Issue Type: Bug
Reporter: Joel Bernstein
Assignee: Joel Bernstein
Priority: Minor
Fix For: 5.0
Attachments: SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch,
SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch,
SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch,
SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch,
renames.diff
The 4x implementation of the CollapsingQParserPlugin and the ExpandComponent
are optimized to work with a top level FieldCache. Top level FieldCaches have
a very fast docID to top-level ordinal lookup. Fast access to the top-level
ordinals allows for very high performance field collapsing on high
cardinality fields.
LUCENE-5666 unified the DocValues and FieldCache api's so that the top level
FieldCache is no longer in regular use. Instead all top level caches are
accessed through MultiDocValues.
This ticket does the following:
1) Optimizes Collapse and Expand to use MultiDocValues and makes this the
default approach when collapsing on String fields
2) Provides an option to use a top level FieldCache if the performance of
MultiDocValues is a blocker. The mechanism for switching to the FieldCache is
a new hint parameter. If the hint parameter is set to FAST_QUERY then the
top-level FieldCache would be used for both Collapse and Expand.
Example syntax:
{code}
fq={!collapse field=x hint=FAST_QUERY}
{code}
3) Adds numeric collapse field implementations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6581) Efficient DocValues support and numeric collapse field implementations for Collapse and Expand
[
https://issues.apache.org/jira/browse/SOLR-6581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14273182#comment-14273182
]
Joel Bernstein commented on SOLR-6581:
--
top_fc sounds good to me. I'll make the change.
Efficient DocValues support and numeric collapse field implementations for
Collapse and Expand
--
Key: SOLR-6581
URL: https://issues.apache.org/jira/browse/SOLR-6581
Project: Solr
Issue Type: Bug
Reporter: Joel Bernstein
Assignee: Joel Bernstein
Priority: Minor
Fix For: 5.0
Attachments: SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch,
SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch,
SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch,
SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch,
renames.diff
The 4x implementation of the CollapsingQParserPlugin and the ExpandComponent
are optimized to work with a top level FieldCache. Top level FieldCaches have
a very fast docID to top-level ordinal lookup. Fast access to the top-level
ordinals allows for very high performance field collapsing on high
cardinality fields.
LUCENE-5666 unified the DocValues and FieldCache api's so that the top level
FieldCache is no longer in regular use. Instead all top level caches are
accessed through MultiDocValues.
This ticket does the following:
1) Optimizes Collapse and Expand to use MultiDocValues and makes this the
default approach when collapsing on String fields
2) Provides an option to use a top level FieldCache if the performance of
MultiDocValues is a blocker. The mechanism for switching to the FieldCache is
a new hint parameter. If the hint parameter is set to FAST_QUERY then the
top-level FieldCache would be used for both Collapse and Expand.
Example syntax:
{code}
fq={!collapse field=x hint=FAST_QUERY}
{code}
3) Adds numeric collapse field implementations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6581) Efficient DocValues support and numeric collapse field implementations for Collapse and Expand
[
https://issues.apache.org/jira/browse/SOLR-6581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14273162#comment-14273162
]
Yonik Seeley commented on SOLR-6581:
FAST_QUERY doesn't give much of an idea of what's going on under the covers,
and a more descriptive name would probably be better if more
methods/optimizations may be added in the future. Maybe something like
top_fc? Probably best to stick to lower case too...
Efficient DocValues support and numeric collapse field implementations for
Collapse and Expand
--
Key: SOLR-6581
URL: https://issues.apache.org/jira/browse/SOLR-6581
Project: Solr
Issue Type: Bug
Reporter: Joel Bernstein
Assignee: Joel Bernstein
Priority: Minor
Fix For: 5.0
Attachments: SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch,
SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch,
SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch,
SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch, SOLR-6581.patch,
renames.diff
The 4x implementation of the CollapsingQParserPlugin and the ExpandComponent
are optimized to work with a top level FieldCache. Top level FieldCaches have
a very fast docID to top-level ordinal lookup. Fast access to the top-level
ordinals allows for very high performance field collapsing on high
cardinality fields.
LUCENE-5666 unified the DocValues and FieldCache api's so that the top level
FieldCache is no longer in regular use. Instead all top level caches are
accessed through MultiDocValues.
This ticket does the following:
1) Optimizes Collapse and Expand to use MultiDocValues and makes this the
default approach when collapsing on String fields
2) Provides an option to use a top level FieldCache if the performance of
MultiDocValues is a blocker. The mechanism for switching to the FieldCache is
a new hint parameter. If the hint parameter is set to FAST_QUERY then the
top-level FieldCache would be used for both Collapse and Expand.
Example syntax:
{code}
fq={!collapse field=x hint=FAST_QUERY}
{code}
3) Adds numeric collapse field implementations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
