[jira] [Commented] (SOLR-9592) decorateDocValues cause serious performance issue because of using slowCompositeReaderWrapper
[ https://issues.apache.org/jira/browse/SOLR-9592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15550289#comment-15550289 ] Takahiro Ishikawa commented on SOLR-9592: - Thanks for commit! [~yo...@apache.org]! > decorateDocValues cause serious performance issue because of using > slowCompositeReaderWrapper > - > > Key: SOLR-9592 > URL: https://issues.apache.org/jira/browse/SOLR-9592 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: Response Writers, search >Affects Versions: 6.0, 6.1, 6.2 >Reporter: Takahiro Ishikawa >Assignee: Yonik Seeley > Labels: performance > Fix For: 6.3, master (7.0) > > Attachments: SOLR-9592.patch, SOLR-9592.patch, SOLR-9592_6x.patch > > > I have serious performance issue using AtomicUpdate (and RealtimeGet) with > non stored docValues. > Because decorateDocValues try to merge each leafLeader on the fly via > slowCompositeReaderWrapper and it’s extremely slow (> 10sec). > Simply access docValues via nonCompositeReader could resolve this > issue.(patch) > AtomicUpdate performance(or RealtimeGet performance) > * Environment > ** solr version : 6.0.0 > ** schema ~ 100 fields(90% docValues, some of those are multi valued) > ** index : 5,000,000 > * Performance > ** original : > 10sec per query > ** patched : at least 100msec per query > This patch will also enhance search performance, because DocStreamer also > fetch docValues via decorateDocValues. > Though it depends on each environment, I could take 20% search performance > gain. > (This patch originally written for solr 6.0.0, and now rewritten for master) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9592) decorateDocValues cause serious performance issue because of using slowCompositeReaderWrapper
[ https://issues.apache.org/jira/browse/SOLR-9592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15549690#comment-15549690 ] ASF subversion and git services commented on SOLR-9592: --- Commit 59d83f57e12f72ff8db503a0c01b77cb5df354fd in lucene-solr's branch refs/heads/branch_6x from [~yo...@apache.org] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=59d83f5 ] SOLR-9592: use correct leaf reader rather than top-level reader in SolrIndexReaderm.decorateDocValues > decorateDocValues cause serious performance issue because of using > slowCompositeReaderWrapper > - > > Key: SOLR-9592 > URL: https://issues.apache.org/jira/browse/SOLR-9592 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: Response Writers, search >Affects Versions: 6.0, 6.1, 6.2 >Reporter: Takahiro Ishikawa >Assignee: Yonik Seeley > Labels: performance > Fix For: 6.3, master (7.0) > > Attachments: SOLR-9592.patch, SOLR-9592.patch, SOLR-9592_6x.patch > > > I have serious performance issue using AtomicUpdate (and RealtimeGet) with > non stored docValues. > Because decorateDocValues try to merge each leafLeader on the fly via > slowCompositeReaderWrapper and it’s extremely slow (> 10sec). > Simply access docValues via nonCompositeReader could resolve this > issue.(patch) > AtomicUpdate performance(or RealtimeGet performance) > * Environment > ** solr version : 6.0.0 > ** schema ~ 100 fields(90% docValues, some of those are multi valued) > ** index : 5,000,000 > * Performance > ** original : > 10sec per query > ** patched : at least 100msec per query > This patch will also enhance search performance, because DocStreamer also > fetch docValues via decorateDocValues. > Though it depends on each environment, I could take 20% search performance > gain. > (This patch originally written for solr 6.0.0, and now rewritten for master) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9592) decorateDocValues cause serious performance issue because of using slowCompositeReaderWrapper
[ https://issues.apache.org/jira/browse/SOLR-9592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15549673#comment-15549673 ] ASF subversion and git services commented on SOLR-9592: --- Commit cae6b49a065bfbc5789d8bdcf3706c158c0fc10d in lucene-solr's branch refs/heads/master from [~yo...@apache.org] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=cae6b49 ] SOLR-9592: use correct leaf reader rather than top-level reader in SolrIndexReaderm.decorateDocValues > decorateDocValues cause serious performance issue because of using > slowCompositeReaderWrapper > - > > Key: SOLR-9592 > URL: https://issues.apache.org/jira/browse/SOLR-9592 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: Response Writers, search >Affects Versions: 6.0, 6.1, 6.2 >Reporter: Takahiro Ishikawa >Assignee: Yonik Seeley > Labels: performance > Fix For: 6.3, master (7.0) > > Attachments: SOLR-9592.patch, SOLR-9592.patch, SOLR-9592_6x.patch > > > I have serious performance issue using AtomicUpdate (and RealtimeGet) with > non stored docValues. > Because decorateDocValues try to merge each leafLeader on the fly via > slowCompositeReaderWrapper and it’s extremely slow (> 10sec). > Simply access docValues via nonCompositeReader could resolve this > issue.(patch) > AtomicUpdate performance(or RealtimeGet performance) > * Environment > ** solr version : 6.0.0 > ** schema ~ 100 fields(90% docValues, some of those are multi valued) > ** index : 5,000,000 > * Performance > ** original : > 10sec per query > ** patched : at least 100msec per query > This patch will also enhance search performance, because DocStreamer also > fetch docValues via decorateDocValues. > Though it depends on each environment, I could take 20% search performance > gain. > (This patch originally written for solr 6.0.0, and now rewritten for master) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9592) decorateDocValues cause serious performance issue because of using slowCompositeReaderWrapper
[
https://issues.apache.org/jira/browse/SOLR-9592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15547449#comment-15547449
]
Takahiro Ishikawa commented on SOLR-9592:
-
bq. I supposed weakly agree
Thank you. I'll keep it(getLeafReader) renamed :)
{quote}
Right. Sometimes you need both a global view and a segment view to do it right.
See something like FacetFieldProcessorByArrayDV, where we use both top level
and segment level.
{quote}
Yes, I got what you are saying. If I have a time, I'll go into detail for
MultiDocValues problem.
Now all my work is done. Is there any other comments?
> decorateDocValues cause serious performance issue because of using
> slowCompositeReaderWrapper
> -
>
> Key: SOLR-9592
> URL: https://issues.apache.org/jira/browse/SOLR-9592
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: Response Writers, search
>Affects Versions: 6.0, 6.1, 6.2
>Reporter: Takahiro Ishikawa
> Labels: performance
> Fix For: 6.3, master (7.0)
>
> Attachments: SOLR-9592.patch, SOLR-9592.patch, SOLR-9592_6x.patch
>
>
> I have serious performance issue using AtomicUpdate (and RealtimeGet) with
> non stored docValues.
> Because decorateDocValues try to merge each leafLeader on the fly via
> slowCompositeReaderWrapper and it’s extremely slow (> 10sec).
> Simply access docValues via nonCompositeReader could resolve this
> issue.(patch)
> AtomicUpdate performance(or RealtimeGet performance)
> * Environment
> ** solr version : 6.0.0
> ** schema ~ 100 fields(90% docValues, some of those are multi valued)
> ** index : 5,000,000
> * Performance
> ** original : > 10sec per query
> ** patched : at least 100msec per query
> This patch will also enhance search performance, because DocStreamer also
> fetch docValues via decorateDocValues.
> Though it depends on each environment, I could take 20% search performance
> gain.
> (This patch originally written for solr 6.0.0, and now rewritten for master)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9592) decorateDocValues cause serious performance issue because of using slowCompositeReaderWrapper
[ https://issues.apache.org/jira/browse/SOLR-9592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15546847#comment-15546847 ] Yonik Seeley commented on SOLR-9592: bq. Are you disagree with renaming(-1) or weakly agree with(+0)? I supposed weakly agree :-) bq. Because, at first glance, getLeafReader Yeah, I really don't care for that name. bq. If my understanding is clear, we should use MultiDocValues in cases where you essencially need global view and decorateDocValues usage is not the case right? Right. Sometimes you need both a global view and a segment view to do it right. See something like FacetFieldProcessorByArrayDV, where we use both top level and segment level. decorateDocValues would seem to only need segment level access. > decorateDocValues cause serious performance issue because of using > slowCompositeReaderWrapper > - > > Key: SOLR-9592 > URL: https://issues.apache.org/jira/browse/SOLR-9592 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: Response Writers, search >Affects Versions: 6.0, 6.1, 6.2 >Reporter: Takahiro Ishikawa > Labels: performance > Fix For: 6.3, master (7.0) > > Attachments: SOLR-9592.patch, SOLR-9592.patch, SOLR-9592_6x.patch > > > I have serious performance issue using AtomicUpdate (and RealtimeGet) with > non stored docValues. > Because decorateDocValues try to merge each leafLeader on the fly via > slowCompositeReaderWrapper and it’s extremely slow (> 10sec). > Simply access docValues via nonCompositeReader could resolve this > issue.(patch) > AtomicUpdate performance(or RealtimeGet performance) > * Environment > ** solr version : 6.0.0 > ** schema ~ 100 fields(90% docValues, some of those are multi valued) > ** index : 5,000,000 > * Performance > ** original : > 10sec per query > ** patched : at least 100msec per query > This patch will also enhance search performance, because DocStreamer also > fetch docValues via decorateDocValues. > Though it depends on each environment, I could take 20% search performance > gain. > (This patch originally written for solr 6.0.0, and now rewritten for master) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9592) decorateDocValues cause serious performance issue because of using slowCompositeReaderWrapper
[
https://issues.apache.org/jira/browse/SOLR-9592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15545623#comment-15545623
]
Takahiro Ishikawa commented on SOLR-9592:
-
Thanks for comments, Yonik!
I'm glad to resolve your concern!
bq. The method name change might help a little
Are you disagree with renaming(-1) or weakly agree with(+0)?
I agree with renaming(Varun suggested). Because, at first glance, getLeafReader
does not imply internally calling SlowCompositeLeafReader and this make people
find performance bottlenecks difficult.
{quote}
but the real issue is knowing how to use things like MultiDocValues (i.e. you
generally want to use them for the ord mapping, but not the other stuff!)
We should really cache the MultiDocValues created as well... but that can be a
different JIRA.
{quote}
I may not catch the meaning here. If my understanding is clear, we should use
MultiDocValues in cases where you essencially need global view and
decorateDocValues usage is not the case right?
How to handle things like MultiDocValues in those cases(caching) is interesting
problem and might be other JIRA.
> decorateDocValues cause serious performance issue because of using
> slowCompositeReaderWrapper
> -
>
> Key: SOLR-9592
> URL: https://issues.apache.org/jira/browse/SOLR-9592
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: Response Writers, search
>Affects Versions: 6.0, 6.1, 6.2
>Reporter: Takahiro Ishikawa
> Labels: performance
> Fix For: 6.3, master (7.0)
>
> Attachments: SOLR-9592.patch, SOLR-9592.patch, SOLR-9592_6x.patch
>
>
> I have serious performance issue using AtomicUpdate (and RealtimeGet) with
> non stored docValues.
> Because decorateDocValues try to merge each leafLeader on the fly via
> slowCompositeReaderWrapper and it’s extremely slow (> 10sec).
> Simply access docValues via nonCompositeReader could resolve this
> issue.(patch)
> AtomicUpdate performance(or RealtimeGet performance)
> * Environment
> ** solr version : 6.0.0
> ** schema ~ 100 fields(90% docValues, some of those are multi valued)
> ** index : 5,000,000
> * Performance
> ** original : > 10sec per query
> ** patched : at least 100msec per query
> This patch will also enhance search performance, because DocStreamer also
> fetch docValues via decorateDocValues.
> Though it depends on each environment, I could take 20% search performance
> gain.
> (This patch originally written for solr 6.0.0, and now rewritten for master)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9592) decorateDocValues cause serious performance issue because of using slowCompositeReaderWrapper
[ https://issues.apache.org/jira/browse/SOLR-9592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15545222#comment-15545222 ] Yonik Seeley commented on SOLR-9592: Nice catch! I knew there had to be reasons why /select was so much slower than /xport for docValues! The method name change might help a little, but the real issue is knowing how to use things like MultiDocValues (i.e. you generally want to use them for the ord mapping, but not the other stuff!) We should really cache the MultiDocValues created as well... > decorateDocValues cause serious performance issue because of using > slowCompositeReaderWrapper > - > > Key: SOLR-9592 > URL: https://issues.apache.org/jira/browse/SOLR-9592 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: Response Writers, search >Affects Versions: 6.0, 6.1, 6.2 >Reporter: Takahiro Ishikawa > Labels: performance > Fix For: 6.3, master (7.0) > > Attachments: SOLR-9592.patch, SOLR-9592.patch > > > I have serious performance issue using AtomicUpdate (and RealtimeGet) with > non stored docValues. > Because decorateDocValues try to merge each leafLeader on the fly via > slowCompositeReaderWrapper and it’s extremely slow (> 10sec). > Simply access docValues via nonCompositeReader could resolve this > issue.(patch) > AtomicUpdate performance(or RealtimeGet performance) > * Environment > ** solr version : 6.0.0 > ** schema ~ 100 fields(90% docValues, some of those are multi valued) > ** index : 5,000,000 > * Performance > ** original : > 10sec per query > ** patched : at least 100msec per query > This patch will also enhance search performance, because DocStreamer also > fetch docValues via decorateDocValues. > Though it depends on each environment, I could take 20% search performance > gain. > (This patch originally written for solr 6.0.0, and now rewritten for master) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9592) decorateDocValues cause serious performance issue because of using slowCompositeReaderWrapper
[ https://issues.apache.org/jira/browse/SOLR-9592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15544884#comment-15544884 ] Takahiro Ishikawa commented on SOLR-9592: - Thanks for comments, Varun! Yes! Renaming to getSlowAtomicReader is great idea. I'll add a patch soon. > decorateDocValues cause serious performance issue because of using > slowCompositeReaderWrapper > - > > Key: SOLR-9592 > URL: https://issues.apache.org/jira/browse/SOLR-9592 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: Response Writers, search >Affects Versions: 6.0, 6.1, 6.2 >Reporter: Takahiro Ishikawa > Labels: performance > Fix For: 6.3, master (7.0) > > Attachments: SOLR-9592.patch > > > I have serious performance issue using AtomicUpdate (and RealtimeGet) with > non stored docValues. > Because decorateDocValues try to merge each leafLeader on the fly via > slowCompositeReaderWrapper and it’s extremely slow (> 10sec). > Simply access docValues via nonCompositeReader could resolve this > issue.(patch) > AtomicUpdate performance(or RealtimeGet performance) > * Environment > ** solr version : 6.0.0 > ** schema ~ 100 fields(90% docValues, some of those are multi valued) > ** index : 5,000,000 > * Performance > ** original : > 10sec per query > ** patched : at least 100msec per query > This patch will also enhance search performance, because DocStreamer also > fetch docValues via decorateDocValues. > Though it depends on each environment, I could take 20% search performance > gain. > (This patch originally written for solr 6.0.0, and now rewritten for master) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9592) decorateDocValues cause serious performance issue because of using slowCompositeReaderWrapper
[
https://issues.apache.org/jira/browse/SOLR-9592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15544642#comment-15544642
]
Varun Thacker commented on SOLR-9592:
-
Thanks for the patch Takahiro!
Additionally should we rename {{SolrIndexSearcher#getLeafReader}} to
{{SolrIndexSearcher#getSlowAtomicReader}} to make it more accurate? Also we
could add Javadocs encouraging people to use {{leafContexts}} ?
> decorateDocValues cause serious performance issue because of using
> slowCompositeReaderWrapper
> -
>
> Key: SOLR-9592
> URL: https://issues.apache.org/jira/browse/SOLR-9592
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: Response Writers, search
>Affects Versions: 6.0, 6.1, 6.2
>Reporter: Takahiro Ishikawa
> Labels: performance
> Fix For: 6.3, master (7.0)
>
> Attachments: SOLR-9592.patch
>
>
> I have serious performance issue using AtomicUpdate (and RealtimeGet) with
> non stored docValues.
> Because decorateDocValues try to merge each leafLeader on the fly via
> slowCompositeReaderWrapper and it’s extremely slow (> 10sec).
> Simply access docValues via nonCompositeReader could resolve this
> issue.(patch)
> AtomicUpdate performance(or RealtimeGet performance)
> * Environment
> ** solr version : 6.0.0
> ** schema ~ 100 fields(90% docValues, some of those are multi valued)
> ** index : 5,000,000
> * Performance
> ** original : > 10sec per query
> ** patched : at least 100msec per query
> This patch will also enhance search performance, because DocStreamer also
> fetch docValues via decorateDocValues.
> Though it depends on each environment, I could take 20% search performance
> gain.
> (This patch originally written for solr 6.0.0, and now rewritten for master)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
