Re: faceting is unusable slow since upgrade to 5.3.0

Thanks Jamie. On Sat, Dec 19, 2015 at 11:31 PM, Jamie Johnson wrote: > Bill, > > Check out the patch attached to > https://issues.apache.org/jira/browse/SOLR-8096. I had considered making > the method uif after I had done most of the work, it would be trivial to > change and

Re: faceting is unusable slow since upgrade to 5.3.0

Can we add method=uif back when not using the JSON Facet API too? That would help a lot of people. On Thu, Dec 17, 2015 at 7:17 AM, Yonik Seeley wrote: > On Wed, Dec 16, 2015 at 4:57 AM, Vincenzo D'Amore > wrote: > > Hi all, > > > > given that solr 5.4

Re: faceting is unusable slow since upgrade to 5.3.0

Bill, Check out the patch attached to https://issues.apache.org/jira/browse/SOLR-8096. I had considered making the method uif after I had done most of the work, it would be trivial to change and would probably be more aligned with not adding unexpected changes to people that are currently using

Re: faceting is unusable slow since upgrade to 5.3.0

Can we still specify the cache implementation for the field cache? When this change occurred to faceting (uninverting reader vs field ) it prevented us from moving to 5.x but if we can get the 4.x functionality using that api we could look to port to the latest. Jamie On Dec 17, 2015 9:18 AM,

Re: faceting is unusable slow since upgrade to 5.3.0

Also can we get the capability to choose the method of faceting in the older faceting component? I'm not looking for complete feature parity just the ability to specify the method. As always thanks. On Fri, Dec 18, 2015 at 8:04 AM, Jamie Johnson wrote: > Can we still

Re: faceting is unusable slow since upgrade to 5.3.0

This fix definitely help for facet.field over docvalues field on mult-segment index since 5.4. I suppose it's irrelevant to JSON Facets, non-dv field, and pre 5.4. I can not comment about comparing perfomance of dv and non-dv fields, because "it depends" (с) benchmarking and profiler are the only

Re: faceting is unusable slow since upgrade to 5.3.0

On Wed, Dec 16, 2015 at 4:57 AM, Vincenzo D'Amore wrote: > Hi all, > > given that solr 5.4 is finally released, is this what's more stable and > efficient version of solrcloud ? > > I have a website which receives many search requests. It serve normally > about 2000 concurrent

Re: faceting is unusable slow since upgrade to 5.3.0

Same question here Wondering if faceting performance is fixed and how to take advantage of it ? On Wed, Dec 16, 2015 at 2:57 AM, Vincenzo D'Amore wrote: > Hi all, > > given that solr 5.4 is finally released, is this what's more stable and > efficient version of

Re: faceting is unusable slow since upgrade to 5.3.0

Hi all, given that solr 5.4 is finally released, is this what's more stable and efficient version of solrcloud ? I have a website which receives many search requests. It serve normally about 2000 concurrent requests, but sometime there are peak from 4000 to 1 requests in few seconds. On

Re: faceting is unusable slow since upgrade to 5.3.0

Sorry for the delay. I had an ugly flu. SOLR-7730 seems to work fine. Using docValues with Solr 5.4.0-2015-09-29_08-29-55 1705813 makes my faceted queries fast again. (90ms vs. 2ms) :-) Thanks Uwe Am 27.09.2015 um 20:32 schrieb Mikhail Khludnev: On Sun, Sep 27, 2015 at 2:00 PM, Uwe

Re: faceting is unusable slow since upgrade to 5.3.0

Uwe, it's good to know! I mean that you've recovered. Take care! On Thu, Oct 8, 2015 at 1:24 PM, Uwe Reh wrote: > Sorry for the delay. I had an ugly flu. > > SOLR-7730 seems to work fine. Using docValues with Solr > 5.4.0-2015-09-29_08-29-55 1705813 makes my faceted

Re: faceting is unusable slow since upgrade to 5.3.0

On Sun, 2015-09-27 at 14:47 +0200, Uwe Reh wrote: > Like Walter Underwood wrote, in technical sense faceting on authors > isn't a good idea. In a technical sense, there is no good or bad about faceting on high-cardinality fields in Solr. The faceting code is fairly efficient (modulo the newly

Re: faceting is unusable slow since upgrade to 5.3.0

Hi Mikhail, is this, what you've requested? lookups: 34084 hits: 34067 hitratio: 1 inserts: 34 evictions: 0 ... item_author_facet: {field=author_facet,memSize=104189615,tindexSize=789195,time=16901,phase1=16534,nTerms=3989851,bigTerms=0,termInstances=16214154,uses=4065} item_topic_facet:

Re: faceting is unusable slow since upgrade to 5.3.0

On Sun, Sep 27, 2015 at 2:00 PM, Uwe Reh wrote: > When 5.4 with SOLR-7730 will be released, I will start to use docValues. > Going this way, seems more straight forward to me. Sure. Giving your answers docValues facets has a really good chance to perform in your

Re: faceting is unusable slow since upgrade to 5.3.0

Hi Mikhail, thanks for the hint, and "no" it wasn't obvious for me. :-) But I think, for us it's better to remain at 4.10.3 and observe the evolution of SOLR-8096. When 5.4 with SOLR-7730 will be released, I will start to use docValues. Going this way, seems more straight forward to me. Uwe

Re: faceting is unusable slow since upgrade to 5.3.0

Uwe, Would you mind to provide a few details about your case? I wonder about number of bigterms and other stats as well at 'author' field (ant other most expensive facets). It looks like log rows: Sep 13, 2011 2:51:53 PM org.apache.solr.request.UnInvertedField uninvert INFO: UnInverted

Re: faceting is unusable slow since upgrade to 5.3.0

Uwe, As a workaround, can you add facet.threads=Ncores to count fields in parallel? Also, setting fcs method for single value fields runs per segment faceting in parallel. Of course, fields which has small number of terms are beneficial from enum method. Excuse me if it's obvious.

Re: faceting is unusable slow since upgrade to 5.3.0

Clear ! Now I understand the current situation. Hope the issue will be fixed soon and the conference is recorded, good luck! Cheers 2015-09-25 15:22 GMT+01:00 Yonik Seeley : > On Fri, Sep 25, 2015 at 5:07 AM, Alessandro Benedetti > wrote: > >

Re: faceting is unusable slow since upgrade to 5.3.0

There is an undocumented "method" parameter - I need to enable that to > allow switching between the docvalues approach and the UnInvertedField > approach. > Only to clarify, please correct me Yonik if my understanding is wrong or outdated : To calculate facets, without going into the

Re: faceting is unusable slow since upgrade to 5.3.0

Am 25.09.2015 um 05:16 schrieb Yonik Seeley: I did some performance benchmarks and opened an issue. It's bad. https://issues.apache.org/jira/browse/SOLR-8096 Hi Yonik, thanks a lot for your investigation. Using the JSON Facet API is fast and seems to be a usable workaround for new

Re: faceting is unusable slow since upgrade to 5.3.0

On Fri, Sep 25, 2015 at 6:33 AM, Uwe Reh wrote: > Am 25.09.2015 um 05:16 schrieb Yonik Seeley: >> >> I did some performance benchmarks and opened an issue. It's bad. >> https://issues.apache.org/jira/browse/SOLR-8096 > > > Hi Yonik, > thanks a lot for your

Re: faceting is unusable slow since upgrade to 5.3.0

On Fri, Sep 25, 2015 at 5:07 AM, Alessandro Benedetti wrote: >There is an undocumented "method" parameter - I need to enable that to > >> allow switching between the docvalues approach and the UnInvertedField >> approach. >> > > Only to clarify, please correct me

Re: faceting is unusable slow since upgrade to 5.3.0

Am 23.09.2015 um 10:02 schrieb Mikhail Khludnev: ... Accelerating non-DV facets is not so clear so far. Please show profiler snapshot for non-DV facets if you wish to go this way. Hi, attached is a visualvm profile to several times a simplified query (just one facet):

Re: faceting is unusable slow since upgrade to 5.3.0

Uwe Unfortunately fieldValueCache was dropped there https://github.com/apache/lucene-solr/commit/fca4c22da81447867533fb28c0f06150cdc2eb9d#diff-5ac9dc7b128b4dd99b764060759222b2R428 However, I see that it's still available in new JSON facets (thus, you need to amend your app). Otherwise, you can

Re: faceting is unusable slow since upgrade to 5.3.0

Am 22.09.2015 um 18:10 schrieb Walter Underwood: Faceting on an author field is almost always a bad idea. Or at least a slow, expensive idea. Hi Wunder, n a technical context, the 'author'-facet may be suboptimal. In our businesses (library services) it's a core feature. Yes the facet is

Re: faceting is unusable slow since upgrade to 5.3.0

On Mon, Sep 21, 2015 at 8:09 AM, Uwe Reh wrote: > our bibliographic index (~20M entries) runs fine with Solr 4.10.3 > With Solr 5.3 faceted searching is constantly incredibly slow (~ 20 seconds) [...] > > The 'fieldValueCache' seems to be unused (no inserts nor

Re: faceting is unusable slow since upgrade to 5.3.0

Yonik, I am really excited about the Json faceting module. I find it really interesting. Is there any pros/cons in using them, or it's definitely the "approach of the future" ? I saw your benchmarks and seems impressive. I have not read all the topic in details, just briefly, but is Json faceting

Re: faceting is unusable slow since upgrade to 5.3.0

Can we add it back with a parameter at least ? Bill Bell Sent from mobile > On Sep 24, 2015, at 8:58 AM, Yonik Seeley wrote: > >> On Mon, Sep 21, 2015 at 8:09 AM, Uwe Reh wrote: >> our bibliographic index (~20M entries) runs fine with Solr

Re: faceting is unusable slow since upgrade to 5.3.0

On Thu, Sep 24, 2015 at 9:58 AM, Yonik Seeley wrote: > Indeed. Use of the fieldValueCache (UnInvertedField) was secretly > removed as part of LUCENE-5666, causing these performance regressions. I did some performance benchmarks and opened an issue. It's bad.

Re: faceting is unusable slow since upgrade to 5.3.0

On Thu, Sep 24, 2015 at 10:16 AM, Alessandro Benedetti wrote: > Yonik, I am really excited about the Json faceting module. > I find it really interesting. > Is there any pros/cons in using them, or it's definitely the "approach of > the future" ? Thanks! The cons to

Re: faceting is unusable slow since upgrade to 5.3.0

Well done Mikhail, curious to see the performance! Apart the disk usage ( of course building docValues will cost more space), taking in consideration the Field cardinality, in the past when the field cardinality was low ( few unique values in the field), the enum approach was suggested ( so

Re: faceting is unusable slow since upgrade to 5.3.0

Uwe, I'm sorry for confusion https://issues.apache.org/jira/browse/SOLR-7730 goes in 5.4 only. Hence, to get fast DV facets you need to apply patch (it's pretty small). Accelerating non-DV facets is not so clear so far. Please show profiler snapshot for non-DV facets if you wish to go this way.

Re: faceting is unusable slow since upgrade to 5.3.0

Despite docvalues provide NRT faceting with a great performance (since 5.4), enum method is still really important for edge cases (many docs, small num of terms). Also, Solr's UnIvertedField had a really smart BigTerms strategy, when fattest terms were counted by enum and remaining ones with fc.

Re: faceting is unusable slow since upgrade to 5.3.0

Faceting on an author field is almost always a bad idea. Or at least a slow, expensive idea. Faceting makes big in-memory lists. More values, bigger lists. An author field usually has many, many values, so you will need a lot of memory. wunder Walter Underwood wun...@wunderwood.org

Re: faceting is unusable slow since upgrade to 5.3.0

Am 22.09.2015 um 02:12 schrieb Joel Bernstein: Have you looked at your Solr instance with a cpu profiler like YourKit? It would be useful to see the hotspots which should be really obvious with 20 second response times. No, until now I have done no profiling. I thought the unused

Re: faceting is unusable slow since upgrade to 5.3.0

The exact version as shown by the UI is: - solr-impl 5.3.0 1696229 - noble - 2015-08-17 17:10:43 - lucene-impl 5.3.0 1696229 - noble - 2015-08-17 16:59:03 Unfortunately my skills in debugging are limited. So I'm not sure about a 'deeper caller stack'. Did you mean the attached snapshot from

Re: faceting is unusable slow since upgrade to 5.3.0

It's quite strange https://issues.apache.org/jira/browse/SOLR-7730 significantly optimized DV facets at 5.3.0 exactly by avoiding FileInfo merge. Would you mind to provide deeper caller stack for org.apache.lucene.index.FileInfos.MultibleFields.getMergedFieldInfos()? Or a time spend in

Re: faceting is unusable slow since upgrade to 5.3.0

here is my try to detect with VirtualVM some hot spots with VirtualVM. Enviroment: A newly started node with ~15 times the query: http://yxz/solr/hebis/select/?q=darwin=true=1=30=material_access=department_3=rvk_facet=author_facet=material_brief=language==count=all=true Ordered by self time

Re: faceting is unusable slow since upgrade to 5.3.0 (missing attachment)

virtualvm_snapshot_solr5.3_facetting.csv Description: MS-Excel spreadsheet

Re: faceting is unusable slow since upgrade to 5.3.0

Can you also try testing with one facet at a time and see if we hit a particular facet that is slow? Joel Bernstein http://joelsolr.blogspot.com/ On Tue, Sep 22, 2015 at 9:36 AM, Uwe Reh wrote: > The exact version as shown by the UI is: > - solr-impl 5.3.0

Re: faceting is unusable slow since upgrade to 5.3.0

Am 21.09.2015 um 15:16 schrieb Shalin Shekhar Mangar: Can you post your complete facet request as well as the schema definition of the field on which you are faceting? Query:

Re: faceting is unusable slow since upgrade to 5.3.0

Can you post your complete facet request as well as the schema definition of the field on which you are faceting? On Mon, Sep 21, 2015 at 5:39 PM, Uwe Reh wrote: > Hi, > > our bibliographic index (~20M entries) runs fine with Solr 4.10.3 > With Solr 5.3 faceted

Re: faceting is unusable slow since upgrade to 5.3.0

Have you looked at your Solr instance with a cpu profiler like YourKit? It would be useful to see the hotspots which should be really obvious with 20 second response times. Also are you running in distributed mode or on a single Solr instance? Joel Bernstein http://joelsolr.blogspot.com/ On