Re: Solr cloud facet query returning incorrect results

[Erick Erickson]

This should work as you expect, so the first thing I’d do 
is add =query and see the parsed query in both cases.

If that doesn’t show anything, please post the 
full debug response in both cases.

Best,
Erick

> On Dec 21, 2020, at 4:31 AM, Alok Bhandari  wrote:
> 
> Hello All ,
> 
> we are using Solr6.2 , in schema that we use we have an integer field. For
> a given query we want to know how many documents have duplicate value for
> the field , for an example how many documents have same doc_id=10.
> 
> So to find this information we fire a query to solr-cloud with following
> parameters
> 
> "q":"organization:abc",
>  "facet.limit":"10",
>  "facet.field":"doc_id",
>  "indent":"on",
>  "fl":"archive_id",
>  "facet.mincount":"2",
>"facet":"true",
> 
> 
> But in response we get that there are no documents having duplicate
> doc_id as in facet query response we are not getting any facet_counts
> , but if I change the query to "q":"organization:abc AND doc_id:10"
> then in response I can see that there are 3 docs with doc_id=10.
> 
> This behavior seems contrary to how facets behave , so wanted to know
> if there is any possible reason for this type of behavior.

Solr cloud facet query returning incorrect results

[Alok Bhandari]

Hello All ,

we are using Solr6.2 , in schema that we use we have an integer field. For
a given query we want to know how many documents have duplicate value for
the field , for an example how many documents have same doc_id=10.

So to find this information we fire a query to solr-cloud with following
parameters

"q":"organization:abc",
  "facet.limit":"10",
  "facet.field":"doc_id",
  "indent":"on",
  "fl":"archive_id",
  "facet.mincount":"2",
"facet":"true",


But in response we get that there are no documents having duplicate
doc_id as in facet query response we are not getting any facet_counts
, but if I change the query to "q":"organization:abc AND doc_id:10"
then in response I can see that there are 3 docs with doc_id=10.

This behavior seems contrary to how facets behave , so wanted to know
if there is any possible reason for this type of behavior.

Re: Facet Query performance

[Shawn Heisey]

On 7/8/2019 12:00 PM, Midas A wrote:

Number of Docs :50+ docs
Index Size: 300 GB
RAM: 256 GB
JVM: 32 GB


Half a million documents producing an index size of 300GB suggests 
*very* large documents.  That typically produces an index with fields 
that have very high cardinality, due to text tokenization.


Is Solr the only thing running on this machine, or does it have other 
memory-hungry software running on it?


The screenshot described at the following URL may provide more insight. 
It will be important to get the sort correct.  If the columns have been 
customized to show information other than the examples, it may need to 
be adjusted:


https://wiki.apache.org/solr/SolrPerformanceProblems#Asking_for_help_on_a_memory.2Fperformance_issue

Assuming that Solr is the only thing on the machine, then it means you 
have about 224 GB of memory available to cache your index data, which is 
at least 300GB.  Normally I would think being able to cache two thirds 
of the index should be enough for good performance, but it's always 
possible that there is something about your setup that means you don't 
have enough memory.


Are you sure that you need a 32GB heap?  Half a million documents should 
NOT require anywhere near that much heap.



Cardinality:
cat=44
rol=1005
ind=504
cl=2000


These cardinality values are VERY low.  If you are certain about those 
numbers, it is not likely that these fields are significant contributors 
to query time, either with or without docValues.  How did you obtain 
those numbers?


Those are not the only fields referenced in your query.  I also see these:

hemp
cEmp
pEmp
is_udis
id
is_resume
upt_date
country
exp
ctc
contents
currdesig
predesig
lng
ttl
kw_sql
kw_it


QTime:  2988 ms


Three seconds for a query with so many facets is something I would 
probably be pretty happy to get.



Our 35% queries takes more than 10 sec.


I have no idea what this sentence means.

Please suggest the ways to improve response time . Attached queries and 
schema.xml and solrconfig.xml


1. Is there any other ways to rewrite queries that improve our query 
performance .?


With the information available, the only suggestion I have currently is 
to replace "q=*" with "q=*:*" -- assuming that the intent is to match 
all documents with the main query.  According to what you attached 
(which I am very surprised to see -- attachments usually don't make it 
to the list), your df parameter is "ttl" ... a field that is heavily 
tokenized.  That means that the cardinality of the ttl field is probably 
VERY high, which would make the wildcard query VERY slow.


2. can we see the DocValues cache in plugin/ stats->cache-> section on 
solr UI panel ?


The admin UI only shows Solr caches.  If Lucene even has a docValues 
cache (and I do not know whether it does), it will not be available in 
Solr's statistics.  I am unaware of any cache in Solr for docValues. 
The entire point of docValues is to avoid the need to generate and cache 
large amounts of data, so I suspect there is not going to be anything 
available in this regard.


Thanks,
Shawn

Re: Facet Query performance

[Shawn Heisey]

On 7/8/2019 3:08 AM, Midas A wrote:

I have enabled docvalues on facet field but query is still taking time.

How i can improve the Query time .
docValues="true" multiValued="true" termVectors="true" /> 


*Query: *




There's very little information here -- only a single field definition 
and the query URL.  No information about how many documents, what sort 
of cardinality there is in the fields being used in the query, no 
information about memory and settings, etc.  You haven't even told us 
how long the query takes.


Your main query is a single * wildcard.  A wildcard query is typically 
quite slow.  If you are aiming for all documents, change that to q=*:* 
instead -- this is special syntax that the query parser understands, and 
is normally executed very quickly.


When a field has DocValues defined, it will automatically be used for 
field-based sorting, field-based facets, and field-based grouping. 
DocValues should not be relied on for queries, because indexed data is 
far faster for that usage.  Queries *can* be done with docValues, but it 
would be VERY slow.  Solr will avoid that usage if it can.


I'm reasonably certain that docValues will NOT be used for facet.query 
as long as the field is indexed.


You do have three-field based facets -- using the facet.field parameter. 
 If docValues was present on cat for ALL of the indexing that has 
happened, then they will work for that field, but you have not told us 
whether rol and pref have them defined.


You have a lot of faceting in this query.  That can cause things to be slow.

Thanks,
Shawn

Re: Facet Query performance

[Midas A]

Hi
How i can know whether DocValues are getting used or not ?
Please help me here .

On Mon, Jul 8, 2019 at 2:38 PM Midas A  wrote:

> Hi ,
>
> I have enabled docvalues on facet field but query is still taking time.
>
> How i can improve the Query time .
>  docValues="true" multiValued="true" termVectors="true" /> 
>
> *Query: *
> http://X.X.X.X:
> /solr/search/select?df=ttl=0=true=id,upt=1=true=1=OR=NOT+hemp:(%22xgidx29760%22+%22xmwxmonster%22+%22xmwxmonsterindia%22+%22xmwxcom%22+%22xswxmonster+com%22+%22xswxmonster%22+%22xswxmonsterindia+com%22+%22xswxmonsterindia%22)=NOT+cEmp:(%
> 22nomster.com%22+OR+%22utyu%22)=NOT+pEmp:(%22nomster.com
> %22+OR+%22utyu%22)=ind:(5)=NOT+is_udis:2=NOT+id:(92197+OR+240613+OR+249717+OR+1007148+OR+2500513+OR+2534675+OR+2813498+OR+9401682)=true=0=is_resume:0^-1000=upt_date:[*+TO+NOW/DAY-36MONTHS]^2=upt_date:[NOW/DAY-36MONTHS+TO+NOW/DAY-24MONTHS]^3=upt_date:[NOW/DAY-24MONTHS+TO+NOW/DAY-12MONTHS]^4=upt_date:[NOW/DAY-12MONTHS+TO+NOW/DAY-9MONTHS]^5=upt_date:[NOW/DAY-9MONTHS+TO+NOW/DAY-6MONTHS]^10=upt_date:[NOW/DAY-6MONTHS+TO+NOW/DAY-3MONTHS]^15=upt_date:[NOW/DAY-3MONTHS+TO+*]^20=NOT+country:isoin^-10=exp:[+10+TO+11+]=exp:[+11+TO+13+]=exp:[+13+TO+15+]=exp:[+15+TO+17+]=exp:[+17+TO+20+]=exp:[+20+TO+25+]=exp:[+25+TO+109+]=ctc:[+100+TO+101+]=ctc:[+101+TO+101.5+]=ctc:[+101.5+TO+102+]=ctc:[+102+TO+103+]=ctc:[+103+TO+104+]=ctc:[+104+TO+105+]=ctc:[+105+TO+107.5+]=ctc:[+107.5+TO+110+]=ctc:[+110+TO+115+]=ctc:[+115+TO+10100+]=0=contents^0.05+currdesig^1.5+predesig^1.5+lng^2+ttl+kw_skl+kw_it=1=false=ttl,kw_skl,kw_it,contents=json=1=0=ind=cat=rol=cl=pref=timing=/resumesearch=1=0=40=2=*=10=id==1=id=id=true=false
>
>

Facet Query performance

[Midas A]

Hi ,

I have enabled docvalues on facet field but query is still taking time.

How i can improve the Query time .
 

*Query: *
http://X.X.X.X:
/solr/search/select?df=ttl=0=true=id,upt=1=true=1=OR=NOT+hemp:(%22xgidx29760%22+%22xmwxmonster%22+%22xmwxmonsterindia%22+%22xmwxcom%22+%22xswxmonster+com%22+%22xswxmonster%22+%22xswxmonsterindia+com%22+%22xswxmonsterindia%22)=NOT+cEmp:(%
22nomster.com%22+OR+%22utyu%22)=NOT+pEmp:(%22nomster.com
%22+OR+%22utyu%22)=ind:(5)=NOT+is_udis:2=NOT+id:(92197+OR+240613+OR+249717+OR+1007148+OR+2500513+OR+2534675+OR+2813498+OR+9401682)=true=0=is_resume:0^-1000=upt_date:[*+TO+NOW/DAY-36MONTHS]^2=upt_date:[NOW/DAY-36MONTHS+TO+NOW/DAY-24MONTHS]^3=upt_date:[NOW/DAY-24MONTHS+TO+NOW/DAY-12MONTHS]^4=upt_date:[NOW/DAY-12MONTHS+TO+NOW/DAY-9MONTHS]^5=upt_date:[NOW/DAY-9MONTHS+TO+NOW/DAY-6MONTHS]^10=upt_date:[NOW/DAY-6MONTHS+TO+NOW/DAY-3MONTHS]^15=upt_date:[NOW/DAY-3MONTHS+TO+*]^20=NOT+country:isoin^-10=exp:[+10+TO+11+]=exp:[+11+TO+13+]=exp:[+13+TO+15+]=exp:[+15+TO+17+]=exp:[+17+TO+20+]=exp:[+20+TO+25+]=exp:[+25+TO+109+]=ctc:[+100+TO+101+]=ctc:[+101+TO+101.5+]=ctc:[+101.5+TO+102+]=ctc:[+102+TO+103+]=ctc:[+103+TO+104+]=ctc:[+104+TO+105+]=ctc:[+105+TO+107.5+]=ctc:[+107.5+TO+110+]=ctc:[+110+TO+115+]=ctc:[+115+TO+10100+]=0=contents^0.05+currdesig^1.5+predesig^1.5+lng^2+ttl+kw_skl+kw_it=1=false=ttl,kw_skl,kw_it,contents=json=1=0=ind=cat=rol=cl=pref=timing=/resumesearch=1=0=40=2=*=10=id==1=id=id=true=false

Re: Intermittent error 401 with JSON Facet query to retrieve count all collections

[Zheng Lin Edwin Yeo]

Hi Jason,

Yes. I am using the latest Solr 8.1.1.

The query which I'm using is the JSON Facet query which I faced the error
initially.

Regards,
Edwin

On Tue, 4 Jun 2019 at 20:15, Jason Gerlowski  wrote:

> Hi Edwin,
>
> Thanks for the additional datapoint.  It seemed to work for me, but we
> don't really understand the problem yet, so maybe it's not a solid
> work around like I'd hoped.  I'm curious to hear whether it works for
> Colvin.
>
> To double check though: forwardCredentials is only supported in Solr >
> 8.0.  You're using an 8.x version, right?
>
> Jason
>
> On Tue, Jun 4, 2019 at 2:45 AM Zheng Lin Edwin Yeo 
> wrote:
> >
> > Hi Jason,
> >
> > Thanks for your reply.
> >
> > I have tried to add the "forwardCredentials": true in the security.json,
> > but I still get the same error.
> >
> > Regards,
> > Edwin
> >
> > On Mon, 3 Jun 2019 at 22:19, Colvin Cowie 
> > wrote:
> >
> > > Hi, thanks I'll give that a go when I get a chance.
> > >
> > > I was trying to reply to an older thread (
> > >
> > >
> http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201904.mbox/%3CCAF2DzVXeVZqnixnkbzw0La1ui5N5-RG9PwfMBHG9vmkfBSMzJA%40mail.gmail.com%3E
> > > ),
> > > which I don't have in my mailbox, so obviously didn't reply to the
> right
> > > address to get my response threaded so mine has appeared on its own.
> Oops.
> > >
> > > A JIRA issue was raised on that thread
> > > https://issues.apache.org/jira/browse/SOLR-13421 but it's not had any
> > > attention.
> > >
> > >
> > > On Mon, 3 Jun 2019 at 14:46, Jason Gerlowski 
> > > wrote:
> > >
> > > > Hi Colvin,
> > > >
> > > > We're still taking a look at fixing the bug, but as a workaround in
> > > > the meantime, you can look into adding a "forwardCredentials":true
> > > > property under the "authentication" section of security.json.  That
> > > > seems to fix the issue in my reproduction at least.
> > > >
> > > > e.g.
> > > >
> > > > {
> > > > "authentication": {
> > > > "blockUnknown": true,
> > > > "class": "solr.BasicAuthPlugin",
> > > > "credentials": {
> > > > "solradmin": ""
> > > > },
> > > > "forwardCredentials": true
> > > > },
> > > > ...
> > > > }
> > > >
> > > > Jason
> > > >
> > > > On Mon, Jun 3, 2019 at 9:31 AM Jason Gerlowski <
> gerlowsk...@gmail.com>
> > > > wrote:
> > > > >
> > > > > One last note: as far as I can tell, nothing about this issue is
> > > > > specific to JSON Faceting or the JSON request API.  It can be
> > > > > triggered just as easily with "/select?q=*:*".
> > > > >
> > > > > The bug created for this is: SOLR-13510
> > > > >
> > > > > On Mon, Jun 3, 2019 at 9:17 AM Jason Gerlowski <
> gerlowsk...@gmail.com>
> > > > wrote:
> > > > > >
> > > > > > I'm also able to reproduce this bug on master.  A few more notes
> > > about
> > > > > > the bad behavior:
> > > > > >
> > > > > > - the behavior occurs regardless of the specific permissions
> > > > > > configured in security.json.  (i.e. whether the top permission is
> > > > > > "all", or "security-edit", or there are no permissions at all.)
> > > > > > - I tried looking for a pattern in which requests saw the 401s,
> but
> > > > > > didn't have any luck.  The 401 occurs when talking to the whole
> > > > > > collection or targeting individual cores directly.  It occurs
> when
> > > > > > curl hits a host containing a replica for the collection in
> question,
> > > > > > and when it doesnt. etc.  This distinguishes it from SOLR-13472,
> > > which
> > > > > > seems more specific to collection structure/layout.
> > > > > >
> > > > > > I'll create a bug for this in JIRA.
> > > > > >
> > > > > > On Sun, Jun 2, 2019 at 9:53 AM Colvin Cowie <
> > > > colvin.cowie@gmail.com> wrote:
> >

Re: Intermittent error 401 with JSON Facet query to retrieve count all collections

[Jason Gerlowski]

Hi Edwin,

Thanks for the additional datapoint.  It seemed to work for me, but we
don't really understand the problem yet, so maybe it's not a solid
work around like I'd hoped.  I'm curious to hear whether it works for
Colvin.

To double check though: forwardCredentials is only supported in Solr >
8.0.  You're using an 8.x version, right?

Jason

On Tue, Jun 4, 2019 at 2:45 AM Zheng Lin Edwin Yeo  wrote:
>
> Hi Jason,
>
> Thanks for your reply.
>
> I have tried to add the "forwardCredentials": true in the security.json,
> but I still get the same error.
>
> Regards,
> Edwin
>
> On Mon, 3 Jun 2019 at 22:19, Colvin Cowie 
> wrote:
>
> > Hi, thanks I'll give that a go when I get a chance.
> >
> > I was trying to reply to an older thread (
> >
> > http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201904.mbox/%3CCAF2DzVXeVZqnixnkbzw0La1ui5N5-RG9PwfMBHG9vmkfBSMzJA%40mail.gmail.com%3E
> > ),
> > which I don't have in my mailbox, so obviously didn't reply to the right
> > address to get my response threaded so mine has appeared on its own. Oops.
> >
> > A JIRA issue was raised on that thread
> > https://issues.apache.org/jira/browse/SOLR-13421 but it's not had any
> > attention.
> >
> >
> > On Mon, 3 Jun 2019 at 14:46, Jason Gerlowski 
> > wrote:
> >
> > > Hi Colvin,
> > >
> > > We're still taking a look at fixing the bug, but as a workaround in
> > > the meantime, you can look into adding a "forwardCredentials":true
> > > property under the "authentication" section of security.json.  That
> > > seems to fix the issue in my reproduction at least.
> > >
> > > e.g.
> > >
> > > {
> > > "authentication": {
> > > "blockUnknown": true,
> > > "class": "solr.BasicAuthPlugin",
> > > "credentials": {
> > > "solradmin": ""
> > > },
> > > "forwardCredentials": true
> > > },
> > > ...
> > > }
> > >
> > > Jason
> > >
> > > On Mon, Jun 3, 2019 at 9:31 AM Jason Gerlowski 
> > > wrote:
> > > >
> > > > One last note: as far as I can tell, nothing about this issue is
> > > > specific to JSON Faceting or the JSON request API.  It can be
> > > > triggered just as easily with "/select?q=*:*".
> > > >
> > > > The bug created for this is: SOLR-13510
> > > >
> > > > On Mon, Jun 3, 2019 at 9:17 AM Jason Gerlowski 
> > > wrote:
> > > > >
> > > > > I'm also able to reproduce this bug on master.  A few more notes
> > about
> > > > > the bad behavior:
> > > > >
> > > > > - the behavior occurs regardless of the specific permissions
> > > > > configured in security.json.  (i.e. whether the top permission is
> > > > > "all", or "security-edit", or there are no permissions at all.)
> > > > > - I tried looking for a pattern in which requests saw the 401s, but
> > > > > didn't have any luck.  The 401 occurs when talking to the whole
> > > > > collection or targeting individual cores directly.  It occurs when
> > > > > curl hits a host containing a replica for the collection in question,
> > > > > and when it doesnt. etc.  This distinguishes it from SOLR-13472,
> > which
> > > > > seems more specific to collection structure/layout.
> > > > >
> > > > > I'll create a bug for this in JIRA.
> > > > >
> > > > > On Sun, Jun 2, 2019 at 9:53 AM Colvin Cowie <
> > > colvin.cowie@gmail.com> wrote:
> > > > > >
> > > > > > Hello. I encountered this issue too and wrote this up before I
> > found
> > > this
> > > > > > thread, but I thought I might as well post it still, if it helps...
> > > > > >
> > > > > > Currently I'm trying to move our product on to Solr 8.1.1. We are
> > > currently
> > > > > > using 6.6.6, so things have definitely moved on.
> > > > > >
> > > > > > We use the BasicAuthPlugin + RuleBasedAuthorizationPlugin to lock
> > > down Solr
> > > > > > (and we also secure our zookeeper). Here's an example for solradmin
> > > as the
> > > > > > user and password
> > > > > >
> > > > > > {
> > > > > > "authentication": {
> > > > > > "blockUnknown": true,
> > > > > > "class": "solr.BasicAuthPlugin",
> > > > > > "credentials": {
> > > > > > "solradmin":
> > > "PIWZwkGnEKxKnqUs3X08xmbmYBaYyAeP3FiKp7fmeHc=
> > > > > > Lnbp6bEbE7Ap8lXvQDKkUX2Xw53QDgP6Ae8QRT0P5/A="
> > > > > > }
> > > > > > },
> > > > > > "authorization": {
> > > > > > "class": "solr.RuleBasedAuthorizationPlugin",
> > > > > > "permissions": [
> > > > > > {
> > > > > > "name": "all",
> > > > > > "role": "admin"
> > > > > > }
> > > > > > ],
> > > > > > "user-role": {
> > > > > > "solradmin": "admin"
> > > > > > }
> > > > > > }
> > > > > > }
> > > > > >
> > > > > >
> > > > > > On Solr 8.1.1, using our previously working security.json, running
> > > queries
> > > > > > (through the admin UI currently) I non-deterministically get 401
> > > responses
> > > > > > on queries when a collection has more than 1 shard. Increasing the
> > > number
> > > > > > of 

Re: Intermittent error 401 with JSON Facet query to retrieve count all collections

[Zheng Lin Edwin Yeo]

Hi Jason,

Thanks for your reply.

I have tried to add the "forwardCredentials": true in the security.json,
but I still get the same error.

Regards,
Edwin

On Mon, 3 Jun 2019 at 22:19, Colvin Cowie 
wrote:

> Hi, thanks I'll give that a go when I get a chance.
>
> I was trying to reply to an older thread (
>
> http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201904.mbox/%3CCAF2DzVXeVZqnixnkbzw0La1ui5N5-RG9PwfMBHG9vmkfBSMzJA%40mail.gmail.com%3E
> ),
> which I don't have in my mailbox, so obviously didn't reply to the right
> address to get my response threaded so mine has appeared on its own. Oops.
>
> A JIRA issue was raised on that thread
> https://issues.apache.org/jira/browse/SOLR-13421 but it's not had any
> attention.
>
>
> On Mon, 3 Jun 2019 at 14:46, Jason Gerlowski 
> wrote:
>
> > Hi Colvin,
> >
> > We're still taking a look at fixing the bug, but as a workaround in
> > the meantime, you can look into adding a "forwardCredentials":true
> > property under the "authentication" section of security.json.  That
> > seems to fix the issue in my reproduction at least.
> >
> > e.g.
> >
> > {
> > "authentication": {
> > "blockUnknown": true,
> > "class": "solr.BasicAuthPlugin",
> > "credentials": {
> > "solradmin": ""
> > },
> > "forwardCredentials": true
> > },
> > ...
> > }
> >
> > Jason
> >
> > On Mon, Jun 3, 2019 at 9:31 AM Jason Gerlowski 
> > wrote:
> > >
> > > One last note: as far as I can tell, nothing about this issue is
> > > specific to JSON Faceting or the JSON request API.  It can be
> > > triggered just as easily with "/select?q=*:*".
> > >
> > > The bug created for this is: SOLR-13510
> > >
> > > On Mon, Jun 3, 2019 at 9:17 AM Jason Gerlowski 
> > wrote:
> > > >
> > > > I'm also able to reproduce this bug on master.  A few more notes
> about
> > > > the bad behavior:
> > > >
> > > > - the behavior occurs regardless of the specific permissions
> > > > configured in security.json.  (i.e. whether the top permission is
> > > > "all", or "security-edit", or there are no permissions at all.)
> > > > - I tried looking for a pattern in which requests saw the 401s, but
> > > > didn't have any luck.  The 401 occurs when talking to the whole
> > > > collection or targeting individual cores directly.  It occurs when
> > > > curl hits a host containing a replica for the collection in question,
> > > > and when it doesnt. etc.  This distinguishes it from SOLR-13472,
> which
> > > > seems more specific to collection structure/layout.
> > > >
> > > > I'll create a bug for this in JIRA.
> > > >
> > > > On Sun, Jun 2, 2019 at 9:53 AM Colvin Cowie <
> > colvin.cowie@gmail.com> wrote:
> > > > >
> > > > > Hello. I encountered this issue too and wrote this up before I
> found
> > this
> > > > > thread, but I thought I might as well post it still, if it helps...
> > > > >
> > > > > Currently I'm trying to move our product on to Solr 8.1.1. We are
> > currently
> > > > > using 6.6.6, so things have definitely moved on.
> > > > >
> > > > > We use the BasicAuthPlugin + RuleBasedAuthorizationPlugin to lock
> > down Solr
> > > > > (and we also secure our zookeeper). Here's an example for solradmin
> > as the
> > > > > user and password
> > > > >
> > > > > {
> > > > > "authentication": {
> > > > > "blockUnknown": true,
> > > > > "class": "solr.BasicAuthPlugin",
> > > > > "credentials": {
> > > > > "solradmin":
> > "PIWZwkGnEKxKnqUs3X08xmbmYBaYyAeP3FiKp7fmeHc=
> > > > > Lnbp6bEbE7Ap8lXvQDKkUX2Xw53QDgP6Ae8QRT0P5/A="
> > > > > }
> > > > > },
> > > > > "authorization": {
> > > > > "class": "solr.RuleBasedAuthorizationPlugin",
> > > > > "permissions": [
> > > > > {
> > > > > "name": "all",
> > > > > "role": "admin"
> > > > > }
> > > > > ],
> > > > > "user-role": {
> > > > > "solradmin": "admin"
> > > > > }
> > > > > }
> > > > > }
> > > > >
> > > > >
> > > > > On Solr 8.1.1, using our previously working security.json, running
> > queries
> > > > > (through the admin UI currently) I non-deterministically get 401
> > responses
> > > > > on queries when a collection has more than 1 shard. Increasing the
> > number
> > > > > of shards in the collection makes the errors more likely.
> > > > >
> > > > > {
> > > > >   "responseHeader":{
> > > > > "zkConnected":true,
> > > > > "status":401,
> > > > > "QTime":30,
> > > > > "params":{
> > > > >   "q":"*:*",
> > > > >   "_":"1559474550365"}},
> > > > >   "error":{
> > > > > "metadata":[
> > > > >
> > > > >
> >
> "error-class","org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteSolrException",
> > > > >
> > > > >
> >
> "root-error-class","org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteSolrException"],
> > > > > "msg":"Error from server at null: Expected mime type
> > > > > 

Re: Intermittent error 401 with JSON Facet query to retrieve count all collections

[Colvin Cowie]

Hi, thanks I'll give that a go when I get a chance.

I was trying to reply to an older thread (
http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201904.mbox/%3CCAF2DzVXeVZqnixnkbzw0La1ui5N5-RG9PwfMBHG9vmkfBSMzJA%40mail.gmail.com%3E),
which I don't have in my mailbox, so obviously didn't reply to the right
address to get my response threaded so mine has appeared on its own. Oops.

A JIRA issue was raised on that thread
https://issues.apache.org/jira/browse/SOLR-13421 but it's not had any
attention.


On Mon, 3 Jun 2019 at 14:46, Jason Gerlowski  wrote:

> Hi Colvin,
>
> We're still taking a look at fixing the bug, but as a workaround in
> the meantime, you can look into adding a "forwardCredentials":true
> property under the "authentication" section of security.json.  That
> seems to fix the issue in my reproduction at least.
>
> e.g.
>
> {
> "authentication": {
> "blockUnknown": true,
> "class": "solr.BasicAuthPlugin",
> "credentials": {
> "solradmin": ""
> },
> "forwardCredentials": true
> },
> ...
> }
>
> Jason
>
> On Mon, Jun 3, 2019 at 9:31 AM Jason Gerlowski 
> wrote:
> >
> > One last note: as far as I can tell, nothing about this issue is
> > specific to JSON Faceting or the JSON request API.  It can be
> > triggered just as easily with "/select?q=*:*".
> >
> > The bug created for this is: SOLR-13510
> >
> > On Mon, Jun 3, 2019 at 9:17 AM Jason Gerlowski 
> wrote:
> > >
> > > I'm also able to reproduce this bug on master.  A few more notes about
> > > the bad behavior:
> > >
> > > - the behavior occurs regardless of the specific permissions
> > > configured in security.json.  (i.e. whether the top permission is
> > > "all", or "security-edit", or there are no permissions at all.)
> > > - I tried looking for a pattern in which requests saw the 401s, but
> > > didn't have any luck.  The 401 occurs when talking to the whole
> > > collection or targeting individual cores directly.  It occurs when
> > > curl hits a host containing a replica for the collection in question,
> > > and when it doesnt. etc.  This distinguishes it from SOLR-13472, which
> > > seems more specific to collection structure/layout.
> > >
> > > I'll create a bug for this in JIRA.
> > >
> > > On Sun, Jun 2, 2019 at 9:53 AM Colvin Cowie <
> colvin.cowie@gmail.com> wrote:
> > > >
> > > > Hello. I encountered this issue too and wrote this up before I found
> this
> > > > thread, but I thought I might as well post it still, if it helps...
> > > >
> > > > Currently I'm trying to move our product on to Solr 8.1.1. We are
> currently
> > > > using 6.6.6, so things have definitely moved on.
> > > >
> > > > We use the BasicAuthPlugin + RuleBasedAuthorizationPlugin to lock
> down Solr
> > > > (and we also secure our zookeeper). Here's an example for solradmin
> as the
> > > > user and password
> > > >
> > > > {
> > > > "authentication": {
> > > > "blockUnknown": true,
> > > > "class": "solr.BasicAuthPlugin",
> > > > "credentials": {
> > > > "solradmin":
> "PIWZwkGnEKxKnqUs3X08xmbmYBaYyAeP3FiKp7fmeHc=
> > > > Lnbp6bEbE7Ap8lXvQDKkUX2Xw53QDgP6Ae8QRT0P5/A="
> > > > }
> > > > },
> > > > "authorization": {
> > > > "class": "solr.RuleBasedAuthorizationPlugin",
> > > > "permissions": [
> > > > {
> > > > "name": "all",
> > > > "role": "admin"
> > > > }
> > > > ],
> > > > "user-role": {
> > > > "solradmin": "admin"
> > > > }
> > > > }
> > > > }
> > > >
> > > >
> > > > On Solr 8.1.1, using our previously working security.json, running
> queries
> > > > (through the admin UI currently) I non-deterministically get 401
> responses
> > > > on queries when a collection has more than 1 shard. Increasing the
> number
> > > > of shards in the collection makes the errors more likely.
> > > >
> > > > {
> > > >   "responseHeader":{
> > > > "zkConnected":true,
> > > > "status":401,
> > > > "QTime":30,
> > > > "params":{
> > > >   "q":"*:*",
> > > >   "_":"1559474550365"}},
> > > >   "error":{
> > > > "metadata":[
> > > >
> > > >
> "error-class","org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteSolrException",
> > > >
> > > >
> "root-error-class","org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteSolrException"],
> > > > "msg":"Error from server at null: Expected mime type
> > > > application/octet-stream but got text/html. \n\n > > > http-equiv=\"Content-Type\"
> > > > content=\"text/html;charset=utf-8\"/>\nError 401 require
> > > > authentication\n\nHTTP ERROR
> 401\nProblem
> > > > accessing /solr/gettingstarted_shard4_replica_n6/select.
> Reason:\n
> > > >  require authentication\n\n\n",
> > > > "code":401}}
> > > >
> > > > The security stats indicate this is happening because the requests
> do not
> > > > have credentials with them, e.g.
> > > >
> 

Re: Intermittent error 401 with JSON Facet query to retrieve count all collections

[Jason Gerlowski]

Hi Colvin,

We're still taking a look at fixing the bug, but as a workaround in
the meantime, you can look into adding a "forwardCredentials":true
property under the "authentication" section of security.json.  That
seems to fix the issue in my reproduction at least.

e.g.

{
"authentication": {
"blockUnknown": true,
"class": "solr.BasicAuthPlugin",
"credentials": {
"solradmin": ""
},
"forwardCredentials": true
},
...
}

Jason

On Mon, Jun 3, 2019 at 9:31 AM Jason Gerlowski  wrote:
>
> One last note: as far as I can tell, nothing about this issue is
> specific to JSON Faceting or the JSON request API.  It can be
> triggered just as easily with "/select?q=*:*".
>
> The bug created for this is: SOLR-13510
>
> On Mon, Jun 3, 2019 at 9:17 AM Jason Gerlowski  wrote:
> >
> > I'm also able to reproduce this bug on master.  A few more notes about
> > the bad behavior:
> >
> > - the behavior occurs regardless of the specific permissions
> > configured in security.json.  (i.e. whether the top permission is
> > "all", or "security-edit", or there are no permissions at all.)
> > - I tried looking for a pattern in which requests saw the 401s, but
> > didn't have any luck.  The 401 occurs when talking to the whole
> > collection or targeting individual cores directly.  It occurs when
> > curl hits a host containing a replica for the collection in question,
> > and when it doesnt. etc.  This distinguishes it from SOLR-13472, which
> > seems more specific to collection structure/layout.
> >
> > I'll create a bug for this in JIRA.
> >
> > On Sun, Jun 2, 2019 at 9:53 AM Colvin Cowie  
> > wrote:
> > >
> > > Hello. I encountered this issue too and wrote this up before I found this
> > > thread, but I thought I might as well post it still, if it helps...
> > >
> > > Currently I'm trying to move our product on to Solr 8.1.1. We are 
> > > currently
> > > using 6.6.6, so things have definitely moved on.
> > >
> > > We use the BasicAuthPlugin + RuleBasedAuthorizationPlugin to lock down 
> > > Solr
> > > (and we also secure our zookeeper). Here's an example for solradmin as the
> > > user and password
> > >
> > > {
> > > "authentication": {
> > > "blockUnknown": true,
> > > "class": "solr.BasicAuthPlugin",
> > > "credentials": {
> > > "solradmin": "PIWZwkGnEKxKnqUs3X08xmbmYBaYyAeP3FiKp7fmeHc=
> > > Lnbp6bEbE7Ap8lXvQDKkUX2Xw53QDgP6Ae8QRT0P5/A="
> > > }
> > > },
> > > "authorization": {
> > > "class": "solr.RuleBasedAuthorizationPlugin",
> > > "permissions": [
> > > {
> > > "name": "all",
> > > "role": "admin"
> > > }
> > > ],
> > > "user-role": {
> > > "solradmin": "admin"
> > > }
> > > }
> > > }
> > >
> > >
> > > On Solr 8.1.1, using our previously working security.json, running queries
> > > (through the admin UI currently) I non-deterministically get 401 responses
> > > on queries when a collection has more than 1 shard. Increasing the number
> > > of shards in the collection makes the errors more likely.
> > >
> > > {
> > >   "responseHeader":{
> > > "zkConnected":true,
> > > "status":401,
> > > "QTime":30,
> > > "params":{
> > >   "q":"*:*",
> > >   "_":"1559474550365"}},
> > >   "error":{
> > > "metadata":[
> > >
> > > "error-class","org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteSolrException",
> > >
> > > "root-error-class","org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteSolrException"],
> > > "msg":"Error from server at null: Expected mime type
> > > application/octet-stream but got text/html. \n\n > > http-equiv=\"Content-Type\"
> > > content=\"text/html;charset=utf-8\"/>\nError 401 require
> > > authentication\n\nHTTP ERROR 401\nProblem
> > > accessing /solr/gettingstarted_shard4_replica_n6/select. Reason:\n
> > >  require authentication\n\n\n",
> > > "code":401}}
> > >
> > > The security stats indicate this is happening because the requests do not
> > > have credentials with them, e.g.
> > > http://localhost:8983/solr/#/gettingstarted_shard4_replica_n6/plugins?type=security=org.apache.solr.security.BasicAuthPlugin
> > >
> > >  org.apache.solr.security.BasicAuthPlugin
> > > class:
> > > org.apache.solr.security.BasicAuthPlugin
> > > description:
> > > Authentication Plugin org.apache.solr.security.BasicAuthPlugin
> > > stats
> > > SECURITY./authentication.authenticated:
> > > 182
> > > SECURITY./authentication.errors.count:
> > > 0
> > > SECURITY./authentication.failMissingCredentials:
> > > 58
> > > SECURITY./authentication.failWrongCredentials:
> > > 0
> > > SECURITY./authentication.passThrough:
> > > 0
> > > SECURITY./authentication.requestTimes.meanRate:
> > > 0.4183414110946125
> > > 

Re: Intermittent error 401 with JSON Facet query to retrieve count all collections

[Jason Gerlowski]

One last note: as far as I can tell, nothing about this issue is
specific to JSON Faceting or the JSON request API.  It can be
triggered just as easily with "/select?q=*:*".

The bug created for this is: SOLR-13510

On Mon, Jun 3, 2019 at 9:17 AM Jason Gerlowski  wrote:
>
> I'm also able to reproduce this bug on master.  A few more notes about
> the bad behavior:
>
> - the behavior occurs regardless of the specific permissions
> configured in security.json.  (i.e. whether the top permission is
> "all", or "security-edit", or there are no permissions at all.)
> - I tried looking for a pattern in which requests saw the 401s, but
> didn't have any luck.  The 401 occurs when talking to the whole
> collection or targeting individual cores directly.  It occurs when
> curl hits a host containing a replica for the collection in question,
> and when it doesnt. etc.  This distinguishes it from SOLR-13472, which
> seems more specific to collection structure/layout.
>
> I'll create a bug for this in JIRA.
>
> On Sun, Jun 2, 2019 at 9:53 AM Colvin Cowie  
> wrote:
> >
> > Hello. I encountered this issue too and wrote this up before I found this
> > thread, but I thought I might as well post it still, if it helps...
> >
> > Currently I'm trying to move our product on to Solr 8.1.1. We are currently
> > using 6.6.6, so things have definitely moved on.
> >
> > We use the BasicAuthPlugin + RuleBasedAuthorizationPlugin to lock down Solr
> > (and we also secure our zookeeper). Here's an example for solradmin as the
> > user and password
> >
> > {
> > "authentication": {
> > "blockUnknown": true,
> > "class": "solr.BasicAuthPlugin",
> > "credentials": {
> > "solradmin": "PIWZwkGnEKxKnqUs3X08xmbmYBaYyAeP3FiKp7fmeHc=
> > Lnbp6bEbE7Ap8lXvQDKkUX2Xw53QDgP6Ae8QRT0P5/A="
> > }
> > },
> > "authorization": {
> > "class": "solr.RuleBasedAuthorizationPlugin",
> > "permissions": [
> > {
> > "name": "all",
> > "role": "admin"
> > }
> > ],
> > "user-role": {
> > "solradmin": "admin"
> > }
> > }
> > }
> >
> >
> > On Solr 8.1.1, using our previously working security.json, running queries
> > (through the admin UI currently) I non-deterministically get 401 responses
> > on queries when a collection has more than 1 shard. Increasing the number
> > of shards in the collection makes the errors more likely.
> >
> > {
> >   "responseHeader":{
> > "zkConnected":true,
> > "status":401,
> > "QTime":30,
> > "params":{
> >   "q":"*:*",
> >   "_":"1559474550365"}},
> >   "error":{
> > "metadata":[
> >
> > "error-class","org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteSolrException",
> >
> > "root-error-class","org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteSolrException"],
> > "msg":"Error from server at null: Expected mime type
> > application/octet-stream but got text/html. \n\n > http-equiv=\"Content-Type\"
> > content=\"text/html;charset=utf-8\"/>\nError 401 require
> > authentication\n\nHTTP ERROR 401\nProblem
> > accessing /solr/gettingstarted_shard4_replica_n6/select. Reason:\n
> >  require authentication\n\n\n",
> > "code":401}}
> >
> > The security stats indicate this is happening because the requests do not
> > have credentials with them, e.g.
> > http://localhost:8983/solr/#/gettingstarted_shard4_replica_n6/plugins?type=security=org.apache.solr.security.BasicAuthPlugin
> >
> >  org.apache.solr.security.BasicAuthPlugin
> > class:
> > org.apache.solr.security.BasicAuthPlugin
> > description:
> > Authentication Plugin org.apache.solr.security.BasicAuthPlugin
> > stats
> > SECURITY./authentication.authenticated:
> > 182
> > SECURITY./authentication.errors.count:
> > 0
> > SECURITY./authentication.failMissingCredentials:
> > 58
> > SECURITY./authentication.failWrongCredentials:
> > 0
> > SECURITY./authentication.passThrough:
> > 0
> > SECURITY./authentication.requestTimes.meanRate:
> > 0.4183414110946125
> > SECURITY./authentication.requests:
> > 240
> > SECURITY./authentication.totalTime:
> > 117791100
> >
> > I assume that this is connected to the changes around
> > https://issues.apache.org/jira/browse/SOLR-7896 and
> > https://issues.apache.org/jira/browse/SOLR-13344 I've tested with Solr
> > 7.6.0 and it appears to be unaffected
> >
> > Repro steps:
> ># Extract solr 8.1.1.
> ># bin\solr start -e cloud
> > 1 node / [default port] / [default collection name] / 4 shards / 1
> > replica / [_default configuration]
> ># server\scripts\cloud-scripts\zkcli -zkhost localhost:9983 -cmd putfile
> > /security.json 
> >
> ># Execute repeated GETS to
> > http://localhost:8983/solr/gettingstarted/select?q=*%3A* - a lot of 

Re: Intermittent error 401 with JSON Facet query to retrieve count all collections

[Jason Gerlowski]

I'm also able to reproduce this bug on master.  A few more notes about
the bad behavior:

- the behavior occurs regardless of the specific permissions
configured in security.json.  (i.e. whether the top permission is
"all", or "security-edit", or there are no permissions at all.)
- I tried looking for a pattern in which requests saw the 401s, but
didn't have any luck.  The 401 occurs when talking to the whole
collection or targeting individual cores directly.  It occurs when
curl hits a host containing a replica for the collection in question,
and when it doesnt. etc.  This distinguishes it from SOLR-13472, which
seems more specific to collection structure/layout.

I'll create a bug for this in JIRA.

On Sun, Jun 2, 2019 at 9:53 AM Colvin Cowie  wrote:
>
> Hello. I encountered this issue too and wrote this up before I found this
> thread, but I thought I might as well post it still, if it helps...
>
> Currently I'm trying to move our product on to Solr 8.1.1. We are currently
> using 6.6.6, so things have definitely moved on.
>
> We use the BasicAuthPlugin + RuleBasedAuthorizationPlugin to lock down Solr
> (and we also secure our zookeeper). Here's an example for solradmin as the
> user and password
>
> {
> "authentication": {
> "blockUnknown": true,
> "class": "solr.BasicAuthPlugin",
> "credentials": {
> "solradmin": "PIWZwkGnEKxKnqUs3X08xmbmYBaYyAeP3FiKp7fmeHc=
> Lnbp6bEbE7Ap8lXvQDKkUX2Xw53QDgP6Ae8QRT0P5/A="
> }
> },
> "authorization": {
> "class": "solr.RuleBasedAuthorizationPlugin",
> "permissions": [
> {
> "name": "all",
> "role": "admin"
> }
> ],
> "user-role": {
> "solradmin": "admin"
> }
> }
> }
>
>
> On Solr 8.1.1, using our previously working security.json, running queries
> (through the admin UI currently) I non-deterministically get 401 responses
> on queries when a collection has more than 1 shard. Increasing the number
> of shards in the collection makes the errors more likely.
>
> {
>   "responseHeader":{
> "zkConnected":true,
> "status":401,
> "QTime":30,
> "params":{
>   "q":"*:*",
>   "_":"1559474550365"}},
>   "error":{
> "metadata":[
>
> "error-class","org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteSolrException",
>
> "root-error-class","org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteSolrException"],
> "msg":"Error from server at null: Expected mime type
> application/octet-stream but got text/html. \n\n http-equiv=\"Content-Type\"
> content=\"text/html;charset=utf-8\"/>\nError 401 require
> authentication\n\nHTTP ERROR 401\nProblem
> accessing /solr/gettingstarted_shard4_replica_n6/select. Reason:\n
>  require authentication\n\n\n",
> "code":401}}
>
> The security stats indicate this is happening because the requests do not
> have credentials with them, e.g.
> http://localhost:8983/solr/#/gettingstarted_shard4_replica_n6/plugins?type=security=org.apache.solr.security.BasicAuthPlugin
>
>  org.apache.solr.security.BasicAuthPlugin
> class:
> org.apache.solr.security.BasicAuthPlugin
> description:
> Authentication Plugin org.apache.solr.security.BasicAuthPlugin
> stats
> SECURITY./authentication.authenticated:
> 182
> SECURITY./authentication.errors.count:
> 0
> SECURITY./authentication.failMissingCredentials:
> 58
> SECURITY./authentication.failWrongCredentials:
> 0
> SECURITY./authentication.passThrough:
> 0
> SECURITY./authentication.requestTimes.meanRate:
> 0.4183414110946125
> SECURITY./authentication.requests:
> 240
> SECURITY./authentication.totalTime:
> 117791100
>
> I assume that this is connected to the changes around
> https://issues.apache.org/jira/browse/SOLR-7896 and
> https://issues.apache.org/jira/browse/SOLR-13344 I've tested with Solr
> 7.6.0 and it appears to be unaffected
>
> Repro steps:
># Extract solr 8.1.1.
># bin\solr start -e cloud
> 1 node / [default port] / [default collection name] / 4 shards / 1
> replica / [_default configuration]
># server\scripts\cloud-scripts\zkcli -zkhost localhost:9983 -cmd putfile
> /security.json 
>
># Execute repeated GETS to
> http://localhost:8983/solr/gettingstarted/select?q=*%3A* - a lot of them,
> but not all, will fail with 401s
>
>
> Also as a side note, because the authentication is now done through the
> form login rather than the browser basic auth, if you go directly to a non
> UI url (e.g. http://localhost:8983/solr/main_index/select?q=*%3A*) you have
> to authenticate to it using the browser's basic auth prompt. Which is
> slightly annoying since the query page in the Admin UI generates links to
> it for the queries you run, and you've already authenticated to get there.
> But it's not a 

Re: Intermittent error 401 with JSON Facet query to retrieve count all collections

[Colvin Cowie]

Hello. I encountered this issue too and wrote this up before I found this
thread, but I thought I might as well post it still, if it helps...

Currently I'm trying to move our product on to Solr 8.1.1. We are currently
using 6.6.6, so things have definitely moved on.

We use the BasicAuthPlugin + RuleBasedAuthorizationPlugin to lock down Solr
(and we also secure our zookeeper). Here's an example for solradmin as the
user and password

{
"authentication": {
"blockUnknown": true,
"class": "solr.BasicAuthPlugin",
"credentials": {
"solradmin": "PIWZwkGnEKxKnqUs3X08xmbmYBaYyAeP3FiKp7fmeHc=
Lnbp6bEbE7Ap8lXvQDKkUX2Xw53QDgP6Ae8QRT0P5/A="
}
},
"authorization": {
"class": "solr.RuleBasedAuthorizationPlugin",
"permissions": [
{
"name": "all",
"role": "admin"
}
],
"user-role": {
"solradmin": "admin"
}
}
}


On Solr 8.1.1, using our previously working security.json, running queries
(through the admin UI currently) I non-deterministically get 401 responses
on queries when a collection has more than 1 shard. Increasing the number
of shards in the collection makes the errors more likely.

{
  "responseHeader":{
"zkConnected":true,
"status":401,
"QTime":30,
"params":{
  "q":"*:*",
  "_":"1559474550365"}},
  "error":{
"metadata":[

"error-class","org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteSolrException",

"root-error-class","org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteSolrException"],
"msg":"Error from server at null: Expected mime type
application/octet-stream but got text/html. \n\n\nError 401 require
authentication\n\nHTTP ERROR 401\nProblem
accessing /solr/gettingstarted_shard4_replica_n6/select. Reason:\n
 require authentication\n\n\n",
"code":401}}

The security stats indicate this is happening because the requests do not
have credentials with them, e.g.
http://localhost:8983/solr/#/gettingstarted_shard4_replica_n6/plugins?type=security=org.apache.solr.security.BasicAuthPlugin

 org.apache.solr.security.BasicAuthPlugin
class:
org.apache.solr.security.BasicAuthPlugin
description:
Authentication Plugin org.apache.solr.security.BasicAuthPlugin
stats
SECURITY./authentication.authenticated:
182
SECURITY./authentication.errors.count:
0
SECURITY./authentication.failMissingCredentials:
58
SECURITY./authentication.failWrongCredentials:
0
SECURITY./authentication.passThrough:
0
SECURITY./authentication.requestTimes.meanRate:
0.4183414110946125
SECURITY./authentication.requests:
240
SECURITY./authentication.totalTime:
117791100

I assume that this is connected to the changes around
https://issues.apache.org/jira/browse/SOLR-7896 and
https://issues.apache.org/jira/browse/SOLR-13344 I've tested with Solr
7.6.0 and it appears to be unaffected

Repro steps:
   # Extract solr 8.1.1.
   # bin\solr start -e cloud
1 node / [default port] / [default collection name] / 4 shards / 1
replica / [_default configuration]
   # server\scripts\cloud-scripts\zkcli -zkhost localhost:9983 -cmd putfile
/security.json 

   # Execute repeated GETS to
http://localhost:8983/solr/gettingstarted/select?q=*%3A* - a lot of them,
but not all, will fail with 401s


Also as a side note, because the authentication is now done through the
form login rather than the browser basic auth, if you go directly to a non
UI url (e.g. http://localhost:8983/solr/main_index/select?q=*%3A*) you have
to authenticate to it using the browser's basic auth prompt. Which is
slightly annoying since the query page in the Admin UI generates links to
it for the queries you run, and you've already authenticated to get there.
But it's not a massive burden or anything... I guess I just preferred
having the browser BA prompt.

Thanks

Re: Intermittent error 401 with JSON Facet query to retrieve count all collections

[Erick Erickson]

 8.0.0, this query can only work occasionally.
>>>>> Most of the time, we will get the following error of 'Error 401 require
>>>>> authentication':
>>>>> 
>>>>> {
>>>>> "responseHeader":
>>>>> { "zkConnected":true, "status":401, "QTime":11}
>>>>> 
>>>>> ,
>>>>> "error":{
>>>>>   "metadata":[
>>>>> 
>>>>> 
>>> "error-class","org.apache.solr.client.solrj.impl.Http2SolrClient$RemoteSolrException",
>>>>> 
>>>>> 
>>> "root-error-class","org.apache.solr.client.solrj.impl.Http2SolrClient$RemoteSolrException"],
>>>>>   "msg":"Error from server at null: Expected mime type
>>>>> application/octet-stream but got text/html. \n\n>>>> http-equiv=\"Content-Type\"
>>>>> content=\"text/html;charset=utf-8\"/>\nError 401 require
>>>>> authentication\n\nHTTP ERROR
>>> 401\nProblem
>>>>> accessing /solr/collection6/select. Reason:\nrequire
>>>>> authentication\n\n\n",
>>>>>   "code":401}}
>>>>> 
>>>>> This issue does not occur in Solr 7.6 and Solr 7.7, even though I have
>>> set
>>>>> up the same authentication for all the versions.
>>>>> 
>>>>> What could be the issue that causes this?
>>>>> 
>>>>> 
>>>>> Below is the format of my security.json:
>>>>> 
>>>>> {
>>>>> "authentication":
>>>>> 
>>>>> {"blockUnknown": true,"class":"solr.BasicAuthPlugin",
>>>>> "credentials":
>>>>> {"user1":"hyHXXuJSqcZdNgdSTGUvrQZRpqrYFUQ2ffmlWQ4GUTk=
>>>>> E0w3/2FD+rlxulbPm2G7i9HZqT+2gMBzcyJCcGcMWwA="}
>>>>> 
>>>>> },
>>>>> "authorization":
>>>>> 
>>>>> {"class":"solr.RuleBasedAuthorizationPlugin","user-role":
>>>>> {"user1":"admin"}
>>>>> 
>>>>> ,
>>>>>  "permissions":[
>>>>> {"name":"security-edit",   "role":"admin"}
>>>>> 
>>>>> ]
>>>>> }}
>>>>> 
>>>>> 
>>>>> Regards,
>>>>> Edwin
>>>>> 
>>>>> 
>>>>> On Mon, 22 Apr 2019 at 09:37, Zheng Lin Edwin Yeo <
>>> edwinye...@gmail.com>
>>>>> wrote:
>>>>> 
>>>>>> Hi,
>>>>>> 
>>>>>> Anyone has experienced this or have any insights of this?
>>>>>> 
>>>>>> Regards,
>>>>>> Edwin
>>>>>> 
>>>>>> On Thu, 18 Apr 2019 at 18:04, Zheng Lin Edwin Yeo <
>>> edwinye...@gmail.com>
>>>>>> wrote:
>>>>>> 
>>>>>>> Is there possibility that this could be a bug in the new Solr 8.0.0?
>>>>>>> 
>>>>>>> Since I do not face the issue in the earlier version, and I have not
>>>>>>> changed any configuration in this new version. My data in Solr 8.0.0
>>> is
>>>>>>> freshly re-index directly in Solr 8.0.0, not upgraded from earlier
>>> version.
>>>>>>> 
>>>>>>> Regards,
>>>>>>> Edwin
>>>>>>> 
>>>>>>> On Thu, 18 Apr 2019 at 10:10, Zheng Lin Edwin Yeo <
>>> edwinye...@gmail.com>
>>>>>>> wrote:
>>>>>>> 
>>>>>>>> Hi Jason,
>>>>>>>> 
>>>>>>>> The same problem still persist after restarting my Solr nodes. The
>>> only
>>>>>>>> time the problem didn't occur is when I disabled the basic
>>> authentication.
>>>>>>>> 
>>>>>>>> I have tried with a few "/select?q=*:*", and they do not exhibit the
>>>>>>>> same problem. Even the similar query with only 1 shard does not
>>> have the
>>>>>>>> problem.
>>>>>>>> 
>>>>>&g

Re: Intermittent error 401 with JSON Facet query to retrieve count all collections

[Zheng Lin Edwin Yeo]

rror-class","org.apache.solr.client.solrj.impl.Http2SolrClient$RemoteSolrException",
>> >>
>> >>
>> "root-error-class","org.apache.solr.client.solrj.impl.Http2SolrClient$RemoteSolrException"],
>> >>"msg":"Error from server at null: Expected mime type
>> >> application/octet-stream but got text/html. \n\n> >> http-equiv=\"Content-Type\"
>> >> content=\"text/html;charset=utf-8\"/>\nError 401 require
>> >> authentication\n\nHTTP ERROR
>> 401\nProblem
>> >> accessing /solr/collection6/select. Reason:\nrequire
>> >> authentication\n\n\n",
>> >>"code":401}}
>> >>
>> >> This issue does not occur in Solr 7.6 and Solr 7.7, even though I have
>> set
>> >> up the same authentication for all the versions.
>> >>
>> >> What could be the issue that causes this?
>> >>
>> >>
>> >> Below is the format of my security.json:
>> >>
>> >> {
>> >> "authentication":
>> >>
>> >> {"blockUnknown": true,"class":"solr.BasicAuthPlugin",
>> >> "credentials":
>> >> {"user1":"hyHXXuJSqcZdNgdSTGUvrQZRpqrYFUQ2ffmlWQ4GUTk=
>> >> E0w3/2FD+rlxulbPm2G7i9HZqT+2gMBzcyJCcGcMWwA="}
>> >>
>> >> },
>> >> "authorization":
>> >>
>> >> {"class":"solr.RuleBasedAuthorizationPlugin","user-role":
>> >> {"user1":"admin"}
>> >>
>> >> ,
>> >>   "permissions":[
>> >> {"name":"security-edit",   "role":"admin"}
>> >>
>> >> ]
>> >> }}
>> >>
>> >>
>> >> Regards,
>> >> Edwin
>> >>
>> >>
>> >> On Mon, 22 Apr 2019 at 09:37, Zheng Lin Edwin Yeo <
>> edwinye...@gmail.com>
>> >> wrote:
>> >>
>> >>> Hi,
>> >>>
>> >>> Anyone has experienced this or have any insights of this?
>> >>>
>> >>> Regards,
>> >>> Edwin
>> >>>
>> >>> On Thu, 18 Apr 2019 at 18:04, Zheng Lin Edwin Yeo <
>> edwinye...@gmail.com>
>> >>> wrote:
>> >>>
>> >>>> Is there possibility that this could be a bug in the new Solr 8.0.0?
>> >>>>
>> >>>> Since I do not face the issue in the earlier version, and I have not
>> >>>> changed any configuration in this new version. My data in Solr 8.0.0
>> is
>> >>>> freshly re-index directly in Solr 8.0.0, not upgraded from earlier
>> version.
>> >>>>
>> >>>> Regards,
>> >>>> Edwin
>> >>>>
>> >>>> On Thu, 18 Apr 2019 at 10:10, Zheng Lin Edwin Yeo <
>> edwinye...@gmail.com>
>> >>>> wrote:
>> >>>>
>> >>>>> Hi Jason,
>> >>>>>
>> >>>>> The same problem still persist after restarting my Solr nodes. The
>> only
>> >>>>> time the problem didn't occur is when I disabled the basic
>> authentication.
>> >>>>>
>> >>>>> I have tried with a few "/select?q=*:*", and they do not exhibit the
>> >>>>> same problem. Even the similar query with only 1 shard does not
>> have the
>> >>>>> problem.
>> >>>>>
>> >>>>>
>> >>>>>
>> https://localhost:8983/solr/collection1/select?q=testing=https://localhost:8983/solr/collection1=0={categories
>> >>>>> : {type : terms,field : content_type,limit : 100}}
>> >>>>>
>> >>>>>
>> >>>>> It is only when there are 2 or more shards, that the problem occur.
>> >>>>>
>> >>>>>
>> >>>>>
>> https://localhost:8983/solr/collection1/select?q=testing=https://localhost:8983/solr/collection1,https://localhost:8983/solr/collection2=0={categories
>> >>>>> : {type : terms,field : content_type,limit : 100}}
>> >>>>>
>> >>>>>
>> >>>>> Regards,
>> >>

Re: Intermittent error 401 with JSON Facet query to retrieve count all collections

[Zheng Lin Edwin Yeo]

eason:\nrequire
> >> authentication\n\n\n",
> >>"code":401}}
> >>
> >> This issue does not occur in Solr 7.6 and Solr 7.7, even though I have
> set
> >> up the same authentication for all the versions.
> >>
> >> What could be the issue that causes this?
> >>
> >>
> >> Below is the format of my security.json:
> >>
> >> {
> >> "authentication":
> >>
> >> {"blockUnknown": true,"class":"solr.BasicAuthPlugin",
> >> "credentials":
> >> {"user1":"hyHXXuJSqcZdNgdSTGUvrQZRpqrYFUQ2ffmlWQ4GUTk=
> >> E0w3/2FD+rlxulbPm2G7i9HZqT+2gMBzcyJCcGcMWwA="}
> >>
> >> },
> >> "authorization":
> >>
> >> {"class":"solr.RuleBasedAuthorizationPlugin","user-role":
> >> {"user1":"admin"}
> >>
> >> ,
> >>   "permissions":[
> >> {"name":"security-edit",   "role":"admin"}
> >>
> >> ]
> >> }}
> >>
> >>
> >> Regards,
> >> Edwin
> >>
> >>
> >> On Mon, 22 Apr 2019 at 09:37, Zheng Lin Edwin Yeo  >
> >> wrote:
> >>
> >>> Hi,
> >>>
> >>> Anyone has experienced this or have any insights of this?
> >>>
> >>> Regards,
> >>> Edwin
> >>>
> >>> On Thu, 18 Apr 2019 at 18:04, Zheng Lin Edwin Yeo <
> edwinye...@gmail.com>
> >>> wrote:
> >>>
> >>>> Is there possibility that this could be a bug in the new Solr 8.0.0?
> >>>>
> >>>> Since I do not face the issue in the earlier version, and I have not
> >>>> changed any configuration in this new version. My data in Solr 8.0.0
> is
> >>>> freshly re-index directly in Solr 8.0.0, not upgraded from earlier
> version.
> >>>>
> >>>> Regards,
> >>>> Edwin
> >>>>
> >>>> On Thu, 18 Apr 2019 at 10:10, Zheng Lin Edwin Yeo <
> edwinye...@gmail.com>
> >>>> wrote:
> >>>>
> >>>>> Hi Jason,
> >>>>>
> >>>>> The same problem still persist after restarting my Solr nodes. The
> only
> >>>>> time the problem didn't occur is when I disabled the basic
> authentication.
> >>>>>
> >>>>> I have tried with a few "/select?q=*:*", and they do not exhibit the
> >>>>> same problem. Even the similar query with only 1 shard does not have
> the
> >>>>> problem.
> >>>>>
> >>>>>
> >>>>>
> https://localhost:8983/solr/collection1/select?q=testing=https://localhost:8983/solr/collection1=0={categories
> >>>>> : {type : terms,field : content_type,limit : 100}}
> >>>>>
> >>>>>
> >>>>> It is only when there are 2 or more shards, that the problem occur.
> >>>>>
> >>>>>
> >>>>>
> https://localhost:8983/solr/collection1/select?q=testing=https://localhost:8983/solr/collection1,https://localhost:8983/solr/collection2=0={categories
> >>>>> : {type : terms,field : content_type,limit : 100}}
> >>>>>
> >>>>>
> >>>>> Regards,
> >>>>> Edwin
> >>>>>
> >>>>>
> >>>>> On Thu, 18 Apr 2019 at 01:15, Jason Gerlowski  >
> >>>>> wrote:
> >>>>>
> >>>>>> Agreed, I'd be surprised if this behavior was specific to JSON
> >>>>>> Faceting.  Though I'm surprised it's happening at all, so...
> >>>>>>
> >>>>>> Anyway, that's easy for you to test though.  Try a few
> "/select?q=*:*"
> >>>>>> queries and see whether they also exhibits this behavior.  One other
> >>>>>> question: does the behavior persist after restarting your Solr
> nodes?
> >>>>>>
> >>>>>> Good luck,
> >>>>>>
> >>>>>> Jason
> >>>>>>
> >>>>>> On Wed, Apr 17, 2019 at 4:05 AM Zheng Lin Edwin Yeo
> >>>>>>  wrote:
> >>>>>>>
> >>>>>>> Hi,
>

Re: Intermittent error 401 with JSON Facet query to retrieve count all collections

[Jan Høydahl]

> 
>> },
>> "authorization":
>> 
>> {"class":"solr.RuleBasedAuthorizationPlugin","user-role":
>> {"user1":"admin"}
>> 
>> ,
>>   "permissions":[
>> {"name":"security-edit",   "role":"admin"}
>> 
>> ]
>> }}
>> 
>> 
>> Regards,
>> Edwin
>> 
>> 
>> On Mon, 22 Apr 2019 at 09:37, Zheng Lin Edwin Yeo 
>> wrote:
>> 
>>> Hi,
>>> 
>>> Anyone has experienced this or have any insights of this?
>>> 
>>> Regards,
>>> Edwin
>>> 
>>> On Thu, 18 Apr 2019 at 18:04, Zheng Lin Edwin Yeo 
>>> wrote:
>>> 
>>>> Is there possibility that this could be a bug in the new Solr 8.0.0?
>>>> 
>>>> Since I do not face the issue in the earlier version, and I have not
>>>> changed any configuration in this new version. My data in Solr 8.0.0 is
>>>> freshly re-index directly in Solr 8.0.0, not upgraded from earlier version.
>>>> 
>>>> Regards,
>>>> Edwin
>>>> 
>>>> On Thu, 18 Apr 2019 at 10:10, Zheng Lin Edwin Yeo 
>>>> wrote:
>>>> 
>>>>> Hi Jason,
>>>>> 
>>>>> The same problem still persist after restarting my Solr nodes. The only
>>>>> time the problem didn't occur is when I disabled the basic authentication.
>>>>> 
>>>>> I have tried with a few "/select?q=*:*", and they do not exhibit the
>>>>> same problem. Even the similar query with only 1 shard does not have the
>>>>> problem.
>>>>> 
>>>>> 
>>>>> https://localhost:8983/solr/collection1/select?q=testing=https://localhost:8983/solr/collection1=0={categories
>>>>> : {type : terms,field : content_type,limit : 100}}
>>>>> 
>>>>> 
>>>>> It is only when there are 2 or more shards, that the problem occur.
>>>>> 
>>>>> 
>>>>> https://localhost:8983/solr/collection1/select?q=testing=https://localhost:8983/solr/collection1,https://localhost:8983/solr/collection2=0={categories
>>>>> : {type : terms,field : content_type,limit : 100}}
>>>>> 
>>>>> 
>>>>> Regards,
>>>>> Edwin
>>>>> 
>>>>> 
>>>>> On Thu, 18 Apr 2019 at 01:15, Jason Gerlowski 
>>>>> wrote:
>>>>> 
>>>>>> Agreed, I'd be surprised if this behavior was specific to JSON
>>>>>> Faceting.  Though I'm surprised it's happening at all, so...
>>>>>> 
>>>>>> Anyway, that's easy for you to test though.  Try a few "/select?q=*:*"
>>>>>> queries and see whether they also exhibits this behavior.  One other
>>>>>> question: does the behavior persist after restarting your Solr nodes?
>>>>>> 
>>>>>> Good luck,
>>>>>> 
>>>>>> Jason
>>>>>> 
>>>>>> On Wed, Apr 17, 2019 at 4:05 AM Zheng Lin Edwin Yeo
>>>>>>  wrote:
>>>>>>> 
>>>>>>> Hi,
>>>>>>> 
>>>>>>> For your info, I have enabled basic authentication and SSL in all
>>>>>> the 3
>>>>>>> versions, and I'm not sure if the issue is more on the
>>>>>> authentication side
>>>>>>> instead of the JSON Facet query?
>>>>>>> 
>>>>>>> Regards,
>>>>>>> Edwin
>>>>>>> 
>>>>>>> On Wed, 17 Apr 2019 at 06:54, Zheng Lin Edwin Yeo <
>>>>>> edwinye...@gmail.com>
>>>>>>> wrote:
>>>>>>> 
>>>>>>>> Hi Jason,
>>>>>>>> 
>>>>>>>> Yes, that is correct.
>>>>>>>> 
>>>>>>>> Below is the format of my security.json. I have changed the masked
>>>>>>>> password for security purposes.
>>>>>>>> 
>>>>>>>> {
>>>>>>>> "authentication":{
>>>>>>>>   "blockUnknown": true,
>>>>>>>>   "class":"solr.BasicAuthPlugin",
>>>>>>>> 
>>>>&

Re: Intermittent error 401 with JSON Facet query to retrieve count all collections

[Zheng Lin Edwin Yeo]

sion, and I have not
>>> changed any configuration in this new version. My data in Solr 8.0.0 is
>>> freshly re-index directly in Solr 8.0.0, not upgraded from earlier version.
>>>
>>> Regards,
>>> Edwin
>>>
>>> On Thu, 18 Apr 2019 at 10:10, Zheng Lin Edwin Yeo 
>>> wrote:
>>>
>>>> Hi Jason,
>>>>
>>>> The same problem still persist after restarting my Solr nodes. The only
>>>> time the problem didn't occur is when I disabled the basic authentication.
>>>>
>>>> I have tried with a few "/select?q=*:*", and they do not exhibit the
>>>> same problem. Even the similar query with only 1 shard does not have the
>>>> problem.
>>>>
>>>>
>>>> https://localhost:8983/solr/collection1/select?q=testing=https://localhost:8983/solr/collection1=0={categories
>>>> : {type : terms,field : content_type,limit : 100}}
>>>>
>>>>
>>>> It is only when there are 2 or more shards, that the problem occur.
>>>>
>>>>
>>>> https://localhost:8983/solr/collection1/select?q=testing=https://localhost:8983/solr/collection1,https://localhost:8983/solr/collection2=0={categories
>>>> : {type : terms,field : content_type,limit : 100}}
>>>>
>>>>
>>>> Regards,
>>>> Edwin
>>>>
>>>>
>>>> On Thu, 18 Apr 2019 at 01:15, Jason Gerlowski 
>>>> wrote:
>>>>
>>>>> Agreed, I'd be surprised if this behavior was specific to JSON
>>>>> Faceting.  Though I'm surprised it's happening at all, so...
>>>>>
>>>>> Anyway, that's easy for you to test though.  Try a few "/select?q=*:*"
>>>>> queries and see whether they also exhibits this behavior.  One other
>>>>> question: does the behavior persist after restarting your Solr nodes?
>>>>>
>>>>> Good luck,
>>>>>
>>>>> Jason
>>>>>
>>>>> On Wed, Apr 17, 2019 at 4:05 AM Zheng Lin Edwin Yeo
>>>>>  wrote:
>>>>> >
>>>>> > Hi,
>>>>> >
>>>>> > For your info, I have enabled basic authentication and SSL in all
>>>>> the 3
>>>>> > versions, and I'm not sure if the issue is more on the
>>>>> authentication side
>>>>> > instead of the JSON Facet query?
>>>>> >
>>>>> > Regards,
>>>>> > Edwin
>>>>> >
>>>>> > On Wed, 17 Apr 2019 at 06:54, Zheng Lin Edwin Yeo <
>>>>> edwinye...@gmail.com>
>>>>> > wrote:
>>>>> >
>>>>> > > Hi Jason,
>>>>> > >
>>>>> > > Yes, that is correct.
>>>>> > >
>>>>> > > Below is the format of my security.json. I have changed the masked
>>>>> > > password for security purposes.
>>>>> > >
>>>>> > > {
>>>>> > > "authentication":{
>>>>> > >"blockUnknown": true,
>>>>> > >"class":"solr.BasicAuthPlugin",
>>>>> > >
>>>>> "credentials":{"user1":"hyHXXuJSqcZdNgdSTGUvrQZRpqrYFUQ2ffmlWQ4GUTk=
>>>>> > > E0w3/2FD+rlxulbPm2G7i9HZqT+2gMBzcyJCcGcMWwA="}
>>>>> > > },
>>>>> > > "authorization":{
>>>>> > >"class":"solr.RuleBasedAuthorizationPlugin",
>>>>> > >"user-role":{"user1":"admin"},
>>>>> > >"permissions":[{"name":"security-edit",
>>>>> > >   "role":"admin"}]
>>>>> > > }}
>>>>> > >
>>>>> > > Regards,
>>>>> > > Edwin
>>>>> > >
>>>>> > > On Tue, 16 Apr 2019 at 23:12, Jason Gerlowski <
>>>>> gerlowsk...@gmail.com>
>>>>> > > wrote:
>>>>> > >
>>>>> > >> Hi Edwin,
>>>>> > >>
>>>>> > >> To clarify what you're running into:
>>>>> > >>
>>>>> > >> - on 7.6, this query works all the time
>>>>

Intermittent error 401 with JSON Facet query to retrieve count all collections

[Zheng Lin Edwin Yeo]

Hi,


I am using the below JSON Facet to retrieve the count of all the different
collections in one query.

https://localhost:8983/solr/collection1/select?q=testing=https://localhost:8983/solr/collection1,https://localhost:8983/solr/collection2,https://localhost:8983/solr/collection3,https://localhost:8983/solr/collection4,https://localhost:8983/solr/collection5,https://localhost:8983/solr/collection6=0={categories
:
{type : terms,field : content_type,limit : 100}}


Previously, in Solr 7.6 and Solr 7.7, this query can work correctly and we
are able to produce the correct output.

{
  "responseHeader":
{ "zkConnected":true, "status":0, "QTime":24}

,
  "response":
{"numFound":41200,"start":0,"maxScore":12.993215,"docs":[]   }

,
  "facets":{
"count":41200,
"categories":{
  "buckets":[
{   "val":"collection1",   "count":26213}

,

{   "val":"collection2",   "count":12075}

,

{   "val":"collection3",   "count":1947}

,

{   "val":"collection4",   "count":850}

,

{   "val":"collection5",   "count":111}

,

{   "val":"collection6",   "count":4}

]}}}


However, in the new Solr 8.0.0, this query can only work occasionally. Most
of the time, we will get the following error of 'Error 401 require
authentication':

{
  "responseHeader":
{ "zkConnected":true, "status":401, "QTime":11}

,
  "error":{
"metadata":[

"error-class","org.apache.solr.client.solrj.impl.Http2SolrClient$RemoteSolrException",

"root-error-class","org.apache.solr.client.solrj.impl.Http2SolrClient$RemoteSolrException"],
"msg":"Error from server at null: Expected mime type
application/octet-stream but got text/html. \n\n\nError 401 require
authentication\n\nHTTP ERROR 401\nProblem
accessing /solr/collection6/select. Reason:\nrequire
authentication\n\n\n",
"code":401}}

This issue does not occur in Solr 7.6 and Solr 7.7, even though I have set
up the same authentication for all the versions.

What could be the issue that causes this?


Below is the format of my security.json:

{
"authentication":

{"blockUnknown": true,"class":"solr.BasicAuthPlugin",
 "credentials":
{"user1":"hyHXXuJSqcZdNgdSTGUvrQZRpqrYFUQ2ffmlWQ4GUTk=
E0w3/2FD+rlxulbPm2G7i9HZqT+2gMBzcyJCcGcMWwA="}

},
"authorization":

{"class":"solr.RuleBasedAuthorizationPlugin","user-role":
{"user1":"admin"}

,
   "permissions":[
{"name":"security-edit",   "role":"admin"}

]
}}


Regards,
Edwin


On Mon, 22 Apr 2019 at 09:37, Zheng Lin Edwin Yeo 
wrote:

> Hi,
>
> Anyone has experienced this or have any insights of this?
>
> Regards,
> Edwin
>
> On Thu, 18 Apr 2019 at 18:04, Zheng Lin Edwin Yeo 
> wrote:
>
>> Is there possibility that this could be a bug in the new Solr 8.0.0?
>>
>> Since I do not face the issue in the earlier version, and I have not
>> changed any configuration in this new version. My data in Solr 8.0.0 is
>> freshly re-index directly in Solr 8.0.0, not upgraded from earlier version.
>>
>> Regards,
>> Edwin
>>
>> On Thu, 18 Apr 2019 at 10:10, Zheng Lin Edwin Yeo 
>> wrote:
>>
>>> Hi Jason,
>>>
>>> The same problem still persist after restarting my Solr nodes. The only
>>> time the problem didn't occur is when I disabled the basic authentication.
>>>
>>> I have tried with a few "/select?q=*:*", and they do not exhibit the
>>> same problem. Even the similar query with only 1 shard does not have the
>>> problem.
>>>
>>>
>>> https://localhost:8983/solr/collection1/select?q=testing=https://localhost:8983/solr/collection1=0={categories
>>> : {type : terms,field : content_type,limit : 100}}
>>>
>>>
>>> It is only when there are 2 or more shards, that the problem occur.
>>>
>>>
>>> https://localhost:8983/solr/collection1/select?q=testing=https://localhost:8983/solr/collection1,https://localhost:8983/solr/collection2=0={categories
>>> : {type : terms,field : content_type,limit : 100}}
>>>
>>>
>>> Regards,
>>> Edwin
>>>
>>>
>>> On Thu, 18 Apr 

Re: JSON Facet query to retrieve count all collections in Solr 8.0.0

[Zheng Lin Edwin Yeo]

Hi,

Anyone has experienced this or have any insights of this?

Regards,
Edwin

On Thu, 18 Apr 2019 at 18:04, Zheng Lin Edwin Yeo 
wrote:

> Is there possibility that this could be a bug in the new Solr 8.0.0?
>
> Since I do not face the issue in the earlier version, and I have not
> changed any configuration in this new version. My data in Solr 8.0.0 is
> freshly re-index directly in Solr 8.0.0, not upgraded from earlier version.
>
> Regards,
> Edwin
>
> On Thu, 18 Apr 2019 at 10:10, Zheng Lin Edwin Yeo 
> wrote:
>
>> Hi Jason,
>>
>> The same problem still persist after restarting my Solr nodes. The only
>> time the problem didn't occur is when I disabled the basic authentication.
>>
>> I have tried with a few "/select?q=*:*", and they do not exhibit the same
>> problem. Even the similar query with only 1 shard does not have the problem.
>>
>>
>> https://localhost:8983/solr/collection1/select?q=testing=https://localhost:8983/solr/collection1=0={categories
>> : {type : terms,field : content_type,limit : 100}}
>>
>>
>> It is only when there are 2 or more shards, that the problem occur.
>>
>>
>> https://localhost:8983/solr/collection1/select?q=testing=https://localhost:8983/solr/collection1,https://localhost:8983/solr/collection2=0={categories
>> : {type : terms,field : content_type,limit : 100}}
>>
>>
>> Regards,
>> Edwin
>>
>>
>> On Thu, 18 Apr 2019 at 01:15, Jason Gerlowski 
>> wrote:
>>
>>> Agreed, I'd be surprised if this behavior was specific to JSON
>>> Faceting.  Though I'm surprised it's happening at all, so...
>>>
>>> Anyway, that's easy for you to test though.  Try a few "/select?q=*:*"
>>> queries and see whether they also exhibits this behavior.  One other
>>> question: does the behavior persist after restarting your Solr nodes?
>>>
>>> Good luck,
>>>
>>> Jason
>>>
>>> On Wed, Apr 17, 2019 at 4:05 AM Zheng Lin Edwin Yeo
>>>  wrote:
>>> >
>>> > Hi,
>>> >
>>> > For your info, I have enabled basic authentication and SSL in all the 3
>>> > versions, and I'm not sure if the issue is more on the authentication
>>> side
>>> > instead of the JSON Facet query?
>>> >
>>> > Regards,
>>> > Edwin
>>> >
>>> > On Wed, 17 Apr 2019 at 06:54, Zheng Lin Edwin Yeo <
>>> edwinye...@gmail.com>
>>> > wrote:
>>> >
>>> > > Hi Jason,
>>> > >
>>> > > Yes, that is correct.
>>> > >
>>> > > Below is the format of my security.json. I have changed the masked
>>> > > password for security purposes.
>>> > >
>>> > > {
>>> > > "authentication":{
>>> > >"blockUnknown": true,
>>> > >"class":"solr.BasicAuthPlugin",
>>> > >
>>> "credentials":{"user1":"hyHXXuJSqcZdNgdSTGUvrQZRpqrYFUQ2ffmlWQ4GUTk=
>>> > > E0w3/2FD+rlxulbPm2G7i9HZqT+2gMBzcyJCcGcMWwA="}
>>> > > },
>>> > > "authorization":{
>>> > >"class":"solr.RuleBasedAuthorizationPlugin",
>>> > >"user-role":{"user1":"admin"},
>>> > >"permissions":[{"name":"security-edit",
>>> > >   "role":"admin"}]
>>> > > }}
>>> > >
>>> > > Regards,
>>> > > Edwin
>>> > >
>>> > > On Tue, 16 Apr 2019 at 23:12, Jason Gerlowski >> >
>>> > > wrote:
>>> > >
>>> > >> Hi Edwin,
>>> > >>
>>> > >> To clarify what you're running into:
>>> > >>
>>> > >> - on 7.6, this query works all the time
>>> > >> - on 7.7 this query works all the time
>>> > >> - on 8.0, this query works the first time you run it, but subsequent
>>> > >> runs return a 401 error?
>>> > >>
>>> > >> Is that correct?  It might be helpful for others if you could share
>>> > >> your security.json.
>>> > >>
>>> > >> Best,
>>> > >>
>>> > >> Jason
>>> > >>
>>> > >> On Mon, Apr 15, 2019 at 10:40 PM

Re: JSON Facet query to retrieve count all collections in Solr 8.0.0

[Zheng Lin Edwin Yeo]

Is there possibility that this could be a bug in the new Solr 8.0.0?

Since I do not face the issue in the earlier version, and I have not
changed any configuration in this new version. My data in Solr 8.0.0 is
freshly re-index directly in Solr 8.0.0, not upgraded from earlier version.

Regards,
Edwin

On Thu, 18 Apr 2019 at 10:10, Zheng Lin Edwin Yeo 
wrote:

> Hi Jason,
>
> The same problem still persist after restarting my Solr nodes. The only
> time the problem didn't occur is when I disabled the basic authentication.
>
> I have tried with a few "/select?q=*:*", and they do not exhibit the same
> problem. Even the similar query with only 1 shard does not have the problem.
>
>
> https://localhost:8983/solr/collection1/select?q=testing=https://localhost:8983/solr/collection1=0={categories
> : {type : terms,field : content_type,limit : 100}}
>
>
> It is only when there are 2 or more shards, that the problem occur.
>
>
> https://localhost:8983/solr/collection1/select?q=testing=https://localhost:8983/solr/collection1,https://localhost:8983/solr/collection2=0={categories
> : {type : terms,field : content_type,limit : 100}}
>
>
> Regards,
> Edwin
>
>
> On Thu, 18 Apr 2019 at 01:15, Jason Gerlowski 
> wrote:
>
>> Agreed, I'd be surprised if this behavior was specific to JSON
>> Faceting.  Though I'm surprised it's happening at all, so...
>>
>> Anyway, that's easy for you to test though.  Try a few "/select?q=*:*"
>> queries and see whether they also exhibits this behavior.  One other
>> question: does the behavior persist after restarting your Solr nodes?
>>
>> Good luck,
>>
>> Jason
>>
>> On Wed, Apr 17, 2019 at 4:05 AM Zheng Lin Edwin Yeo
>>  wrote:
>> >
>> > Hi,
>> >
>> > For your info, I have enabled basic authentication and SSL in all the 3
>> > versions, and I'm not sure if the issue is more on the authentication
>> side
>> > instead of the JSON Facet query?
>> >
>> > Regards,
>> > Edwin
>> >
>> > On Wed, 17 Apr 2019 at 06:54, Zheng Lin Edwin Yeo > >
>> > wrote:
>> >
>> > > Hi Jason,
>> > >
>> > > Yes, that is correct.
>> > >
>> > > Below is the format of my security.json. I have changed the masked
>> > > password for security purposes.
>> > >
>> > > {
>> > > "authentication":{
>> > >"blockUnknown": true,
>> > >"class":"solr.BasicAuthPlugin",
>> > >
>> "credentials":{"user1":"hyHXXuJSqcZdNgdSTGUvrQZRpqrYFUQ2ffmlWQ4GUTk=
>> > > E0w3/2FD+rlxulbPm2G7i9HZqT+2gMBzcyJCcGcMWwA="}
>> > > },
>> > > "authorization":{
>> > >"class":"solr.RuleBasedAuthorizationPlugin",
>> > >"user-role":{"user1":"admin"},
>> > >"permissions":[{"name":"security-edit",
>> > >   "role":"admin"}]
>> > > }}
>> > >
>> > > Regards,
>> > > Edwin
>> > >
>> > > On Tue, 16 Apr 2019 at 23:12, Jason Gerlowski 
>> > > wrote:
>> > >
>> > >> Hi Edwin,
>> > >>
>> > >> To clarify what you're running into:
>> > >>
>> > >> - on 7.6, this query works all the time
>> > >> - on 7.7 this query works all the time
>> > >> - on 8.0, this query works the first time you run it, but subsequent
>> > >> runs return a 401 error?
>> > >>
>> > >> Is that correct?  It might be helpful for others if you could share
>> > >> your security.json.
>> > >>
>> > >> Best,
>> > >>
>> > >> Jason
>> > >>
>> > >> On Mon, Apr 15, 2019 at 10:40 PM Zheng Lin Edwin Yeo
>> > >>  wrote:
>> > >> >
>> > >> > Hi,
>> > >> >
>> > >> > I am using the below JSON Facet to retrieve the count of all the
>> > >> different
>> > >> > collections in one query.
>> > >> >
>> > >> >
>> > >>
>> https://localhost:8983/solr/collection1/select?q=testing=https://localhost:8983/solr/collection1,https://localhost:8983/solr/collection2,https://localhost:8983/solr/collection3,https://localhost:8983/solr/collection4,https://localhost:8983/solr/co

Re: JSON Facet query to retrieve count all collections in Solr 8.0.0

[Zheng Lin Edwin Yeo]

Hi Jason,

The same problem still persist after restarting my Solr nodes. The only
time the problem didn't occur is when I disabled the basic authentication.

I have tried with a few "/select?q=*:*", and they do not exhibit the same
problem. Even the similar query with only 1 shard does not have the problem.

https://localhost:8983/solr/collection1/select?q=testing=https://localhost:8983/solr/collection1=0={categories
: {type : terms,field : content_type,limit : 100}}


It is only when there are 2 or more shards, that the problem occur.

https://localhost:8983/solr/collection1/select?q=testing=https://localhost:8983/solr/collection1,https://localhost:8983/solr/collection2=0={categories
: {type : terms,field : content_type,limit : 100}}


Regards,
Edwin


On Thu, 18 Apr 2019 at 01:15, Jason Gerlowski  wrote:

> Agreed, I'd be surprised if this behavior was specific to JSON
> Faceting.  Though I'm surprised it's happening at all, so...
>
> Anyway, that's easy for you to test though.  Try a few "/select?q=*:*"
> queries and see whether they also exhibits this behavior.  One other
> question: does the behavior persist after restarting your Solr nodes?
>
> Good luck,
>
> Jason
>
> On Wed, Apr 17, 2019 at 4:05 AM Zheng Lin Edwin Yeo
>  wrote:
> >
> > Hi,
> >
> > For your info, I have enabled basic authentication and SSL in all the 3
> > versions, and I'm not sure if the issue is more on the authentication
> side
> > instead of the JSON Facet query?
> >
> > Regards,
> > Edwin
> >
> > On Wed, 17 Apr 2019 at 06:54, Zheng Lin Edwin Yeo 
> > wrote:
> >
> > > Hi Jason,
> > >
> > > Yes, that is correct.
> > >
> > > Below is the format of my security.json. I have changed the masked
> > > password for security purposes.
> > >
> > > {
> > > "authentication":{
> > >"blockUnknown": true,
> > >"class":"solr.BasicAuthPlugin",
> > >"credentials":{"user1":"hyHXXuJSqcZdNgdSTGUvrQZRpqrYFUQ2ffmlWQ4GUTk=
> > > E0w3/2FD+rlxulbPm2G7i9HZqT+2gMBzcyJCcGcMWwA="}
> > > },
> > > "authorization":{
> > >"class":"solr.RuleBasedAuthorizationPlugin",
> > >"user-role":{"user1":"admin"},
> > >"permissions":[{"name":"security-edit",
> > >   "role":"admin"}]
> > > }}
> > >
> > > Regards,
> > > Edwin
> > >
> > > On Tue, 16 Apr 2019 at 23:12, Jason Gerlowski 
> > > wrote:
> > >
> > >> Hi Edwin,
> > >>
> > >> To clarify what you're running into:
> > >>
> > >> - on 7.6, this query works all the time
> > >> - on 7.7 this query works all the time
> > >> - on 8.0, this query works the first time you run it, but subsequent
> > >> runs return a 401 error?
> > >>
> > >> Is that correct?  It might be helpful for others if you could share
> > >> your security.json.
> > >>
> > >> Best,
> > >>
> > >> Jason
> > >>
> > >> On Mon, Apr 15, 2019 at 10:40 PM Zheng Lin Edwin Yeo
> > >>  wrote:
> > >> >
> > >> > Hi,
> > >> >
> > >> > I am using the below JSON Facet to retrieve the count of all the
> > >> different
> > >> > collections in one query.
> > >> >
> > >> >
> > >>
> https://localhost:8983/solr/collection1/select?q=testing=https://localhost:8983/solr/collection1,https://localhost:8983/solr/collection2,https://localhost:8983/solr/collection3,https://localhost:8983/solr/collection4,https://localhost:8983/solr/collection5,https://localhost:8983/solr/collection6=0={categories
> > >> > : {type : terms,field : content_type,limit : 100}}
> > >> >
> > >> >
> > >> > Previously, in Solr 7.6 and Solr 7.7, this query can work correctly
> and
> > >> we
> > >> > are able to produce the correct output.
> > >> >
> > >> > {
> > >> >   "responseHeader":{
> > >> > "zkConnected":true,
> > >> > "status":0,
> > >> > "QTime":24},
> > >> >
>  "response":{"numFound":41200,"start":0,"maxScore":12.993215,"docs":[]
> > >> >   },
> 

Re: JSON Facet query to retrieve count all collections in Solr 8.0.0

[Jason Gerlowski]

Agreed, I'd be surprised if this behavior was specific to JSON
Faceting.  Though I'm surprised it's happening at all, so...

Anyway, that's easy for you to test though.  Try a few "/select?q=*:*"
queries and see whether they also exhibits this behavior.  One other
question: does the behavior persist after restarting your Solr nodes?

Good luck,

Jason

On Wed, Apr 17, 2019 at 4:05 AM Zheng Lin Edwin Yeo
 wrote:
>
> Hi,
>
> For your info, I have enabled basic authentication and SSL in all the 3
> versions, and I'm not sure if the issue is more on the authentication side
> instead of the JSON Facet query?
>
> Regards,
> Edwin
>
> On Wed, 17 Apr 2019 at 06:54, Zheng Lin Edwin Yeo 
> wrote:
>
> > Hi Jason,
> >
> > Yes, that is correct.
> >
> > Below is the format of my security.json. I have changed the masked
> > password for security purposes.
> >
> > {
> > "authentication":{
> >"blockUnknown": true,
> >"class":"solr.BasicAuthPlugin",
> >"credentials":{"user1":"hyHXXuJSqcZdNgdSTGUvrQZRpqrYFUQ2ffmlWQ4GUTk=
> > E0w3/2FD+rlxulbPm2G7i9HZqT+2gMBzcyJCcGcMWwA="}
> > },
> > "authorization":{
> >"class":"solr.RuleBasedAuthorizationPlugin",
> >"user-role":{"user1":"admin"},
> >"permissions":[{"name":"security-edit",
> >   "role":"admin"}]
> > }}
> >
> > Regards,
> > Edwin
> >
> > On Tue, 16 Apr 2019 at 23:12, Jason Gerlowski 
> > wrote:
> >
> >> Hi Edwin,
> >>
> >> To clarify what you're running into:
> >>
> >> - on 7.6, this query works all the time
> >> - on 7.7 this query works all the time
> >> - on 8.0, this query works the first time you run it, but subsequent
> >> runs return a 401 error?
> >>
> >> Is that correct?  It might be helpful for others if you could share
> >> your security.json.
> >>
> >> Best,
> >>
> >> Jason
> >>
> >> On Mon, Apr 15, 2019 at 10:40 PM Zheng Lin Edwin Yeo
> >>  wrote:
> >> >
> >> > Hi,
> >> >
> >> > I am using the below JSON Facet to retrieve the count of all the
> >> different
> >> > collections in one query.
> >> >
> >> >
> >> https://localhost:8983/solr/collection1/select?q=testing=https://localhost:8983/solr/collection1,https://localhost:8983/solr/collection2,https://localhost:8983/solr/collection3,https://localhost:8983/solr/collection4,https://localhost:8983/solr/collection5,https://localhost:8983/solr/collection6=0={categories
> >> > : {type : terms,field : content_type,limit : 100}}
> >> >
> >> >
> >> > Previously, in Solr 7.6 and Solr 7.7, this query can work correctly and
> >> we
> >> > are able to produce the correct output.
> >> >
> >> > {
> >> >   "responseHeader":{
> >> > "zkConnected":true,
> >> > "status":0,
> >> > "QTime":24},
> >> >   "response":{"numFound":41200,"start":0,"maxScore":12.993215,"docs":[]
> >> >   },
> >> >   "facets":{
> >> > "count":41200,
> >> > "categories":{
> >> >   "buckets":[{
> >> >   "val":"collection1",
> >> >   "count":26213},
> >> > {
> >> >   "val":"collection2",
> >> >   "count":12075},
> >> > {
> >> >   "val":"collection3",
> >> >   "count":1947},
> >> > {
> >> >   "val":"collection4",
> >> >   "count":850},
> >> > {
> >> >   "val":"collection5",
> >> >   "count":111},
> >> > {
> >> >   "val":"collection6",
> >> >   "count":4}]}}}
> >> >
> >> >
> >> > However, in the new Solr 8.0.0, this query can only work once.
> >> > Subsequently, we will get the following error of 'require
> >> authentication':
> >> >
> >> > {
> >> >   "responseHeader":{
> >> > "zkConnected":true,
> >> > "status":401,
> >> > "QTime":11},
> >> >   "error":{
> >> > "metadata":[
> >> >
> >> >
> >> "error-class","org.apache.solr.client.solrj.impl.Http2SolrClient$RemoteSolrException",
> >> >
> >> >
> >> "root-error-class","org.apache.solr.client.solrj.impl.Http2SolrClient$RemoteSolrException"],
> >> > "msg":"Error from server at null: Expected mime type
> >> > application/octet-stream but got text/html. \n\n >> > http-equiv=\"Content-Type\"
> >> > content=\"text/html;charset=utf-8\"/>\nError 401 require
> >> > authentication\n\nHTTP ERROR
> >> 401\nProblem
> >> > accessing /solr/collection6/select. Reason:\nrequire
> >> > authentication\n\n\n",
> >> > "code":401}}
> >> >
> >> > This issue does not occur in Solr 7.6 and Solr 7.7, even though I have
> >> set
> >> > up the same authentication for all the versions.
> >> >
> >> > What could be the issue that causes this?
> >> >
> >> > Regards,
> >> > Edwin
> >>
> >

Re: JSON Facet query to retrieve count all collections in Solr 8.0.0

[Zheng Lin Edwin Yeo]

Hi,

For your info, I have enabled basic authentication and SSL in all the 3
versions, and I'm not sure if the issue is more on the authentication side
instead of the JSON Facet query?

Regards,
Edwin

On Wed, 17 Apr 2019 at 06:54, Zheng Lin Edwin Yeo 
wrote:

> Hi Jason,
>
> Yes, that is correct.
>
> Below is the format of my security.json. I have changed the masked
> password for security purposes.
>
> {
> "authentication":{
>"blockUnknown": true,
>"class":"solr.BasicAuthPlugin",
>"credentials":{"user1":"hyHXXuJSqcZdNgdSTGUvrQZRpqrYFUQ2ffmlWQ4GUTk=
> E0w3/2FD+rlxulbPm2G7i9HZqT+2gMBzcyJCcGcMWwA="}
> },
> "authorization":{
>"class":"solr.RuleBasedAuthorizationPlugin",
>"user-role":{"user1":"admin"},
>"permissions":[{"name":"security-edit",
>   "role":"admin"}]
> }}
>
> Regards,
> Edwin
>
> On Tue, 16 Apr 2019 at 23:12, Jason Gerlowski 
> wrote:
>
>> Hi Edwin,
>>
>> To clarify what you're running into:
>>
>> - on 7.6, this query works all the time
>> - on 7.7 this query works all the time
>> - on 8.0, this query works the first time you run it, but subsequent
>> runs return a 401 error?
>>
>> Is that correct?  It might be helpful for others if you could share
>> your security.json.
>>
>> Best,
>>
>> Jason
>>
>> On Mon, Apr 15, 2019 at 10:40 PM Zheng Lin Edwin Yeo
>>  wrote:
>> >
>> > Hi,
>> >
>> > I am using the below JSON Facet to retrieve the count of all the
>> different
>> > collections in one query.
>> >
>> >
>> https://localhost:8983/solr/collection1/select?q=testing=https://localhost:8983/solr/collection1,https://localhost:8983/solr/collection2,https://localhost:8983/solr/collection3,https://localhost:8983/solr/collection4,https://localhost:8983/solr/collection5,https://localhost:8983/solr/collection6=0={categories
>> > : {type : terms,field : content_type,limit : 100}}
>> >
>> >
>> > Previously, in Solr 7.6 and Solr 7.7, this query can work correctly and
>> we
>> > are able to produce the correct output.
>> >
>> > {
>> >   "responseHeader":{
>> > "zkConnected":true,
>> > "status":0,
>> > "QTime":24},
>> >   "response":{"numFound":41200,"start":0,"maxScore":12.993215,"docs":[]
>> >   },
>> >   "facets":{
>> > "count":41200,
>> > "categories":{
>> >   "buckets":[{
>> >   "val":"collection1",
>> >   "count":26213},
>> > {
>> >   "val":"collection2",
>> >   "count":12075},
>> > {
>> >   "val":"collection3",
>> >   "count":1947},
>> > {
>> >   "val":"collection4",
>> >   "count":850},
>> > {
>> >   "val":"collection5",
>> >   "count":111},
>> > {
>> >   "val":"collection6",
>> >   "count":4}]}}}
>> >
>> >
>> > However, in the new Solr 8.0.0, this query can only work once.
>> > Subsequently, we will get the following error of 'require
>> authentication':
>> >
>> > {
>> >   "responseHeader":{
>> > "zkConnected":true,
>> > "status":401,
>> > "QTime":11},
>> >   "error":{
>> > "metadata":[
>> >
>> >
>> "error-class","org.apache.solr.client.solrj.impl.Http2SolrClient$RemoteSolrException",
>> >
>> >
>> "root-error-class","org.apache.solr.client.solrj.impl.Http2SolrClient$RemoteSolrException"],
>> > "msg":"Error from server at null: Expected mime type
>> > application/octet-stream but got text/html. \n\n> > http-equiv=\"Content-Type\"
>> > content=\"text/html;charset=utf-8\"/>\nError 401 require
>> > authentication\n\nHTTP ERROR
>> 401\nProblem
>> > accessing /solr/collection6/select. Reason:\nrequire
>> > authentication\n\n\n",
>> > "code":401}}
>> >
>> > This issue does not occur in Solr 7.6 and Solr 7.7, even though I have
>> set
>> > up the same authentication for all the versions.
>> >
>> > What could be the issue that causes this?
>> >
>> > Regards,
>> > Edwin
>>
>

Re: JSON Facet query to retrieve count all collections in Solr 8.0.0

[Jason Gerlowski]

Hi Edwin,

To clarify what you're running into:

- on 7.6, this query works all the time
- on 7.7 this query works all the time
- on 8.0, this query works the first time you run it, but subsequent
runs return a 401 error?

Is that correct?  It might be helpful for others if you could share
your security.json.

Best,

Jason

On Mon, Apr 15, 2019 at 10:40 PM Zheng Lin Edwin Yeo
 wrote:
>
> Hi,
>
> I am using the below JSON Facet to retrieve the count of all the different
> collections in one query.
>
> https://localhost:8983/solr/collection1/select?q=testing=https://localhost:8983/solr/collection1,https://localhost:8983/solr/collection2,https://localhost:8983/solr/collection3,https://localhost:8983/solr/collection4,https://localhost:8983/solr/collection5,https://localhost:8983/solr/collection6=0={categories
> : {type : terms,field : content_type,limit : 100}}
>
>
> Previously, in Solr 7.6 and Solr 7.7, this query can work correctly and we
> are able to produce the correct output.
>
> {
>   "responseHeader":{
> "zkConnected":true,
> "status":0,
> "QTime":24},
>   "response":{"numFound":41200,"start":0,"maxScore":12.993215,"docs":[]
>   },
>   "facets":{
> "count":41200,
> "categories":{
>   "buckets":[{
>   "val":"collection1",
>   "count":26213},
> {
>   "val":"collection2",
>   "count":12075},
> {
>   "val":"collection3",
>   "count":1947},
> {
>   "val":"collection4",
>   "count":850},
> {
>   "val":"collection5",
>   "count":111},
> {
>   "val":"collection6",
>   "count":4}]}}}
>
>
> However, in the new Solr 8.0.0, this query can only work once.
> Subsequently, we will get the following error of 'require authentication':
>
> {
>   "responseHeader":{
> "zkConnected":true,
> "status":401,
> "QTime":11},
>   "error":{
> "metadata":[
>
> "error-class","org.apache.solr.client.solrj.impl.Http2SolrClient$RemoteSolrException",
>
> "root-error-class","org.apache.solr.client.solrj.impl.Http2SolrClient$RemoteSolrException"],
> "msg":"Error from server at null: Expected mime type
> application/octet-stream but got text/html. \n\n http-equiv=\"Content-Type\"
> content=\"text/html;charset=utf-8\"/>\nError 401 require
> authentication\n\nHTTP ERROR 401\nProblem
> accessing /solr/collection6/select. Reason:\nrequire
> authentication\n\n\n",
> "code":401}}
>
> This issue does not occur in Solr 7.6 and Solr 7.7, even though I have set
> up the same authentication for all the versions.
>
> What could be the issue that causes this?
>
> Regards,
> Edwin

JSON Facet query to retrieve count all collections in Solr 8.0.0

[Zheng Lin Edwin Yeo]

Hi,

I am using the below JSON Facet to retrieve the count of all the different
collections in one query.

https://localhost:8983/solr/collection1/select?q=testing=https://localhost:8983/solr/collection1,https://localhost:8983/solr/collection2,https://localhost:8983/solr/collection3,https://localhost:8983/solr/collection4,https://localhost:8983/solr/collection5,https://localhost:8983/solr/collection6=0={categories
: {type : terms,field : content_type,limit : 100}}


Previously, in Solr 7.6 and Solr 7.7, this query can work correctly and we
are able to produce the correct output.

{
  "responseHeader":{
"zkConnected":true,
"status":0,
"QTime":24},
  "response":{"numFound":41200,"start":0,"maxScore":12.993215,"docs":[]
  },
  "facets":{
"count":41200,
"categories":{
  "buckets":[{
  "val":"collection1",
  "count":26213},
{
  "val":"collection2",
  "count":12075},
{
  "val":"collection3",
  "count":1947},
{
  "val":"collection4",
  "count":850},
{
  "val":"collection5",
  "count":111},
{
  "val":"collection6",
  "count":4}]}}}


However, in the new Solr 8.0.0, this query can only work once.
Subsequently, we will get the following error of 'require authentication':

{
  "responseHeader":{
"zkConnected":true,
"status":401,
"QTime":11},
  "error":{
"metadata":[

"error-class","org.apache.solr.client.solrj.impl.Http2SolrClient$RemoteSolrException",

"root-error-class","org.apache.solr.client.solrj.impl.Http2SolrClient$RemoteSolrException"],
"msg":"Error from server at null: Expected mime type
application/octet-stream but got text/html. \n\n\nError 401 require
authentication\n\nHTTP ERROR 401\nProblem
accessing /solr/collection6/select. Reason:\nrequire
authentication\n\n\n",
"code":401}}

This issue does not occur in Solr 7.6 and Solr 7.7, even though I have set
up the same authentication for all the versions.

What could be the issue that causes this?

Regards,
Edwin

Re: SolrCloud become unresponsive after huge pivot facet query

[Shawn Heisey]

On 1/31/2019 12:11 PM, Ruchir Choudhry wrote:

Wanted to start working on Solr bugs, will appreciate if you or some can
allocate me with some minor bugs.


It doesn't work like that.  Issues are not handed out, it's a strictly 
volunteer system.


You'll need to find the issues you want to work on.  It's a good idea to 
check the issue to see if somebody else might be already working on it, 
and if not, add a comment saying you intend to work on it.


There are some labels that we try to give to issues that we think a new 
developer could tackle.  The following search URL should show you open 
issues tagged this way, sorted so the oldest ones are first:


https://issues.apache.org/jira/issues/?jql=project%20%3D%20SOLR%20AND%20labels%20in%20(newdev%2Cbeginner%2Cbeginners)%20AND%20status%20in%20(Open%2CReopened)%20ORDER%20BY%20key%20ASC%20

It's a big URL, and it might get mangled by either my mail client or the 
mailing list.  If that happens, hopefully you can reconstruct it.


You can also do manual searches for topics that interest you.

Thanks,
Shawn

Re: SolrCloud become unresponsive after huge pivot facet query

[Ruchir Choudhry]

Hello Erick,

Wanted to start working on Solr bugs, will appreciate if you or some can
allocate me with some minor bugs.

Warm Regards,
Ruchir




On Wed, Jan 30, 2019 at 8:53 AM Erick Erickson 
wrote:

> My suggestion is "don't do that" ;).
>
> Ok, seriously. Conceptually what you have is an N-dimnensional matrix.
> Each "dimension" is
> one of your pivot fields with one cell for each unique value in the
> field. So the size is
> (cardinality of field 1) x (cardinality of field 2) * (cardinality of
> field 3) .
>
> To make matters worse, the results from each shard need to be
> aggregated, so you're
> correct that you're shoving potentially a _very_ large set of data
> across your network that
> then has to be sorted into the final packet.
>
> You don't indicate that you have OOM errors so what I suspect is
> happening is you're
> in "GC hell". Each GC cycle recovers just enough memory to continue
> for a very short
> time, then stops for another GC cycle. Rinse, repeat. Timeout.
>
> For more concrete suggestions.
> 1> You can use the "Luke request handler" to find the cardinality of
> the fields and then
>  have a blacklist of fields so you wind up rejecting these queries up
> front.
>
> 2> Consider the streaming capabilities. "Rollup" can be used for
>  high cardinality fields.
>  see: https://lucene.apache.org/solr/guide/6_6/stream-decorators.html.
> NOTE:
> "Facet" streams push the faceting down to the replicas, which you don't
> want to use in this case as it'll be the same problem. The facet
> streams
> are faster when they can be used, but I doubt you can in your case.
> BTW, as chance would have it, Joel B. just explained this to me ;).
>
> Best,
> Erick
>
> On Wed, Jan 30, 2019 at 3:41 AM Matteo Diarena 
> wrote:
> >
> > Dear all,
> > we have a solrcloud cluster with the following features:
> > - 3 zookeeper nodes
> > - 4 solr nodes with:
> >- 4 CPU
> >- 16GB RAM
> >
> > Each solr instance is configured as follow:
> > SOLR_JAVA_MEM="-Xms2g -Xmx8g"
> > SOLR_OPTS="$SOLR_OPTS -Dlucene.cms.override_spins=false
> -Dlucene.cms.override_core_count=4"
> >
> > On the cluster we created a collection with 5 shards each with 2
> replicas for a total of 10 replicas.
> >
> > The full index size is less than 2 GB and under normal usage the used
> heap space is between 200MB and 500MB.
> >
> > Unfortunately if we try to perform a query like the this:
> >
> >
> .../select?q=*:*=ActionType:MAILOPENING=true=0=FIELD_ObjectId,FIELD_MailId_ObjectId.facet.pivot.mincount=0_ObjectId.facet.limit=-1_ObjectId.facet.pivot.mincount=0_ObjectId.facet.limit=-1
> >
> > where FIELD_ObjectId and FIELD_MailId are high cardinality fields all
> the heap space is used and the entire solr cluster becomes really slow and
> unresponsive.
> > The solr instance is not killed and the heap space is never released so
> the only way is to get the cluster up again is to restart all the solr
> instances.
> >
> > I know that the problem is the wrong query but I'd like to know how I
> can avoid this kind of problems.
> > Is there a way to limit the memory usage during query execution to avoid
> a single query to hang a cluster?
> >
> > I tried to disable all caches and to investigate the heap dump but I
> didn't manage to find any good solution.
> > I also thought that an issue could be the really big search response
> exchange between shards. Is it possible?
> >
> > Actually the cluster is not in production so I can easily perform tests
> or get all needed data.
> >
> > Any suggestion is welcome.
> >
> > Thanks a lot,
> > Matteo Diarena
> > Direttore Innovazione
> >
> > Volo.com S.r.l. (www.volocom.it - volo...@pec.it
> )
> > Via Luigi Rizzo, 8/1 - 20151 MILANO
> > Via Leone XIII, 95 - 00165 ROMA
> >
> > Tel +39 02 89453024 / +39 02 89453023
> > Fax +39 02 89453500
> > Mobile +39 345 2129244
> > m.diar...@volocom.it
> >
>

R: SolrCloud become unresponsive after huge pivot facet query

[Matteo Diarena]

Hi Erick,
first of all thanks a lot for your response! 

I suppose that in my case is happening exactly what you describe as "GC Hell" 
because I see continuous GC cycles and solr is not showing OOM errors.

I absolutely agree with you that this is a bad query but I was wondering if 
there is any setting (both solr side or jvm side) to avoid that a single query 
can tear down an entire cluster without the possibility of an automatic 
recovery.
I will implement in our APIs a check on luke request handler to prevent the 
execution of expensive queries.

Anyway, I my opinion it could be useful a parameter like a timeAllowed working 
on the whole query process or even better something like a memoryAllowed 
parameter to kill a query if it's too expensive in terms of timing or memory 
consumption.

I'll study a little bit better streaming capabilities, it seems really powerful.

Best regards,
Matteo Diarena
Direttore Innovazione

Volo.com S.r.l. (www.volocom.it - volo...@pec.it)
Via Luigi Rizzo, 8/1 - 20151 MILANO 
Via Leone XIII, 95 - 00165 ROMA

Tel +39 02 89453024 / +39 02 89453023
Fax +39 02 89453500
Mobile +39 345 2129244
m.diar...@volocom.it

-Messaggio originale-
Da: Erick Erickson  
Inviato: 30 January 2019 17:44
A: solr-user 
Oggetto: Re: SolrCloud become unresponsive after huge pivot facet query

My suggestion is "don't do that" ;).

Ok, seriously. Conceptually what you have is an N-dimnensional matrix.
Each "dimension" is
one of your pivot fields with one cell for each unique value in the field. So 
the size is (cardinality of field 1) x (cardinality of field 2) * (cardinality 
of field 3) .

To make matters worse, the results from each shard need to be aggregated, so 
you're correct that you're shoving potentially a _very_ large set of data 
across your network that then has to be sorted into the final packet.

You don't indicate that you have OOM errors so what I suspect is happening is 
you're in "GC hell". Each GC cycle recovers just enough memory to continue for 
a very short time, then stops for another GC cycle. Rinse, repeat. Timeout.

For more concrete suggestions.
1> You can use the "Luke request handler" to find the cardinality of
the fields and then
 have a blacklist of fields so you wind up rejecting these queries up front.

2> Consider the streaming capabilities. "Rollup" can be used for
 high cardinality fields.
 see: https://lucene.apache.org/solr/guide/6_6/stream-decorators.html.
NOTE:
"Facet" streams push the faceting down to the replicas, which you don't
want to use in this case as it'll be the same problem. The facet streams
are faster when they can be used, but I doubt you can in your case.
BTW, as chance would have it, Joel B. just explained this to me ;).

Best,
Erick

On Wed, Jan 30, 2019 at 3:41 AM Matteo Diarena  wrote:
>
> Dear all,
> we have a solrcloud cluster with the following features:
> - 3 zookeeper nodes
> - 4 solr nodes with:
>- 4 CPU
>- 16GB RAM
>
> Each solr instance is configured as follow:
> SOLR_JAVA_MEM="-Xms2g -Xmx8g"
> SOLR_OPTS="$SOLR_OPTS -Dlucene.cms.override_spins=false 
> -Dlucene.cms.override_core_count=4"
>
> On the cluster we created a collection with 5 shards each with 2 replicas for 
> a total of 10 replicas.
>
> The full index size is less than 2 GB and under normal usage the used heap 
> space is between 200MB and 500MB.
>
> Unfortunately if we try to perform a query like the this:
>
> .../select?q=*:*=ActionType:MAILOPENING=true=0
> ot=FIELD_ObjectId,FIELD_MailId_ObjectId.facet.pivot.mincount=0
> _ObjectId.facet.limit=-1_ObjectId.facet.pivot.mincount
> =0_ObjectId.facet.limit=-1
>
> where FIELD_ObjectId and FIELD_MailId are high cardinality fields all the 
> heap space is used and the entire solr cluster becomes really slow and 
> unresponsive.
> The solr instance is not killed and the heap space is never released so the 
> only way is to get the cluster up again is to restart all the solr instances.
>
> I know that the problem is the wrong query but I'd like to know how I can 
> avoid this kind of problems.
> Is there a way to limit the memory usage during query execution to avoid a 
> single query to hang a cluster?
>
> I tried to disable all caches and to investigate the heap dump but I didn't 
> manage to find any good solution.
> I also thought that an issue could be the really big search response exchange 
> between shards. Is it possible?
>
> Actually the cluster is not in production so I can easily perform tests or 
> get all needed data.
>
> Any suggestion is welcome.
>
> Thanks a lot,
> Matteo Diarena
> Direttore Innovazione
>
> Volo.com S.r.l. (

Re: SolrCloud become unresponsive after huge pivot facet query

[Erick Erickson]

My suggestion is "don't do that" ;).

Ok, seriously. Conceptually what you have is an N-dimnensional matrix.
Each "dimension" is
one of your pivot fields with one cell for each unique value in the
field. So the size is
(cardinality of field 1) x (cardinality of field 2) * (cardinality of
field 3) .

To make matters worse, the results from each shard need to be
aggregated, so you're
correct that you're shoving potentially a _very_ large set of data
across your network that
then has to be sorted into the final packet.

You don't indicate that you have OOM errors so what I suspect is
happening is you're
in "GC hell". Each GC cycle recovers just enough memory to continue
for a very short
time, then stops for another GC cycle. Rinse, repeat. Timeout.

For more concrete suggestions.
1> You can use the "Luke request handler" to find the cardinality of
the fields and then
 have a blacklist of fields so you wind up rejecting these queries up front.

2> Consider the streaming capabilities. "Rollup" can be used for
 high cardinality fields.
 see: https://lucene.apache.org/solr/guide/6_6/stream-decorators.html.
NOTE:
"Facet" streams push the faceting down to the replicas, which you don't
want to use in this case as it'll be the same problem. The facet streams
are faster when they can be used, but I doubt you can in your case.
BTW, as chance would have it, Joel B. just explained this to me ;).

Best,
Erick

On Wed, Jan 30, 2019 at 3:41 AM Matteo Diarena  wrote:
>
> Dear all,
> we have a solrcloud cluster with the following features:
> - 3 zookeeper nodes
> - 4 solr nodes with:
>- 4 CPU
>- 16GB RAM
>
> Each solr instance is configured as follow:
> SOLR_JAVA_MEM="-Xms2g -Xmx8g"
> SOLR_OPTS="$SOLR_OPTS -Dlucene.cms.override_spins=false 
> -Dlucene.cms.override_core_count=4"
>
> On the cluster we created a collection with 5 shards each with 2 replicas for 
> a total of 10 replicas.
>
> The full index size is less than 2 GB and under normal usage the used heap 
> space is between 200MB and 500MB.
>
> Unfortunately if we try to perform a query like the this:
>
> .../select?q=*:*=ActionType:MAILOPENING=true=0=FIELD_ObjectId,FIELD_MailId_ObjectId.facet.pivot.mincount=0_ObjectId.facet.limit=-1_ObjectId.facet.pivot.mincount=0_ObjectId.facet.limit=-1
>
> where FIELD_ObjectId and FIELD_MailId are high cardinality fields all the 
> heap space is used and the entire solr cluster becomes really slow and 
> unresponsive.
> The solr instance is not killed and the heap space is never released so the 
> only way is to get the cluster up again is to restart all the solr instances.
>
> I know that the problem is the wrong query but I'd like to know how I can 
> avoid this kind of problems.
> Is there a way to limit the memory usage during query execution to avoid a 
> single query to hang a cluster?
>
> I tried to disable all caches and to investigate the heap dump but I didn't 
> manage to find any good solution.
> I also thought that an issue could be the really big search response exchange 
> between shards. Is it possible?
>
> Actually the cluster is not in production so I can easily perform tests or 
> get all needed data.
>
> Any suggestion is welcome.
>
> Thanks a lot,
> Matteo Diarena
> Direttore Innovazione
>
> Volo.com S.r.l. (www.volocom.it - 
> volo...@pec.it)
> Via Luigi Rizzo, 8/1 - 20151 MILANO
> Via Leone XIII, 95 - 00165 ROMA
>
> Tel +39 02 89453024 / +39 02 89453023
> Fax +39 02 89453500
> Mobile +39 345 2129244
> m.diar...@volocom.it
>

SolrCloud become unresponsive after huge pivot facet query

[Matteo Diarena]

Dear all,
we have a solrcloud cluster with the following features:
- 3 zookeeper nodes
- 4 solr nodes with:
   - 4 CPU
   - 16GB RAM

Each solr instance is configured as follow:
SOLR_JAVA_MEM="-Xms2g -Xmx8g"
SOLR_OPTS="$SOLR_OPTS -Dlucene.cms.override_spins=false 
-Dlucene.cms.override_core_count=4"

On the cluster we created a collection with 5 shards each with 2 replicas for a 
total of 10 replicas.

The full index size is less than 2 GB and under normal usage the used heap 
space is between 200MB and 500MB.

Unfortunately if we try to perform a query like the this:

.../select?q=*:*=ActionType:MAILOPENING=true=0=FIELD_ObjectId,FIELD_MailId_ObjectId.facet.pivot.mincount=0_ObjectId.facet.limit=-1_ObjectId.facet.pivot.mincount=0_ObjectId.facet.limit=-1

where FIELD_ObjectId and FIELD_MailId are high cardinality fields all the heap 
space is used and the entire solr cluster becomes really slow and unresponsive.
The solr instance is not killed and the heap space is never released so the 
only way is to get the cluster up again is to restart all the solr instances.

I know that the problem is the wrong query but I'd like to know how I can avoid 
this kind of problems.
Is there a way to limit the memory usage during query execution to avoid a 
single query to hang a cluster?

I tried to disable all caches and to investigate the heap dump but I didn't 
manage to find any good solution.
I also thought that an issue could be the really big search response exchange 
between shards. Is it possible?

Actually the cluster is not in production so I can easily perform tests or get 
all needed data.

Any suggestion is welcome.

Thanks a lot,
Matteo Diarena
Direttore Innovazione

Volo.com S.r.l. (www.volocom.it - 
volo...@pec.it)
Via Luigi Rizzo, 8/1 - 20151 MILANO
Via Leone XIII, 95 - 00165 ROMA

Tel +39 02 89453024 / +39 02 89453023
Fax +39 02 89453500
Mobile +39 345 2129244
m.diar...@volocom.it

Re: Json Facet Query Stripping Field Name with Hyphen

[RAUNAK AGRAWAL]

Hi Erick/Yonik,

Thank you guys. I am going to rename the fields.

On Thu, Jan 4, 2018 at 10:04 PM, Yonik Seeley  wrote:

> The JSON Facet API uses the function query parser for something like
> sum(week_-91) so you'll probably have problems with any function that
> uses these fields as well.
> As Erick says, you're better off renaming the fields.  There is a
> workaround for wonky field names via the "field" function:
> sum(field(week_-91))
>
> -Yonik
>
>
> On Thu, Jan 4, 2018 at 10:02 AM, RAUNAK AGRAWAL
>  wrote:
> > Hi Guys,
> >
> > I am facing issue where I am trying to follow the JSON facet API. I have
> > data in my collection and field names are like "week_0", "week_-1" which
> > means current week and previous week respectively.
> >
> > When I am querying for week_0 summation using the following query I am
> able
> > to get the result.
> >
> > http://localhost:8983/solr/collection1/query?q=*:*.
> facet={week_0_sum:'sum(week_0)'}=0
> >
> >
> > But when I am trying to do the same for any field "week_-*", it is break.
> >
> > For example when I am trying:
> > http://localhost:8983/solr/collection1/query?q=*:*.
> facet={week_-91_sum:%27sum(week_-91)%27}=0
> >
> >
> > I am getting the exception as* "msg": "undefined field: \"week_\"''*
> >
> >
> > That means solr is stripping field name after hyphen (-). Is there
> > workaround to fix this. I tried adding escape character (\) but it is of
> no
> > help.
> >
> > With escape:
> > http://localhost:8983/solr/collection1/query?q=*:*.
> facet={week_-91_sum:%27sum(week_\-91)%27}=0
> >
> >
> > Please help me regarding this.
> >
> > Thanks
>

Re: Json Facet Query Stripping Field Name with Hyphen

[Yonik Seeley]

The JSON Facet API uses the function query parser for something like
sum(week_-91) so you'll probably have problems with any function that
uses these fields as well.
As Erick says, you're better off renaming the fields.  There is a
workaround for wonky field names via the "field" function:
sum(field(week_-91))

-Yonik


On Thu, Jan 4, 2018 at 10:02 AM, RAUNAK AGRAWAL
 wrote:
> Hi Guys,
>
> I am facing issue where I am trying to follow the JSON facet API. I have
> data in my collection and field names are like "week_0", "week_-1" which
> means current week and previous week respectively.
>
> When I am querying for week_0 summation using the following query I am able
> to get the result.
>
> http://localhost:8983/solr/collection1/query?q=*:*={week_0_sum:'sum(week_0)'}=0
>
>
> But when I am trying to do the same for any field "week_-*", it is break.
>
> For example when I am trying:
> http://localhost:8983/solr/collection1/query?q=*:*={week_-91_sum:%27sum(week_-91)%27}=0
>
>
> I am getting the exception as* "msg": "undefined field: \"week_\"''*
>
>
> That means solr is stripping field name after hyphen (-). Is there
> workaround to fix this. I tried adding escape character (\) but it is of no
> help.
>
> With escape:
> http://localhost:8983/solr/collection1/query?q=*:*={week_-91_sum:%27sum(week_\-91)%27}=0
>
>
> Please help me regarding this.
>
> Thanks

Re: Json Facet Query Stripping Field Name with Hyphen

[Erick Erickson]

>From the ref guide:

"Field names should consist of alphanumeric or underscore characters
only and not start with a digit."

While field naming isn't strictly enforced, having field names like
week_-1 is also not guaranteed to be supported. You should change your
field name.

I raised SOLR-11819 for one place I see in the ref guide where a
hyphen is used, if you see any others please add the location to the
JIRA (SOLR-11819).

Best,
Erick

On Thu, Jan 4, 2018 at 7:02 AM, RAUNAK AGRAWAL  wrote:
> Hi Guys,
>
> I am facing issue where I am trying to follow the JSON facet API. I have
> data in my collection and field names are like "week_0", "week_-1" which
> means current week and previous week respectively.
>
> When I am querying for week_0 summation using the following query I am able
> to get the result.
>
> http://localhost:8983/solr/collection1/query?q=*:*={week_0_sum:'sum(week_0)'}=0
>
>
> But when I am trying to do the same for any field "week_-*", it is break.
>
> For example when I am trying:
> http://localhost:8983/solr/collection1/query?q=*:*={week_-91_sum:%27sum(week_-91)%27}=0
>
>
> I am getting the exception as* "msg": "undefined field: \"week_\"''*
>
>
> That means solr is stripping field name after hyphen (-). Is there
> workaround to fix this. I tried adding escape character (\) but it is of no
> help.
>
> With escape:
> http://localhost:8983/solr/collection1/query?q=*:*={week_-91_sum:%27sum(week_\-91)%27}=0
>
>
> Please help me regarding this.
>
> Thanks

Json Facet Query Stripping Field Name with Hyphen

[RAUNAK AGRAWAL]

Hi Guys,

I am facing issue where I am trying to follow the JSON facet API. I have
data in my collection and field names are like "week_0", "week_-1" which
means current week and previous week respectively.

When I am querying for week_0 summation using the following query I am able
to get the result.

http://localhost:8983/solr/collection1/query?q=*:*={week_0_sum:'sum(week_0)'}=0


But when I am trying to do the same for any field "week_-*", it is break.

For example when I am trying:
http://localhost:8983/solr/collection1/query?q=*:*={week_-91_sum:%27sum(week_-91)%27}=0


I am getting the exception as* "msg": "undefined field: \"week_\"''*


That means solr is stripping field name after hyphen (-). Is there
workaround to fix this. I tried adding escape character (\) but it is of no
help.

With escape:
http://localhost:8983/solr/collection1/query?q=*:*={week_-91_sum:%27sum(week_\-91)%27}=0


Please help me regarding this.

Thanks

Re: facet query when using word 'AND'

[Erick Erickson]

I typed the below then noticed that the field that has "I have a pen
AND an apple" is called "body" and you're faceting and searching on a
field called "suggest". The below is still relevant if there is still
a problem though ;)

Your problem isn't faceting, right? It's that you aren't matching any
docs at all when you expect to. Once we solve that then we can move on
to faceting ;)

First, I'd add =query and see what the parsed query looks like.
Does it fit your expectations? See "parsedQuery_toString" in the debug
output.

Next, use the admin/analysis page to see how the input is analyzed at
index and query time. The first thing to verify is that the "and" is
not being removed since it's often considered a stopword. Do you get
facets when searching for "apple" (no quotes)? If you do is "and"
included as a facet? If not then you need to modify the stopwords
associated with the field in your schema.

You can use the admin//schema browser to see the actual
terms in the index that'll help you verify that the _tokens_ in the
index include what you expect. Faceting works on indexed tokens BTW.

This is different than returning the field, since when you specify
fl=suggest you're getting back the _stored_ values, not the searchable
indexed tokens. This fools everybody at first. Stored values are a
verbatim copy of the input, no analysis at all. However, what you can
search/facet/etc. on is the tokenized version, which are _not_
returned via the "fl" field.

Best,
Erick

P.S. My compliments for including enough data to offer a diagnosis the
first time. It's refreshing not to have to go through 3 or 4 exchanges
before having enough information to say anything even potentially
useful.



On Wed, Sep 6, 2017 at 6:13 AM, Shawn Heisey <apa...@elyograg.org> wrote:
> On 9/6/2017 3:48 AM, Noriyuki TAKEI wrote:
>> I use facet query,but I found it dose not work when using 'AND'.
>>
>> I woud like to use facet query using 'AND' as not Operator but simple word.
>
> With the standard or edismax parser, AND in all uppercase is interpreted
> as an operator.  There are two ways to deal with this.  One is to change
> the word to lowercase, which might not do what you want depending on
> your text analysis, the other is to escape part of it -- use A\ND
> instead of AND.  I have verified that escaping one of the letters in the
> word *does* work.
>
> If you're using the edismax query parser and the lowercase option, be
> sure that the lowercaseOperators parameter is set to false.  The default
> setting depends on luceneMatchVersion.  It's false when that's 7.0.0 or
> later, true if it's lower.  Which means that until version 7.0 is
> released, it will default to true.
>
> https://issues.apache.org/jira/browse/SOLR-4646
>
> Thanks,
> Shawn
>

Re: facet query when using word 'AND'

[Shawn Heisey]

On 9/6/2017 3:48 AM, Noriyuki TAKEI wrote:
> I use facet query,but I found it dose not work when using 'AND'.
>
> I woud like to use facet query using 'AND' as not Operator but simple word.

With the standard or edismax parser, AND in all uppercase is interpreted
as an operator.  There are two ways to deal with this.  One is to change
the word to lowercase, which might not do what you want depending on
your text analysis, the other is to escape part of it -- use A\ND
instead of AND.  I have verified that escaping one of the letters in the
word *does* work.

If you're using the edismax query parser and the lowercase option, be
sure that the lowercaseOperators parameter is set to false.  The default
setting depends on luceneMatchVersion.  It's false when that's 7.0.0 or
later, true if it's lower.  Which means that until version 7.0 is
released, it will default to true.

https://issues.apache.org/jira/browse/SOLR-4646

Thanks,
Shawn

facet query when using word 'AND'

[Noriyuki TAKEI]

Hi,all

I use facet query,but I found it dose not work when using 'AND'.

I woud like to use facet query using 'AND' as not Operator but simple word.

At first,Solr Config is as below.



  suggest_dict
  solr.Suggester
  AnalyzingLookupFactory
  suggest
  suggest_ja
  true
  true
  text_ja_romaji

  

  

  suggest
  AND
  0
  true

  true
  suggest
  1000
  1

  true
  suggest_dict
  10
  true
  30
  10
  true


  suggest_ja

  

I executed the query below,but Solr gave unexpected result.

$ curl
"http://localhost:8983/solr/kms/suggest_ja?wt=json=true=\"AND\"=true;
{
  "response":{"numFound":0,"start":0,"maxScore":0.0,"docs":[]
  },
  "facet_counts":{
"facet_queries":{},
"facet_fields":{
  "suggest":[]},
"facet_ranges":{},
"facet_intervals":{},
"facet_heatmaps":{}},
  "spellcheck":{
"suggestions":[],
"collations":[]}}

I'd like to use facet search including the word "AND",so I surrounded "AND"
by
double quotes and then  appended the escappe parameter befoe
dobule quote as below.

\"AND\"

The Document included word "AND"(I have a pen AND an apple) is already
indexed.The evidence is as below.

$ curl "http://localhost:8983/solr/kms/select?wt=json=true=\"AND\";

{

  "responseHeader":{
"zkConnected":true,
"status":0,
"QTime":9,
"params":{
  "q":"\"AND\"",
  "indent":"true",
  "wt":"json"}},
  "response":{"numFound":1,"start":0,"maxScore":0.2770272,"docs":[
  {
"pub_date":"2017-03-09T12:34:56.789Z",
"body":"I have a pen AND an apple",
"title":"test",
"url":"http://10.16.44.180:8080/#/management/two/;,
"system_cd":"hoge",
"document_id":"001",
"id":"hoge001",
"content_type":"doc",
"_version_":157862221496320}]
  }}


Therefore I expected the result as below.

{
  "response":{"numFound":1,"start":0,"maxScore":0.66747504,"docs":[]
  },
  "facet_counts":{
"facet_queries":{},
"facet_fields":{
  "suggest":[
"AND",1,
"apple",1,
"pen",1,
]},
"facet_ranges":{},
"facet_intervals":{},
"facet_heatmaps":{}},
  "spellcheck":{
"suggestions":[],
"collations":[]}}


Actually facet fields includes nothing.

How do I solve this?



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Filter Facet Query

[Furkan KAMACI]

Hi Alex,

I found the reason, thanks for the help. Facet shows all possible values
including 0.

Could you help on my last question:

I have facet results like:

"", 9
"research",6
"development",3


I want to filter empty string from my facet "" (I don't want to add it to
fq, just filter from facets). How can I do that?

On Tue, Apr 18, 2017 at 11:52 AM, Alexandre Rafalovitch 
wrote:

> Are you saying that all the values in the facet are zero with that
> query? The query you gave seems to be the super-basic faceting code,
> so maybe something super-basic is missing.
>
> E.g.
> *) Did you check that the documents you get back actually have any
> values in that field to facet on?
> *) Did you try making a query just by ID for a document that
> definitely has the value in that field?
> *) Did you do the query with echoParams=all to see that you are not
> having any hidden extra parameters that get appended?
>
> Regards,
>Alex.
>
> 
> http://www.solr-start.com/ - Resources for Solr users, new and experienced
>
>
> On 18 April 2017 at 11:43, Furkan KAMACI  wrote:
> > OK, it returns 0 results every time.
> >
> > So,
> >
> > I want to filter out research values with empty string ("") from facet
> > result. How can I do that?
> >
> >
> > On Tue, Apr 18, 2017 at 8:53 AM, Furkan KAMACI 
> > wrote:
> >
> >> First problem is they do not match with main query.
> >>
> >> 18 Nis 2017 Sal, saat 01:54 tarihinde Dave <
> hastings.recurs...@gmail.com>
> >> şunu yazdı:
> >>
> >>> Min.count is what you're looking for to get non 0 facets
> >>>
> >>> > On Apr 17, 2017, at 6:51 PM, Furkan KAMACI 
> >>> wrote:
> >>> >
> >>> > My query:
> >>> >
> >>> > /select?facet.field=research=on=content:test
> >>> >
> >>> > Q1) Facet returns research values with 0 counts which has a research
> >>> value
> >>> > that is not from a document matched by main query (content:test). Is
> >>> that
> >>> > usual?
> >>> >
> >>> > Q2) I want to filter out research values with empty string ("") from
> >>> facet
> >>> > result. How can I do that?
> >>> >
> >>> > Kind Regards,
> >>> > Furkan KAMACI
> >>>
> >>
>

Re: Filter Facet Query

[Alexandre Rafalovitch]

Are you saying that all the values in the facet are zero with that
query? The query you gave seems to be the super-basic faceting code,
so maybe something super-basic is missing.

E.g.
*) Did you check that the documents you get back actually have any
values in that field to facet on?
*) Did you try making a query just by ID for a document that
definitely has the value in that field?
*) Did you do the query with echoParams=all to see that you are not
having any hidden extra parameters that get appended?

Regards,
   Alex.


http://www.solr-start.com/ - Resources for Solr users, new and experienced


On 18 April 2017 at 11:43, Furkan KAMACI  wrote:
> OK, it returns 0 results every time.
>
> So,
>
> I want to filter out research values with empty string ("") from facet
> result. How can I do that?
>
>
> On Tue, Apr 18, 2017 at 8:53 AM, Furkan KAMACI 
> wrote:
>
>> First problem is they do not match with main query.
>>
>> 18 Nis 2017 Sal, saat 01:54 tarihinde Dave 
>> şunu yazdı:
>>
>>> Min.count is what you're looking for to get non 0 facets
>>>
>>> > On Apr 17, 2017, at 6:51 PM, Furkan KAMACI 
>>> wrote:
>>> >
>>> > My query:
>>> >
>>> > /select?facet.field=research=on=content:test
>>> >
>>> > Q1) Facet returns research values with 0 counts which has a research
>>> value
>>> > that is not from a document matched by main query (content:test). Is
>>> that
>>> > usual?
>>> >
>>> > Q2) I want to filter out research values with empty string ("") from
>>> facet
>>> > result. How can I do that?
>>> >
>>> > Kind Regards,
>>> > Furkan KAMACI
>>>
>>

Re: Filter Facet Query

[Furkan KAMACI]

OK, it returns 0 results every time.

So,

I want to filter out research values with empty string ("") from facet
result. How can I do that?


On Tue, Apr 18, 2017 at 8:53 AM, Furkan KAMACI 
wrote:

> First problem is they do not match with main query.
>
> 18 Nis 2017 Sal, saat 01:54 tarihinde Dave 
> şunu yazdı:
>
>> Min.count is what you're looking for to get non 0 facets
>>
>> > On Apr 17, 2017, at 6:51 PM, Furkan KAMACI 
>> wrote:
>> >
>> > My query:
>> >
>> > /select?facet.field=research=on=content:test
>> >
>> > Q1) Facet returns research values with 0 counts which has a research
>> value
>> > that is not from a document matched by main query (content:test). Is
>> that
>> > usual?
>> >
>> > Q2) I want to filter out research values with empty string ("") from
>> facet
>> > result. How can I do that?
>> >
>> > Kind Regards,
>> > Furkan KAMACI
>>
>

Re: Filter Facet Query

[Furkan KAMACI]

First problem is they do not match with main query.

18 Nis 2017 Sal, saat 01:54 tarihinde Dave 
şunu yazdı:

> Min.count is what you're looking for to get non 0 facets
>
> > On Apr 17, 2017, at 6:51 PM, Furkan KAMACI 
> wrote:
> >
> > My query:
> >
> > /select?facet.field=research=on=content:test
> >
> > Q1) Facet returns research values with 0 counts which has a research
> value
> > that is not from a document matched by main query (content:test). Is that
> > usual?
> >
> > Q2) I want to filter out research values with empty string ("") from
> facet
> > result. How can I do that?
> >
> > Kind Regards,
> > Furkan KAMACI
>

Re: Filter Facet Query

[Dave]

Min.count is what you're looking for to get non 0 facets

> On Apr 17, 2017, at 6:51 PM, Furkan KAMACI  wrote:
> 
> My query:
> 
> /select?facet.field=research=on=content:test
> 
> Q1) Facet returns research values with 0 counts which has a research value
> that is not from a document matched by main query (content:test). Is that
> usual?
> 
> Q2) I want to filter out research values with empty string ("") from facet
> result. How can I do that?
> 
> Kind Regards,
> Furkan KAMACI

Filter Facet Query

[Furkan KAMACI]

My query:

/select?facet.field=research=on=content:test

Q1) Facet returns research values with 0 counts which has a research value
that is not from a document matched by main query (content:test). Is that
usual?

Q2) I want to filter out research values with empty string ("") from facet
result. How can I do that?

Kind Regards,
Furkan KAMACI

Re: Facet query - exlude main query

[Jacques du Rand]

Oh right right ! Sorry late night :)
Thank You Chris


On 21 February 2017 at 20:30, Chris Hostetter 
wrote:

>
> : Maybe I'm doing something wrong ?
> : /select?q.op=OR=2={!tag=mq}nissan=name%20name_
> raw=json=0=20={!ex=tag_mq}feature_s_1_make
>
> that url still contains "ex=tag_mq" .. which is looking for a query with a
> tag named "tag_mq" .. in your q param you are using a tag named "mq"
>
> Use "facet.field={!ex=mq}feature_s_1_make" as i mentioned before
>
>
> >>> You are attempting to exclude tags named "tag_make", "tag_model", and
> >>> "tag_mq" -- but the name of the tag you are using in the query is "mq"
> >>>
> >>> if you include "mq" in the list of excluded tags (the "ex" local
> >>> param) it should work fine...
> >>>
> >>> facet.field={!ex=mq}feature_s_1_make"
>
>
>
>
> -Hoss
> http://www.lucidworks.com/
>



-- 
Jacques du Rand
Senior R  Programmer

T: +27214688017
F: +27862160617
E: jacq...@pricecheck.co.za

Re: Facet query - exlude main query

[Chris Hostetter]

: Maybe I'm doing something wrong ?
: 
/select?q.op=OR=2={!tag=mq}nissan=name%20name_raw=json=0=20={!ex=tag_mq}feature_s_1_make

that url still contains "ex=tag_mq" .. which is looking for a query with a 
tag named "tag_mq" .. in your q param you are using a tag named "mq"

Use "facet.field={!ex=mq}feature_s_1_make" as i mentioned before


>>> You are attempting to exclude tags named "tag_make", "tag_model", and 
>>> "tag_mq" -- but the name of the tag you are using in the query is "mq"
>>>
>>> if you include "mq" in the list of excluded tags (the "ex" local 
>>> param) it should work fine...
>>> 
>>> facet.field={!ex=mq}feature_s_1_make"




-Hoss
http://www.lucidworks.com/

Re: Facet query - exlude main query

[Jacques du Rand]

Maybe I'm doing something wrong ?
/select?q.op=OR=2={!tag=mq}nissan=name%20name_raw=json=0=20={!ex=tag_mq}feature_s_1_make

Still only getting ONE facet value ?


{
status: 0,
QTime: 5,
params: {
mm: "2",
facet.field: [
"{!ex=tag_mq}feature_s_1_make",
"{!ex=tag_model}feature_s_2_model"
],
qs: "5",
components: [
"query",
"stats",
"facet",
"debug"
],
df: "text",
ps: "5",
echoParams: "all",
start: "0",
q.op: "OR",
rows: "20",
q: "{!tag=mq}nissan",
tie: "0.1",
facet.limit: "200",
defType: "edismax",
qf: "name name_raw",
facet.method: "fcs",
facet.threads: "6",
facet.mincount: "1",
pf3: "",
wt: "json",
facet: "true",
tr: "example.xsl"
}
},
response: {
numFound: 55,
start: 0,
docs: []
},
facet_counts: {
facet_queries: { },
facet_fields: {
feature_s_1_make: [
"nissan",
55
],

On 21 February 2017 at 19:55, Jacques du Rand 
wrote:

> Sure..
>
> So this is for a car search-application
>
>
> If i change:
>
> q={!tag=mq}nissan
>
> to
>
> q=*:*
>
> I get all the makes.
>
> VAR ECHO:
>
> responseHeader: {
> status: 0,
> QTime: 4,
> params: {
> mm: "2",
> facet.field: [
> "{!ex=tag_make,tag_model,tag_mq}feature_s_1_make",
> "{!ex=tag_model}feature_s_2_model"
>
> ],
> qs: "5",
> components: [
> "query",
> "stats",
> "facet",
> "debug"
> ],
> df: "text",
> ps: "5",
> echoParams: "all",
> start: "0",
> q.op: "OR",
> rows: "20",
> q: "{!tag=mq}nissan",
> tie: "0.1",
> facet.limit: "200",
> defType: "edismax",
> qf: "name name_raw ",
> facet.method: "fcs",
> facet.threads: "6",
> facet.mincount: "1",
> pf3: "",
> wt: "json",
> facet: "true",
> tr: "example.xsl"
>
>
> --->RESPONSE - expecting ALL car makes...
> facet_fields: {
> feature_s_1_make: [
> "nissan",
> 55
> ],
>
>
> On 21 February 2017 at 19:36, Chris Hostetter 
> wrote:
>
>>
>> :  Solr3.1  Starting with Solr
>> 3.1, the
>>
>> : primary relevance query (i.e. the one normally specified by the *q*
>> parameter)
>> : may also be excluded.
>> :
>> : But doesnt show me how to exlude it ??
>>
>> same tag + ex local params, as you had in your example...
>>
>> : 2. q={!tag=mq}blah={!ex=tag_man,mq}manufacturer
>> ...
>> : But  I still only get the facets WITH the main query  of "blah"
>>
>> Can you give us a more more complete picture of what your data & full
>> request & response looks like? ... it's working fine for me with Solr 6
>> and the techproducts sample data (see below).
>>
>> https://wiki.apache.org/solr/UsingMailingLists
>>
>>
>> $ curl 'http://localhost:8983/solr/techproducts/query?rows=0=%7B
>> !tag=y%7dinStock:true=%7B!tag=x%7dname:ipod=true
>> et.field=inStock=%7b!key=exclude_x+ex=x%7dinStoc
>> k=%7b!key=exclude_xy+ex=x,y%7dinStock'
>> {
>>   "responseHeader":{
>> "status":0,
>> "QTime":2,
>> "params":{
>>   "q":"{!tag=x}name:ipod",
>>   "facet.field":["inStock",
>> "{!key=exclude_x ex=x}inStock",
>> "{!key=exclude_xy ex=x,y}inStock"],
>>   "fq":"{!tag=y}inStock:true",
>>   "rows":"0",
>>   "facet":"true"}},
>>   "response":{"numFound":1,"start":0,"docs":[]
>>   },
>>   "facet_counts":{
>> "facet_queries":{},
>> "facet_fields":{
>>   "inStock":[
>> "true",1,
>> "false",0],
>>   "exclude_x":[
>> "true",17,
>> "false",0],
>>   "exclude_xy":[
>> "true",17,
>> "false",4]},
>> "facet_ranges":{},
>> "facet_intervals":{},
>> "facet_heatmaps":{}}}
>>
>>
>>
>>
>>
>> -Hoss
>> http://www.lucidworks.com/
>>
>
>
>
> --
> Jacques du Rand
> Senior R  Programmer
>
> T: +27214688017
> F: +27862160617
> E: jacq...@pricecheck.co.za
> 
>



-- 
Jacques du Rand
Senior R  Programmer

T: +27214688017
F: +27862160617
E: jacq...@pricecheck.co.za

Re: Facet query - exlude main query

[Chris Hostetter]

: facet.field: [
: "{!ex=tag_make,tag_model,tag_mq}feature_s_1_make",
: "{!ex=tag_model}feature_s_2_model"
...
: q: "{!tag=mq}nissan",

You are attempting to exclude tags named "tag_make", "tag_model", and 
"tag_mq" -- but the name of the tag you are using in the query is "mq"

if you include "mq" in the list of excluded tags (the "ex" local param) it 
should work fine...

facet.field={!ex=mq}feature_s_1_make"



-Hoss
http://www.lucidworks.com/

Re: Facet query - exlude main query

[Jacques du Rand]

Sure..

So this is for a car search-application


If i change:

q={!tag=mq}nissan

to

q=*:*

I get all the makes.

VAR ECHO:

responseHeader: {
status: 0,
QTime: 4,
params: {
mm: "2",
facet.field: [
"{!ex=tag_make,tag_model,tag_mq}feature_s_1_make",
"{!ex=tag_model}feature_s_2_model"

],
qs: "5",
components: [
"query",
"stats",
"facet",
"debug"
],
df: "text",
ps: "5",
echoParams: "all",
start: "0",
q.op: "OR",
rows: "20",
q: "{!tag=mq}nissan",
tie: "0.1",
facet.limit: "200",
defType: "edismax",
qf: "name name_raw ",
facet.method: "fcs",
facet.threads: "6",
facet.mincount: "1",
pf3: "",
wt: "json",
facet: "true",
tr: "example.xsl"


--->RESPONSE - expecting ALL car makes...
facet_fields: {
feature_s_1_make: [
"nissan",
55
],


On 21 February 2017 at 19:36, Chris Hostetter 
wrote:

>
> :  Solr3.1  Starting with Solr 3.1,
> the
> : primary relevance query (i.e. the one normally specified by the *q*
> parameter)
> : may also be excluded.
> :
> : But doesnt show me how to exlude it ??
>
> same tag + ex local params, as you had in your example...
>
> : 2. q={!tag=mq}blah={!ex=tag_man,mq}manufacturer
> ...
> : But  I still only get the facets WITH the main query  of "blah"
>
> Can you give us a more more complete picture of what your data & full
> request & response looks like? ... it's working fine for me with Solr 6
> and the techproducts sample data (see below).
>
> https://wiki.apache.org/solr/UsingMailingLists
>
>
> $ curl 'http://localhost:8983/solr/techproducts/query?rows=0=%
> 7B!tag=y%7dinStock:true=%7B!tag=x%7dname:ipod=true&
> facet.field=inStock=%7b!key=exclude_x+ex=x%
> 7dinStock=%7b!key=exclude_xy+ex=x,y%7dinStock'
> {
>   "responseHeader":{
> "status":0,
> "QTime":2,
> "params":{
>   "q":"{!tag=x}name:ipod",
>   "facet.field":["inStock",
> "{!key=exclude_x ex=x}inStock",
> "{!key=exclude_xy ex=x,y}inStock"],
>   "fq":"{!tag=y}inStock:true",
>   "rows":"0",
>   "facet":"true"}},
>   "response":{"numFound":1,"start":0,"docs":[]
>   },
>   "facet_counts":{
> "facet_queries":{},
> "facet_fields":{
>   "inStock":[
> "true",1,
> "false",0],
>   "exclude_x":[
> "true",17,
> "false",0],
>   "exclude_xy":[
> "true",17,
> "false",4]},
> "facet_ranges":{},
> "facet_intervals":{},
> "facet_heatmaps":{}}}
>
>
>
>
>
> -Hoss
> http://www.lucidworks.com/
>



-- 
Jacques du Rand
Senior R  Programmer

T: +27214688017
F: +27862160617
E: jacq...@pricecheck.co.za

Re: Facet query - exlude main query

[Chris Hostetter]

:  Solr3.1  Starting with Solr 3.1, the
: primary relevance query (i.e. the one normally specified by the *q* parameter)
: may also be excluded.
: 
: But doesnt show me how to exlude it ??

same tag + ex local params, as you had in your example...

: 2. q={!tag=mq}blah={!ex=tag_man,mq}manufacturer
...
: But  I still only get the facets WITH the main query  of "blah"

Can you give us a more more complete picture of what your data & full 
request & response looks like? ... it's working fine for me with Solr 6 
and the techproducts sample data (see below).

https://wiki.apache.org/solr/UsingMailingLists


$ curl 
'http://localhost:8983/solr/techproducts/query?rows=0=%7B!tag=y%7dinStock:true=%7B!tag=x%7dname:ipod=true=inStock=%7b!key=exclude_x+ex=x%7dinStock=%7b!key=exclude_xy+ex=x,y%7dinStock'
{
  "responseHeader":{
"status":0,
"QTime":2,
"params":{
  "q":"{!tag=x}name:ipod",
  "facet.field":["inStock",
"{!key=exclude_x ex=x}inStock",
"{!key=exclude_xy ex=x,y}inStock"],
  "fq":"{!tag=y}inStock:true",
  "rows":"0",
  "facet":"true"}},
  "response":{"numFound":1,"start":0,"docs":[]
  },
  "facet_counts":{
"facet_queries":{},
"facet_fields":{
  "inStock":[
"true",1,
"false",0],
  "exclude_x":[
"true",17,
"false",0],
  "exclude_xy":[
"true",17,
"false",4]},
"facet_ranges":{},
"facet_intervals":{},
"facet_heatmaps":{}}}





-Hoss
http://www.lucidworks.com/

Facet query - exlude main query

[Jacques du Rand]

HI Guys
The Wiki says

 Solr3.1  Starting with Solr 3.1, the
primary relevance query (i.e. the one normally specified by the *q* parameter)
may also be excluded.

But doesnt show me how to exlude it ??

i've tried:
1 .={!ex=tag_man,mainquery}manufacturer

2. q={!tag=mq}blah={!ex=tag_man,mq}manufacturer

3. ...=manufacturer:*


But  I still only get the facets WITH the main query  of "blah"

Any Ideas ?

-- 
Jacques du Rand
Senior R  Programmer

T: +27214688017
F: +27862160617
E: jacq...@pricecheck.co.za

Re: How to alter the facet query limit default

[alessandro.benedetti]

It is already possible[1].

I still don't understand why your default stopped working.
You said that when you changed the name of your fields ( that were used in
the copy fields as destination), the default value ( coming from the source
field) stopped to work.
How did you solve that issue ? Have you found any reason/bug ?

Cheers


[1]
https://cwiki.apache.org/confluence/display/solr/RequestHandlers+and+SearchComponents+in+SolrConfig




--
View this message in context: 
http://lucene.472066.n3.nabble.com/How-to-alter-the-facet-query-limit-default-tp4315939p4317994.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: How to alter the facet query limit default

[Comcast]

I solved it by changing the field names to camelcase and adding the facet limit 
parameter to the query.

Sent from my iPhone

> On Jan 31, 2017, at 4:19 AM, alessandro.benedetti <abenede...@apache.org> 
> wrote:
> 
> What was the problem with the default in the end ?
> 
> Cheers
> 
> 
> 
> --
> View this message in context: 
> http://lucene.472066.n3.nabble.com/How-to-alter-the-facet-query-limit-default-tp4315939p4317964.html
> Sent from the Solr - User mailing list archive at Nabble.com.

Re: How to alter the facet query limit default

[Comcast]

It would be nice if facets and all the parameters could be configured in a 
config/xml file.

Sent from my iPhone

> On Jan 31, 2017, at 4:19 AM, alessandro.benedetti <abenede...@apache.org> 
> wrote:
> 
> What was the problem with the default in the end ?
> 
> Cheers
> 
> 
> 
> --
> View this message in context: 
> http://lucene.472066.n3.nabble.com/How-to-alter-the-facet-query-limit-default-tp4315939p4317964.html
> Sent from the Solr - User mailing list archive at Nabble.com.

Re: How to alter the facet query limit default

[alessandro.benedetti]

What was the problem with the default in the end ?

Cheers



--
View this message in context: 
http://lucene.472066.n3.nabble.com/How-to-alter-the-facet-query-limit-default-tp4315939p4317964.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: How to alter the facet query limit default

[Erick Erickson]

Alexandre:

We recommend only using alphanum and underscore here:
https://cwiki.apache.org/confluence/display/solr/Defining+Fields. WDYT
about adding an explicit warning about this problem there? Or a JIRA
about "field names with periods cannot use field qualifiers like
f..facet.limit where  contains periods"?

Erick

On Mon, Jan 30, 2017 at 10:35 AM, KRIS MUSSHORN <mussho...@comcast.net> wrote:
>
> Alexandre..
>
> Once i converted to fieldnames without dots it worked.. thank you
>
> Kris
> - Original Message -
>
> From: "Alexandre Rafalovitch" <arafa...@gmail.com>
> To: "solr-user" <solr-user@lucene.apache.org>
> Sent: Thursday, January 26, 2017 11:40:49 AM
> Subject: Re: How to alter the facet query limit default
>
> facet.limit?
> f..facet.limit? (not sure how that would work with field
> name that contains dots)
>
> Docs are at: https://cwiki.apache.org/confluence/display/solr/Faceting
>
> Regards,
> Alex.
> 
> http://www.solr-start.com/ - Resources for Solr users, new and experienced
>
>
> On 26 January 2017 at 10:36, KRIS MUSSHORN <mussho...@comcast.net> wrote:
>> SOLR 5.4.1 i am running a query with multiple facet fields.
>> _snip_ 
>> select?q=*%3A*=metatag.date.prefix4+DESC=7910=metatag.date.prefix7=json=true=true=metatag.date.prefix7
>>  =metatag.date.prefix4=metatag.doctype
>>
>> field metatag.date.prefix7 has way more facets than the default of 100.
>>
>> How would I set up solr, or modify my query, to ensure that the facets 
>> return all values.
>>
>> TIA,
>>
>> Kris
>>
>

Re: How to alter the facet query limit default

[KRIS MUSSHORN]

Alexandre.. 

Once i converted to fieldnames without dots it worked.. thank you 

Kris 
- Original Message -

From: "Alexandre Rafalovitch" <arafa...@gmail.com> 
To: "solr-user" <solr-user@lucene.apache.org> 
Sent: Thursday, January 26, 2017 11:40:49 AM 
Subject: Re: How to alter the facet query limit default 

facet.limit? 
f..facet.limit? (not sure how that would work with field 
name that contains dots) 

Docs are at: https://cwiki.apache.org/confluence/display/solr/Faceting 

Regards, 
Alex. 
 
http://www.solr-start.com/ - Resources for Solr users, new and experienced 


On 26 January 2017 at 10:36, KRIS MUSSHORN <mussho...@comcast.net> wrote: 
> SOLR 5.4.1 i am running a query with multiple facet fields. 
> _snip_ 
> select?q=*%3A*=metatag.date.prefix4+DESC=7910=metatag.date.prefix7=json=true=true=metatag.date.prefix7
>  =metatag.date.prefix4=metatag.doctype 
> 
> field metatag.date.prefix7 has way more facets than the default of 100. 
> 
> How would I set up solr, or modify my query, to ensure that the facets return 
> all values. 
> 
> TIA, 
> 
> Kris 
> 

Re: How to alter the facet query limit default

[KRIS MUSSHORN]

Alexandre, 
Thanks. 
I will refactor my schema to eliminate the period seperated values in a field 
names and try your suggestion. 
I'll let you know how it goes. 

Kris 
- Original Message -

From: "Alexandre Rafalovitch" <arafa...@gmail.com> 
To: "solr-user" <solr-user@lucene.apache.org> 
Sent: Thursday, January 26, 2017 11:40:49 AM 
Subject: Re: How to alter the facet query limit default 

facet.limit? 
f..facet.limit? (not sure how that would work with field 
name that contains dots) 

Docs are at: https://cwiki.apache.org/confluence/display/solr/Faceting 

Regards, 
Alex. 
 
http://www.solr-start.com/ - Resources for Solr users, new and experienced 


On 26 January 2017 at 10:36, KRIS MUSSHORN <mussho...@comcast.net> wrote: 
> SOLR 5.4.1 i am running a query with multiple facet fields. 
> _snip_ 
> select?q=*%3A*=metatag.date.prefix4+DESC=7910=metatag.date.prefix7=json=true=true=metatag.date.prefix7
>  =metatag.date.prefix4=metatag.doctype 
> 
> field metatag.date.prefix7 has way more facets than the default of 100. 
> 
> How would I set up solr, or modify my query, to ensure that the facets return 
> all values. 
> 
> TIA, 
> 
> Kris 
> 

Re: How to alter the facet query limit default

[Alexandre Rafalovitch]

facet.limit?
f..facet.limit? (not sure how that would work with field
name that contains dots)

Docs are at: https://cwiki.apache.org/confluence/display/solr/Faceting

Regards,
   Alex.

http://www.solr-start.com/ - Resources for Solr users, new and experienced


On 26 January 2017 at 10:36, KRIS MUSSHORN  wrote:
> SOLR 5.4.1 i am running a query with multiple facet fields.
> _snip_ 
> select?q=*%3A*=metatag.date.prefix4+DESC=7910=metatag.date.prefix7=json=true=true=metatag.date.prefix7
>  =metatag.date.prefix4=metatag.doctype
>
> field metatag.date.prefix7 has way more facets than the default of 100.
>
> How would I set up solr, or modify my query, to ensure that the facets return 
> all values.
>
> TIA,
>
> Kris
>

How to alter the facet query limit default

[KRIS MUSSHORN]

SOLR 5.4.1 i am running a query with multiple facet fields. 
_snip_ 
select?q=*%3A*=metatag.date.prefix4+DESC=7910=metatag.date.prefix7=json=true=true=metatag.date.prefix7
 =metatag.date.prefix4=metatag.doctype 

field metatag.date.prefix7 has way more facets than the default of 100. 

How would I set up solr, or modify my query, to ensure that the facets return 
all values. 

TIA, 

Kris 

range facet query with /sql handler

[radha krishnan]

HI,

can we do a range facet query with /sql handler in solr 6.3 ?

something like the below one


json.facet: {
my_histogram: {
type: range,
field: timestamp,
start: "2017-01-06T20:04:19.884Z",
end: "2017-01-06T22:07:37.778Z",
gap: "+1MINUTE",
mincount: 0
}
}
q: timestamp: ["2017-01-06T20:04:19.884Z"
TO "2017-01-06T22:07:37.778Z"
]
rows: 0
//shards:
http://localhost:8983/solr/core4,http://localhost:8983/solr/core3,http://localhost:8983/solr/core2
//fl:-t_message
sort: timestamp desc


Thanks,
Radhakrishnan D

Re: facet query performance

[Toke Eskildsen]

On Mon, 2016-11-14 at 11:36 +0530, Midas A wrote:
> How to improve facet query performance

1) Don't shard unless you really need to. Replicas are fine.

2) If the problem is the first facet call, then enable DocValues and
re-index.

3) Keep facet.limit <= 100, especially if you shard.

and most important

4) Describe in detail what you have, how you facet and what you expect.
Give us something to work.


- Toke Eskildsen, State and University Library, Denmark

facet query performance

[Midas A]

How to improve facet query performance

Re: normal solr query vs facet query performance

[Shawn Heisey]

On 4/18/2016 5:06 AM, Mugeesh Husain wrote:
> 1.)solr normal query(q=*:*) vs facet query(facet.query="abc") ?
> 2.)solr normal query(q=*:*) vs facet
> search(facet=tru=coullumn_name) ?
> 3.)solr filter query(q=Column:some value) vs facet query(facet.query="abc")
> ?
> 4.)solr normal query(q=*:*) vs filter query(q=column:some value) ?

This is a question that is nearly impossible to answer without your
actual index, and even then only you can answer it.  You need to *try*
these queries and see what happens.

https://lucidworks.com/blog/sizing-hardware-in-the-abstract-why-we-dont-have-a-definitive-answer/

Note that there is a performance bug with the *:* (MatchAllDocuments)
query on 5.x versions, which is only solved in 5.5.0 and later.  This
query runs quite a bit slower than it should.

https://issues.apache.org/jira/browse/SOLR-8251

Thanks,
Shawn

normal solr query vs facet query performance

[Mugeesh Husain]

Hello,

I am looking for which query will be fast in term of performance,

1.)solr normal query(q=*:*) vs facet query(facet.query="abc") ?
2.)solr normal query(q=*:*) vs facet
search(facet=tru=coullumn_name) ?
3.)solr filter query(q=Column:some value) vs facet query(facet.query="abc")
?
4.)solr normal query(q=*:*) vs filter query(q=column:some value) ?



Also provide some good tutorial for above these things.


Thanks



--
View this message in context: 
http://lucene.472066.n3.nabble.com/normal-solr-query-vs-facet-query-performance-tp4270907.html
Sent from the Solr - User mailing list archive at Nabble.com.

Json facet query error "null:java.lang.IllegalArgumentException"

[Yago Riveiro]

Hi,

I'm hitting an error when a try to run a json facet query in a node that
doesn't have any shard that belongs to collection. The same query using
using the legacy facet method works.

http://devel-16:8983/solr/collection-perf/query?rows=0=*:*={label:{type:terms,field:url,limit:-1,sort:{index:asc},numBuckets:true}}

Error:
HTTP ERROR 500

Problem accessing /solr/collection-perf/select. Reason:

{msg=Illegal character in query at index 102:
http://devel-15:8983/solr/collection-perf/select?rows=0=date:20150101={label:{type:terms,field:url_encoded,limit:-1,sort:{index:asc}}}=json,trace=java.lang.IllegalArgumentException:
Illegal character in query at index 102:
http://devel-15:8983/solr/collection-perf/select?rows=0=date:20150101={label:{type:terms,field:url_encoded,limit:-1,sort:{index:asc}}}=json
at java.net.URI.create(URI.java:859)
at org.apache.http.client.methods.HttpGet.(HttpGet.java:69)
at 
org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:535)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:446)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:214)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:179)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:499)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
at
org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.URISyntaxException: Illegal character in query at index
102:
http://devel-15:8983/solr/collection-perf/select?rows=0=date:20150101={label:{type:terms,field:url_encoded,limit:-1,sort:{index:asc}}}=json
at java.net.URI$Parser.fail(URI.java:2829)
at java.net.URI$Parser.checkChars(URI.java:3002)
at java.net.URI$Parser.parseHierarchical(URI.java:3092)
at java.net.URI$Parser.parse(URI.java:3034)
at java.net.URI.(URI.java:595)
at java.net.URI.create(URI.java:857)
... 25 more
,code=500}
Powered by Jetty://



-
Best regards
--
View this message in context: 
http://lucene.472066.n3.nabble.com/Json-facet-query-error-null-java-lang-IllegalArgumentException-tp4246523.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Json facet query error "null:java.lang.IllegalArgumentException"

[Yonik Seeley]

This was a generic query-forwarding bug in Solr, that was recently fixed.
Not sure the JIRA now... what version are you using?
-Yonik


On Tue, Dec 22, 2015 at 10:11 AM, Yago Riveiro <yago.rive...@gmail.com> wrote:
> Hi,
>
> I'm hitting an error when a try to run a json facet query in a node that
> doesn't have any shard that belongs to collection. The same query using
> using the legacy facet method works.
>
> http://devel-16:8983/solr/collection-perf/query?rows=0=*:*={label:{type:terms,field:url,limit:-1,sort:{index:asc},numBuckets:true}}
>
> Error:
> HTTP ERROR 500
>
> Problem accessing /solr/collection-perf/select. Reason:
>
> {msg=Illegal character in query at index 102:
> http://devel-15:8983/solr/collection-perf/select?rows=0=date:20150101={label:{type:terms,field:url_encoded,limit:-1,sort:{index:asc}}}=json,trace=java.lang.IllegalArgumentException:
> Illegal character in query at index 102:
> http://devel-15:8983/solr/collection-perf/select?rows=0=date:20150101={label:{type:terms,field:url_encoded,limit:-1,sort:{index:asc}}}=json
> at java.net.URI.create(URI.java:859)
> at org.apache.http.client.methods.HttpGet.(HttpGet.java:69)
> at 
> org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:535)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:446)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:214)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:179)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
> at org.eclipse.jetty.server.Server.handle(Server.java:499)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
> at
> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.net.URISyntaxException: Illegal character in query at index
> 102:
> http://devel-15:8983/solr/collection-perf/select?rows=0=date:20150101={label:{type:terms,field:url_encoded,limit:-1,sort:{index:asc}}}=json
> at java.net.URI$Parser.fail(URI.java:2829)
> at java.net.URI$Parser.checkChars(URI.java:3002)
> at java.net.URI$Parser.parseHierarchical(URI.java:3092)
> at java.net.URI$Parser.parse(URI.java:3034)
> at java.net.URI.(URI.java:595)
> at java.net.URI.create(URI.java:857)
> ... 25 more
> ,code=500}
> Powered by Jetty://
>
>
>
> -
> Best regards
> --
> View this message in context: 
> http://lucene.472066.n3.nabble.com/Json-facet-query-error-null-java-lang-IllegalArgumentException-tp4246523.html
> Sent from the Solr - User mailing list archive at Nabble.com.

Re: Json facet query error "null:java.lang.IllegalArgumentException"

[Yonik Seeley]

OK found the issue:
 https://issues.apache.org/jira/browse/SOLR-5971
Fixed in 5.4
-Yonik


On Tue, Dec 22, 2015 at 10:15 AM, Yonik Seeley <ysee...@gmail.com> wrote:
> This was a generic query-forwarding bug in Solr, that was recently fixed.
> Not sure the JIRA now... what version are you using?
> -Yonik
>
>
> On Tue, Dec 22, 2015 at 10:11 AM, Yago Riveiro <yago.rive...@gmail.com> wrote:
>> Hi,
>>
>> I'm hitting an error when a try to run a json facet query in a node that
>> doesn't have any shard that belongs to collection. The same query using
>> using the legacy facet method works.
>>
>> http://devel-16:8983/solr/collection-perf/query?rows=0=*:*={label:{type:terms,field:url,limit:-1,sort:{index:asc},numBuckets:true}}
>>
>> Error:
>> HTTP ERROR 500
>>
>> Problem accessing /solr/collection-perf/select. Reason:
>>
>> {msg=Illegal character in query at index 102:
>> http://devel-15:8983/solr/collection-perf/select?rows=0=date:20150101={label:{type:terms,field:url_encoded,limit:-1,sort:{index:asc}}}=json,trace=java.lang.IllegalArgumentException:
>> Illegal character in query at index 102:
>> http://devel-15:8983/solr/collection-perf/select?rows=0=date:20150101={label:{type:terms,field:url_encoded,limit:-1,sort:{index:asc}}}=json
>> at java.net.URI.create(URI.java:859)
>> at org.apache.http.client.methods.HttpGet.(HttpGet.java:69)
>> at 
>> org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:535)
>> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:446)
>> at
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:214)
>> at
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:179)
>> at
>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
>> at
>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
>> at
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
>> at
>> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
>> at
>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
>> at
>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
>> at
>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
>> at
>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
>> at
>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
>> at
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>> at
>> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
>> at
>> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
>> at
>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
>> at org.eclipse.jetty.server.Server.handle(Server.java:499)
>> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
>> at
>> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
>> at
>> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
>> at
>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
>> at
>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
>> at java.lang.Thread.run(Thread.java:745)
>> Caused by: java.net.URISyntaxException: Illegal character in query at index
>> 102:
>> http://devel-15:8983/solr/collection-perf/select?rows=0=date:20150101={label:{type:terms,field:url_encoded,limit:-1,sort:{index:asc}}}=json
>> at java.net.URI$Parser.fail(URI.java:2829)
>> at java.net.URI$Parser.checkChars(URI.java:3002)
>> at java.net.URI$Parser.parseHierarchical(URI.java:3092)
>> at java.net.URI$Parser.parse(URI.java:3034)
>> at java.net.URI.(URI.java:595)
>> at java.net.URI.create(URI.java:857)
>> ... 25 more
>> ,code=500}
>> Powered by Jetty://
>>
>>
>>
>> -
>> Best regards
>> --
>> View this message in context: 
>> http://lucene.472066.n3.nabble.com/Json-facet-query-error-null-java-lang-IllegalArgumentException-tp4246523.html
>> Sent from the Solr - User mailing list archive at Nabble.com.

Re: Json facet query error "null:java.lang.IllegalArgumentException"

[Yago Riveiro]

I’m in 5.3.1.




I’m waiting some time to upgrade to 5.4 to see if some nasty bug is reported. 
But after hitting this issue I think that I should upgrade ...


—/Yago Riveiro

On Tue, Dec 22, 2015 at 3:17 PM, Yonik Seeley <ysee...@gmail.com> wrote:

> OK found the issue:
>  https://issues.apache.org/jira/browse/SOLR-5971
> Fixed in 5.4
> -Yonik
> On Tue, Dec 22, 2015 at 10:15 AM, Yonik Seeley <ysee...@gmail.com> wrote:
>> This was a generic query-forwarding bug in Solr, that was recently fixed.
>> Not sure the JIRA now... what version are you using?
>> -Yonik
>>
>>
>> On Tue, Dec 22, 2015 at 10:11 AM, Yago Riveiro <yago.rive...@gmail.com> 
>> wrote:
>>> Hi,
>>>
>>> I'm hitting an error when a try to run a json facet query in a node that
>>> doesn't have any shard that belongs to collection. The same query using
>>> using the legacy facet method works.
>>>
>>> http://devel-16:8983/solr/collection-perf/query?rows=0=*:*={label:{type:terms,field:url,limit:-1,sort:{index:asc},numBuckets:true}}
>>>
>>> Error:
>>> HTTP ERROR 500
>>>
>>> Problem accessing /solr/collection-perf/select. Reason:
>>>
>>> {msg=Illegal character in query at index 102:
>>> http://devel-15:8983/solr/collection-perf/select?rows=0=date:20150101={label:{type:terms,field:url_encoded,limit:-1,sort:{index:asc}}}=json,trace=java.lang.IllegalArgumentException:
>>> Illegal character in query at index 102:
>>> http://devel-15:8983/solr/collection-perf/select?rows=0=date:20150101={label:{type:terms,field:url_encoded,limit:-1,sort:{index:asc}}}=json
>>> at java.net.URI.create(URI.java:859)
>>> at org.apache.http.client.methods.HttpGet.(HttpGet.java:69)
>>> at 
>>> org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:535)
>>> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:446)
>>> at
>>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:214)
>>> at
>>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:179)
>>> at
>>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
>>> at
>>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
>>> at
>>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
>>> at
>>> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
>>> at
>>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
>>> at
>>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
>>> at
>>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
>>> at
>>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
>>> at
>>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
>>> at
>>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>>> at
>>> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
>>> at
>>> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
>>> at
>>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
>>> at org.eclipse.jetty.server.Server.handle(Server.java:499)
>>> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
>>> at
>>> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
>>> at
>>> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
>>> at
>>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
>>> at
>>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
>>> at java.lang.Thread.run(Thread.java:745)
>>> Caused by: java.net.URISyntaxException: Illegal character in query at index
>>> 102:
>>> http://devel-15:8983/solr/collection-perf/select?rows=0=date:20150101={label:{type:terms,field:url_encoded,limit:-1,sort:{index:asc}}}=json
>>> at java.net.URI$Parser.fail(URI.java:2829)
>>> at java.net.URI$Parser.checkChars(URI.java:3002)
>>> at java.net.URI$Parser.parseHierarchical(URI.java:3092)
>>> at java.net.URI$Parser.parse(URI.java:3034)
>>> at java.net.URI.(URI.java:595)
>>> at java.net.URI.create(URI.java:857)
>>> ... 25 more
>>> ,code=500}
>>> Powered by Jetty://
>>>
>>>
>>>
>>> -
>>> Best regards
>>> --
>>> View this message in context: 
>>> http://lucene.472066.n3.nabble.com/Json-facet-query-error-null-java-lang-IllegalArgumentException-tp4246523.html
>>> Sent from the Solr - User mailing list archive at Nabble.com.

Re: Solr facet query(critical solr query response)

[Alessandro Benedetti]

I suggest you to take a look to Yonik blog and in particular to the new
Json Faceting approach and Nested document modelling.
Hope this can help :

http://yonik.com/solr-nested-objects/ ( nested objects )

http://yonik.com/json-facet-api/ ( nested facets)

Cheers

On 4 November 2015 at 12:15, Mugeesh Husain <muge...@gmail.com> wrote:

> Thanks Erik
>
>  >>Shouldn't td have s2=1 also?  yes was my mistake
> I have also a search concern
>
> I have a 3 table Resturant, Review,User
>
> retuarant-[restid, restname,location...]
> User-[userid, uname,]
> Review-[reviewID, userid,restid,comment...]
>
> I am searching based on restaurant, how could i will find column of
> user
>
> Please give suggestion
>
> Thanks,
> Mugeesh
>
>
>
>
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Solr-facet-query-critical-solr-query-response-tp4238135p4238180.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>



-- 
--

Benedetti Alessandro
Visiting card : http://about.me/alessandro_benedetti

"Tyger, tyger burning bright
In the forests of the night,
What immortal hand or eye
Could frame thy fearful symmetry?"

William Blake - Songs of Experience -1794 England

Re: Solr facet query(critical solr query response)

[Erik Hatcher]

Shouldn't td have s2=1 also?  Try this:

  facet.pivot=vi,vk



> On Nov 4, 2015, at 03:23, Mugeesh Husain <muge...@gmail.com> wrote:
> 
> i am facing issue of how to get below response from solr query.
> 
> In my table, i have a two column vi and vk which has below values:
> 
> row1: vi:["ta"] ,vk:["s1"]
> row2: vi:["tb"] ,vk:["s2"]
> row3: vi:["tc"] ,vk:["s0"]
> row4: vi:["td"] ,vk:["s2"]
> row5: vi:["ta"] ,vk:["s1"] 
> row5: vi:["tc"] ,vk:["s0"] and so on.
> 
> I want to facet in below way
> 
> ta-[so=0,s1=2,s2=0]
> tb-[so=0,s1=1,s2=0]
> tc-[so=2,s1=0,s2=0]
> td-[so=0,s1=0,s2=0]
> 
> how i create query which i can get such response ?
> 
> Please suggestion 
> 
> Thanks,
> Mugeesh
> 
> 
> 
> 
> 
> 
> 
> 
> 
> --
> View this message in context: 
> http://lucene.472066.n3.nabble.com/Solr-facet-query-critical-solr-query-response-tp4238135.html
> Sent from the Solr - User mailing list archive at Nabble.com.

Re: Solr facet query(critical solr query response)

[Mugeesh Husain]

Thanks Erik  

 >>Shouldn't td have s2=1 also?  yes was my mistake
I have also a search concern

I have a 3 table Resturant, Review,User

retuarant-[restid, restname,location...]
User-[userid, uname,]
Review-[reviewID, userid,restid,comment...]

I am searching based on restaurant, how could i will find column of
user 

Please give suggestion 

Thanks, 
Mugeesh 
 






--
View this message in context: 
http://lucene.472066.n3.nabble.com/Solr-facet-query-critical-solr-query-response-tp4238135p4238180.html
Sent from the Solr - User mailing list archive at Nabble.com.

Solr facet query(critical solr query response)

[Mugeesh Husain]

i am facing issue of how to get below response from solr query.

In my table, i have a two column vi and vk which has below values:

row1: vi:["ta"] ,vk:["s1"]
row2: vi:["tb"] ,vk:["s2"]
row3: vi:["tc"] ,vk:["s0"]
row4: vi:["td"] ,vk:["s2"]
row5: vi:["ta"] ,vk:["s1"] 
row5: vi:["tc"] ,vk:["s0"] and so on.

I want to facet in below way

ta-[so=0,s1=2,s2=0]
tb-[so=0,s1=1,s2=0]
tc-[so=2,s1=0,s2=0]
td-[so=0,s1=0,s2=0]

how i create query which i can get such response ?

Please suggestion 

Thanks,
Mugeesh









--
View this message in context: 
http://lucene.472066.n3.nabble.com/Solr-facet-query-critical-solr-query-response-tp4238135.html
Sent from the Solr - User mailing list archive at Nabble.com.

facet query is not working

[Midas A]

http://localhost:8983/solr/col/select?q=*:*sfield=geolocationpt=26.697,83.1876facet.query={!frange%20l=0%20u=50}geodist()facet.query={!frange%20l=50.001%20u=100}geodist()wt=json


I am not getting facet results .

schema:
field name=geolocation type=location indexed=true stored=true/ 
dynamicField name=*_coordinate type=tdouble indexed=true stored=
false/

Re: facet query is not working

[Mikhail Khludnev]

isn't facet=true necessary?

On Thu, Jun 18, 2015 at 12:03 PM, Midas A test.mi...@gmail.com wrote:


 http://localhost:8983/solr/col/select?q=*:*sfield=geolocationpt=26.697,83.1876facet.query={!frange%20l=0%20u=50}geodist()facet.query={!frange%20l=50.001%20u=100}geodist()wt=json


 I am not getting facet results .

 schema:
 field name=geolocation type=location indexed=true stored=true/ 
 dynamicField name=*_coordinate type=tdouble indexed=true stored=
 false/




-- 
Sincerely yours
Mikhail Khludnev
Principal Engineer,
Grid Dynamics

http://www.griddynamics.com
mkhlud...@griddynamics.com

Re: facet query is not working

[Alessandro Benedetti]

If he has not put any appends or invariant in the request handler,
facet=true is mandatory to activate the facets.

I haven't tried those specific facet queries .

I hope the problem was not simply he didn't activate faceting ...

2015-06-18 10:35 GMT+01:00 Mikhail Khludnev mkhlud...@griddynamics.com:

 isn't facet=true necessary?

 On Thu, Jun 18, 2015 at 12:03 PM, Midas A test.mi...@gmail.com wrote:

 
 
 http://localhost:8983/solr/col/select?q=*:*sfield=geolocationpt=26.697,83.1876facet.query={!frange%20l=0%20u=50}geodist()facet.query={!frange%20l=50.001%20u=100}geodist()wt=json
 
 
  I am not getting facet results .
 
  schema:
  field name=geolocation type=location indexed=true stored=true/
 
  dynamicField name=*_coordinate type=tdouble indexed=true stored=
  false/
 



 --
 Sincerely yours
 Mikhail Khludnev
 Principal Engineer,
 Grid Dynamics

 http://www.griddynamics.com
 mkhlud...@griddynamics.com




-- 
--

Benedetti Alessandro
Visiting card : http://about.me/alessandro_benedetti

Tyger, tyger burning bright
In the forests of the night,
What immortal hand or eye
Could frame thy fearful symmetry?

William Blake - Songs of Experience -1794 England

Re: Global query parameters to facet query

[Chris Hostetter]

: when disabling the facet, the query works perfectly. I understand that
: either defType parameter or q.myalias.qf parameters, do not affect the
: facet query (which always runs with lucene parser??)

a top level defType param only applies to the q param.  This is by 
design.  

defType is special and only applies to hte main query where it is 
being evaluated, in order to make it easier to specify the type of parser 
to use on input straight from end users -- things like fq, facet.query, 
and any recursively specified query strings (ie: via other nested local 
params), etc, are not affected by defType.  These other params are almost 
always programatically generated, so it's typically easy to use an 
explicit type local param to change the parser they use (if they looked 
at defType it would cause all sorts of confusing behavior for most users 
who just wanted to cahnge the parser for the main query)


-Hoss
http://www.lucidworks.com/

Re: Global query parameters to facet query

[Isaac Hebsh]

created SOLR-5542.
Anyone else want it?


On Thu, Dec 5, 2013 at 8:55 PM, Isaac Hebsh isaac.he...@gmail.com wrote:

 Hi,

 It seems that a facet query does not use the global query parameters (for
 example, field aliasing for edismax parser).
 We have an intensive use of facet queries (in some cases, we have a lot of
 facet.query for a single q), and the using of LocalParams for each
 facet.query is not convenient.

 Did I miss a normal way to solve it?
 Did anyone else encountered this requirement?

Re: Global query parameters to facet query

[Chris Hostetter]

: It seems that a facet query does not use the global query parameters (for
: example, field aliasing for edismax parser).

can you please give a specific example of a query that isn't working for 
you?

Using this query against the examle data, things work exactly as i would 
expect showing that the QParsers used for facet.queries inherit the 
global params (unless overridden by local params of course)...

http://localhost:8983/solr/select?q=*:*wt=jsonindent=truefacet=truefacet.query={!dismax}solr+bogusfacet.query={!dismax%20mm=1}solr+bogusfacet.query={!dismax%20mm=1%20qf=%27foo_t%27}solr+bogusrows=0mm=2qf=name
{
  responseHeader:{
status:0,
QTime:2,
params:{
  mm:2,
  facet:true,
  indent:true,
  facet.query:[{!dismax}solr bogus,
{!dismax mm=1}solr bogus,
{!dismax mm=1 qf='foo_t'}solr bogus],
  q:*:*,
  qf:name,
  wt:json,
  rows:0}},
  response:{numFound:32,start:0,docs:[]
  },
  facet_counts:{
facet_queries:{
  {!dismax}solr bogus:0,
  {!dismax mm=1}solr bogus:1,
  {!dismax mm=1 qf='foo_t'}solr bogus:0},
facet_fields:{},
facet_dates:{},
facet_ranges:{}}}






-Hoss
http://www.lucidworks.com/

Global query parameters to facet query

[Isaac Hebsh]

Hi,

It seems that a facet query does not use the global query parameters (for
example, field aliasing for edismax parser).
We have an intensive use of facet queries (in some cases, we have a lot of
facet.query for a single q), and the using of LocalParams for each
facet.query is not convenient.

Did I miss a normal way to solve it?
Did anyone else encountered this requirement?

Re: solr facet query on multiple search term

[Chris Hostetter]

: Actual requirement is to get day wise total no. of counts for multiple
: terms.

the approach you described in your original mail (using multiple fqs to 
identify your terms, and using multiple facet.range requests on the date 
field with differnet exclusions) will work fine.

the confusing part was simply that you didn't clarify how the q param in 
your final example related ot the different q params you had in your two 
originla queries.

assuming you only want these counts aggregated over docs that match 
q=col2:1 the facet exclusions example you posted should do exactly 
what you want.


-Hoss

Re: solr facet query on multiple search term

[vrparekh]

Thanks Erick,

yes example url i provided is bit confusing, sorry for that.

Actual requirement is to get day wise total no. of counts for multiple
terms.

if we use q=(firstterm OR
secondterm)facet.query=firsttermfacet.query=secondTerm. It will provide
total no. of records count for both search term, but not day wise
(facet.range will have combine results of both.)

need something like below (just sample),

lst name=facet_queries
  lst name=firstterm
 lst name=facet_ranges
   lst name=datefield
 lst name=counts
  int name=2013-06-09T00:00:00Z10551/int
  int name=2013-06-10T00:00:00Z20802/int
  /lst
/lst
 /lst
/lst
   lst name=secondterm
 lst name=facet_ranges
   lst name=datefield
 lst name=counts
  int name=2013-06-09T00:00:00Z100/int
  int name=2013-06-10T00:00:00Z5/int
  /lst
/lst
 /lst
/lst
/lst




--
View this message in context: 
http://lucene.472066.n3.nabble.com/solr-facet-query-on-multiple-search-term-tp4068856p4069259.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: solr facet query on multiple search term

[Erick Erickson]

There's nothing like that built in that I know
of, the closest in concept is pivot faceting
but that doesn't work in this case.

Best
Erick

On Mon, Jun 10, 2013 at 2:13 AM, vrparekh vrpar...@gmail.com wrote:
 Thanks Erick,

 yes example url i provided is bit confusing, sorry for that.

 Actual requirement is to get day wise total no. of counts for multiple
 terms.

 if we use q=(firstterm OR
 secondterm)facet.query=firsttermfacet.query=secondTerm. It will provide
 total no. of records count for both search term, but not day wise
 (facet.range will have combine results of both.)

 need something like below (just sample),

 lst name=facet_queries
   lst name=firstterm
  lst name=facet_ranges
lst name=datefield
  lst name=counts
   int name=2013-06-09T00:00:00Z10551/int
   int name=2013-06-10T00:00:00Z20802/int
   /lst
 /lst
  /lst
 /lst
lst name=secondterm
  lst name=facet_ranges
lst name=datefield
  lst name=counts
   int name=2013-06-09T00:00:00Z100/int
   int name=2013-06-10T00:00:00Z5/int
   /lst
 /lst
  /lst
 /lst
 /lst




 --
 View this message in context: 
 http://lucene.472066.n3.nabble.com/solr-facet-query-on-multiple-search-term-tp4068856p4069259.html
 Sent from the Solr - User mailing list archive at Nabble.com.

solr facet query on multiple search term

[vrparekh]

Hello All,

I required facet counts for multiple SearchTerms.
Currently I am doing two separate facet query on each search term with
facet.range=dateField

e.g.

 http://solrserver/select?q=1stsearchTermfq=onfacet-parameters 

 http://solrserver/select?q=2ndsearchTermfq=onfacet-parameters

Note :: SearchTerm field will be text_en_splitting

Now I have found another way to do facet query on multiple search term by
tagging and excluding

e.g.

http://solrurl/select?start=0rows=10hl=off;
facet=on
facet.range.start=2013-06-06T16%3a00%3a00Z
facet.range.end=2013-06-07T16%3a00%3a01Z
facet.range.gap=%2B1HOUR
wt=xml
sort=dateField+desc
facet.range={!key=music+ex=movie}dateField
   
fq={!tag=music}content:musicfacet.range={!key=movie+ex=music}dateField
fq={!tag=movie}content:movieq=(col2:1+)
   
fq=+dateField:[2013-06-05T16:00:00Z+TO+2013-06-07T16:00:00Z]+AND+(+Col1:test+)
fl=col1,col2,col3


I have tested for few search term , It is providing same result as different
query for each search term.
Is this the proper way (with results and performance)?



--
View this message in context: 
http://lucene.472066.n3.nabble.com/solr-facet-query-on-multiple-search-term-tp4068856.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: solr facet query on multiple search term

[Erick Erickson]

I'm a little confused here. Faceting is about counting docs that meet
your query restrictions. I.e. the q= and fq= clauses. So your original
problem statement simply cannot be combined into a single query
since your q= clauses are different. You could do something like
q=(firstterm OR secondterm)facet.query=firsttermfacet.query=secondTerm
That would give you accurate facet counts for the terms, but it
certainly doesn't preserve the original intent of
q=firsttermfacet.query=blahblah.

But facet.query is only counted over the docs that match
the q= clause (well, the q= clause and any fq clauses). So perhaps
you can supply a few example input docs and desired counts on the other side.

Best
Erick

On Fri, Jun 7, 2013 at 8:01 AM, vrparekh vrpar...@gmail.com wrote:
 Hello All,

 I required facet counts for multiple SearchTerms.
 Currently I am doing two separate facet query on each search term with
 facet.range=dateField

 e.g.

  http://solrserver/select?q=1stsearchTermfq=onfacet-parameters

  http://solrserver/select?q=2ndsearchTermfq=onfacet-parameters

 Note :: SearchTerm field will be text_en_splitting

 Now I have found another way to do facet query on multiple search term by
 tagging and excluding

 e.g.

 http://solrurl/select?start=0rows=10hl=off;
 facet=on
 facet.range.start=2013-06-06T16%3a00%3a00Z
 facet.range.end=2013-06-07T16%3a00%3a01Z
 facet.range.gap=%2B1HOUR
 wt=xml
 sort=dateField+desc
 facet.range={!key=music+ex=movie}dateField

 fq={!tag=music}content:musicfacet.range={!key=movie+ex=music}dateField
 fq={!tag=movie}content:movieq=(col2:1+)

 fq=+dateField:[2013-06-05T16:00:00Z+TO+2013-06-07T16:00:00Z]+AND+(+Col1:test+)
 fl=col1,col2,col3


 I have tested for few search term , It is providing same result as different
 query for each search term.
 Is this the proper way (with results and performance)?



 --
 View this message in context: 
 http://lucene.472066.n3.nabble.com/solr-facet-query-on-multiple-search-term-tp4068856.html
 Sent from the Solr - User mailing list archive at Nabble.com.

Re: Maximum number of facet query ina single query

[Jack Krupansky]

You mean 6000 filter queries? Or do they really have 6000 faceted fields in 
a single query?!


Even so, I wouldn't recommend that an average new Solr developer should have 
either 6000 fields in a single document or 6000 query terms or even 6000 
parameters. I mean, sure, you can try it and if it does happen to work, 
great, go for it, but if it doesn't work for 6000 or 100 or even more than 
about 20 facets/filters, I wouldn't complain too loudly about Solr (or 
Lucene) if it doesn't achieve sub-100 ms query time.


-- Jack Krupansky

-Original Message- 
From: Otis Gospodnetic

Sent: Tuesday, April 30, 2013 10:15 PM
To: solr-user@lucene.apache.org
Subject: Re: Maximum number of facet query ina single query

FWIW, one of our current clients runs queries with 6000 facet queries...

Otis
Solr  ElasticSearch Support
http://sematext.com/


On Apr 30, 2013 5:22 AM, vicky desai vicky.de...@germinait.com wrote:


Hi,

Is there any upper limit on the number of facet queries I can include in a
single query. Also is there any performance hit if I include too many 
facet

queries in a single query

Any help would be appreciated



--
View this message in context:
http://lucene.472066.n3.nabble.com/Maximum-number-of-facet-query-ina-single-query-tp4059926.html
Sent from the Solr - User mailing list archive at Nabble.com.

SolrCloud facet query repeatably fails with No live SolrServers for some terms, not all

[Brett Hoerner]

An example:
https://gist.github.com/bretthoerner/2ffc362450bcd4c2487a

I'll note that all shards and replicas show as Up (green) in the Admin UI.

Does anyone know how this could happen? I can repeat this over and over
with the same terms. It was my understanding that something like a facet
query would need to go to *all* shards for any query (I'm using the default
SolrCloud sharding mechanism, nothing special).

How could a text field search for 'happy' always work and 'austin' always
return an error, shouldn't that down server be hit for a 'happy' query
also?

Thanks,
Brett

Re: facet.offset issue (previosly: [solr 3.4] anomaly during distributed facet query with 102 shards)

[Toke Eskildsen]

On Mon, 2013-04-29 at 18:22 +0200, Dmitry Kan wrote:
 Does it even make sense to paginate in facet searches, if we require deep
 paging?

Whether it makes sense logically is up to you.

Technically deep paging in facets could be implemented in approximately
the same way as deep paging in search results: By providing previously
received maximum count and value, the faceting module could skip ahead
to the subsequent entries and only return those.

It would be fairly simple to implement for single core, but tricky for
distributed search. I am not aware of any initiatives in that direction.

- Toke Eskildsen, State and University Library

Re: facet.offset issue (previosly: [solr 3.4] anomaly during distributed facet query with 102 shards)

[Dmitry Kan]

It is logical to use facet pagination for us, if it would work. It just
doesn't, probably due the data amount we store and imposed RAM settings.




On Tue, Apr 30, 2013 at 10:39 AM, Toke Eskildsen 
t...@statsbiblioteket.dkwrote:

 On Mon, 2013-04-29 at 18:22 +0200, Dmitry Kan wrote:
  Does it even make sense to paginate in facet searches, if we require deep
  paging?

 Whether it makes sense logically is up to you.

 Technically deep paging in facets could be implemented in approximately
 the same way as deep paging in search results: By providing previously
 received maximum count and value, the faceting module could skip ahead
 to the subsequent entries and only return those.

 It would be fairly simple to implement for single core, but tricky for
 distributed search. I am not aware of any initiatives in that direction.

 - Toke Eskildsen, State and University Library

Maximum number of facet query ina single query

[vicky desai]

Hi,

Is there any upper limit on the number of facet queries I can include in a
single query. Also is there any performance hit if I include too many facet
queries in a single query

Any help would be appreciated



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Maximum-number-of-facet-query-ina-single-query-tp4059926.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Maximum number of facet query ina single query

[Erik Hatcher]

There's no fixed limit of facet.query's.  But certainly there is a performance 
impact, which is often mitigated by warming and caching.  You'll need to test 
the impact in your environment.

Erik

On Apr 30, 2013, at 02:22 , vicky desai wrote:

 Hi,
 
 Is there any upper limit on the number of facet queries I can include in a
 single query. Also is there any performance hit if I include too many facet
 queries in a single query
 
 Any help would be appreciated
 
 
 
 --
 View this message in context: 
 http://lucene.472066.n3.nabble.com/Maximum-number-of-facet-query-ina-single-query-tp4059926.html
 Sent from the Solr - User mailing list archive at Nabble.com.

Re: Maximum number of facet query ina single query

[Otis Gospodnetic]

FWIW, one of our current clients runs queries with 6000 facet queries...

Otis
Solr  ElasticSearch Support
http://sematext.com/


On Apr 30, 2013 5:22 AM, vicky desai vicky.de...@germinait.com wrote:

 Hi,

 Is there any upper limit on the number of facet queries I can include in a
 single query. Also is there any performance hit if I include too many facet
 queries in a single query

 Any help would be appreciated



 --
 View this message in context:
 http://lucene.472066.n3.nabble.com/Maximum-number-of-facet-query-ina-single-query-tp4059926.html
 Sent from the Solr - User mailing list archive at Nabble.com.

Re: [solr 3.4] anomaly during distributed facet query with 102 shards

[Dmitry Kan]

Hi,

1. Ruled out possibility to test 4.2.1 router against 3.4 shard farm for
obvious reasons (java.lang.RuntimeException: Invalid version (expected 2,
but 60) or the data in not in 'javabin' format).

2. Tried jetty, but same result.


On Thu, Apr 25, 2013 at 5:16 PM, Dmitry Kan solrexp...@gmail.com wrote:

 Thanks, Yonik. Yes, I supposed that. We are in the pre-release phase, so
 we have the pressure.

 Solr 3.4.

 Would setting up 4.2.1 router work with 3.4 shards?
 On 25 Apr 2013 17:11, Yonik Seeley yo...@lucidworks.com wrote:

 On Thu, Apr 25, 2013 at 8:32 AM, Dmitry Kan solrexp...@gmail.com wrote:
  Are there any distrib facet gurus on the list? I would be ready to try
  sensible ideas, including on the source code level, if someone of you
 could
  give me a hand.

 The Lucene/Solr Revolution conference is coming up next week, so I
 think many are busy creating their presentations.
 What version of Solr are you using?  Have you tried using a newer
 version?  Is it reproducable with a smaller cluster?  If so, you could
 try using the included Jetty server instead of Tomcat to rule out that
 factor.

 -Yonik
 http://lucidworks.com

facet.offset issue (previosly: [solr 3.4] anomaly during distributed facet query with 102 shards)

[Dmitry Kan]

Hi list,

We have encountered a weird bug related to the facet.offset parameter. In
short: the more general query is, that generates lots of hits, the higher
the risk of the facet.offset parameter to stop working.

In more detail:

1. Since getting all facets we need (facet.limit=1000) from around 100
shards didn't work for some broad query terms, like the (yes, we index
and search those too), we decided to paginate.

2. The facet page size is set to 100 for all pages starting the second one.
We start with: facet.offset=0facet.limit=30, then continue with
facet.offset=30facet.limit=100, then facet.offset=100facet.limit=100 and
so on, until we get facet.offset=900.

All facets work just fine, until we hit facet.offset=700.

Debugging showed, that in the class HttpCommComponent static Executor
instance is created with a setting to terminate idle threads after 5 sec.
Our belief, is that this setting way too low for our billion document
scenario and broad searches. Setting this to 5 min seems to improve the
situation a bit, but not solve fully. This same class is no longer used in
4.2.1 (can anyone tell what's used instead in distributed faceting?) so it
isn't easy to compare these parts of the code.

Anyhow, playing now with this value in the hope to see some light in the
tunnel (would be good, if it is not the train).

One more question: can this be related to RAM allocation on the router and
/ or shards? If RAM isn't enough for some operations, why the router or
shards wouldn't just crash with OOM?

If anyone has other ideas for what to try / look into, that'll be much
appreciated.

Dmitry

Re: [solr 3.4] anomaly during distributed facet query with 102 shards

[Dmitry Kan]

Are there any distrib facet gurus on the list? I would be ready to try
sensible ideas, including on the source code level, if someone of you could
give me a hand.

Dmitry


On Wed, Apr 24, 2013 at 3:08 PM, Dmitry Kan solrexp...@gmail.com wrote:

 Hello list,

 We deal with an anomaly when doing a distributed facet query against 102
 shards.

 The problem manifests itself in both the frontend solr (router) and a
 shard. Each time the request is executed, always different shard is
 affected (at random, hence the anomaly).

 The query is: http://router_host:router_port
 /solr/select?q=testfacet=truefacet.field=field_of_type_longfacet.limit=1330facet.mincount=1rows=1facet.sort=indexfacet.zeros=falsefacet.offset=0
 I have omitted the shards parameter.

 The router log:

 request: http://10.155.244.181:9150/solr/select
 at 
 org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:430)
 at 
 org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:244)
 at 
 org.apache.solr.handler.component.HttpCommComponent$1.call(SearchHandler.java:421)
 at 
 org.apache.solr.handler.component.HttpCommComponent$1.call(SearchHandler.java:393)
 at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
 at java.util.concurrent.FutureTask.run(FutureTask.java:166)
 at 
 java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
 at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
 at java.util.concurrent.FutureTask.run(FutureTask.java:166)
 at 
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
 at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
 at java.lang.Thread.run(Thread.java:722)

 Notice the port of a shard, that is affected. That port changes all the
 time, even for the same request
 The log entry is prepended with lines:

 SEVERE: org.apache.solr.common.SolrException: Internal Server Error

 Internal Server Error

 (they are not in the pastebin link)

 The shard log:

 Apr 24, 2013 11:08:49 AM org.apache.solr.common.SolrException log
 SEVERE: java.lang.NullPointerException
 at java.io.StringReader.init(StringReader.java:50)
 at 
 org.apache.lucene.queryParser.QueryParser.parse(QueryParser.java:203)
 at 
 org.apache.solr.search.LuceneQParser.parse(LuceneQParserPlugin.java:80)
 at org.apache.solr.search.QParser.getQuery(QParser.java:142)
 at 
 org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:81)
 at 
 org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:173)
 at 
 org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
 at org.apache.solr.core.SolrCore.execute(SolrCore.java:1368)
 at 
 org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:356)
 at 
 org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:252)
 at 
 org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
 at 
 org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
 at 
 org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
 at 
 org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
 at 
 org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
 at 
 org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
 at 
 org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
 at 
 org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
 at 
 org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:849)
 at 
 org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
 at 
 org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:454)
 at java.lang.Thread.run(Thread.java:722)

 Apr 24, 2013 11:08:49 AM org.apache.solr.core.SolrCore execute
 INFO: [] webapp=/solr path=/select params={} status=500 QTime=2
 Apr 24, 2013 11:08:49 AM org.apache.solr.common.SolrException log
 SEVERE: java.lang.NullPointerException
 at java.io.StringReader.init(StringReader.java:50)
 at 
 org.apache.lucene.queryParser.QueryParser.parse(QueryParser.java:203)
 at 
 org.apache.solr.search.LuceneQParser.parse(LuceneQParserPlugin.java:80)
 at org.apache.solr.search.QParser.getQuery(QParser.java:142)
 at 
 org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:81)
 at 
 org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:173

Re: [solr 3.4] anomaly during distributed facet query with 102 shards

[Yonik Seeley]

On Thu, Apr 25, 2013 at 8:32 AM, Dmitry Kan solrexp...@gmail.com wrote:
 Are there any distrib facet gurus on the list? I would be ready to try
 sensible ideas, including on the source code level, if someone of you could
 give me a hand.

The Lucene/Solr Revolution conference is coming up next week, so I
think many are busy creating their presentations.
What version of Solr are you using?  Have you tried using a newer
version?  Is it reproducable with a smaller cluster?  If so, you could
try using the included Jetty server instead of Tomcat to rule out that
factor.

-Yonik
http://lucidworks.com

Re: [solr 3.4] anomaly during distributed facet query with 102 shards

[Dmitry Kan]

Thanks, Yonik. Yes, I supposed that. We are in the pre-release phase, so we
have the pressure.

Solr 3.4.

Would setting up 4.2.1 router work with 3.4 shards?
On 25 Apr 2013 17:11, Yonik Seeley yo...@lucidworks.com wrote:

 On Thu, Apr 25, 2013 at 8:32 AM, Dmitry Kan solrexp...@gmail.com wrote:
  Are there any distrib facet gurus on the list? I would be ready to try
  sensible ideas, including on the source code level, if someone of you
 could
  give me a hand.

 The Lucene/Solr Revolution conference is coming up next week, so I
 think many are busy creating their presentations.
 What version of Solr are you using?  Have you tried using a newer
 version?  Is it reproducable with a smaller cluster?  If so, you could
 try using the included Jetty server instead of Tomcat to rule out that
 factor.

 -Yonik
 http://lucidworks.com

Fwd: [solr 3.4] anomaly during distributed facet query with 102 shards

[Dmitry Kan]

Hello list,

We deal with an anomaly when doing a distributed facet query against 102
shards.

The problem manifests itself in both the frontend solr (router) and a
shard. Each time the request is executed, always different shard is
affected (at random, hence the anomaly).

The query is: http://router_host:router_port
/solr/select?q=testfacet=truefacet.field=field_of_type_longfacet.limit=1330facet.mincount=1rows=1facet.sort=indexfacet.zeros=falsefacet.offset=0
I have omitted the shards parameter.

The router log:

request: http://10.155.244.181:9150/solr/select
at 
org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:430)
at 
org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:244)
at 
org.apache.solr.handler.component.HttpCommComponent$1.call(SearchHandler.java:421)
at 
org.apache.solr.handler.component.HttpCommComponent$1.call(SearchHandler.java:393)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)

Notice the port of a shard, that is affected. That port changes all the
time, even for the same request
The log entry is prepended with lines:

SEVERE: org.apache.solr.common.SolrException: Internal Server Error

Internal Server Error

(they are not in the pastebin link)

The shard log:

Apr 24, 2013 11:08:49 AM org.apache.solr.common.SolrException log
SEVERE: java.lang.NullPointerException
at java.io.StringReader.init(StringReader.java:50)
at org.apache.lucene.queryParser.QueryParser.parse(QueryParser.java:203)
at 
org.apache.solr.search.LuceneQParser.parse(LuceneQParserPlugin.java:80)
at org.apache.solr.search.QParser.getQuery(QParser.java:142)
at 
org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:81)
at 
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:173)
at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1368)
at 
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:356)
at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:252)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at 
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at 
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at 
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
at 
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at 
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at 
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
at 
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:849)
at 
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
at 
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:454)
at java.lang.Thread.run(Thread.java:722)

Apr 24, 2013 11:08:49 AM org.apache.solr.core.SolrCore execute
INFO: [] webapp=/solr path=/select params={} status=500 QTime=2
Apr 24, 2013 11:08:49 AM org.apache.solr.common.SolrException log
SEVERE: java.lang.NullPointerException
at java.io.StringReader.init(StringReader.java:50)
at org.apache.lucene.queryParser.QueryParser.parse(QueryParser.java:203)
at 
org.apache.solr.search.LuceneQParser.parse(LuceneQParserPlugin.java:80)
at org.apache.solr.search.QParser.getQuery(QParser.java:142)
at 
org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:81)
at 
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:173)
at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1368)
at 
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:356)
at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:252

Re: Solr 3.6.1 and facet query regular expression

[Erick Erickson]

Have you just tried what you showed? It should work if I understand the
question. If it doesn't work, please show us the query in, the response
out, and what is NOT as you expect.

Best
Erick


On Sat, Mar 16, 2013 at 5:20 AM, xpow swirja...@gmail.com wrote:

 please help ...



 --
 View this message in context:
 http://lucene.472066.n3.nabble.com/Solr-3-6-1-and-facet-query-regular-expression-tp4047628p4047947.html
 Sent from the Solr - User mailing list archive at Nabble.com.