Front end framework

[Serenity Trust]

Dear all,
I've upgraded my application from Lucene to Nutch/Solr. The 
Nutch indexing system looks powerful as does Solr. My app is written in 
java/jsp. Can anyone point me to a working example front a front end in jsp or 
even struts? Is this viable? I can see there is solrj but I was hoping for a 
working example.

Regards
Ed Murray
Manager
--
SERENITY TRUST
Network Administration & I.T. Services
THIS MESSAGE CONTAINS CONFIDENTIAL INFORMATION AND IS INTENDED FOR THE 
ADDRESSED RECIPIENT(S) ONLY. IF YOU ARE NOT THE INTENDED RECIPIENT YOU ARE 
NOTIFIED THAT DISCLOSING, COPYING, DISTRIBUTING OR TAKING ANY ACTION IN 
RELIANCE ON THE CONTENTS OF THIS INFORMATION IS STRICTLY PROHIBITED.

Does CloudMLTQParser support getting related documents from different shards?

[Arnold Bronley]

Hi,

I have one custom Solr plugin that uses MoreLikeThis class. AFAIK,
MoreLikeThis handler does not support distributed mode and the issue is
still open for that - https://issues.apache.org/jira/browse/SOLR-5480.

However, I saw that there is some possibility to use CloudMLTQParser to
work around above issue. CloudMLTQParser claims to work in distributed mode
too -  https://issues.apache.org/jira/browse/SOLR-6248. When I took a look
at the code though,
https://github.com/apache/lucene-solr/blob/2d690885e554dda7b4b4e0f46f2bd9cacdb32df6/solr/core/src/java/org/apache/solr/search/mlt/CloudMLTQParser.java
,
it seems like that for the document for which we are finding out related
documents, that document is getting fetched now with real-time get request
handler.
core.getRequestHandler("/get").handleRequest(request, rsp);
This is good because now that document will get fetched from any shard
existing
in cloud wherever that document exists.

However, the part where the relevant related documents are supposed to be
fetched it still uses the same old sort of code.
e.g. MoreLikeThis mlt = new MoreLikeThis(req.getSearcher().
getIndexReader());

Isn't getSearcher bound to only the particular shard and AFAIK it does not
work across shards in cloud?

So how then, CloudMLTQParser works in SolrCloud mode? Does it even work?

Facet Range by Distance generating count of field - need sum of the field

[Robert Scavilla]

Thank yo in advance for your help.

I need to get the sum of a pivot range field. The following query uses the
stats function to sum the *sumField* values. I'm trying to sum the same
field in a frange subquery and I don't know how.

/select?defType=edismax=*:*={!geofilt}=totalResultsUsers,_dist_:geodist(),score=geodist()
desc=true=0=-1=1=value=true=miles=json=dId:193=Coordinates=40.243919,-74.753489=5={!tag=t1}sumField={!stats=t1}startDate=startDate:[2019-12-01
TO *]={!frange l=0 u=5}geodist()={!frange l=5.001
u=10}geodist()

This query produces the following results:

  "facet_counts":{
"facet_queries":{
  "{!frange l=0 u=5}geodist()":27,
  "{!frange l=5.001 u=10}geodist()":0},
"facet_pivot":{
  "startDate":[{
  "field":"startDate",
  "value":"2019-12-01",
  "count":27,
  "stats":{
"stats_fields":{
  "users":{
"min":1.0,
"max":158.0,
**"count":27,
"missing":0,
"sum":488.0,
"sumOfSquares":40848.0,
"mean":18.074074074074073,
"stddev":35.09758475793535]}},




What I'm going for is line: "*{!frange l=0 u=5}geodist()":27* should be
488.0 which is the sum of the field as opposed to 27 which is the count of
the field.

Thank you!

Re: Solr 8.3

[Erick Erickson]

Well, you can’t do what you want. The Admin UI is
intended as a way for new users to get 
started, it was never intended to support all of the
possible options.

Look at the documentation for the Collections
CREATE API, for instance:
https://lucene.apache.org/solr/guide/8_1/collections-api.html

In particular createNodeSet, and createNodeSet.shuffle. If
you want to control absolutely everything, the special
value EMPTY for createNodeSet won’t create any replicas
and you can place each one individually with ADDREPLICA. Of
course you can script this if you need automation.

What you haven’t explained is _why_ you want to do this. Very
often taking this kind of control is unnecessary and a distraction.
That said, when it _is_ necessary you need to use the Collections
API rather than the admin UI.


Best,
Erick


> On Jan 3, 2020, at 4:53 AM, vishal patel  
> wrote:
> 
> Thanks for the reply. Actually, I don't want to change the IP of the replica 
> but I want to create a collection on a specific shard IP.
> As per my production server configuration, we consider 10.38.33.28 is shard1, 
> 10.38.33.30 is shard2,  10.38.33.31 is replica of shard1 and 10.38.33.29 is 
> replica of shard2.
> We need to create all new collections the same as above.
> 
> 
> When I create a collection(documents) using the admin panel. It created like
> shard1 10.38.33.30
>   replica1 10.38.33.31
> shard2 10.38.33.28
>   replica2 10.38.33.29
> 
> Actually we want like this
> shard1 10.38.33.28
>   replica1 10.38.33.31
> shard2 10.38.33.30
>   replica2 10.38.33.29
> 
> Regards,
> Vishal
> 
> From: Sankar Panda 
> Sent: Friday, January 3, 2020 12:36 PM
> To: solr-user@lucene.apache.org 
> Subject: Re: Solr 8.3
> 
> Hi Vishal,
> 
> You can .go to the collection in admin console.mannually changed the ip 
> address as you want.Remove the replica and add it as per your 
> requirements.This option provides by the solr cloud.
> Thanks
> Sankar panda
> 
> On Fri, Jan 3, 2020, 11:33 vishal patel 
> mailto:vishalpatel200...@outlook.com>> wrote:
> I do not want to change the IP of the existing replica. I want to fix the IP 
> for the first time creating a collection.
> 
> I have 4 machines. my IP of each machine is below
> machine1 10.38.33.28
> machine2 10.38.33.29
> machine3 10.38.33.30
> machine4 10.38.33.31
> 
> I have created solr instance on each machine. when I am creating a first 
> collection(documents) using admin panel my structure becomes like this
> shard1 10.38.33.30
>   replica1 10.38.33.31
> shard2 10.38.33.28
>   replica2 10.38.33.29
> 
> When creating a second collection(forms), my structure becomes like this
> shard1 10.38.33.28
>   replica1 10.38.33.30
> shard2 10.38.33.29
>   replica2 10.38.33.31
> 
> I have attached a screenshot. [https://ibb.co/Yb2TpVX]
> 
> Randomly IP assigned to shard or replica when creating collection but I want 
> to make shard1 on 10.38.33.28, replica1 on 10.38.33.31, shard2 on 10.38.33.30 
> and replica2 on 10.38.33.31 when first time creating a collection using admin 
> panel.
> 
> Can I Fix the IP to shard OR replica when creating a new 
> [cid:4c560b03-a18d-4bc1-b592-6b2f8103f1ae] collection?
> 
> Regards,
> Vishal
> 
> 
> 
> From: Erick Erickson mailto:erickerick...@gmail.com>>
> Sent: Thursday, January 2, 2020 7:40 PM
> To: solr-user@lucene.apache.org 
> mailto:solr-user@lucene.apache.org>>
> Subject: Re: Solr 8.3
> 
> No, you cannot change the IP of an existing replica. Either do as Sankar 
> mentioned when you first create the collection or use the MOVREPLICA 
> collections API command.
> 
> MOVEREPLICA has existed for quite a long time, but if it’s not available, you 
> can do the same with the ADDREPLICA command to add a replica to a specific 
> node, wait for the replica to become fully active, then use DELETEREPLICA on 
> the one you no longer want.
> 
> Best,
> Erick
> 
>> On Jan 2, 2020, at 8:10 AM, Sankar Panda 
>> mailto:panda.san...@gmail.com>> wrote:
>> 
>> Hi Vishal,
>> You can create a empty nodeset and manually configure in the collection as
>> desired in the admin page
>> Thanks
>> Sankar Panda
>> 
>> On Thu, Jan 2, 2020, 14:36 vishal patel 
>> mailto:vishalpatel200...@outlook.com>>
>> wrote:
>> 
>>> My created collection in solr cloud below
>>> 
>>> 10.38.33.24 is shard and its replica is 10.38.33.27.
>>> 10.38.33.227 is shard and its replica is 10.38.33.219.
>>> 
>>> I want to create a new collection on the Same. can not change the shard IP
>>> for the new collection. How can I?
>>> 
>>> 
>>> --
>>> *From:* sudhir kumar 
>>> mailto:gogikarsud...@gmail.com>>
>>> *Sent:* Thursday, January 2, 2020 2:01 PM
>>> *To:* solr-user@lucene.apache.org 
>>> mailto:solr-user@lucene.apache.org>>
>>> *Subject:* Re: Solr 8.3
>>> 
>>> sample url to create collection:
>>> 
>>> 
>>> 

Re: Running Solr 4.8.1 as a 64 bit Windows Service

[Erick Erickson]

Have you tried just removing the startup param for MaxPermSize? That’s not 
something
that Solr _requires_, it’s a recommendation for the version you’re running 
though. That
said, there are about a zillion memory tuning parameters you can set, be 
prepared for
an adventure.

As to why you’re experiencing memory pressure in the first place, there’s 
another
zillion possibilities. One of the biggest is whether fields you try to facet, 
group, or
sort on have docValues set. Although that  won’t particularly help unless 
you’re on a
64 bit JVM and can use the MMapDirectory implementation (which is automatic if
you don’t override), see:
https://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html

That’ll move a huge amount of your index, _especially_ if you have correctly
specified docValues for the relevant fields out into the OS memory space.

So there’s no reason you can’t run Solr under 64 bit operating system, and
every reason you should so there’s something about your setup that’s incorrect
I’d guess.

NOTE: if you change the schema, you _must_ re-index from scratch, preferably
into a new core.

Finally, you mention multiple cores. Assuming you can’t get the 64 bit thing to
work, is there any possibility of moving the cores to separate machines?

Best,
Erick


> On Jan 2, 2020, at 8:24 PM, Michael Tobias  
> wrote:
> 
> Hi guys
> 
> 
> 
> We are still running Solr 4.8.1 (Bitnami Windows version) and cannot upgrade
> at this time.
> 
> 
> 
> Everything has worked fine for a year running as a 32 bit windows service
> (Windows Server 2012R2) but we are now experiencing dreadful memory issues -
> the JVM is being exhausted and I cannot do any sizeable updates.
> 
> 
> 
> We have added RAM to the server and I want to move to 64 bit java to take
> advantage of the increased RAM. BUT I am having real problems.  Either the
> service refuses to start or I can get it started but it won't load my two
> cores.  There was also a java error about java 8 not supporting the
> MaxPermSize parameter.. So I have reverted to 32 bit and Solr runs fine for
> searches but I cannot do updates to the cores.
> 
> 
> 
> Can anybody with experience of this please assist?
> 
> 
> 
> Regards
> 
> 
> 
> Michael
> 
> 
> 
> 
> 
> 
> 
> 
> 

Re: Facing jwt authentication problem using solr 8.1.1

[Jan Høydahl]

As I said, I don’t think you will be successful with a public OAuth provider 
like google or GitHub.
I know little about how or whether they issue access tokens at all. I think 
they just deal with ID tokens?

Jan

> 3. jan. 2020 kl. 08:07 skrev lakshgupta29 :
> 
> Hi Jan,
> 
> I've used Auth2.0 provider for admin UI Dashboard authentication. I've
> passed IdP endpoint and client id as per the provider details. It's logging
> in my dashboard successfully, not able to authenticate API, 
> getting below exception. 
> 
> o.a.s.s.SolrLogAuditLoggerPlugin type="REJECTED" message="Rejected"
> method="GET" status="401" requestType="UNKNOWN" username="null"
> resource="/solr" queryString="_=1578034784015=json" collections=null
> 
> As above exception, username is "null". 
> 
> Could you please help out? 
> 
> Please find the screenshot after provider authentication.
>  
> 
> 
> 
> 
> --
> Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Facing jwt authentication problem using solr 8.1.1

[lakshgupta29]

Hi Jan,

I've used Auth2.0 provider for admin UI Dashboard authentication. I've
passed IdP endpoint and client id as per the provider details. It's logging
in my dashboard successfully, not able to authenticate API, 
getting below exception. 

o.a.s.s.SolrLogAuditLoggerPlugin type="REJECTED" message="Rejected"
method="GET" status="401" requestType="UNKNOWN" username="null"
resource="/solr" queryString="_=1578034784015=json" collections=null

As above exception, username is "null". 

Could you please help out? 

Please find the screenshot after provider authentication.
 




--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Running Solr 4.8.1 as a 64 bit Windows Service

[Michael Tobias]

Hi guys

 

We are still running Solr 4.8.1 (Bitnami Windows version) and cannot upgrade
at this time.

 

Everything has worked fine for a year running as a 32 bit windows service
(Windows Server 2012R2) but we are now experiencing dreadful memory issues -
the JVM is being exhausted and I cannot do any sizeable updates.

 

We have added RAM to the server and I want to move to 64 bit java to take
advantage of the increased RAM. BUT I am having real problems.  Either the
service refuses to start or I can get it started but it won't load my two
cores.  There was also a java error about java 8 not supporting the
MaxPermSize parameter.. So I have reverted to 32 bit and Solr runs fine for
searches but I cannot do updates to the cores.

 

Can anybody with experience of this please assist?

 

Regards

 

Michael

 

 

 

 

Re: Question about the max num of solr node

[Jörn Franke]

Why do you want to set up so many? What are your designs in terms of volumes / 
no of documents etc? 


> Am 03.01.2020 um 10:32 schrieb Hongxu Ma :
> 
> Hi community
> I plan to set up a 128 host cluster: 2 solr nodes on each host.
> But I have a little concern about whether solr can support so many nodes.
> 
> I searched on wiki and found:
> https://cwiki.apache.org/confluence/display/SOLR/2019-11+Meeting+on+SolrCloud+and+project+health
> "If you create thousands of collections, it’ll lock up and become inoperable. 
>  Scott reported that If you boot up a 100+ node cluster, SolrCloud won’t get 
> to a happy state; currently you need to start them gradually."
> 
> I wonder to know:
> Beside the quoted items, does solr have known issues in a big cluster?
> And does solr have a hard limit number of max node?
> 
> Thanks.

Re: Solr 8.3

[vishal patel]

Thanks for the reply. Actually, I don't want to change the IP of the replica 
but I want to create a collection on a specific shard IP.
As per my production server configuration, we consider 10.38.33.28 is shard1, 
10.38.33.30 is shard2,  10.38.33.31 is replica of shard1 and 10.38.33.29 is 
replica of shard2.
We need to create all new collections the same as above.


When I create a collection(documents) using the admin panel. It created like
shard1 10.38.33.30
   replica1 10.38.33.31
shard2 10.38.33.28
   replica2 10.38.33.29

Actually we want like this
shard1 10.38.33.28
   replica1 10.38.33.31
shard2 10.38.33.30
   replica2 10.38.33.29

Regards,
Vishal

From: Sankar Panda 
Sent: Friday, January 3, 2020 12:36 PM
To: solr-user@lucene.apache.org 
Subject: Re: Solr 8.3

Hi Vishal,

You can .go to the collection in admin console.mannually changed the ip address 
as you want.Remove the replica and add it as per your requirements.This option 
provides by the solr cloud.
Thanks
Sankar panda

On Fri, Jan 3, 2020, 11:33 vishal patel 
mailto:vishalpatel200...@outlook.com>> wrote:
I do not want to change the IP of the existing replica. I want to fix the IP 
for the first time creating a collection.

I have 4 machines. my IP of each machine is below
machine1 10.38.33.28
machine2 10.38.33.29
machine3 10.38.33.30
machine4 10.38.33.31

I have created solr instance on each machine. when I am creating a first 
collection(documents) using admin panel my structure becomes like this
shard1 10.38.33.30
   replica1 10.38.33.31
shard2 10.38.33.28
   replica2 10.38.33.29

When creating a second collection(forms), my structure becomes like this
shard1 10.38.33.28
   replica1 10.38.33.30
shard2 10.38.33.29
   replica2 10.38.33.31

I have attached a screenshot. [https://ibb.co/Yb2TpVX]

Randomly IP assigned to shard or replica when creating collection but I want to 
make shard1 on 10.38.33.28, replica1 on 10.38.33.31, shard2 on 10.38.33.30 and 
replica2 on 10.38.33.31 when first time creating a collection using admin panel.

Can I Fix the IP to shard OR replica when creating a new 
[cid:4c560b03-a18d-4bc1-b592-6b2f8103f1ae] collection?

Regards,
Vishal



From: Erick Erickson mailto:erickerick...@gmail.com>>
Sent: Thursday, January 2, 2020 7:40 PM
To: solr-user@lucene.apache.org 
mailto:solr-user@lucene.apache.org>>
Subject: Re: Solr 8.3

No, you cannot change the IP of an existing replica. Either do as Sankar 
mentioned when you first create the collection or use the MOVREPLICA 
collections API command.

MOVEREPLICA has existed for quite a long time, but if it’s not available, you 
can do the same with the ADDREPLICA command to add a replica to a specific 
node, wait for the replica to become fully active, then use DELETEREPLICA on 
the one you no longer want.

Best,
Erick

> On Jan 2, 2020, at 8:10 AM, Sankar Panda 
> mailto:panda.san...@gmail.com>> wrote:
>
> Hi Vishal,
> You can create a empty nodeset and manually configure in the collection as
> desired in the admin page
> Thanks
> Sankar Panda
>
> On Thu, Jan 2, 2020, 14:36 vishal patel 
> mailto:vishalpatel200...@outlook.com>>
> wrote:
>
>> My created collection in solr cloud below
>>
>> 10.38.33.24 is shard and its replica is 10.38.33.27.
>> 10.38.33.227 is shard and its replica is 10.38.33.219.
>>
>> I want to create a new collection on the Same. can not change the shard IP
>> for the new collection. How can I?
>>
>>
>> --
>> *From:* sudhir kumar 
>> mailto:gogikarsud...@gmail.com>>
>> *Sent:* Thursday, January 2, 2020 2:01 PM
>> *To:* solr-user@lucene.apache.org 
>> mailto:solr-user@lucene.apache.org>>
>> *Subject:* Re: Solr 8.3
>>
>> sample url to create collection:
>>
>>
>> http//host:8080/solr/admin/collections?action=CREATE=collectionname=2=3=2=
>> host:8080_solr,host:8080_solr,host:8080_solr,host:8080_solr
>> =collectionconfig
>>
>> On Thu, Jan 2, 2020 at 1:56 PM sudhir kumar 
>> mailto:gogikarsud...@gmail.com>>
>> wrote:
>>
>>> Hey Vishal,
>>>
>>> You can use createNodeSet property while creating collection which will
>>> allows you to create shards on specified IP.
>>>
>>> /admin/collections?action=CREATE=*name*=*number*
>>> =*number*&*maxShardsPerNode*=*number*&*createNodeSet*=
>>> *nodelist*=*configname*
>>>
>>>
>>> Thanks,
>>>
>>> Sudhir
>>>
>>>
>>> On Thu, Jan 2, 2020 at 1:49 PM vishal patel <
>> vishalpatel200...@outlook.com>
>>> wrote:
>>>
 When I am creating 2 shards and 2 replicas using admin panel, automatic
 assign a shard or replica to any IP.
 I want to make the specific shard or replica to solr instance at the
>> time
 of creating a collection. Can I?

 Regards,
 Vishal

>>>
>>

Question about the max num of solr node

[Hongxu Ma]

Hi community
I plan to set up a 128 host cluster: 2 solr nodes on each host.
But I have a little concern about whether solr can support so many nodes.

I searched on wiki and found:
https://cwiki.apache.org/confluence/display/SOLR/2019-11+Meeting+on+SolrCloud+and+project+health
"If you create thousands of collections, it’ll lock up and become inoperable.  
Scott reported that If you boot up a 100+ node cluster, SolrCloud won’t get to 
a happy state; currently you need to start them gradually."

I wonder to know:
Beside the quoted items, does solr have known issues in a big cluster?
And does solr have a hard limit number of max node?

Thanks.