Re: Unbalanced shard requests

[Phill Campbell]

In my previous report I was configured to use as much RAM as possible. With 
that configuration it seemed it was not load balancing.
So, I reconfigured and redeployed to use 1/4 the RAM. What a difference for the 
better!

10.156.112.50   load average: 13.52, 10.56, 6.46
10.156.116.34   load average: 11.23, 12.35, 9.63
10.156.122.13   load average: 10.29, 12.40, 9.69

Very nice.
My tool that tests records RPS. In the “bad” configuration it was less than 1 
RPS.
NOW it is showing 21 RPS.

http://10.156.112.50:10002/solr/admin/metrics?group=core=QUERY./select.requestTimes
 

{
  "responseHeader":{
"status":0,
"QTime":161},
  "metrics":{
"solr.core.BTS.shard1.replica_n2":{
  "QUERY./select.requestTimes":{
"count":5723,
"meanRate":6.8163888639859085,
"1minRate":11.557013215119536,
"5minRate":8.760356217628159,
"15minRate":4.707624230995833,
"min_ms":0.131545,
"max_ms":388.710848,
"mean_ms":30.300492048215947,
"median_ms":6.336654,
"stddev_ms":51.527164088667035,
"p75_ms":35.427943,
"p95_ms":140.025957,
"p99_ms":230.533099,
"p999_ms":388.710848


http://10.156.122.13:10004/solr/admin/metrics?group=core=QUERY./select.requestTimes
 

{
  "responseHeader":{
"status":0,
"QTime":11},
  "metrics":{
"solr.core.BTS.shard2.replica_n8":{
  "QUERY./select.requestTimes":{
"count":6469,
"meanRate":7.502581801189549,
"1minRate":12.211423085368564,
"5minRate":9.445681397767322,
"15minRate":5.216209798637846,
"min_ms":0.154691,
"max_ms":701.657394,
"mean_ms":34.2734699171445,
"median_ms":5.640378,
"stddev_ms":62.27649205954566,
"p75_ms":39.016371,
"p95_ms":156.997982,
"p99_ms":288.883028,
"p999_ms":538.368031

http://10.156.116.34:10002/solr/admin/metrics?group=core=QUERY./select.requestTimes
 

{
  "responseHeader":{
"status":0,
"QTime":67},
  "metrics":{
"solr.core.BTS.shard3.replica_n16":{
  "QUERY./select.requestTimes":{
"count":7109,
"meanRate":7.787524673806184,
"1minRate":11.88519763582083,
"5minRate":9.893315557386755,
"15minRate":5.620178363676527,
"min_ms":0.150887,
"max_ms":472.826462,
"mean_ms":32.184282366621204,
"median_ms":6.977733,
"stddev_ms":55.729908615189196,
"p75_ms":36.655011,
"p95_ms":151.12627,
"p99_ms":251.440162,
"p999_ms":472.826462


Compare that to the previous report and you can see the improvement.
So, note to myself. Figure out the sweet spot for RAM usage. Use too much and 
strange behavior is noticed. While using too much all the load focused on one 
box and query times slowed.
I did not see any OOM errors during any of this.

Regards



> On May 18, 2020, at 3:23 PM, Phill Campbell  
> wrote:
> 
> I have been testing 8.5.2 and it looks like the load has moved but is still 
> on one machine.
> 
> Setup:
> 3 physical machines.
> Each machine hosts 8 instances of Solr.
> Each instance of Solr hosts one replica.
> 
> Another way to say it:
> Number of shards = 8. Replication factor = 3.
> 
> Here is the cluster state. You can see that the leaders are well distributed. 
> 
> {"TEST_COLLECTION":{
>"pullReplicas":"0",
>"replicationFactor":"3",
>"shards":{
>  "shard1":{
>"range":"8000-9fff",
>"state":"active",
>"replicas":{
>  "core_node3":{
>"core":"TEST_COLLECTION_shard1_replica_n1",
>"base_url":"http://10.156.122.13:10007/solr;,
>"node_name":"10.156.122.13:10007_solr",
>"state":"active",
>"type":"NRT",
>"force_set_state":"false"},
>  "core_node5":{
>"core":"TEST_COLLECTION_shard1_replica_n2",
>"base_url":"http://10.156.112.50:10002/solr;,
>"node_name":"10.156.112.50:10002_solr",
>"state":"active",
>"type":"NRT",
>"force_set_state":"false",
>"leader":"true"},
>  "core_node7":{
>"core":"TEST_COLLECTION_shard1_replica_n4",
>"base_url":"http://10.156.112.50:10006/solr;,
>"node_name":"10.156.112.50:10006_solr",
>"state":"active",
>"type":"NRT",
>"force_set_state":"false"}}},
>  "shard2":{
>"range":"a000-bfff",
>"state":"active",
>"replicas":{
>  "core_node9":{
>"core":"TEST_COLLECTION_shard2_replica_n6",
>"base_url":"http://10.156.112.50:10003/solr;,
>"node_name":"10.156.112.50:10003_solr",

Re: Unbalanced shard requests

[Phill Campbell]

I have been testing 8.5.2 and it looks like the load has moved but is still on 
one machine.

Setup:
3 physical machines.
Each machine hosts 8 instances of Solr.
Each instance of Solr hosts one replica.

Another way to say it:
Number of shards = 8. Replication factor = 3.

Here is the cluster state. You can see that the leaders are well distributed. 

{"TEST_COLLECTION":{
"pullReplicas":"0",
"replicationFactor":"3",
"shards":{
  "shard1":{
"range":"8000-9fff",
"state":"active",
"replicas":{
  "core_node3":{
"core":"TEST_COLLECTION_shard1_replica_n1",
"base_url":"http://10.156.122.13:10007/solr;,
"node_name":"10.156.122.13:10007_solr",
"state":"active",
"type":"NRT",
"force_set_state":"false"},
  "core_node5":{
"core":"TEST_COLLECTION_shard1_replica_n2",
"base_url":"http://10.156.112.50:10002/solr;,
"node_name":"10.156.112.50:10002_solr",
"state":"active",
"type":"NRT",
"force_set_state":"false",
"leader":"true"},
  "core_node7":{
"core":"TEST_COLLECTION_shard1_replica_n4",
"base_url":"http://10.156.112.50:10006/solr;,
"node_name":"10.156.112.50:10006_solr",
"state":"active",
"type":"NRT",
"force_set_state":"false"}}},
  "shard2":{
"range":"a000-bfff",
"state":"active",
"replicas":{
  "core_node9":{
"core":"TEST_COLLECTION_shard2_replica_n6",
"base_url":"http://10.156.112.50:10003/solr;,
"node_name":"10.156.112.50:10003_solr",
"state":"active",
"type":"NRT",
"force_set_state":"false"},
  "core_node11":{
"core":"TEST_COLLECTION_shard2_replica_n8",
"base_url":"http://10.156.122.13:10004/solr;,
"node_name":"10.156.122.13:10004_solr",
"state":"active",
"type":"NRT",
"force_set_state":"false",
"leader":"true"},
  "core_node12":{
"core":"TEST_COLLECTION_shard2_replica_n10",
"base_url":"http://10.156.116.34:10008/solr;,
"node_name":"10.156.116.34:10008_solr",
"state":"active",
"type":"NRT",
"force_set_state":"false"}}},
  "shard3":{
"range":"c000-dfff",
"state":"active",
"replicas":{
  "core_node15":{
"core":"TEST_COLLECTION_shard3_replica_n13",
"base_url":"http://10.156.122.13:10008/solr;,
"node_name":"10.156.122.13:10008_solr",
"state":"active",
"type":"NRT",
"force_set_state":"false"},
  "core_node17":{
"core":"TEST_COLLECTION_shard3_replica_n14",
"base_url":"http://10.156.116.34:10005/solr;,
"node_name":"10.156.116.34:10005_solr",
"state":"active",
"type":"NRT",
"force_set_state":"false"},
  "core_node19":{
"core":"TEST_COLLECTION_shard3_replica_n16",
"base_url":"http://10.156.116.34:10002/solr;,
"node_name":"10.156.116.34:10002_solr",
"state":"active",
"type":"NRT",
"force_set_state":"false",
"leader":"true"}}},
  "shard4":{
"range":"e000-",
"state":"active",
"replicas":{
  "core_node20":{
"core":"TEST_COLLECTION_shard4_replica_n18",
"base_url":"http://10.156.122.13:10001/solr;,
"node_name":"10.156.122.13:10001_solr",
"state":"active",
"type":"NRT",
"force_set_state":"false"},
  "core_node23":{
"core":"TEST_COLLECTION_shard4_replica_n21",
"base_url":"http://10.156.116.34:10004/solr;,
"node_name":"10.156.116.34:10004_solr",
"state":"active",
"type":"NRT",
"force_set_state":"false"},
  "core_node25":{
"core":"TEST_COLLECTION_shard4_replica_n22",
"base_url":"http://10.156.112.50:10001/solr;,
"node_name":"10.156.112.50:10001_solr",
"state":"active",
"type":"NRT",
"force_set_state":"false",
"leader":"true"}}},
  "shard5":{
"range":"0-1fff",
"state":"active",
"replicas":{
  "core_node27":{
"core":"TEST_COLLECTION_shard5_replica_n24",
"base_url":"http://10.156.116.34:10007/solr;,
"node_name":"10.156.116.34:10007_solr",
"state":"active",
"type":"NRT",
"force_set_state":"false"},
  "core_node29":{
"core":"TEST_COLLECTION_shard5_replica_n26",
"base_url":"http://10.156.122.13:10006/solr;,

Re: What is the logical order of applying sorts in SOLR?

[Stephen Lewis Bianamara]

I don't see how that would work. A boost would increase the relevance based
on a query, so how would it be able to apply a sort by the unique key field?

On Sat, May 16, 2020 at 3:51 PM David Hastings 
wrote:

> the bq parameter, heres a SO thread for it:
>
>
> https://stackoverflow.com/questions/45150856/how-to-know-when-to-use-solr-bq-vs-bf-and-how-to-apply-query-logic
>
> On Sat, May 16, 2020 at 6:27 PM Stephen Lewis Bianamara <
> stephen.bianam...@gmail.com> wrote:
>
> > Hi Paras,
> >
> > I'm not sure I follow. How would one replace sort by id with boosting?
> >
> > Thanks,
> > Stephen
> >
> > On Fri, May 15, 2020, 7:43 AM Paras Lehana
> >  wrote:
> >
> > > As a workaround, can you try field boosting?
> > >
> > > On Tue, 12 May 2020 at 00:45, Stephen Lewis Bianamara <
> > > stephen.bianam...@gmail.com> wrote:
> > >
> > > > Hi SOLR Community,
> > > >
> > > > What is the order of operations which SOLR applies to sorting? I've
> > > > observed many times and across SOLR versions that a restrictive
> filter
> > > with
> > > > a sort takes an extremely long time to return, suggesting to me that
> > the
> > > > SORT is applied before the filter.
> > > >
> > > > An example situation is querying for fq:Foo=Bar vs querying for
> > > fq:Foo=Bar
> > > > sort by Id desc. I've observed over many SOLR versions and
> collections
> > > that
> > > > the former is orders of magnitude cheaper and quicker to respond,
> even
> > > when
> > > > the result set is tiny (10-100).
> > > >
> > > > Does anyone in this forum know whether this is the default behavior
> and
> > > > whether there is any way through the API or SOLR configuration to
> apply
> > > > sorts after filters?
> > > >
> > > > Thanks,
> > > > Stephen
> > > >
> > >
> > >
> > > --
> > > --
> > > Regards,
> > >
> > > *Paras Lehana* [65871]
> > > Development Engineer, *Auto-Suggest*,
> > > IndiaMART InterMESH Ltd,
> > >
> > > 11th Floor, Tower 2, Assotech Business Cresterra,
> > > Plot No. 22, Sector 135, Noida, Uttar Pradesh, India 201305
> > >
> > > Mob.: +91-9560911996
> > > Work: 0120-4056700 | Extn:
> > > *1196*
> > >
> > > --
> > > *
> > > *
> > >
> > >  
> > >
> >
>

Re: How to determine why solr stops running?

[James Greene]

I usually do a combination of grepping for ERROR in solr logs and checking
journalctl to see if an external program may have killed the process.


Cheers,

/
*   James Austin Greene
*  www.jamesaustingreene.com
*  336-lol-nerd
/


On Mon, May 18, 2020 at 1:39 PM Erick Erickson 
wrote:

> ps aux | grep solr
>
> on a *.nix system will show you all the runtime parameters.
>
> > On May 18, 2020, at 12:46 PM, Ryan W  wrote:
> >
> > Is there a config file containing the start params?  I run solr like...
> >
> > bin/solr start
> >
> > I have not seen anything in the logs that seems informative. When I grep
> in
> > the logs directory for 'memory', I see nothing besides a couple entries
> > like...
> >
> > 2020-05-14 13:05:56.155 INFO  (main) [   ]
> o.a.s.h.a.MetricsHistoryHandler
> > No .system collection, keeping metrics history in memory.
> >
> > I don't know what that entry means, though the date does roughly coincide
> > with the last time solr stopped running.
> >
> > Thank you.
> >
> >
> > On Mon, May 18, 2020 at 12:00 PM Erick Erickson  >
> > wrote:
> >
> >> Probably, but check that you are running with the oom-killer, it'll be
> in
> >> your start params.
> >>
> >> But absent that, something external will be the culprit, Solr doesn't
> stop
> >> by itself. Do look at the Solr log once things stop, it should show if
> >> someone or something stopped it.
> >>
> >> On Mon, May 18, 2020, 10:43 Ryan W  wrote:
> >>
> >>> I don't see any log file with "oom" in the file name.  Does that mean
> >> there
> >>> hasn't been an out-of-memory issue?  Thanks.
> >>>
> >>> On Thu, May 14, 2020 at 10:05 AM James Greene <
> >> ja...@jamesaustingreene.com
> 
> >>> wrote:
> >>>
>  Check the log for for an OOM crash.  Fatal exceptions will be in the
> >> main
>  solr log and out of memory errors will be in their own -oom log.
> 
>  I've encountered quite a few solr crashes and usually it's when
> >> there's a
>  threshold of concurrent users and/or indexing happening.
> 
> 
> 
>  On Thu, May 14, 2020, 9:23 AM Ryan W  wrote:
> 
> > Hi all,
> >
> > I manage a site where solr has stopped running a couple times in the
> >>> past
> > week. The server hasn't been rebooted, so that's not the reason.
> >> What
>  else
> > causes solr to stop running?  How can I investigate why this is
>  happening?
> >
> > Thank you,
> > Ryan
> >
> 
> >>>
> >>
>
>

Re: Fetch related documents from Custom Function

[mganeshs]

Yes. But being inside solr ( I mean code getting executing via Custom
function ), do we have option to read the other solr documents in a easy
way.



--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: How to determine why solr stops running?

[Erick Erickson]

ps aux | grep solr

on a *.nix system will show you all the runtime parameters.

> On May 18, 2020, at 12:46 PM, Ryan W  wrote:
> 
> Is there a config file containing the start params?  I run solr like...
> 
> bin/solr start
> 
> I have not seen anything in the logs that seems informative. When I grep in
> the logs directory for 'memory', I see nothing besides a couple entries
> like...
> 
> 2020-05-14 13:05:56.155 INFO  (main) [   ] o.a.s.h.a.MetricsHistoryHandler
> No .system collection, keeping metrics history in memory.
> 
> I don't know what that entry means, though the date does roughly coincide
> with the last time solr stopped running.
> 
> Thank you.
> 
> 
> On Mon, May 18, 2020 at 12:00 PM Erick Erickson 
> wrote:
> 
>> Probably, but check that you are running with the oom-killer, it'll be in
>> your start params.
>> 
>> But absent that, something external will be the culprit, Solr doesn't stop
>> by itself. Do look at the Solr log once things stop, it should show if
>> someone or something stopped it.
>> 
>> On Mon, May 18, 2020, 10:43 Ryan W  wrote:
>> 
>>> I don't see any log file with "oom" in the file name.  Does that mean
>> there
>>> hasn't been an out-of-memory issue?  Thanks.
>>> 
>>> On Thu, May 14, 2020 at 10:05 AM James Greene <
>> ja...@jamesaustingreene.com
 
>>> wrote:
>>> 
 Check the log for for an OOM crash.  Fatal exceptions will be in the
>> main
 solr log and out of memory errors will be in their own -oom log.
 
 I've encountered quite a few solr crashes and usually it's when
>> there's a
 threshold of concurrent users and/or indexing happening.
 
 
 
 On Thu, May 14, 2020, 9:23 AM Ryan W  wrote:
 
> Hi all,
> 
> I manage a site where solr has stopped running a couple times in the
>>> past
> week. The server hasn't been rebooted, so that's not the reason.
>> What
 else
> causes solr to stop running?  How can I investigate why this is
 happening?
> 
> Thank you,
> Ryan
> 
 
>>> 
>> 

Re: How to determine why solr stops running?

[Ryan W]

Is there a config file containing the start params?  I run solr like...

bin/solr start

I have not seen anything in the logs that seems informative. When I grep in
the logs directory for 'memory', I see nothing besides a couple entries
like...

2020-05-14 13:05:56.155 INFO  (main) [   ] o.a.s.h.a.MetricsHistoryHandler
No .system collection, keeping metrics history in memory.

I don't know what that entry means, though the date does roughly coincide
with the last time solr stopped running.

Thank you.


On Mon, May 18, 2020 at 12:00 PM Erick Erickson 
wrote:

> Probably, but check that you are running with the oom-killer, it'll be in
> your start params.
>
> But absent that, something external will be the culprit, Solr doesn't stop
> by itself. Do look at the Solr log once things stop, it should show if
> someone or something stopped it.
>
> On Mon, May 18, 2020, 10:43 Ryan W  wrote:
>
> > I don't see any log file with "oom" in the file name.  Does that mean
> there
> > hasn't been an out-of-memory issue?  Thanks.
> >
> > On Thu, May 14, 2020 at 10:05 AM James Greene <
> ja...@jamesaustingreene.com
> > >
> > wrote:
> >
> > > Check the log for for an OOM crash.  Fatal exceptions will be in the
> main
> > > solr log and out of memory errors will be in their own -oom log.
> > >
> > > I've encountered quite a few solr crashes and usually it's when
> there's a
> > > threshold of concurrent users and/or indexing happening.
> > >
> > >
> > >
> > > On Thu, May 14, 2020, 9:23 AM Ryan W  wrote:
> > >
> > > > Hi all,
> > > >
> > > > I manage a site where solr has stopped running a couple times in the
> > past
> > > > week. The server hasn't been rebooted, so that's not the reason.
> What
> > > else
> > > > causes solr to stop running?  How can I investigate why this is
> > > happening?
> > > >
> > > > Thank you,
> > > > Ryan
> > > >
> > >
> >
>

Re: How to determine why solr stops running?

[Erick Erickson]

Probably, but check that you are running with the oom-killer, it'll be in
your start params.

But absent that, something external will be the culprit, Solr doesn't stop
by itself. Do look at the Solr log once things stop, it should show if
someone or something stopped it.

On Mon, May 18, 2020, 10:43 Ryan W  wrote:

> I don't see any log file with "oom" in the file name.  Does that mean there
> hasn't been an out-of-memory issue?  Thanks.
>
> On Thu, May 14, 2020 at 10:05 AM James Greene  >
> wrote:
>
> > Check the log for for an OOM crash.  Fatal exceptions will be in the main
> > solr log and out of memory errors will be in their own -oom log.
> >
> > I've encountered quite a few solr crashes and usually it's when there's a
> > threshold of concurrent users and/or indexing happening.
> >
> >
> >
> > On Thu, May 14, 2020, 9:23 AM Ryan W  wrote:
> >
> > > Hi all,
> > >
> > > I manage a site where solr has stopped running a couple times in the
> past
> > > week. The server hasn't been rebooted, so that's not the reason.  What
> > else
> > > causes solr to stop running?  How can I investigate why this is
> > happening?
> > >
> > > Thank you,
> > > Ryan
> > >
> >
>

Re: Login Mechanism in SOLR

[Fiz N]

Yes Thanks and it was helpful.

Thanks
Fiz..

On Mon, May 18, 2020 at 8:14 AM Jan Høydahl  wrote:

> Hi,
>
> Have you read what the reference guide has to say on the topic?
>
>
> https://lucene.apache.org/solr/guide/8_5/authentication-and-authorization-plugins.html
>
> Jan
>
> > 18. mai 2020 kl. 17:00 skrev Fiz N :
> >
> > Hi SOLR Experts,
> >
> > Could you please tell us options for login mechanism in SOLR? Is custom
> or
> > application login possible and also are SSO and LDAP based
> > authentication/authorization possible? What is the best approach I can
> > proceed with?
> >
> > Thanks & Regards
> > Fiz Ahmed.
>
>

Re: Login Mechanism in SOLR

[Jan Høydahl]

Hi,

Have you read what the reference guide has to say on the topic?

https://lucene.apache.org/solr/guide/8_5/authentication-and-authorization-plugins.html

Jan

> 18. mai 2020 kl. 17:00 skrev Fiz N :
> 
> Hi SOLR Experts,
> 
> Could you please tell us options for login mechanism in SOLR? Is custom or
> application login possible and also are SSO and LDAP based
> authentication/authorization possible? What is the best approach I can
> proceed with?
> 
> Thanks & Regards
> Fiz Ahmed.

Login Mechanism in SOLR

[Fiz N]

Hi SOLR Experts,

Could you please tell us options for login mechanism in SOLR? Is custom or
application login possible and also are SSO and LDAP based
authentication/authorization possible? What is the best approach I can
proceed with?

Thanks & Regards
Fiz Ahmed.

Re: How to determine why solr stops running?

[Ryan W]

I don't see any log file with "oom" in the file name.  Does that mean there
hasn't been an out-of-memory issue?  Thanks.

On Thu, May 14, 2020 at 10:05 AM James Greene 
wrote:

> Check the log for for an OOM crash.  Fatal exceptions will be in the main
> solr log and out of memory errors will be in their own -oom log.
>
> I've encountered quite a few solr crashes and usually it's when there's a
> threshold of concurrent users and/or indexing happening.
>
>
>
> On Thu, May 14, 2020, 9:23 AM Ryan W  wrote:
>
> > Hi all,
> >
> > I manage a site where solr has stopped running a couple times in the past
> > week. The server hasn't been rebooted, so that's not the reason.  What
> else
> > causes solr to stop running?  How can I investigate why this is
> happening?
> >
> > Thank you,
> > Ryan
> >
>

Re: Fetch related documents from Custom Function

[Mikhail Khludnev]

Hello,
It sounds either like classic denormalization or (little bit slow and
cumbersome) result transformer [subquery].

On Mon, May 18, 2020 at 4:04 PM mganeshs  wrote:

> Is there a easy possibility of reading the few field from related documents
> from Custom function ?
>
> For ex, Project document contains, project id, project name, Project
> manager
> id  ( which is nothing but employee id ). & Employee document contains
> field
> ( Employee id, Employee name ). Now while querying the Project documents,
> in
> a custom function want to pass project manager id, and would like to read
> employee document of that Project manager and return employee name of that
> project manager.
>
> WE can do Join, but for various reason, for me Join won't work. So would
> like to read the employee document from the custom function. As Custom
> function is getting executed inside SOLR, what's the easy to read the other
> documents in SOLR, instead of establishing new connection via solrj and
> read
> it.
>
> Thanks in advance.
>
>
>
> --
> Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>


-- 
Sincerely yours
Mikhail Khludnev

Re: problem to migrate solr 4.6 to 8.4

[Erick Erickson]

First, did you check the Solr log on the server that’s getting the
docs to index? Sometimes there’s a better error message/stack trace
there.

Second, although people do use DIH for production, I tend to think
of it as a PoC tool. You can pretty easily write a SolrJ program
that does the import, and that gives you a lot more control. Here;s
a sample of doing that:

https://lucidworks.com/post/indexing-with-solrj/

Best,
Erick

> On May 18, 2020, at 8:30 AM, Eric DEWITTE  wrote:
> 
> Hello,
> 
> I need help because I'm stuck on a data import.
> 
> I have an old server under solr 4.6 and I wanted to import data from it to a 
> new server 8.4
> 
> The old server has a pretty big core with 352 million records...
> 
> I managed to import in increments of 100,000 up to 169 million, but since 
> then I've been getting these error messages:
> Exception in thread "main" org.apache.solr.client.solrj.SolrServerException: 
> Timeout occurred while waiting response from server at: 
> http://127.0.0.1:8984/solr/core1
>at 
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:676)
>at 
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:265)
>at 
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
>at 
> org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
>at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:1003)
>at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:1018)
>at Migrator.main(Migrator.java:43)
> Caused by: java.net.SocketTimeoutException: Read timed out
> 
> I tried to lower the slice (50,000 instead of 100,000) but same effects.
> 
> I don't know where I can go to increase the timeout or if I have to do 
> something else to be able to continue and finish my import.
> 
> Thank you
> 
> E.D.

Fetch related documents from Custom Function

[mganeshs]

Is there a easy possibility of reading the few field from related documents
from Custom function ? 

For ex, Project document contains, project id, project name, Project manager
id  ( which is nothing but employee id ). & Employee document contains field
( Employee id, Employee name ). Now while querying the Project documents, in
a custom function want to pass project manager id, and would like to read
employee document of that Project manager and return employee name of that
project manager. 

WE can do Join, but for various reason, for me Join won't work. So would
like to read the employee document from the custom function. As Custom
function is getting executed inside SOLR, what's the easy to read the other
documents in SOLR, instead of establishing new connection via solrj and read
it.

Thanks in advance.



--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html

problem to migrate solr 4.6 to 8.4

[Eric DEWITTE]

Hello,

I need help because I'm stuck on a data import.

I have an old server under solr 4.6 and I wanted to import data from it to a 
new server 8.4

The old server has a pretty big core with 352 million records...

I managed to import in increments of 100,000 up to 169 million, but since then 
I've been getting these error messages:
Exception in thread "main" org.apache.solr.client.solrj.SolrServerException: 
Timeout occurred while waiting response from server at: 
http://127.0.0.1:8984/solr/core1
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:676)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:265)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:1003)
at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:1018)
at Migrator.main(Migrator.java:43)
Caused by: java.net.SocketTimeoutException: Read timed out

I tried to lower the slice (50,000 instead of 100,000) but same effects.

I don't know where I can go to increase the timeout or if I have to do 
something else to be able to continue and finish my import.

Thank you

E.D.

Re: Performance issue in Query execution in Solr 8.3.0 and 8.5.1

[vishal patel]

Any one is looking my issue? Due to this issue I can not upgrade Solr 8.3.0.

regards,
Vishal Patel

From: vishal patel 
Sent: Sunday, May 17, 2020 11:49 AM
To: solr-user 
Subject: Re: Performance issue in Query execution in Solr 8.3.0 and 8.5.1

Solr 6.1.0 : 1881

Here is my thread dump stack trace and log for Solr 6.1.0. It is helpful for 
you.
My threads: qtp557041912-245356 and qtp557041912-245342.
https://drive.google.com/file/d/1owtotYEnJacMiEZyuGLk3AHQ9kQG5rww/view?usp=sharing

Regards
Vishal Patel



From: vishal patel 
Sent: Sunday, May 17, 2020 11:04 AM
To: solr-user 
Subject: Re: Performance issue in Query execution in Solr 8.3.0 and 8.5.1

Thanks for reply.

I know Query field value is large. But same thing is working fine in Solr 6.1.0 
and query executed within 300 milliseconds. Schema.xml and Solrconfig.xml are 
same. Why is it taking lots of time for execution in Solr 8.3.0?

Is there any changes in Solr 8.3.0?

Regards,
Vishal Patel

From: Mikhail Khludnev 
Sent: Saturday, May 16, 2020 6:55 PM
To: solr-user 
Subject: Re: Performance issue in Query execution in Solr 8.3.0 and 8.5.1

It seems this thread is doing heavy work, mind the bottom line.

202.8013ms
124.8008ms
qtp153245266-156 (156)
org.apache.lucene.search.similarities.BM25Similarity$BM25Scorer.(BM25Similarity.java:219)
org.apache.lucene.search.similarities.BM25Similarity.scorer(BM25Similarity.java:192)
org.apache.lucene.search.similarities.PerFieldSimilarityWrapper.scorer(PerFieldSimilarityWrapper.java:47)
org.apache.lucene.search.TermQuery$TermWeight.(TermQuery.java:74)
org.apache.lucene.search.TermQuery.createWeight(TermQuery.java:205)
org.apache.lucene.search.IndexSearcher.createWeight(IndexSearcher.java:726)
org.apache.lucene.search.BooleanWeight.(BooleanWeight.java:63)
org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:231)
org.apache.lucene.search.IndexSearcher.createWeight(IndexSearcher.java:726)
org.apache.lucene.search.TopFieldCollector.populateScores(TopFieldCollector.java:531)
org.apache.solr.search.grouping.distributed.command.TopGroupsFieldCommand.postCollect(TopGroupsFieldCommand.java:178)
org.apache.solr.search.grouping.CommandHandler.execute(CommandHandler.java:168)
org.apache.solr.handler.component.QueryComponent.doProcessGroupedDistributedSearchSecondPhase(QueryComponent.java:1403)
org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:387)
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:328)
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)
org.apache.solr.core.SolrCore.execute(SolrCore.java:2596)


It seems like it ranks groups by query score, that doubtful thing to do.

>From the log. Here's how to recognize query running 25 sec "QTime=25063"


Query itself q=+msg_id:(10519539+10519540+10523575+10523576+ ... is
not what search engines are made for. They are purposed for short
query.

You may

1. leverage {!terms} query parser which might handle such long terms
list more efficiently

2. make sure you don't enable unnecessary grouping features, eg group
ranking in the stack above makes no sense for this kind of query


It's worth to revamp an overall approach in favor of query time
{!join} or index time join see {!parent}/nested docs.



On Sat, May 16, 2020 at 1:46 PM vishal patel 
wrote:

> Thanks for reply.
>
> I have taken a thread dump at the time of query execution. I do not know
> the thread name so send the All threads. I have also send the logs so you
> can get idea.
>
> Thread Dump All Stack Trace:
> https://drive.google.com/file/d/1N4rVXJoaAwNvPIY2aw57gKA9mb4vRTMR/view
> Solr 8.3 shard 1 log:
> https://drive.google.com/file/d/1h5d_eZfQvYET7JKzbNKZwhZ_RmaX7hWf/view
> Solr 8.3 shard 2 log:
> https://drive.google.com/file/d/19CRflzQ7n5BZBNaaC7EFszgzKKlPfIVl/view
>
> I have some questions regarding the thread dump
> - How can I know the my thread name from thread dump? can I get from the
> log?
> - When do I take a thread dump? on query execution or after query
> execution?
>
> Note: I got a thread name from log and checked in thread dump on query
> execution time and after query executed. Both time thread stack trace got
> different.
>
> If any other things are required then let me know I will send.
>
> Regards,
> Vishal Patel
> 
> From: Mikhail Khludnev 
> Sent: Saturday, May 16, 2020 2:23 PM
> To: solr-user 
> Subject: Re: Performance issue in Query execution in Solr 8.3.0 and 8.5.1
>
> Can you check Thread Dump in Solr Admin while Solr 8.3 crunches query for
> 34 seconds? Please share the deepest thread stack. This might give a clue
> what's going on there.
>
> On Sat, May 16, 2020 at 11:46 AM vishal patel <
> vishalpatel200...@outlook.com>
> wrote:
>
> > Any one is looking my issue? Please help me.
> >
> > Sent from Outlook
> > 

Re: Solrcloud 6.6 becomes nuts

[Dominique Bejean]

Hi Shawn,

In fact, I was using logs from a core at WARN log level so with only slow
queries (>500ms).

I just checked in a core at INFO log level with all queries (we set the log
level top INFO for one core after the previous crash) and there is no more
queries with these two facets when the problem starts. There are nearly 150
queries per minute faceting with the 750K unique terms fields during the 3
hours before the problem occurs and no increase during the few minutes
before and when the problem starts.

I can't see anything specific in logs at the time the problem start.

Regards

Dominique




Le lun. 18 mai 2020 à 03:28, Shawn Heisey  a écrit :

> On 5/17/2020 4:18 PM, Dominique Bejean wrote:
> > I was not thinking that queries using facet with fields with high number
> > of unique value but with low hits count can be the origin of this
> problem.
>
> Performance for most things does not depend on numFound (hit count) or
> the rows parameter.  The number of terms in the field and the total
> number of documents in the index matters a lot more.
>
> If you do facets or grouping on a field with 750K unique terms, it's
> going to be very slow and require a LOT of memory.  I would not be
> surprised to see it require more than 4GB.  These features are designed
> to work best with fields that have a relatively small number of possible
> values.
>
> Thanks,
> Shawn
>