Re: How to add MoreLikeThis MLT handler in Solr Cloud

[Vignan Malyala]

Any reference on this Is it possible actually?

On Tue, May 12, 2020 at 2:21 PM Vignan Malyala  wrote:

> How to add mlt handler in Solr Cloud?
>
> There is very limited documentation on this. Using search component with
> mlt=true doesn't include all configurations like boosting and mlt filters.
> Also the results with filters don't seem to work.
> Adding mlt handler seem better, but how to add in solr cloud.
> In standalone solr its easy to add mlt handler which we did, but what
> about solr cloud?
>
> Thanks in advance!
> Regards,
> Sai Vignan M
>

Re: How to add MoreLikeThis MLT handler in Solr Cloud

[Vignan Malyala]

Anyone knows how to add mlt handler in solr cloud?

On Tue, May 12, 2020 at 2:21 PM Vignan Malyala  wrote:

> How to add mlt handler in Solr Cloud?
>
> There is very limited documentation on this. Using search component with
> mlt=true doesn't include all configurations like boosting and mlt filters.
> Also the results with filters don't seem to work.
> Adding mlt handler seem better, but how to add in solr cloud.
> In standalone solr its easy to add mlt handler which we did, but what
> about solr cloud?
>
> Thanks in advance!
> Regards,
> Sai Vignan M
>

How to add MoreLikeThis MLT handler in Solr Cloud

[Vignan Malyala]

How to add mlt handler in Solr Cloud?

There is very limited documentation on this. Using search component with
mlt=true doesn't include all configurations like boosting and mlt filters.
Also the results with filters don't seem to work.
Adding mlt handler seem better, but how to add in solr cloud.
In standalone solr its easy to add mlt handler which we did, but what about
solr cloud?

Thanks in advance!
Regards,
Sai Vignan M

Re: Creating 100000 dynamic fields in solr

[Vignan Malyala]

Thanks Jan! This helps a lot!

Sai Vignan Malyala

On Mon, May 11, 2020 at 5:07 PM Jan Høydahl  wrote:

> Sounds like you are looking for parent/child docs here, see
> https://lucene.apache.org/solr/guide/8_5/indexing-nested-documents.html
>
> {
> "type": "user",
> "name": "user1",
> "products": [
> { "id": "prod_A", "cost": 50},
> { "id": "prod_B", "cost": 200},
> { "id": "prod_D", "cost": 25}
> ]
> }
>
> This will index 4 documents - one user document and three product-cost
> child documents.
>
> You can then search the child docs and return matching parents with e.g.
> q=*:*={!parent which="type:user"}((id:prod_A AND cost:[50 TO 100]) OR
> (id:prod_D AND cost:[0 TO 40]))=[child]
>
> Hope this helps.
>
> Jan
>
> > 11. mai 2020 kl. 11:35 skrev Vignan Malyala :
> >
> > I have around 1M products used by my clients.
> > Client need a filter of these 1M products by their cost filters.
> >
> > Just like:
> > User1 has 5 products (A,B,C,D,E)
> > User2 has 3 products (D,E,F)
> > User3 has 10 products (A,B,C,H,I,J,K,L,M,N,O)
> >
> > ...every customer has different sets.
> >
> > Now they want to search users by filter of product costs:
> > Product_A_cost :  50 TO 100
> > Product_D_cost :  0 TO 40
> >
> > it should return all the users who use products in this filter range.
> >
> > As I have 1M products, do I need to create dynamic fields for all users
> > with filed names as Product_A_cost and product_B_cost. etc to make a
> > search by them? If I should, then I haveto create 1M dynamic fields
> > Or is there any other way?
> >
> > Hope I'm clear here!
> >
> >
> > On Mon, May 11, 2020 at 1:47 PM Jan Høydahl 
> wrote:
> >
> >> Sounds like an anti pattern. Can you explain what search problem you are
> >> trying to solve with this many unique fields?
> >>
> >> Jan Høydahl
> >>
> >>> 11. mai 2020 kl. 07:51 skrev Vignan Malyala :
> >>>
> >>> Hi
> >>> Is it good idea to create 10 dynamic fields of time pint in solr?
> >>> I have that many fields to search on actually which come upon based on
> >>> users.
> >>>
> >>> Thanks in advance!
> >>> And I'm using Solr Cloud in real-time.
> >>>
> >>> Regards,
> >>> Sai Vignan M
> >>
>
>

Re: Creating 100000 dynamic fields in solr

[Vignan Malyala]

Thank you Jan, Vincezo and Joe.
This helps us a lot.

On Mon, May 11, 2020 at 10:03 PM Joe Obernberger <
joseph.obernber...@gmail.com> wrote:

> Could you use a multi-valued field for user in each of your products?
>
> So productA and a field User that is a list of all the users that have
> productA.  Then you could do a search like:
>
> user:User1 AND Product_A_cost:[5 TO 10]
> user:(User1 User5...) AND Product_B_cost[0 TO 40]
>
> -Joe
>
> On 5/11/2020 5:35 AM, Vignan Malyala wrote:
> > I have around 1M products used by my clients.
> > Client need a filter of these 1M products by their cost filters.
> >
> > Just like:
> > User1 has 5 products (A,B,C,D,E)
> > User2 has 3 products (D,E,F)
> > User3 has 10 products (A,B,C,H,I,J,K,L,M,N,O)
> >
> > ...every customer has different sets.
> >
> > Now they want to search users by filter of product costs:
> > Product_A_cost :  50 TO 100
> > Product_D_cost :  0 TO 40
> >
> > it should return all the users who use products in this filter range.
> >
> > As I have 1M products, do I need to create dynamic fields for all users
> > with filed names as Product_A_cost and product_B_cost. etc to make a
> > search by them? If I should, then I haveto create 1M dynamic fields
> > Or is there any other way?
> >
> > Hope I'm clear here!
> >
> >
> > On Mon, May 11, 2020 at 1:47 PM Jan Høydahl 
> wrote:
> >
> >> Sounds like an anti pattern. Can you explain what search problem you are
> >> trying to solve with this many unique fields?
> >>
> >> Jan Høydahl
> >>
> >>> 11. mai 2020 kl. 07:51 skrev Vignan Malyala :
> >>>
> >>> Hi
> >>> Is it good idea to create 10 dynamic fields of time pint in solr?
> >>> I have that many fields to search on actually which come upon based on
> >>> users.
> >>>
> >>> Thanks in advance!
> >>> And I'm using Solr Cloud in real-time.
> >>>
> >>> Regards,
> >>> Sai Vignan M
> >
>

Re: Creating 100000 dynamic fields in solr

[Vignan Malyala]

I have around 1M products used by my clients.
Client need a filter of these 1M products by their cost filters.

Just like:
User1 has 5 products (A,B,C,D,E)
User2 has 3 products (D,E,F)
User3 has 10 products (A,B,C,H,I,J,K,L,M,N,O)

...every customer has different sets.

Now they want to search users by filter of product costs:
Product_A_cost :  50 TO 100
Product_D_cost :  0 TO 40

it should return all the users who use products in this filter range.

As I have 1M products, do I need to create dynamic fields for all users
with filed names as Product_A_cost and product_B_cost. etc to make a
search by them? If I should, then I haveto create 1M dynamic fields
Or is there any other way?

Hope I'm clear here!


On Mon, May 11, 2020 at 1:47 PM Jan Høydahl  wrote:

> Sounds like an anti pattern. Can you explain what search problem you are
> trying to solve with this many unique fields?
>
> Jan Høydahl
>
> > 11. mai 2020 kl. 07:51 skrev Vignan Malyala :
> >
> > Hi
> > Is it good idea to create 10 dynamic fields of time pint in solr?
> > I have that many fields to search on actually which come upon based on
> > users.
> >
> > Thanks in advance!
> > And I'm using Solr Cloud in real-time.
> >
> > Regards,
> > Sai Vignan M
>

Creating 100000 dynamic fields in solr

[Vignan Malyala]

Hi
Is it good idea to create 10 dynamic fields of time pint in solr?
I have that many fields to search on actually which come upon based on
users.

Thanks in advance!
And I'm using Solr Cloud in real-time.

Regards,
Sai Vignan M

Re: SolrCloud - Underlying core creation failed while creating collection with new configset

[Vignan Malyala]

Hi Erick,
Did see any extra error in solr logs. Its the same error I mentioned
earlier.
I'm using SolrCloud by the way.

On Wed, Mar 4, 2020 at 8:06 PM Erick Erickson 
wrote:

> You need to look at the solr logs on the machine where the attempt was
> made to create the replica...
>
> Best,
> Erick
>
> > On Mar 4, 2020, at 03:24, Vignan Malyala  wrote:
> >
> > Hi
> > I created a new config set as mentioned in Solr Cloud documentation using
> > upload zip.
> > I get this error when I try to create a collection using my new
> configset.
> >
> > Error from shard: http://X.X.X.X:8983/solr
> >
> > OverseerCollectionMessageHandler Cleaning up collection [test5].
> >
> > Collection: test5 operation: create
> > failed:org.apache.solr.common.SolrException: Underlying core creation
> > failed while creating collection: test5
> >   at
> org.apache.solr.cloud.api.collections.CreateCollectionCmd.call(CreateCollectionCmd.java:303)
> >   at
> org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler.processMessage(OverseerCollectionMessageHandler.java:263)
> >   at
> org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:505)
> >   at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)
> >   at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> >   at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> >   at java.lang.Thread.run(Thread.java:748)
> >
> >
> >
> > Please help me out with this.
> >
> > Regards,
> >
> > Sai Vignan
>

SolrCloud - Underlying core creation failed while creating collection with new configset

[Vignan Malyala]

Hi
I created a new config set as mentioned in Solr Cloud documentation using
upload zip.
I get this error when I try to create a collection using my new configset.

Error from shard: http://X.X.X.X:8983/solr

OverseerCollectionMessageHandler Cleaning up collection [test5].

Collection: test5 operation: create
failed:org.apache.solr.common.SolrException: Underlying core creation
failed while creating collection: test5
at 
org.apache.solr.cloud.api.collections.CreateCollectionCmd.call(CreateCollectionCmd.java:303)
at 
org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler.processMessage(OverseerCollectionMessageHandler.java:263)
at 
org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:505)
at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)



Please help me out with this.

Regards,

Sai Vignan

Re: Can I create 1000 cores in SOLR CLOUD

[Vignan Malyala]

Guys,
Did anyone work on this type of thing?
Can you please help with this? For real time deployment and issues?

On Mon, Jan 27, 2020 at 5:29 PM Vignan Malyala  wrote:

> Hi all,
>
> We are currently using solr without cloud with 500 cores. It works good.
>
> Now we are planning to expand it using solr cloud with 1000 cores, (2
> cores for each of my client with different domain data).
>
> I'm planning to put all fields as "stored".
>
> Is it the right thought? Will it have any issues?  Will it become slow??
> How should I take care in production?
> Please help!
>
> Thanks in advance!
>
> Regards,
> VIgnan
>

Can I create 1000 cores in SOLR CLOUD

[Vignan Malyala]

Hi all,

We are currently using solr without cloud with 500 cores. It works good.

Now we are planning to expand it using solr cloud with 1000 cores, (2 cores
for each of my client with different domain data).

I'm planning to put all fields as "stored".

Is it the right thought? Will it have any issues?  Will it become slow??
How should I take care in production?
Please help!

Thanks in advance!

Regards,
VIgnan

Re: How to add a new field to already an existing index in Solr 6.6 ?

[Vignan Malyala]

How do I update my schema with new field?


On Tue, Dec 3, 2019 at 6:51 PM Erick Erickson 
wrote:

> Update your schema to include the new field and reload your collection.
>
> Then updating your field should work.
>
> Best,
> Erick
>
> > On Dec 3, 2019, at 4:40 AM, Vignan Malyala  wrote:
> >
> > How to add a new field to already an existing index in Solr 6.6 ?
> >
> > I tried to use set for this, but it shows error as undefined field. But
> > however I could create a new index with set.
> > But, how to add new filed to already indexed data?
> > Is it possible?
> >
> > Thank you!
> >
> > Regards,
> > Sai
>
>

Re: From solr to solr cloud

[Vignan Malyala]

Hi Shawn,

Thanks for your response!

Yes! 500 collections.
Each collection/core has around 50k to 50L documents/jsons (depending upon
the client). We made one core for each client. Each json has 15 fields.
It already in production as as Solr stand alone server.
We want to use SolrCloud for it now, so as to make it scalable for future
safety. How do I make it possible?

As per your response, I understood that, I have to create 3 zookeeper
instances and some machines that house 1 solr node each.
Is that the optimized solution? *And how many machines do I need to build
to house solr nodes keeping in mind 500 collections?*

Thanks in advance!

On Fri, Dec 6, 2019 at 11:44 AM Shawn Heisey  wrote:

> On 12/5/2019 12:28 PM, Vignan Malyala wrote:
> > I currently have 500 collections in my stand alone solr. Bcoz of day by
> day
> > increase in Data, I want to convert it into solr cloud.
> > Can you suggest me how to do it successfully.
> > How many shards should be there?
> > How many nodes should be there?
> > Are so called nodes different machines i should take?
> > How many zoo keeper nodes should be there?
> > Are so called zoo keeper nodes different machines i should take?
> > Total how many machines i have to take to implement scalable solr cloud?
>
> 500 collections is large enough that running it in SolrCloud is likely
> to encounter scalability issues.  SolrCloud's design does not do well
> with that many collections in the cluster, even if there are a lot of
> machines.
>
> There's a lot of comment history on this issue:
>
> https://issues.apache.org/jira/browse/SOLR-7191
>
> Generally speaking, each machine should only house one Solr node,
> whether you're running cloud or not.  If each one requires a really huge
> heap, it might be worthwhile to split it, but that's the only time I
> would do so.  And I would generally prefer to add more machines than to
> run multiple Solr nodes on one machine.
>
> One thing you might do, if the way your data is divided will permit it,
> is to run multiple SolrCloud clusters.  Multiple clusters can all use
> one ZooKeeper ensemble.
>
> ZooKeeper requires a minimum of three machines for fault tolerance.
> With 3 or 4 machines in the ensemble, you can survive one machine
> failure.  To survive two failures requires at least 5 machines.
>
> Thanks,
> Shawn
>

Re: From solr to solr cloud

[Vignan Malyala]

Yes! 500 collections.
Each collection/core has around 50k to 50L documents/jsons (depending upon
the client). We made one core for each client. Each json has 15 fields.
It already in production as as Solr stand alone server.

We want to use SolrCloud for it now, so as to make it scalable for future
safety. How do I make it possible (obviously with minimum cost)?

On Fri, Dec 6, 2019 at 11:14 AM Paras Lehana 
wrote:

> Do you mean 500 cores? Tell us about the data more. How many documents per
> core do you have or what performance issues are you facing?
>
> On Fri, 6 Dec 2019 at 01:01, David Hastings 
> wrote:
>
> > are you noticing performance decreases in stand alone solr as of now?
> >
> > On Thu, Dec 5, 2019 at 2:29 PM Vignan Malyala 
> > wrote:
> >
> > > Hi
> > > I currently have 500 collections in my stand alone solr. Bcoz of day by
> > day
> > > increase in Data, I want to convert it into solr cloud.
> > > Can you suggest me how to do it successfully.
> > > How many shards should be there?
> > > How many nodes should be there?
> > > Are so called nodes different machines i should take?
> > > How many zoo keeper nodes should be there?
> > > Are so called zoo keeper nodes different machines i should take?
> > > Total how many machines i have to take to implement scalable solr
> cloud?
> > >
> > > Plz detail these questions. Any of documents on web aren't clear for
> > > production environments.
> > > Thanks in advance.
> > >
> >
>
>
> --
> --
> Regards,
>
> *Paras Lehana* [65871]
> Development Engineer, Auto-Suggest,
> IndiaMART Intermesh Ltd.
>
> 8th Floor, Tower A, Advant-Navis Business Park, Sector 142,
> Noida, UP, IN - 201303
>
> Mob.: +91-9560911996
> Work: 01203916600 | Extn:  *8173*
>
> --
> *
> *
>
>  <https://www.facebook.com/IndiaMART/videos/578196442936091/>
>

From solr to solr cloud

[Vignan Malyala]

Hi
I currently have 500 collections in my stand alone solr. Bcoz of day by day
increase in Data, I want to convert it into solr cloud.
Can you suggest me how to do it successfully.
How many shards should be there?
How many nodes should be there?
Are so called nodes different machines i should take?
How many zoo keeper nodes should be there?
Are so called zoo keeper nodes different machines i should take?
Total how many machines i have to take to implement scalable solr cloud?

Plz detail these questions. Any of documents on web aren't clear for
production environments.
Thanks in advance.

How to add a new field to already an existing index in Solr 6.6 ?

[Vignan Malyala]

How to add a new field to already an existing index in Solr 6.6 ?

I tried to use set for this, but it shows error as undefined field. But
however I could create a new index with set.
But, how to add new filed to already indexed data?
Is it possible?

Thank you!

Regards,
Sai

Re: Production Issue: cannot connect to solr server suddenly

[Vignan Malyala]

Sure!

Error:
Failed to connect to server at '
http://127.0.0.1:8983/solr/my_core/update/?commit=true
<https://www.google.com/url?q=http://solradmin:Red8891@127.0.0.1:8983/solr/tenant_311/update/?commit%3Dtrue=D=hangouts=157492189995=AFQjCNFYp6NcfnmMNwEZxoY3WjXJXH_Q6g>',
are you sure that URL is correct? Checking it in a browser might help:
HTTPConnectionPool(host='127.0.0.1', port=8983): Max retries exceeded with
url: /solr/my_core/update/?commit=true (Caused by
NewConnectionError(': Failed to establish a new connection: [Errno 111]
Connection refused',))

I'm using python with pysolr to connect to Solr Instance.
In my production server, I get this issue at certain times in my python
logs while it tries to connect to solr to index data or search results.
When I check the solr UI, it will be working fine, but this issue of solr
refusing to connect at certain times occurs and so my users are not able to
index documents or search data.
Please help with this issue.

Should I increase threads and how to increase threads ? Should I increase
physical memory ?
whats the solution?

On Tue, Nov 26, 2019 at 3:51 PM Paras Lehana 
wrote:

> Hi Sai,
>
> Please elaborate. What language is the code written in? Why is there
> google.com in the query?
>
> Max retries exceeded with url
>
>
> This happens when you make too many requests on a server than allowed.
> Check with server at solradmin in case you have DoS or related policy
> preventing this.
>
> On Mon, 25 Nov 2019 at 16:39, Vignan Malyala  wrote:
>
> > I don't get this error always. At certain times, I get this error with my
> > Solr suddenly.
> > However, If I check my Solr url, it will be working but. When I want to
> > update via code, it will not work.
> > Please help me out with this.
> >
> > ERROR:
> > *Failed to connect to server at
> > 'http://127.0.0.1:8983/solr/my_core/update/?commit=true
> > <
> >
> https://www.google.com/url?q=http://solradmin:Red8891@127.0.0.1:8983/solr/tenant_311/update/?commit%3Dtrue=D=hangouts=1574765671451000=AFQjCNGE326wW7hZNwLUH2dEw8scCTyEXw
> > >',
> > are you sure that URL is correct? Checking it in a browser might help:
> > HTTPConnectionPool(host='127.0.0.1', port=8983): Max retries exceeded
> with
> > url: /solr/my_core/update/?commit=true (Caused by
> > NewConnectionError(' > 0x7efd7be78a98>: Failed to establish a new connection: [Errno 111]
> > Connection refused',))*
> >
> >
> >
> >
> > Regards,
> > Sai Vignan
> >
>
>
> --
> --
> Regards,
>
> *Paras Lehana* [65871]
> Development Engineer, Auto-Suggest,
> IndiaMART Intermesh Ltd.
>
> 8th Floor, Tower A, Advant-Navis Business Park, Sector 142,
> Noida, UP, IN - 201303
>
> Mob.: +91-9560911996
> Work: 01203916600 | Extn:  *8173*
>
> --
> IMPORTANT:
> NEVER share your IndiaMART OTP/ Password with anyone.
>

Production Issue: cannot connect to solr server suddenly

[Vignan Malyala]

I don't get this error always. At certain times, I get this error with my
Solr suddenly.
However, If I check my Solr url, it will be working but. When I want to
update via code, it will not work.
Please help me out with this.

ERROR:
*Failed to connect to server at
'http://127.0.0.1:8983/solr/my_core/update/?commit=true
',
are you sure that URL is correct? Checking it in a browser might help:
HTTPConnectionPool(host='127.0.0.1', port=8983): Max retries exceeded with
url: /solr/my_core/update/?commit=true (Caused by
NewConnectionError(': Failed to establish a new connection: [Errno 111]
Connection refused',))*




Regards,
Sai Vignan

Re: Solr Prod stopped yesterday - saya "insufficient memory for the Java Runtime Environment"

[Vignan Malyala]

Ok. I have around 500 cores in my solr. So, how much heap I should allocate
in solr and jvm?
(Currently as I see, in solr.in.sh shows heap as  - Xms 20g -Xmx 20g.
And my system jvm heap shows -Xms 528m -Xmx 8g. I've re-checked it.)


On Wed 23 Oct, 2019, 7:52 PM Shawn Heisey,  wrote:

> On 10/23/2019 4:09 AM, Vignan Malyala wrote:
> > *Solr prod stopped yesterday. How to prevent this.*
> >
> > Solr heap info is :  -Xms20g -Xmx20g
> > JVM Heap info. : -Xms528m -Xmx8g
>
> There is no such thing as a Solr heap separate from the JVM heap.  There
> are multiple environment variables that can specify the heap size ...
> only one of those settings is actually going to take effect.  I have not
> done any investigation to determine which one it will be.
>
> > Physical Ram - 32GB
> > Solr version - 6.6.1
> > Swap memory - 8g
> >
> > *hc_err_pid.log got created with following info in it:*
> > #
> > # There is insufficient memory for the Java Runtime Environment to
> continue.
> > # Native memory allocation (mmap) failed to map 16106127360 bytes for
> > committing reserved memory.
>
> This sounds like there is insufficient memory available when running
> Solr for the system to start Java with the configured settings.  Based
> on this number, which is about 16GB, I'm betting that the heap size
> which took effect is the 20GB one, or maybe it got set to 16GB by
> another setting that you did not mention above.
>
> Your information says that there is 32GB total memory ... maybe there
> are other programs that are using up some of that memory before Solr
> attempts to start, and there is not enough memory left for Solr.
>
> Thanks,
> Shawn
>

Solr Prod stopped yesterday - saya "insufficient memory for the Java Runtime Environment"

[Vignan Malyala]

*Solr prod stopped yesterday. How to prevent this.*

Solr heap info is :  -Xms20g -Xmx20g
JVM Heap info. : -Xms528m -Xmx8g
Physical Ram - 32GB
Solr version - 6.6.1
Swap memory - 8g

*hc_err_pid.log got created with following info in it:*
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 16106127360 bytes for
committing reserved memory.
# Possible reasons:
#   The system is out of physical RAM or swap space
#   The process is running with CompressedOops enabled, and the Java Heap
may be blocking the growth of the native heap
# Possible solutions:
#   Reduce memory load on the system
#   Increase physical memory or swap space
#   Check if swap backing store is full
#   Decrease Java heap size (-Xmx/-Xms)
#   Decrease number of Java threads
#   Decrease Java thread stack sizes (-Xss)
#   Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
#  Out of Memory Error (os_linux.cpp:2749), pid=86291,
tid=0x7f8822e47700
#
# JRE version:  (8.0_211-b12) (build )
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.211-b12 mixed mode
linux-amd64 compressed oops)
# Failed to write core dump. Core dumps have been disabled. To enable core
dumping, try "ulimit -c unlimited" before starting Java again
#


*solr_gc.log shows following:*
2019-10-23T01:24:23.688+0530: 644433.457: Total time for which application
threads were stopped: 0.0028257 seconds, Stopping threads took: 0.0016863
seconds
Heap
 par new generation   total 4369088K, used 736363K [0x0002c000,
0x0004, 0x0004)
  eden space 3495296K,  13% used [0x0002c000, 0x0002dc9d5ed8,
0x00039556)
  from space 873792K,  30% used [0x0003caab, 0x0003daff4d80,
0x0004)
  to   space 873792K,   0% used [0x00039556, 0x00039556,
0x0003caab)
 concurrent mark-sweep generation total 15728640K, used 3807001K
[0x0004, 0x0007c000, 0x0007c000)
 Metaspace   used 45325K, capacity 47047K, committed 47324K, reserved
1091584K
  class spaceused 4821K, capacity 5230K, committed 5340K, reserved
1048576K

Max number of cores in solr

[Vignan Malyala]

Hi
Im planning to create separate core for each of my client in solr.
Can I create around 500 cores in solr. Is it a good idea?
For each client i have around 10 records on average currently.

How much physical memory it might consume. Plz help with this.
Thank you

Re: Solr is very slow with term vectors

[Vignan Malyala]

How do I check that in solr? Can anyone share link on implementation of
threads in solr?

On Fri 16 Aug, 2019, 4:52 PM Jörn Franke,  wrote:

> Is your custom query parser multithreaded and leverages all cores?
>
> > Am 16.08.2019 um 13:12 schrieb Vignan Malyala :
> >
> > I want response time below 3 seconds.
> > And fyi I'm already using 32 cores.
> > My cache is already full too and obviously same requests don't occur in
> my
> > case.
> >
> >
> >> On Fri 16 Aug, 2019, 11:47 AM Jörn Franke, 
> wrote:
> >>
> >> How much response time do you require?
> >> I think you have to solve the issue in your code by introducing higher
> >> parallelism during calculation and potentially more cores.
> >>
> >> Maybe you can also precalculate what you do, cache it and use during
> >> request the precalculated values.
> >>
> >>> Am 16.08.2019 um 05:08 schrieb Vignan Malyala :
> >>>
> >>> Hi
> >>> Any solution for this? Taking around 50 seconds to get response.
> >>>
> >>>> On Mon 12 Aug, 2019, 3:28 PM Vignan Malyala, 
> >> wrote:
> >>>>
> >>>> Hi Doug / Walter,
> >>>>
> >>>> I'm just using this methodology.
> >>>> PFB link of my sample code.
> >>>> https://github.com/saaay71/solr-vector-scoring
> >>>>
> >>>> The only issue is speed of response for 1M records.
> >>>>
> >>>> On Mon, Aug 12, 2019 at 12:24 AM Walter Underwood <
> >> wun...@wunderwood.org>
> >>>> wrote:
> >>>>
> >>>>> tf.idf was invented because cosine similarity is too much
> computation.
> >>>>> tf.idf gives similar results much, much faster than cosine distance.
> >>>>>
> >>>>> I would expect cosine similarity to be slow. I would also expect
> >>>>> retrieving 1 million records to be slow. Doing both of those in one
> >> minute
> >>>>> is pretty good.
> >>>>>
> >>>>> As Kernighan and Paugher said in 1978, "Don’t diddle code to make it
> >>>>> faster—find a better algorithm.”
> >>>>>
> >>>>> https://en.wikipedia.org/wiki/The_Elements_of_Programming_Style
> >>>>>
> >>>>> wunder
> >>>>> Walter Underwood
> >>>>> wun...@wunderwood.org
> >>>>> http://observer.wunderwood.org/  (my blog)
> >>>>>
> >>>>>> On Aug 11, 2019, at 10:40 AM, Doug Turnbull <
> >>>>> dturnb...@opensourceconnections.com> wrote:
> >>>>>>
> >>>>>> Hi Vignan,
> >>>>>>
> >>>>>> We need to see more details / code of what your query parser plugin
> >> does
> >>>>>> exactly with term vectors, we can't really help you without more
> >>>>> details.
> >>>>>> Is it open source? Can you share a minimal example that recreates
> the
> >>>>>> problem?
> >>>>>>
> >>>>>> On Sun, Aug 11, 2019 at 1:19 PM Vignan Malyala <
> dsmsvig...@gmail.com>
> >>>>> wrote:
> >>>>>>
> >>>>>>> Hi guys,
> >>>>>>>
> >>>>>>> I made my custom qparser plugin in Solr for scoring. The plugin
> only
> >>>>> does
> >>>>>>> cosine similarity of vectors for each record. I use term vectors
> >> here.
> >>>>>>> Results are fine!
> >>>>>>>
> >>>>>>> BUT, Solr response is very slow with term vectors. It takes around
> 55
> >>>>>>> seconds for each request for 100 records.
> >>>>>>> How do I make it faster to get my results in ms ?
> >>>>>>> Please respond soon as its lil urgent.
> >>>>>>>
> >>>>>>> Note: All my values are stored and indexed. I am not using Solr
> >> Cloud.
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> *Doug Turnbull **| CTO* | OpenSource Connections
> >>>>>> <http://opensourceconnections.com>, LLC | 240.476.9983
> >>>>>> Author: Relevant Search <http://manning.com/turnbull>
> >>>>>> This e-mail and all contents, including attachments, is considered
> to
> >> be
> >>>>>> Company Confidential unless explicitly stated otherwise, regardless
> >>>>>> of whether attachments are marked as such.
> >>>>>
> >>>>>
> >>
>

Re: Solr is very slow with term vectors

[Vignan Malyala]

I want response time below 3 seconds.
And fyi I'm already using 32 cores.
My cache is already full too and obviously same requests don't occur in my
case.


On Fri 16 Aug, 2019, 11:47 AM Jörn Franke,  wrote:

> How much response time do you require?
> I think you have to solve the issue in your code by introducing higher
> parallelism during calculation and potentially more cores.
>
> Maybe you can also precalculate what you do, cache it and use during
> request the precalculated values.
>
> > Am 16.08.2019 um 05:08 schrieb Vignan Malyala :
> >
> > Hi
> > Any solution for this? Taking around 50 seconds to get response.
> >
> >> On Mon 12 Aug, 2019, 3:28 PM Vignan Malyala, 
> wrote:
> >>
> >> Hi Doug / Walter,
> >>
> >> I'm just using this methodology.
> >> PFB link of my sample code.
> >> https://github.com/saaay71/solr-vector-scoring
> >>
> >> The only issue is speed of response for 1M records.
> >>
> >> On Mon, Aug 12, 2019 at 12:24 AM Walter Underwood <
> wun...@wunderwood.org>
> >> wrote:
> >>
> >>> tf.idf was invented because cosine similarity is too much computation.
> >>> tf.idf gives similar results much, much faster than cosine distance.
> >>>
> >>> I would expect cosine similarity to be slow. I would also expect
> >>> retrieving 1 million records to be slow. Doing both of those in one
> minute
> >>> is pretty good.
> >>>
> >>> As Kernighan and Paugher said in 1978, "Don’t diddle code to make it
> >>> faster—find a better algorithm.”
> >>>
> >>> https://en.wikipedia.org/wiki/The_Elements_of_Programming_Style
> >>>
> >>> wunder
> >>> Walter Underwood
> >>> wun...@wunderwood.org
> >>> http://observer.wunderwood.org/  (my blog)
> >>>
> >>>> On Aug 11, 2019, at 10:40 AM, Doug Turnbull <
> >>> dturnb...@opensourceconnections.com> wrote:
> >>>>
> >>>> Hi Vignan,
> >>>>
> >>>> We need to see more details / code of what your query parser plugin
> does
> >>>> exactly with term vectors, we can't really help you without more
> >>> details.
> >>>> Is it open source? Can you share a minimal example that recreates the
> >>>> problem?
> >>>>
> >>>> On Sun, Aug 11, 2019 at 1:19 PM Vignan Malyala 
> >>> wrote:
> >>>>
> >>>>> Hi guys,
> >>>>>
> >>>>> I made my custom qparser plugin in Solr for scoring. The plugin only
> >>> does
> >>>>> cosine similarity of vectors for each record. I use term vectors
> here.
> >>>>> Results are fine!
> >>>>>
> >>>>> BUT, Solr response is very slow with term vectors. It takes around 55
> >>>>> seconds for each request for 100 records.
> >>>>> How do I make it faster to get my results in ms ?
> >>>>> Please respond soon as its lil urgent.
> >>>>>
> >>>>> Note: All my values are stored and indexed. I am not using Solr
> Cloud.
> >>>>>
> >>>>
> >>>>
> >>>> --
> >>>> *Doug Turnbull **| CTO* | OpenSource Connections
> >>>> <http://opensourceconnections.com>, LLC | 240.476.9983
> >>>> Author: Relevant Search <http://manning.com/turnbull>
> >>>> This e-mail and all contents, including attachments, is considered to
> be
> >>>> Company Confidential unless explicitly stated otherwise, regardless
> >>>> of whether attachments are marked as such.
> >>>
> >>>
>

Re: Solr is very slow with term vectors

[Vignan Malyala]

Hi
Any solution for this? Taking around 50 seconds to get response.

On Mon 12 Aug, 2019, 3:28 PM Vignan Malyala,  wrote:

> Hi Doug / Walter,
>
> I'm just using this methodology.
> PFB link of my sample code.
> https://github.com/saaay71/solr-vector-scoring
>
> The only issue is speed of response for 1M records.
>
> On Mon, Aug 12, 2019 at 12:24 AM Walter Underwood 
> wrote:
>
>> tf.idf was invented because cosine similarity is too much computation.
>> tf.idf gives similar results much, much faster than cosine distance.
>>
>> I would expect cosine similarity to be slow. I would also expect
>> retrieving 1 million records to be slow. Doing both of those in one minute
>> is pretty good.
>>
>> As Kernighan and Paugher said in 1978, "Don’t diddle code to make it
>> faster—find a better algorithm.”
>>
>> https://en.wikipedia.org/wiki/The_Elements_of_Programming_Style
>>
>> wunder
>> Walter Underwood
>> wun...@wunderwood.org
>> http://observer.wunderwood.org/  (my blog)
>>
>> > On Aug 11, 2019, at 10:40 AM, Doug Turnbull <
>> dturnb...@opensourceconnections.com> wrote:
>> >
>> > Hi Vignan,
>> >
>> > We need to see more details / code of what your query parser plugin does
>> > exactly with term vectors, we can't really help you without more
>> details.
>> > Is it open source? Can you share a minimal example that recreates the
>> > problem?
>> >
>> > On Sun, Aug 11, 2019 at 1:19 PM Vignan Malyala 
>> wrote:
>> >
>> >> Hi guys,
>> >>
>> >> I made my custom qparser plugin in Solr for scoring. The plugin only
>> does
>> >> cosine similarity of vectors for each record. I use term vectors here.
>> >> Results are fine!
>> >>
>> >> BUT, Solr response is very slow with term vectors. It takes around 55
>> >> seconds for each request for 100 records.
>> >> How do I make it faster to get my results in ms ?
>> >> Please respond soon as its lil urgent.
>> >>
>> >> Note: All my values are stored and indexed. I am not using Solr Cloud.
>> >>
>> >
>> >
>> > --
>> > *Doug Turnbull **| CTO* | OpenSource Connections
>> > <http://opensourceconnections.com>, LLC | 240.476.9983
>> > Author: Relevant Search <http://manning.com/turnbull>
>> > This e-mail and all contents, including attachments, is considered to be
>> > Company Confidential unless explicitly stated otherwise, regardless
>> > of whether attachments are marked as such.
>>
>>

Re: Solr is very slow with term vectors

[Vignan Malyala]

Hi Doug / Walter,

I'm just using this methodology.
PFB link of my sample code.
https://github.com/saaay71/solr-vector-scoring

The only issue is speed of response for 1M records.

On Mon, Aug 12, 2019 at 12:24 AM Walter Underwood 
wrote:

> tf.idf was invented because cosine similarity is too much computation.
> tf.idf gives similar results much, much faster than cosine distance.
>
> I would expect cosine similarity to be slow. I would also expect
> retrieving 1 million records to be slow. Doing both of those in one minute
> is pretty good.
>
> As Kernighan and Paugher said in 1978, "Don’t diddle code to make it
> faster—find a better algorithm.”
>
> https://en.wikipedia.org/wiki/The_Elements_of_Programming_Style
>
> wunder
> Walter Underwood
> wun...@wunderwood.org
> http://observer.wunderwood.org/  (my blog)
>
> > On Aug 11, 2019, at 10:40 AM, Doug Turnbull <
> dturnb...@opensourceconnections.com> wrote:
> >
> > Hi Vignan,
> >
> > We need to see more details / code of what your query parser plugin does
> > exactly with term vectors, we can't really help you without more details.
> > Is it open source? Can you share a minimal example that recreates the
> > problem?
> >
> > On Sun, Aug 11, 2019 at 1:19 PM Vignan Malyala 
> wrote:
> >
> >> Hi guys,
> >>
> >> I made my custom qparser plugin in Solr for scoring. The plugin only
> does
> >> cosine similarity of vectors for each record. I use term vectors here.
> >> Results are fine!
> >>
> >> BUT, Solr response is very slow with term vectors. It takes around 55
> >> seconds for each request for 100 records.
> >> How do I make it faster to get my results in ms ?
> >> Please respond soon as its lil urgent.
> >>
> >> Note: All my values are stored and indexed. I am not using Solr Cloud.
> >>
> >
> >
> > --
> > *Doug Turnbull **| CTO* | OpenSource Connections
> > <http://opensourceconnections.com>, LLC | 240.476.9983
> > Author: Relevant Search <http://manning.com/turnbull>
> > This e-mail and all contents, including attachments, is considered to be
> > Company Confidential unless explicitly stated otherwise, regardless
> > of whether attachments are marked as such.
>
>

Solr is very slow with term vectors

[Vignan Malyala]

Hi guys,

I made my custom qparser plugin in Solr for scoring. The plugin only does
cosine similarity of vectors for each record. I use term vectors here.
Results are fine!

BUT, Solr response is very slow with term vectors. It takes around 55
seconds for each request for 100 records.
How do I make it faster to get my results in ms ?
Please respond soon as its lil urgent.

Note: All my values are stored and indexed. I am not using Solr Cloud.

Solr is very slow with term vectors

[Vignan Malyala]

Hi

I made by custom qparser plugin in Solr for scoring. The plugin only does
cosine similarity of vectors. for each record. Results are fine!

*BUT, Solr response is very slow. It takes around 55 seconds for each
request.*
*How do I make it faster to get my results in ms ?*
*Please respond soon as its lil urgent.*

Note: All my values are stored and indexed. I am not using Solr Cloud.

Regards,
Sai Vignan Malyala