I agree 10B will not be residing on the same machine :)
About the other issue you raised, while submitting the query to Solr I was
keeping a close eye on RAM and JVM consumption on Solr Admin and for
queries at the beginning that were taking most of the time, neither RAM nor
JVM was hitting the limit so I doubt that is the problem. For reference, I
did have an issue with JVM raising an exception of "Out of Memory" when it
was around 500MB but then I raised the machine capacity to 14GB RAM and 4GB
JVM. I have read here (
https://wiki.apache.org/solr/SolrPerformanceProblems#OS_Disk_Cache) that
for best performance I should be able to put my entire collection in
memory. Does that sound reasonable?
As for the logs, I searched for "Salman" with rows=10 and start=1000 and it
took about 29 seconds to complete. However, it took less at each shard as
shown in the log file
INFO - 2015-10-09 16:43:39.170; [c:sabr102 s:shard1 r:core_node4
x:sabr102_shard1_replica2] org.apache.solr.core.SolrCore;
[sabr102_shard1_replica2] webapp=/solr path=/select
params={distrib=false&wt=javabin&version=2&rows=1010&df=text&fl=id&fl=score&shard.url=http://
[MySolrIP]:8983/solr/sabr102_shard1_replica1/|
http://100.114.184.37:7574/solr/sabr102_shard1_replica2/&NOW=1444409019061&start=0&shards.purpose=4&q=(content_text:Salman)&isShard=true&fsv=true&preferLocalShards=false}
hits=1819 status=0 QTime=91
INFO - 2015-10-09 16:44:08.116; [c:sabr102 s:shard1 r:core_node4
x:sabr102_shard1_replica2] org.apache.solr.core.SolrCore;
[sabr102_shard1_replica2] webapp=/solr path=/select
params={ids=584673511333089281,584680513887010816,584697461744111616,584668540118044672,583299685516984320&distrib=false&wt=javabin&version=2&rows=10&df=text&shard.url=
http://100.114.184.37:8983/solr/sabr102_shard1_replica1/|http://[MySolrIP]:7574/solr/sabr102_shard1_replica2/&NOW=1444409019061&start=1000&shards.purpose=64&q=(content_text:Salman)&isShard=true&preferLocalShards=false}
status=0 QTime=4
the search in the second shard started AFTER 29 seconds. Any logic behind
what I am seeing here?
Moreover, I do understand that everyone's need is different and I do need
to prototype, but there must be strategies to follow even when prototyping,
that is what I am looking forward to hear from you and the community. My
concurrent users are not that much, but I do have a good amount of data to
be stored/indexed in Solr and even if one user is not able to execute
queries quite efficiently, that will be problematic.
Regards,
Salman
On Fri, Oct 9, 2015 at 7:06 PM, Erick Erickson <erickerick...@gmail.com>
wrote:
> bq: 10GB JVM as mentioned here...and they were getting 140 ms response
> time for 10 Billion documents
>
> This simply could _not_ work in a single shard as there's a hard 2B
> doc limit per shard. On slide 14
> it states "both collections are sharded". They are not fitting 10B
> docs in 10G of JVM on a single
> machine. Trust me on this ;). The slides do not state how many shards
> they've
> split their collection into, but I suspect it's a bunch. Each
> application is different enough that the
> numbers wouldn't translate anyway...
>
> 70M docs can fit on a single shard with quite good response time, but
> YMMV. You simply
> have to experiment. Here's a long blog on the subject:
>
> https://lucidworks.com/blog/2012/07/23/sizing-hardware-in-the-abstract-why-we-dont-have-a-definitive-answer/
>
> Start with a profiler and see where you're spending your time. My
> first guess is that
> you're spending a lot of CPU cycles in garbage collection. This
> sometimes happens
> when you are running near your JVM limit, a GC kicks in and recovers a
> tiny bit of memory
> and then initiates another GC cycle immediately. Turn on GC logging
> and take a look
> at the stats provided, see:
> https://lucidworks.com/blog/2011/03/27/garbage-collection-bootcamp-1-0/
>
> Tens of seconds is entirely unexpected though. Do the Solr logs point
> to anything happening?
>
> Best,
> Erick
>
> On Fri, Oct 9, 2015 at 8:51 AM, Salman Ansari <salman.rah...@gmail.com>
> wrote:
> > Thanks Eric for your response. If you find pagination is not the main
> > culprit, what other factors do you guys suggest I need to tweak to test
> > that? As I mentioned, by navigating to 20000 results using start and row
> I
> > am getting time out from Solr.NET and I need a way to fix that.
> >
> > You suggested that 4GB JVM is not enough, I have seen MapQuest going with
> > 10GB JVM as mentioned here
> >
> http://www.slideshare.net/lucidworks/high-performance-solr-and-jvm-tuning-strategies-used-for-map-quests-search-ahead-darren-spehr
> > and they were getting 140 ms response time for 10 Billion documents. Not
> > sure how many shards they had though. With data of around 70M documents,
> > what do you guys suggest as how many shards should I use and how much
> > should I dedicate for RAM and JVM?
> >
> > Regards,
> > Salman
> >
> > On Fri, Oct 9, 2015 at 6:37 PM, Erick Erickson <erickerick...@gmail.com>
> > wrote:
> >
> >> I think paging is something of a red herring. You say:
> >>
> >> bq: but still I get delays of around 16 seconds and sometimes even more.
> >>
> >> Even for a start of 1,000, this is ridiculously long for Solr. All
> >> you're really saving
> >> here is keeping a record of the id and score for a list 1,000 cells
> >> long (or even
> >> 20,000 assuming 1,000 pages and 20 docs/page). that's somewhat wasteful,
> >> but it's still hard to believe it's responsible for what you're seeing.
> >>
> >> Having 4G of RAM for 70M docs is very little memory, assuming this is on
> >> a single shard.
> >>
> >> So my suspicion is that you have something fundamentally slow about
> >> your system, the additional overhead shouldn't be as large as you're
> >> reporting.
> >>
> >> And I'll second Toke's comment. It's very rare that users see anything
> >> _useful_ by navigating that deep. Make them hit next next next and
> they'll
> >> tire out way before that.
> >>
> >> Cursor mark's sweet spot is handling some kind of automated process that
> >> goes through the whole result set. It'll work for what you're trying
> >> to do though.
> >>
> >> Best,
> >> Erick
> >>
> >> On Fri, Oct 9, 2015 at 8:27 AM, Salman Ansari <salman.rah...@gmail.com>
> >> wrote:
> >> > Is this a real problem or a worry? Do you have users that page really
> >> deep
> >> > and if so, have you considered other mechanisms for delivering what
> they
> >> > need?
> >> >
> >> > The issue is that currently I have around 70M documents and some
> generic
> >> > queries are resulting in lots of pages. Now if I try deep navigation
> (to
> >> > page# 1000 for example), a lot of times the query takes so long that
> >> > Solr.NET throws operation time out exception. The first page is
> >> relatively
> >> > faster to load but it does take around few seconds as well. After
> reading
> >> > some documentation I realized that cursors could help and it does. I
> have
> >> > tried to following the test better performance:
> >> >
> >> > 1) Used cursors instead of start and row
> >> > 2) Increased the RAM on my Solr machine to 14GB
> >> > 3) Increase the JVM on that machine to 4GB
> >> > 4) Increased the filterChache
> >> > 5) Increased the docCache
> >> > 6) Run Optimize on the Solr Admin
> >> >
> >> > but still I get delays of around 16 seconds and sometimes even more.
> >> > What other mechanisms do you suggest I should use to handle this
> issue?
> >> >
> >> > While pagination is faster than increasing the start parameter, the
> >> > difference is small as long as you stay below a start of 1000. 10K
> might
> >> > also work for you. Do your users page beyond that?
> >> > I can limit users not to go beyond 10K but still think at that level
> >> > cursors will be much faster than increasing the start variable as
> >> explained
> >> > here (
> >> https://cwiki.apache.org/confluence/display/solr/Pagination+of+Results
> >> > ), have you tried both ways on your collection and it was giving you
> >> > similar results?
> >> >
> >> > On Fri, Oct 9, 2015 at 5:20 PM, Toke Eskildsen <
> t...@statsbiblioteket.dk>
> >> > wrote:
> >> >
> >> >> Salman Ansari <salman.rah...@gmail.com> wrote:
> >> >>
> >> >> [Pagination with cursors]
> >> >>
> >> >> > For example, what happens if the user navigates from page 1 to
> page 2,
> >> >> > does the front end need to store the next cursor at each query?
> >> >>
> >> >> Yes.
> >> >>
> >> >> > What about going to a previous page, do we need to store all
> cursors
> >> >> > that have been navigated up to now at the client side?
> >> >>
> >> >> Yes, if you want to provide that functionality.
> >> >>
> >> >> Is this a real problem or a worry? Do you have users that page really
> >> deep
> >> >> and if so, have you considered other mechanisms for delivering what
> they
> >> >> need?
> >> >>
> >> >> While pagination is faster than increasing the start parameter, the
> >> >> difference is small as long as you stay below a start of 1000. 10K
> might
> >> >> also work for you. Do your users page beyond that?
> >> >>
> >> >> - Toke Eskildsen
> >> >>
> >>
>
