Re: Analyzing .pos files using luke

[KNitin]

Erick,

1. Yes both indices are optimized. They use lucene_40 version for their
solr indices.
2. I discovered a bloat on one of the index. I am trying to figure out what
might have caused the bloat. There are some schema changes on the bloated
one but I was wondering if there is way to identify some smoking guns
purely by using luke/analyzing .pos files.

Thanks,
Nitin

On Mon, Jul 11, 2016 at 9:52 AM Erick Erickson <erickerick...@gmail.com>
wrote:

> Why do you care? Is there a specific problem you're trying
> to diagnose? Since the merge logic wasn't identical
> (almost guaranteed), the sizes of the files are unreliable
> since they include deleted documents which are compacted
> on merging.
>
> So, you can try an optimize and compare immediately
> afterwards. That should give you a general sense.
> Otherwise, be absolutely sure that the schema definitions
> haven't changed. What versions of Solr? Be sure
> DocValues isn't different (that's recently become a
> default, and you haven't told us _which_ versions of
> Solr you're comparing).
>
> Best,
> Erick
>
> On Sun, Jul 10, 2016 at 9:27 PM, KNitin <nitin.t...@gmail.com> wrote:
> > Hi,
> >
> >  I am trying to diff between 2 versions of solr index. Both the indices
> > have similar .doc, .pay file sizes but their .pos files are extremely
> > different. How do i dig deeper to understand what could be causing this
> > difference?  Is there  a way to just open/analyze .pos file/compare 2
> .pos
> > files?
> >
> > Thanks in advance,
> > Nitin
>

Analyzing .pos files using luke

[KNitin]

Hi,

 I am trying to diff between 2 versions of solr index. Both the indices
have similar .doc, .pay file sizes but their .pos files are extremely
different. How do i dig deeper to understand what could be causing this
difference?  Is there  a way to just open/analyze .pos file/compare 2 .pos
files?

Thanks in advance,
Nitin

Solr /Lucene Payload loading

[KNitin]

Hi,

  I am indexing a bunch of payloads with terms in solr. I notice during
query time that the IO reads increase a lot everytime i require the payload
to be fetched. Does solr load payload from the disk all the time? Is there
anyway to force it to be loaded into mem?

Thanks,
Nitin

SolrCloud shards marked as down and Does not recovery connection to zk

[KNitin]

Hi,

I am  indexing about 5M docs in a 4 shard and 1 replica setup. During
indexing one of the shards is marked as down in zookeeper but when i tail
the logs all the updates are received in the shard and a hard commit at the
end of the job also succeeds.  (The auto commit is set to trigger every 10
mins or 150K documents).  The shard does not recover until i force restart
solr on that node.  The mem/cpu/load on solr is very less during this time.

How and when does solr try to reconnect to zk?

Thanks,
Nitin

Re: SolrCloud shard marked as down and "reloading" collection doesnt restore it

[KNitin]

After  more debugging, I figured out that it is related to this:
https://issues.apache.org/jira/browse/SOLR-3274

Is there a recommended fix (apart from running a zk ensemble?)

On Thu, Feb 11, 2016 at 10:29 AM, KNitin <nitin.t...@gmail.com> wrote:

> Hi,
>
>  I noticed while running an indexing job (2M docs but per doc size could
> be 2-3 MB) that one of the shards goes down just after the commit.  (Not
> related to OOM or high cpu/load).  This marks the shard as "down" in zk and
> even a reload of the collection does not recover the state.
>
> There are no exceptions in the logs and the stack trace indicates jetty
> threads in blocked state.
>
> The last few lines in the logs are as follows:
>
> trib=TOLEADER=javabin=2} {add=[1552605 (1525453861590925312)]}
> 0 5
> INFO  - 2016-02-06 19:17:47.658;
> org.apache.solr.update.DirectUpdateHandler2; start
> commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
> INFO  - 2016-02-06 19:18:02.209; org.apache.solr.core.SolrDeletionPolicy;
> SolrDeletionPolicy.onCommit: commits: num=2
> INFO  - 2016-02-06 19:18:02.209; org.apache.solr.core.SolrDeletionPolicy;
> newest commit generation = 6
> INFO  - 2016-02-06 19:18:02.233; org.apache.solr.search.SolrIndexSearcher;
> Opening Searcher@321a0cc9 main
> INFO  - 2016-02-06 19:18:02.296; org.apache.solr.core.QuerySenderListener;
> QuerySenderListener sending requests to Searcher@321a0cc9
> main{StandardDirectoryReader(segments_6:180:nrt
> _20(4.6):C15155/216:delGen=1 _w(4.6):C1538/63:delGen=2
> _16(4.6):C279/20:delGen=2 _e(4.6):C11386/514:delGen=3
> _g(4.6):C4434/204:delGen=3 _p(4.6):C418/5:delGen=1 _v(4.6):C1
> _x(4.6):C17583/316:delGen=2 _y(4.6):C9783/112:delGen=2
> _z(4.6):C4736/47:delGen=2 _12(4.6):C705/2:delGen=1 _13(4.6):C275/4:delGen=1
> _1b(4.6):C619 _26(4.6):C318/13:delGen=1 _1e(4.6):C25356/763:delGen=3
> _1f(4.6):C13024/426:delGen=2 _1g(4.6):C5368/142:delGen=2
> _1j(4.6):C499/16:delGen=2 _1m(4.6):C448/23:delGen=2
> _1p(4.6):C236/17:delGen=2 _1k(4.6):C173/5:delGen=1
> _1s(4.6):C1082/78:delGen=2 _1t(4.6):C195/17:delGen=2 _1u(4.6):C2
> _21(4.6):C16494/1278:delGen=1 _22(4.6):C5193/398:delGen=1
> _23(4.6):C1361/102:delGen=1 _24(4.6):C475/36:delGen=1
> _29(4.6):C126/11:delGen=1 _2d(4.6):C97/3:delGen=1 _27(4.6):C59/7:delGen=1
> _28(4.6):C26/6:delGen=1 _2b(4.6):C40 _25(4.6):C39/1:delGen=1
> _2c(4.6):C139/9:delGen=1 _2a(4.6):C26/6:delGen=1)}
>
>
> The only solution is to restart the cluster. Why does a reload not work
> and is this a known bug (for which there is a patch i can apply)?
>
> Any pointers are much appreciated
>
> Thanks!
> Nitin
>

SolrCloud shard marked as down and "reloading" collection doesnt restore it

[KNitin]

Hi,

 I noticed while running an indexing job (2M docs but per doc size could be
2-3 MB) that one of the shards goes down just after the commit.  (Not
related to OOM or high cpu/load).  This marks the shard as "down" in zk and
even a reload of the collection does not recover the state.

There are no exceptions in the logs and the stack trace indicates jetty
threads in blocked state.

The last few lines in the logs are as follows:

trib=TOLEADER=javabin=2} {add=[1552605 (1525453861590925312)]} 0
5
INFO  - 2016-02-06 19:17:47.658;
org.apache.solr.update.DirectUpdateHandler2; start
commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
INFO  - 2016-02-06 19:18:02.209; org.apache.solr.core.SolrDeletionPolicy;
SolrDeletionPolicy.onCommit: commits: num=2
INFO  - 2016-02-06 19:18:02.209; org.apache.solr.core.SolrDeletionPolicy;
newest commit generation = 6
INFO  - 2016-02-06 19:18:02.233; org.apache.solr.search.SolrIndexSearcher;
Opening Searcher@321a0cc9 main
INFO  - 2016-02-06 19:18:02.296; org.apache.solr.core.QuerySenderListener;
QuerySenderListener sending requests to Searcher@321a0cc9
main{StandardDirectoryReader(segments_6:180:nrt
_20(4.6):C15155/216:delGen=1 _w(4.6):C1538/63:delGen=2
_16(4.6):C279/20:delGen=2 _e(4.6):C11386/514:delGen=3
_g(4.6):C4434/204:delGen=3 _p(4.6):C418/5:delGen=1 _v(4.6):C1
_x(4.6):C17583/316:delGen=2 _y(4.6):C9783/112:delGen=2
_z(4.6):C4736/47:delGen=2 _12(4.6):C705/2:delGen=1 _13(4.6):C275/4:delGen=1
_1b(4.6):C619 _26(4.6):C318/13:delGen=1 _1e(4.6):C25356/763:delGen=3
_1f(4.6):C13024/426:delGen=2 _1g(4.6):C5368/142:delGen=2
_1j(4.6):C499/16:delGen=2 _1m(4.6):C448/23:delGen=2
_1p(4.6):C236/17:delGen=2 _1k(4.6):C173/5:delGen=1
_1s(4.6):C1082/78:delGen=2 _1t(4.6):C195/17:delGen=2 _1u(4.6):C2
_21(4.6):C16494/1278:delGen=1 _22(4.6):C5193/398:delGen=1
_23(4.6):C1361/102:delGen=1 _24(4.6):C475/36:delGen=1
_29(4.6):C126/11:delGen=1 _2d(4.6):C97/3:delGen=1 _27(4.6):C59/7:delGen=1
_28(4.6):C26/6:delGen=1 _2b(4.6):C40 _25(4.6):C39/1:delGen=1
_2c(4.6):C139/9:delGen=1 _2a(4.6):C26/6:delGen=1)}


The only solution is to restart the cluster. Why does a reload not work and
is this a known bug (for which there is a patch i can apply)?

Any pointers are much appreciated

Thanks!
Nitin

Re: Monitor backup progress when location parameter is used.

[KNitin]

You can also checkout : https://github.com/bloomreach/solrcloud-haft for
doing backup and restore of your solrcloud collections.

On Fri, Jan 15, 2016 at 12:23 AM, Gian Maria Ricci - aka Alkampfer <
alkamp...@nablasoft.com> wrote:

> Ok thanks, I also think that it's worth a jira, because for restore
> operation we have a convenient restorestatus command that tells exactly the
> status of the restore operation, I think that a backupstatus command could
> be useful.
>
> --
> Gian Maria Ricci
> Cell: +39 320 0136949
>
>
> -Original Message-
> From: Jack Krupansky [mailto:jack.krupan...@gmail.com]
> Sent: giovedì 14 gennaio 2016 17:00
> To: solr-user@lucene.apache.org
> Subject: Re: Monitor backup progress when location parameter is used.
>
> I think the doc is wrong or at least misleading:
>
> https://cwiki.apache.org/confluence/display/solr/Making+and+Restoring+Backups+of+SolrCores
>
> "The backup operation can be monitored to see if it has completed by
> sending the details command to the /replication handler..."
>
> From reading the code, it looks like the snapshot details are only stored
> and returned after the snapshot completes, either successfully or fails,
> but there is nothing set or reported if a snapshot is in progress. So, if
> you don't see a "backup" section in the response, that means the snapshot
> is in progress.
>
> I think it's worth a Jira - either to improve the doc or improve the code
> to report backup as "inProgress... StartedAt...".
>
> You can also look at the log... "Creating backup snapshot" indicates the
> backup has started and "Done creating backup snapshot" indicates success or
> "Exception while creating snapshot" indicates failure. If only that first
> message appeals, it means the backup is still in progress.
>
>
> -- Jack Krupansky
>
> On Thu, Jan 14, 2016 at 9:23 AM, Gian Maria Ricci - aka Alkampfer <
> alkamp...@nablasoft.com> wrote:
>
> > If I start a backup operation using the location parameter
> >
> >
> >
> > *http://localhost:8983/solr/mycore/replication?command=backup=myc
> > ore&
> >  > ore&>location=z:\temp\backupmycore*
> >
> >
> >
> > How can I monitor when the backup operation is finished? Issuing a
> > standard *details* operation
> >
> >
> >
> > *http://localhost:8983/solr/  mycore
> > /replication?command=details*
> >
> >
> >
> > does not gives me useful information, because there are no information
> > on backup on returning data.
> >
> >
> >
> >
> >
> > 
> >
> >
> >
> > 
> >
> > 0
> >
> > 1
> >
> > 
> >
> > 
> >
> > 57.62 GB
> >
> >  > name="indexPath">X:\NoSql\Solr\solr-5.3.1\server\solr\mycore\data\inde
> > x/
> >
> > 
> >
> > 
> >
> > 1452534703494
> >
> > 1509
> >
> > 
> >
> > _2cw.fdt
> >
> > _2cw.fdx
> >
> > _2cw.fnm
> >
> > _2cw.nvd
> >
> > _2cw.nvm
> >
> > _2cw.si
> >
> > _2cw_Lucene50_0.doc
> >
> > _2cw_Lucene50_0.dvd
> >
> > _2cw_Lucene50_0.dvm
> >
> > _2cw_Lucene50_0.pos
> >
> > _2cw_Lucene50_0.tim
> >
> > _2cw_Lucene50_0.tip
> >
> > segments_15x
> >
> > 
> >
> > 
> >
> > 
> >
> > true
> >
> > false
> >
> > 1452534703494
> >
> > 1509
> >
> > 
> >
> >  > name="confFiles">schema.xml,stopwords.txt,elevate.xml
> >
> > 
> >
> > optimize
> >
> > 
> >
> > true
> >
> > 
> >
> > 
> >
> >
> >
> > 
> >
> > --
> > Gian Maria Ricci
> > Cell: +39 320 0136949
> >
> > [image:
> > https://ci5.googleusercontent.com/proxy/5oNMOYAeFXZ_LDKanNfoLRHC37mAZk
> > VVhkPN7QxMdA0K5JW2m0bm8azJe7oWZMNt8fKHNX1bzrUTd-kIyE40CmwT2Mlf8OI=s0-d
> > -e1-ft#http://www.codewrecks.com/files/signature/mvp.png]
> > 
> [image:
> > https://ci3.googleusercontent.com/proxy/f-unQbmk6NtkHFspO5Y6x4jlIf_xrm
> > GLUT3fU9y_7VUHSFUjLs7aUIMdZQYTh3eWIA0sBnvNX3WGXCU59chKXLuAHi2ArWdAcBcl
> > KA=s0-d-e1-ft#http://www.codewrecks.com/files/signature/linkedin.jpg]
> >  [image:
> > https://ci3.googleusercontent.com/proxy/gjapMzu3KEakBQUstx_-cN7gHJ_Gpc
> > IZNEPjCzOYMrPl-r1DViPE378qNAQyEWbXMTj6mcduIAGaApe9qHG1KN_hyFxQAIkdNSVT
> > =s0-d-e1-ft#http://www.codewreck
> s.com/files/signature/twitter.jpg]
> >  [image:
> > https://ci5.googleusercontent.com/proxy/iuDOD2sdaxRDvTwS8MO7-CcXchpNJX
> > 96uaWuvagoVLcjpAPsJi88XeOonE4vHT6udVimo7yL9ZtdrYueEfH7jXnudmi_Vvw=s0-d
> > -e1-ft#http://www.codewrecks.com/files/signature/rss.jpg]
> > 

Transaction Log rotation /retention setup

[KNitin]

Hi,

I was wondering if txn logs obey any log rotation setup rules. Sometimes
indexing can get pretty large and txn logs grow upto tens of
gigabytes(occupying disk which eventually needs to be cleaned up) or as
indexing is progressing and a commit had been made, I want to delete old
txn log to save some space.

Is this supported in Solr already?

Kindly advise,

Thanks
Nitin

Re: Specifying a different txn log directory

[KNitin]

Hi,

Eric:

 I changed updateLog as follows.

 /mnt/nitin_test/ 

I made this change after the collection was created and then updated zk and
reloaded the collection.

Mark: Ok that might be the issue. I will try doing this without the reload.

Thanks,
Nitin

On Sat, Jan 9, 2016 at 2:32 PM, Mark Miller <markrmil...@gmail.com> wrote:

> dataDir and tlog dir cannot be changed with a core reload.
>
> - Mark
>
> On Sat, Jan 9, 2016 at 1:20 PM Erick Erickson <erickerick...@gmail.com>
> wrote:
>
> > Please show us exactly what you did. and exactly
> > what you saw to say that "does not seem to work".
> >
> > Best,
> > Erick
> >
> > On Fri, Jan 8, 2016 at 7:47 PM, KNitin <nitin.t...@gmail.com> wrote:
> > > Hi,
> > >
> > > How do I specify a different directory for transaction logs? I tried
> > using
> > > the updatelog entry in solrconfig.xml and reloaded the collection but
> > that
> > > does not seem to work.
> > >
> > > Is there another setting I need to change?
> > >
> > > Thanks
> > > Nitin
> >
> --
> - Mark
> about.me/markrmiller
>

Specifying a different txn log directory

[KNitin]

Hi,

How do I specify a different directory for transaction logs? I tried using
the updatelog entry in solrconfig.xml and reloaded the collection but that
does not seem to work.

Is there another setting I need to change?

Thanks
Nitin

Re: Field Size per document in Solr

[KNitin]

I want to get the field size (in kb or mb) as is It is stored on disk. That
approach might not give that info.

On Monday, January 4, 2016, Upayavira <u...@odoko.co.uk> wrote:

>
> Solr does store the term positions, but you won't find it easy to
> extract them, as they are stored against terms not fields.
>
> Your best bet is to index field lengths into Solr alongside the field
> values. You could use an UpdateProcessor to do this if you want to do it
> in Solr.
>
> Upayavira
>
> On Tue, Jan 5, 2016, at 12:39 AM, KNitin wrote:
> > Hi,
> >
> >  I want to get the size of individual fields per document (or per index)
> >  in
> > solrcloud. Is there a way to do this using exiting solr or lucene api?
> >
> > *Use case*: I have a few dynamic fields which may or may not be populated
> > everyday depending on certain conditions. I also do faceting and some
> > custom processing on these fields (using custom solr components). I want
> > to
> > be able to plot the per field size of an index in realtime so that I can
> > try to identify the trend between fields & latencies.
> >
> > Thanks a lot in advance!
> > Nitin
>

Field Size per document in Solr

[KNitin]

Hi,

 I want to get the size of individual fields per document (or per index) in
solrcloud. Is there a way to do this using exiting solr or lucene api?

*Use case*: I have a few dynamic fields which may or may not be populated
everyday depending on certain conditions. I also do faceting and some
custom processing on these fields (using custom solr components). I want to
be able to plot the per field size of an index in realtime so that I can
try to identify the trend between fields & latencies.

Thanks a lot in advance!
Nitin

Re: Max indexing threads & RamBuffered size

[KNitin]

Thanks Eric. I will profile and check it out.

On Saturday, December 5, 2015, Erick Erickson <erickerick...@gmail.com>
wrote:

> bq: What adds bottleneck in the indexing flow? Is it the buffering and
> flushing
> out to disk ?
>
> It Depends (tm). What do the Solr logs show when one of these two things
> happens?
>
> You pretty much have to put a profiler on the Solr instance to see where
> it's
> spending the time, but timeouts are very often caused by:
> 1> having a very large heap
> 2> hitting a stop-the-world garbage collection that exceeds your timeouts.
>
> Best,
> Erick
>
> On Sat, Dec 5, 2015 at 8:07 PM, KNitin <nitin.t...@gmail.com
> <javascript:;>> wrote:
> > I have an extremely large indexing load (per doc size of 4-5 Mb with over
> > 100M docs). I have auto commit settings to flush to disk (with open
> > searcher as false) every 20 seconds. Even with that the update sometime
> > fails or timed out. The goal is to improve the indexing throughput and
> > hence trying to experiment and see if tweaking any of these can speed up.
> >
> > What adds bottleneck in the indexing flow? Is it the buffering and
> flushing
> > out to disk ?
> >
> > On Sat, Dec 5, 2015 at 11:15 AM, Erick Erickson <erickerick...@gmail.com
> <javascript:;>>
> > wrote:
> >
> >> I'm pretty sure that max indexing threads is per core, but just looked
> >> and it's not supported in Solr 5.3 and above so I wouldn't worry about
> >> it at all.
> >>
> >> I've never seen much in the way of benefit for bumping this past 128M
> >> or maybe 256M. This is just how much memory is filled up before the
> >> buffer is flushed to disk. Unless you have very high indexing loads or
> >> really long autocommit times, you'll rarely hit it anyway since this
> >> memory is also flushed when you do any flavor of hard commit.
> >>
> >> Best,
> >> Erick
> >>
> >> On Fri, Dec 4, 2015 at 4:55 PM, KNitin <nitin.t...@gmail.com
> <javascript:;>> wrote:
> >> > Hi,
> >> >
> >> > The max indexing threads in the solrconfig.xml is set to 8 by default.
> >> Does
> >> > this mean only 8 concurrent indexing threads will be allowed per
> >> collection
> >> > level? or per core level?
> >> >
> >> > Buffered size : This seems to be set at 64Mb. If we have beefier
> machine
> >> > that can take more load, can we set this to a higher limit say 1 or 2
> Gb?
> >> > What will be downside of doing so? (apart from commits taking longer).
> >> >
> >> > Thanks in advance!
> >> > Nitin
> >>
>

Re: Max indexing threads & RamBuffered size

[KNitin]

I have an extremely large indexing load (per doc size of 4-5 Mb with over
100M docs). I have auto commit settings to flush to disk (with open
searcher as false) every 20 seconds. Even with that the update sometime
fails or timed out. The goal is to improve the indexing throughput and
hence trying to experiment and see if tweaking any of these can speed up.

What adds bottleneck in the indexing flow? Is it the buffering and flushing
out to disk ?

On Sat, Dec 5, 2015 at 11:15 AM, Erick Erickson <erickerick...@gmail.com>
wrote:

> I'm pretty sure that max indexing threads is per core, but just looked
> and it's not supported in Solr 5.3 and above so I wouldn't worry about
> it at all.
>
> I've never seen much in the way of benefit for bumping this past 128M
> or maybe 256M. This is just how much memory is filled up before the
> buffer is flushed to disk. Unless you have very high indexing loads or
> really long autocommit times, you'll rarely hit it anyway since this
> memory is also flushed when you do any flavor of hard commit.
>
> Best,
> Erick
>
> On Fri, Dec 4, 2015 at 4:55 PM, KNitin <nitin.t...@gmail.com> wrote:
> > Hi,
> >
> > The max indexing threads in the solrconfig.xml is set to 8 by default.
> Does
> > this mean only 8 concurrent indexing threads will be allowed per
> collection
> > level? or per core level?
> >
> > Buffered size : This seems to be set at 64Mb. If we have beefier machine
> > that can take more load, can we set this to a higher limit say 1 or 2 Gb?
> > What will be downside of doing so? (apart from commits taking longer).
> >
> > Thanks in advance!
> > Nitin
>

Max indexing threads & RamBuffered size

[KNitin]

Hi,

The max indexing threads in the solrconfig.xml is set to 8 by default. Does
this mean only 8 concurrent indexing threads will be allowed per collection
level? or per core level?

Buffered size : This seems to be set at 64Mb. If we have beefier machine
that can take more load, can we set this to a higher limit say 1 or 2 Gb?
What will be downside of doing so? (apart from commits taking longer).

Thanks in advance!
Nitin

Re: Generating Index offline and loading into solrcloud

[KNitin]

Great. Thanks!

On Thu, Nov 19, 2015 at 11:24 AM, Sameer Maggon <sam...@measuredsearch.com>
wrote:

> If you are trying to create a large index and want speedups there, you
> could use the MapReduceTool -
> https://github.com/cloudera/search/tree/cdh5-1.0.0_5.2.1/search-mr. At a
> high level, it takes your files (csv, json, etc) as input can create either
> a single or a sharded index that you can either copy it to your Solr
> Servers. I've used this to create indexes that include hundreds of millions
> of documents in fairly decent amount of time.
>
> Thanks,
> --
> *Sameer Maggon*
> Measured Search
> www.measuredsearch.com <http://measuredsearch.com/>
>
> On Thu, Nov 19, 2015 at 11:17 AM, KNitin <nitin.t...@gmail.com> wrote:
>
> > Hi,
> >
> >  I was wondering if there are existing tools that will generate solr
> index
> > offline (in solrcloud mode)  that can be later on loaded into solrcloud,
> > before I decide to implement my own. I found some tools that do only solr
> > based index loading (non-zk mode). Is there one with zk mode enabled?
> >
> >
> > Thanks in advance!
> > Nitin
> >
>

Generating Index offline and loading into solrcloud

[KNitin]

Hi,

 I was wondering if there are existing tools that will generate solr index
offline (in solrcloud mode)  that can be later on loaded into solrcloud,
before I decide to implement my own. I found some tools that do only solr
based index loading (non-zk mode). Is there one with zk mode enabled?


Thanks in advance!
Nitin

Re: Generating Index offline and loading into solrcloud

[KNitin]

Ah got it. Another generic question, is there too much of a difference
between generating files in map reduce and loading into solrcloud vs using
solr NRT api? Has any one run any test of that sort?

Thanks a ton,
Nitin

On Thu, Nov 19, 2015 at 3:00 PM, Erick Erickson <erickerick...@gmail.com>
wrote:

> Sure, you can use Lucene to create indexes for shards
> if (and only if) you deal with the routing issues
>
> About updates: I'm not talking about atomic updates at all.
> The usual model for Solr is if you have a unique key
> defined, new versions of documents replace old versions
> of documents based on uniqueKey. That process is
> not guaranteed by MRIT is all.
>
> Best,
> Erick
>
> On Thu, Nov 19, 2015 at 12:56 PM, KNitin <nitin.t...@gmail.com> wrote:
> > Thanks, Eric.  Looks like  MRIT uses Embedded solr running per
> > mapper/reducer and uses that to index documents. Is that the recommended
> > model? Can we use raw lucene libraries to generate index and then load
> them
> > into solrcloud? (Barring the complexities for indexing into right shard
> and
> > merging them).
> >
> > I am thinking of using this for regular offline indexing which needs to
> be
> > idempotent.  When you mean update do you mean partial updates using _set?
> > If we add and delete every time for a document that should work, right?
> > (since all docs are indexed by doc id which contains all operational
> > history)? Let me know if I am missing something.
> >
> > On Thu, Nov 19, 2015 at 12:09 PM, Erick Erickson <
> erickerick...@gmail.com>
> > wrote:
> >
> >> Note two things:
> >>
> >> 1> this is running on Hadoop
> >> 2> it is part of the standard Solr release as MapReduceIndexerTool,
> >> look in the contribs...
> >>
> >> If you're trying to do this yourself, you must be very careful to index
> >> docs
> >> to the correct shard then merge the correct shards. MRIT does this all
> >> automatically.
> >>
> >> Additionally, it has the cool feature that if (and only if) your Solr
> >> index is running over
> >> HDFS, the --go-live option will automatically merge the indexes into
> >> the appropriate
> >> running Solr instances.
> >>
> >> One caveat. This tool doesn't handle _updating_ documents. So if you
> >> run it twice
> >> on the same data set, you'll have two copies of every doc. It's
> >> designed as a bulk
> >> initial-load tool.
> >>
> >> Best,
> >> Erick
> >>
> >>
> >>
> >> On Thu, Nov 19, 2015 at 11:45 AM, KNitin <nitin.t...@gmail.com> wrote:
> >> > Great. Thanks!
> >> >
> >> > On Thu, Nov 19, 2015 at 11:24 AM, Sameer Maggon <
> >> sam...@measuredsearch.com>
> >> > wrote:
> >> >
> >> >> If you are trying to create a large index and want speedups there,
> you
> >> >> could use the MapReduceTool -
> >> >> https://github.com/cloudera/search/tree/cdh5-1.0.0_5.2.1/search-mr.
> At
> >> a
> >> >> high level, it takes your files (csv, json, etc) as input can create
> >> either
> >> >> a single or a sharded index that you can either copy it to your Solr
> >> >> Servers. I've used this to create indexes that include hundreds of
> >> millions
> >> >> of documents in fairly decent amount of time.
> >> >>
> >> >> Thanks,
> >> >> --
> >> >> *Sameer Maggon*
> >> >> Measured Search
> >> >> www.measuredsearch.com <http://measuredsearch.com/>
> >> >>
> >> >> On Thu, Nov 19, 2015 at 11:17 AM, KNitin <nitin.t...@gmail.com>
> wrote:
> >> >>
> >> >> > Hi,
> >> >> >
> >> >> >  I was wondering if there are existing tools that will generate
> solr
> >> >> index
> >> >> > offline (in solrcloud mode)  that can be later on loaded into
> >> solrcloud,
> >> >> > before I decide to implement my own. I found some tools that do
> only
> >> solr
> >> >> > based index loading (non-zk mode). Is there one with zk mode
> enabled?
> >> >> >
> >> >> >
> >> >> > Thanks in advance!
> >> >> > Nitin
> >> >> >
> >> >>
> >>
>

Re: Generating Index offline and loading into solrcloud

[KNitin]

Thanks, Eric.  Looks like  MRIT uses Embedded solr running per
mapper/reducer and uses that to index documents. Is that the recommended
model? Can we use raw lucene libraries to generate index and then load them
into solrcloud? (Barring the complexities for indexing into right shard and
merging them).

I am thinking of using this for regular offline indexing which needs to be
idempotent.  When you mean update do you mean partial updates using _set?
If we add and delete every time for a document that should work, right?
(since all docs are indexed by doc id which contains all operational
history)? Let me know if I am missing something.

On Thu, Nov 19, 2015 at 12:09 PM, Erick Erickson <erickerick...@gmail.com>
wrote:

> Note two things:
>
> 1> this is running on Hadoop
> 2> it is part of the standard Solr release as MapReduceIndexerTool,
> look in the contribs...
>
> If you're trying to do this yourself, you must be very careful to index
> docs
> to the correct shard then merge the correct shards. MRIT does this all
> automatically.
>
> Additionally, it has the cool feature that if (and only if) your Solr
> index is running over
> HDFS, the --go-live option will automatically merge the indexes into
> the appropriate
> running Solr instances.
>
> One caveat. This tool doesn't handle _updating_ documents. So if you
> run it twice
> on the same data set, you'll have two copies of every doc. It's
> designed as a bulk
> initial-load tool.
>
> Best,
> Erick
>
>
>
> On Thu, Nov 19, 2015 at 11:45 AM, KNitin <nitin.t...@gmail.com> wrote:
> > Great. Thanks!
> >
> > On Thu, Nov 19, 2015 at 11:24 AM, Sameer Maggon <
> sam...@measuredsearch.com>
> > wrote:
> >
> >> If you are trying to create a large index and want speedups there, you
> >> could use the MapReduceTool -
> >> https://github.com/cloudera/search/tree/cdh5-1.0.0_5.2.1/search-mr. At
> a
> >> high level, it takes your files (csv, json, etc) as input can create
> either
> >> a single or a sharded index that you can either copy it to your Solr
> >> Servers. I've used this to create indexes that include hundreds of
> millions
> >> of documents in fairly decent amount of time.
> >>
> >> Thanks,
> >> --
> >> *Sameer Maggon*
> >> Measured Search
> >> www.measuredsearch.com <http://measuredsearch.com/>
> >>
> >> On Thu, Nov 19, 2015 at 11:17 AM, KNitin <nitin.t...@gmail.com> wrote:
> >>
> >> > Hi,
> >> >
> >> >  I was wondering if there are existing tools that will generate solr
> >> index
> >> > offline (in solrcloud mode)  that can be later on loaded into
> solrcloud,
> >> > before I decide to implement my own. I found some tools that do only
> solr
> >> > based index loading (non-zk mode). Is there one with zk mode enabled?
> >> >
> >> >
> >> > Thanks in advance!
> >> > Nitin
> >> >
> >>
>

Re: Best way to backup and restore an index for a cloud setup in 4.6.1?

[KNitin]

You can use solrcloud haft :
https://github.com/bloomreach/solrcloud-haft

We use it in our production against 4.6.1.

Nitin

On Monday, May 11, 2015, Shalin Shekhar Mangar 
wrote:

> Hi John,
>
> There are a few HTTP APIs for replication, one of which can let you take a
> backup of the index. Restoring can be as simple as just copying over the
> index in the right location on the disk. A new restore API will be released
> with the next version of Solr which will make some of these tasks easier.
>
> See
>
> https://cwiki.apache.org/confluence/display/solr/Index+Replication#IndexReplication-HTTPAPICommandsfortheReplicationHandler
>
> On Fri, May 8, 2015 at 10:26 PM, John Smith  > wrote:
>
> > All,
> >
> > With a cloud setup for a collection in 4.6.1, what is the most elegant
> way
> > to backup and restore an index?
> >
> > We are specifically looking into the application of when doing a full
> > reindex, with the idea of building an index on one set of servers,
> backing
> > up the index, and then restoring that backup on another set of servers.
> Is
> > there a better way to rebuild indexes on another set of servers?
> >
> > We are not sharding if that makes any difference.
> >
> > Thanks,
> > g10vstmoney
> >
>
>
>
> --
> Regards,
> Shalin Shekhar Mangar.
>

Re: Data Import Handler / Backup indexes

[KNitin]

afaik Data import handler does not offer backups. You can try using the
replication handler to backup data as you wish to any custom end point.

You can also try out : https://github.com/bloomreach/solrcloud-haft.  This
helps backup solr indices across clusters.

On Tue, Nov 17, 2015 at 7:08 AM, Brian Narsi  wrote:

> I am using Data Import Handler to retrieve data from a database with
>
> full-import, clean = true, commit = true and optimize = true
>
> This has always worked correctly without any errors.
>
> But just to be on the safe side, I am thinking that we should do a backup
> before initiating Data Import Handler. And just in case something happens
> restore the backup.
>
> Can backup be done automatically (before initiating Data Import Handler)?
>
> Thanks
>

Re: Replication as backup in SolrCloud

[KNitin]

We built and open sourced haft precisely for such use cases.
https://github.com/bloomreach/solrcloud-haft

 You can clone an entire
cluster or selective collections between clusters. It has only been tested
upto solr 4.10.

Let me know if you run into issues
Nitin

On Monday, June 22, 2015, Erick Erickson  wrote:

> Currently, one is best off treating these as two separate clusters and
> having your client send the data to both, or reproducing your
> system-of-record and running your DCs completely separately.
>
> Hopefully soon, though, there'll be what you're asking for
> active/passive DCs, see:
> https://issues.apache.org/jira/browse/SOLR-6273
>
> Best,
> Erick
>
> On Mon, Jun 22, 2015 at 10:16 AM, StrW_dev  > wrote:
> > Hi,
> >
> > I have a SolrCloud cluster in one data center, but as backup I want to
> have
> > a second (replicated) cluster in another data center.
> >
> > What I want is to replicate to this second cluster, but I don't want my
> > queries to go to this cluster. Is this possible within SolrCloud? As now
> it
> > seems to replicate, but also distribute the query request to this
> replicated
> > server.
> >
> > Gr
> >
> >
> >
> > --
> > View this message in context:
> http://lucene.472066.n3.nabble.com/Replication-as-backup-in-SolrCloud-tp4213267.html
> > Sent from the Solr - User mailing list archive at Nabble.com.
>

Re: copy data between collection

[KNitin]

Yes that is correct. https://github.com/bloomreach/solrcloud-haft helps
precisely with that. You can clone an entire cluster or selective
collections between clusters. It has only been tested upto solr 4.10

Let me know if you run into issues
Nitin

On Mon, Oct 26, 2015 at 9:46 AM, Jeff Wartes  wrote:

>
> The “copy” command in this tool automatically does what Upayavira
> describes, including bringing the replicas up to date. (if any)
> https://github.com/whitepages/solrcloud_manager
>
>
> I’ve been using it as a mechanism for copying a collection into a new
> cluster (different ZK), but it should work within
> a cluster too. The same caveats apply - see the entry in the README.
>
> I’ve also been doing some collection backup/restore stuff that could be
> used to copy a collection within a cluster, (back up your collection, then
> restore into a new collection with a different name) but I only just
> pushed that, and haven’t bundled it into a release yet.
>
> In all cases, you’re responsible for managing the actual collection
> definitions yourself.
>
> An alternative tool I’m aware of is this one:
> https://github.com/bloomreach/solrcloud-haft
>
> This says it’s only tested with Solr 4.6, but I’d think it should work.
> The Solr APIs for replication haven’t changed much. I haven’t used it, but
> it looks like it has some stuff around saving ZK data that could be
> useful, and that’s one thing I haven’t focused on myself yet.
>
>
>
> On 10/26/15, 4:46 AM, "Upayavira"  wrote:
>
> >Hi Shani,
> >
> >There isn't a SolrCloud way to do it. A proper 'clone this collection'
> >feature would be a very useful thing.
> >
> >However, I have managed to do it, in a way that involves some caveats:
> > * you should only do this on a collection that has no replicas. Add
> > replicas *after* cloning the index
> > * if you must do it on a sharded index, then you will need to do it
> > once for each shard. No guarantees though
> >
> >All SolrCloud nodes are all already enabled as 'replication masters' so
> >that new replicas can pull a full index from the current leader. We're
> >gonna use this feature to pull our index (assuming single shard):
> >
> >http://
> :8983/solr/_shard1_replica1/replicat
> >ion?command=fetchindex=http://
> :8983/solr/ >lection>_shard1_replica1/replication
> >
> >This basically says to the core behind your new collection: "Go to the
> >core behind the old collection, and pull its entire index".
> >
> >This worked for me. I added a replica afterwards, and the index cloned
> >correctly. However, when I did it against a collection that had a
> >replica already, the replica *didn't* notice, meaning the leader/replica
> >were now out of sync, i.e: Really make sure you do this replication
> >before you add replicas to your new collection.
> >
> >Hope this helps.
> >
> >Upayavira
> >
> >On Mon, Oct 26, 2015, at 11:21 AM, Chaushu, Shani wrote:
> >> Hi,
> >> Is there an API to copy all the documents from one collection to another
> >> collection in the same solr server simply?
> >> I'm using solr cloud 4.10
> >> Thanks,
> >> Shani
> >>
> >> -
> >> Intel Electronics Ltd.
> >>
> >> This e-mail and any attachments may contain confidential material for
> >> the sole use of the intended recipient(s). Any review or distribution
> >> by others is strictly prohibited. If you are not the intended
> >> recipient, please contact the sender and delete all copies.
>
>

Disabling Query result cache at runtime

[KNitin]

Hi,

 Is there a way to make solr not cache the results when we send the query?
(mainly for query result cache). I need to still enable doc and filter
caching.

Let me know if this is possible,

Thanks
Nitin

Re: Disabling Query result cache at runtime

[KNitin]

Yes for worst case analysis. I usually do this by setting it in zk config
but wanted to check if we can do this at runtime. We tried the
q={!cache=false} but it does not help.

On Fri, Nov 13, 2015 at 12:53 PM, Mikhail Khludnev <
mkhlud...@griddynamics.com> wrote:

> Does q={!cache=false}foo:bar&... work in this case?
>
> On Fri, Nov 13, 2015 at 9:31 PM, KNitin <nitin.t...@gmail.com> wrote:
>
> > Hi,
> >
> >  Is there a way to make solr not cache the results when we send the
> query?
> > (mainly for query result cache). I need to still enable doc and filter
> > caching.
> >
> > Let me know if this is possible,
> >
> > Thanks
> > Nitin
> >
>
>
>
> --
> Sincerely yours
> Mikhail Khludnev
> Principal Engineer,
> Grid Dynamics
>
> <http://www.griddynamics.com>
> <mkhlud...@griddynamics.com>
>

Raw lucene query for a given solr query

[KNitin]

Hi,

 We have a few custom solrcloud components that act as value sources inside
solrcloud for boosting items in the index.  I want to get the final raw
lucene query used by solr for querying the index (for debugging purposes).

Is it possible to get that information?

Kindly advise

Thanks,
Nitin

Re: On the fly reloading of solr core properties

[KNitin]

Hi

I would really appreciate it if any of you can share your insights with
such a use case.

Thanks much
Nitin

On Tuesday, April 28, 2015, KNitin nitin.t...@gmail.com wrote:

 Hi

  In Solrcloud (4.6.1) every time a property/value is changed in
 solrcore.properties file, a core/collection reload is needed to pick up the
 new values.

 Core/Collection reloads for large collections (example 100 shards) is very
 expensive (performance wise) and can pose a threat to the collection
 stability (sometimes the reload fails since the timeout is only 60
 seconds).For a RT serving infrastructure, this has the risk of potentially
 bringing down the collection itself (or cause it to be unstable)

 Would adding a Real Time config api (map) inside solrcloud help? Every
 solr core can pick up the core specific configs from this Shared Map (which
 can be changed on the fly). This can help with dynamically changing
 properties without core reloads.

 Is this a common use case and can be achieved without core reloads?

 Kindly advise,

 Nitin

On the fly reloading of solr core properties

[KNitin]

Hi

 In Solrcloud (4.6.1) every time a property/value is changed in
solrcore.properties file, a core/collection reload is needed to pick up the
new values.

Core/Collection reloads for large collections (example 100 shards) is very
expensive (performance wise) and can pose a threat to the collection
stability (sometimes the reload fails since the timeout is only 60
seconds).For a RT serving infrastructure, this has the risk of potentially
bringing down the collection itself (or cause it to be unstable)

Would adding a Real Time config api (map) inside solrcloud help? Every solr
core can pick up the core specific configs from this Shared Map (which can
be changed on the fly). This can help with dynamically changing properties
without core reloads.

Is this a common use case and can be achieved without core reloads?

Kindly advise,

Nitin

Understanding SolrCloud Restart Behavior - 4.6 onwards

[KNitin]

Hi

 I am trying to understand the process/node restart flow in a SolrCloud
Cluster  . What are the exact set of steps occur (like core/collection
recovery, zk interaction etc) when a node is getting restarted?

I am looking to implement some business logic at a collection/node level
when solr is restarted.

Any pointers on which classes to look at would be really helpful.

Thanks
Nitin

Re: Loading an index (generated by map reduce) in SolrCloud

[KNitin]

Thanks for all the responses. I will try copying the corresponding segments
to the corresponding shards

On Wed, Sep 17, 2014 at 8:26 PM, ralph tice ralph.t...@gmail.com wrote:

 If you are updating or deleting from your indexes I don't believe it is
 possible to get a consistent copy of the index from the file system
 directly without monkeying with hard links.  The safest thing is to use the
 ADDREPLICA command in the Collections API and then an UNLOAD from the CORE
 API if you want to take the data offline.  If you don't care to use
 additional servers/JVMs, you can use the replication handler to make backup
 instead.

 This older discussion covers most any backup strategy I can think of:
 http://grokbase.com/t/lucene/solr-user/12c37h0g18/backing-up-solr-4-0

 On Wed, Sep 17, 2014 at 9:01 PM, shushuai zhu ss...@yahoo.com.invalid
 wrote:

  Hi, my case is a little simpler. For example, I have 100 collections now
  in my solr cloud, and I want to backup 20 of them so I can restore them
  later. I think I can just copy the index and log for each shard/core to
  another location, then delete the collections. Later, I can create new
  collections (likely with different names), then copy the index and log
 back
  to the right directory structure on the node. After that, I can either
  reload the collection or core.
 
  However, some testing shows these do not work. I could not reload the
  collection or core. Have not tried re-starting the solr cloud. Can
 someone
  point out the best way to achieve the goal? I prefer not to re-start solr
  cloud.
 
  Shushuai
 
 
  
   From: ralph tice ralph.t...@gmail.com
  To: solr-user@lucene.apache.org
  Sent: Wednesday, September 17, 2014 6:53 PM
  Subject: Re: Loading an index (generated by map reduce) in SolrCloud
 
 
  FWIW, I do a lot of moving Lucene indexes around and as long as the core
 is
  unloaded it's never been an issue for Solr to be running at the same
 time.
 
  If you move a core into the correct hierarchy for a replica, you can call
  the Collections API's CREATESHARD action with the appropriate params
 (make
  sure you use createNodeSet to point to the right server) and Solr will
 load
  the index appropriately.  It's easiest to create a dummy shard and see
  where data lands on your installation than to try to guess.
 
  Ex:
  PORT=8983
  SHARD=myshard
  COLLECTION=mycollection
  SOLR_HOST=box1.mysolr.corp
  curl http://
 
 
 ${SOLR_HOST}:${PORT}/solr/admin/collections?action=CREATESHARDshard=${SHARD}collection=${COLLECTION}createNodeSet=${SOLR_HOST}:${PORT}_solr
 
  One file to watch out for if you are moving cores across machines/JVMs is
  the core.properties file, which you don't want to duplicate to another
  server/location when moving a data directory.  I don't recommend trying
 to
  move transaction logs around either.
 
 
 
 
 
  On Wed, Sep 17, 2014 at 5:22 PM, Erick Erickson erickerick...@gmail.com
 
  wrote:
 
   Details please. You say MapReduce. Is this the
   MapReduceIndexerTool? If so, you can use
   the --go-live option to auto-merge them. Your
   Solr instances need to be running over HDFS
   though.
  
   If you don't have Solr running over HDFS, you can
   just copy the results for each shard to the right place.
   What that means is that you must insure that the
   shards produced via MRIT get copied to the corresponding
   Solr local directory for each shard. If you put the wrong
   one in the wrong place you'll have trouble with multiple
   copies of documents showing up when you re-add any
   doc that already exists in your Solr installation.
  
   BTW, I'd surely stop all my Solr instances while copying
   all this around.
  
   Best,
   Erick
  
   On Wed, Sep 17, 2014 at 1:41 PM, KNitin nitin.t...@gmail.com wrote:
Hello
   
 I have generated a lucene index (with 6 shards) using Map Reduce. I
  want
to load this into a SolrCloud Cluster inside a collection.
   
Is there any out of the box way of doing this?  Any ideas are much
appreciated
   
Thanks
Nitin
  
 

Loading an index (generated by map reduce) in SolrCloud

[KNitin]

Hello

 I have generated a lucene index (with 6 shards) using Map Reduce. I want
to load this into a SolrCloud Cluster inside a collection.

Is there any out of the box way of doing this?  Any ideas are much
appreciated

Thanks
Nitin

Re: Disabling transaction logs

[KNitin]

Thank u all! Yes I want to disable it for testing purposes

The main issue is that rolling restart of solrcloud for 1000 collections is
extremely unreliable and slow. More than 30% of the collections fail to
recover.

What are some good guidelines to follow while restarting a massive cluster
like this ?

Are there any new improvements (post 4.6) in solr that helps restarts to be
more robust ?

Thanks

On Sunday, August 10, 2014, rulinma ruli...@gmail.com wrote:

 good.



 --
 View this message in context:
 http://lucene.472066.n3.nabble.com/Disabling-transaction-logs-tp4151721p415.html
 Sent from the Solr - User mailing list archive at Nabble.com.

Re: Disabling transaction logs

[KNitin]

Thanks, Mark. Yes I keep track of the overseer and restart it in the end.
The only thing that i observe is that as the zookeeper cluster state file
grows, this behavior gets worse. I notice the following issues

   1. Two nodes (different replicas for the same shard) get stuck in
   recovering state without either becoming a leader. I thought zk was meant
   to break ties but doesnt help
   2. If the recovery fails on a replica, it gets stuck retrying for a very
   long time (in the order of tens of minutes) before it finally giving
   up/recovering
   3. There have been cases 1000 collections restart successfully but takes
   over 2 hours (because of #2)

The cluster state json file is continuously being updated as the cluster
restarts (to update core status). Has anyone see this being a big
bottleneck? Does zookeeper locking files for writes cause a huge issue
while restarting solr?

Also a side question: Why do we need to have a global cluster state json?
Is it better to break it down to a per collection state json file?

Thanks for all your help!
Nitin




On Wed, Aug 13, 2014 at 9:15 AM, Mark Miller markrmil...@gmail.com wrote:

 That is good testing :) We should track down what is up with that 30%.
 Might open a JIRA with some logs.

 It can help if you restart the overseer node last.

 There are likely some improvements around this post 4.6.

 --
 Mark Miller
 about.me/markrmiller

 On August 13, 2014 at 12:05:27 PM, KNitin (nitin.t...@gmail.com) wrote:
  Thank u all! Yes I want to disable it for testing purposes
 
  The main issue is that rolling restart of solrcloud for 1000 collections
 is
  extremely unreliable and slow. More than 30% of the collections fail to
  recover.
 
  What are some good guidelines to follow while restarting a massive
 cluster
  like this ?
 
  Are there any new improvements (post 4.6) in solr that helps restarts to
 be
  more robust ?
 
  Thanks
 
  On Sunday, August 10, 2014, rulinma wrote:
 
   good.
  
  
  
   --
   View this message in context:
  
 http://lucene.472066.n3.nabble.com/Disabling-transaction-logs-tp4151721p415.html
   Sent from the Solr - User mailing list archive at Nabble.com.
  
 

Re: Disabling transaction logs

[KNitin]

Thanks Anshum. This is great to know. If any of you can share your
experience with restarting such massive clusters, that will greatly help




On Wed, Aug 13, 2014 at 3:19 PM, Anshum Gupta ans...@anshumgupta.net
wrote:

 Hi Nitin,

 There's already an issue for breaking the clusterstate.json. Here's the
 link:
 https://issues.apache.org/jira/browse/SOLR-5473

 A lot of work has already been done on that one and hopefully, it
 should be in trunk soon.


 On Wed, Aug 13, 2014 at 3:13 PM, KNitin nitin.t...@gmail.com wrote:
  Thanks, Mark. Yes I keep track of the overseer and restart it in the end.
  The only thing that i observe is that as the zookeeper cluster state file
  grows, this behavior gets worse. I notice the following issues
 
 1. Two nodes (different replicas for the same shard) get stuck in
 recovering state without either becoming a leader. I thought zk was
 meant
 to break ties but doesnt help
 2. If the recovery fails on a replica, it gets stuck retrying for a
 very
 long time (in the order of tens of minutes) before it finally giving
 up/recovering
 3. There have been cases 1000 collections restart successfully but
 takes
 over 2 hours (because of #2)
 
  The cluster state json file is continuously being updated as the cluster
  restarts (to update core status). Has anyone see this being a big
  bottleneck? Does zookeeper locking files for writes cause a huge issue
  while restarting solr?
 
  Also a side question: Why do we need to have a global cluster state json?
  Is it better to break it down to a per collection state json file?
 
  Thanks for all your help!
  Nitin
 
 
 
 
  On Wed, Aug 13, 2014 at 9:15 AM, Mark Miller markrmil...@gmail.com
 wrote:
 
  That is good testing :) We should track down what is up with that 30%.
  Might open a JIRA with some logs.
 
  It can help if you restart the overseer node last.
 
  There are likely some improvements around this post 4.6.
 
  --
  Mark Miller
  about.me/markrmiller
 
  On August 13, 2014 at 12:05:27 PM, KNitin (nitin.t...@gmail.com) wrote:
   Thank u all! Yes I want to disable it for testing purposes
  
   The main issue is that rolling restart of solrcloud for 1000
 collections
  is
   extremely unreliable and slow. More than 30% of the collections fail
 to
   recover.
  
   What are some good guidelines to follow while restarting a massive
  cluster
   like this ?
  
   Are there any new improvements (post 4.6) in solr that helps restarts
 to
  be
   more robust ?
  
   Thanks
  
   On Sunday, August 10, 2014, rulinma wrote:
  
good.
   
   
   
--
View this message in context:
   
 
 http://lucene.472066.n3.nabble.com/Disabling-transaction-logs-tp4151721p415.html
Sent from the Solr - User mailing list archive at Nabble.com.
   
  
 
 



 --

 Anshum Gupta
 http://www.anshumgupta.net

Disabling transaction logs

[KNitin]

Hello

I am using solr 4.6.1 with over 1000 collections and 8 nodes. Restarting of
nodes takes a long time (especially if we have indexing running against it)
. I want to see if disabling transaction logs can help with a better robust
restart. However I can't see any docs around disabling txn logs in solrcloud

Can anyone help with info on how to disable transaction logs ?


Thanks
Nitin

Cannot get shard id error - Hitting limits on creating collections

[KNitin]

Hi

 I am running solr cloud 4.3.1 (there is a plan to upgrade to later
versions but that would take a few months). I noticed a very peculiar solr
behavior in solr that beyond *2496* cores I am unable to create any more
collections due to this error

*Could not get shard id for core.*

I also noticed in the solr tree view that the overseer's collections work
queue gets stuck
 ( /overseer
collection-queue-work
qn-000360
qn-000362
qn-000364)

The test results are as follows.

With 8 shards and 2 replicas, I can create 156 collections (and then hit
the above error)
With 4 shards and 2 replicas, I can create 312 collections (and then hit
the above error)
With 2 shards and 2 replicas, I can create 624 collections (and then hit
the above error)

The total no of cores is 2496 in all the above cases.

I am unable to create any more collections after this due to cannot get
shard id error?

Is this a known bug or is there a work around for this? Is it fixed in
future releases?

Thanks much
-Nitin

Re: Cannot get shard id error - Hitting limits on creating collections

[KNitin]

Thanks, Shawn

I have already raised the jute.buffersize to 5Mb on the zookeeper server
side but still hitting the same problem. Should i make any changes on the
solr server side for this (client side changes?)


On Tue, Apr 8, 2014 at 9:09 AM, Shawn Heisey s...@elyograg.org wrote:

 On 4/8/2014 9:48 AM, KNitin wrote:

   I am running solr cloud 4.3.1 (there is a plan to upgrade to later
 versions but that would take a few months). I noticed a very peculiar solr
 behavior in solr that beyond *2496* cores I am unable to create any more

 collections due to this error

 *Could not get shard id for core.*


 I also noticed in the solr tree view that the overseer's collections
 work
 queue gets stuck
   ( /overseer
 collection-queue-work
 qn-000360
 qn-000362
 qn-000364)

 The test results are as follows.

 With 8 shards and 2 replicas, I can create 156 collections (and then hit
 the above error)
 With 4 shards and 2 replicas, I can create 312 collections (and then hit
 the above error)
 With 2 shards and 2 replicas, I can create 624 collections (and then hit
 the above error)

 The total no of cores is 2496 in all the above cases.

 I am unable to create any more collections after this due to cannot get
 shard id error?

 Is this a known bug or is there a work around for this? Is it fixed in
 future releases?


 You're probably hitting configuration limits, which are set high enough
 for typical scalability requirements.  Certain things need to be
 increased for extreme scalability.  I don't know about all of them, so this
 is likely an incomplete list:

 One of them, most likely the one involved here, is the maximum size of the
 zookeeper database - the jute.maxbuffer system property, which defaults to
 one megabyte.  Another is the maximum number of threads allowed by the
 servlet container. In Jetty, this is the maxThreads parameter.  Another is
 the various connection and thread pool settings in the ShardHandler config.

 http://zookeeper.apache.org/doc/r3.4.6/zookeeperAdmin.html#Unsafe+Options
 http://wiki.apache.org/solr/SolrConfigXml#Configuration_
 of_Shard_Handlers_for_Distributed_searches
 https://cwiki.apache.org/confluence/display/solr/
 Moving+to+the+New+solr.xml+Format

 As usual, I could be entirely incorrect about everything I'm saying.

 Thanks,
 Shawn

Re: Cannot get shard id error - Hitting limits on creating collections

[KNitin]

Thanks. I missed the clients part from doc. Will try and update the
results here




On Tue, Apr 8, 2014 at 3:27 PM, Shawn Heisey s...@elyograg.org wrote:

 On 4/8/2014 4:13 PM, KNitin wrote:

 I have already raised the jute.buffersize to 5Mb on the zookeeper server
 side but still hitting the same problem. Should i make any changes on the
 solr server side for this (client side changes?)


 The jute.maxbuffer system property needs to be set on everything that uses
 Zookeeper, which includes Solr itself as well as the server side.  This
 would also include any call to zkCli. Here's that documentation link I gave
 you before where it says this:

 http://zookeeper.apache.org/doc/r3.4.6/zookeeperAdmin.html#Unsafe+Options

 Thanks,
 Shawn

Re: Cannot get shard id error - Hitting limits on creating collections

[KNitin]

Thanks, Shawn. Adding it to all clients and servers worked


On Tue, Apr 8, 2014 at 3:37 PM, KNitin nitin.t...@gmail.com wrote:

 Thanks. I missed the clients part from doc. Will try and update the
 results here




 On Tue, Apr 8, 2014 at 3:27 PM, Shawn Heisey s...@elyograg.org wrote:

 On 4/8/2014 4:13 PM, KNitin wrote:

 I have already raised the jute.buffersize to 5Mb on the zookeeper server
 side but still hitting the same problem. Should i make any changes on the
 solr server side for this (client side changes?)


 The jute.maxbuffer system property needs to be set on everything that
 uses Zookeeper, which includes Solr itself as well as the server side.
  This would also include any call to zkCli. Here's that documentation link
 I gave you before where it says this:

 http://zookeeper.apache.org/doc/r3.4.6/zookeeperAdmin.html#Unsafe+Options

 Thanks,
 Shawn

Race condition in Leader Election

[KNitin]

Hi

 When restarting a node in solrcloud, i run into scenarios where both the
replicas for a shard get into recovering state and never come up causing
the error No servers hosting this shard. To fix this, I either unload one
core or restart one of the nodes again so that one of them becomes the
leader.

Is there a way to force leader election for a shard for solrcloud? Is
there a way to break ties automatically (without restarting nodes) to make
a node as the leader for the shard?


Thanks
Nitin

Re: Race condition in Leader Election

[KNitin]

I am using 4.3.1.


On Thu, Mar 6, 2014 at 11:48 AM, Mark Miller markrmil...@gmail.com wrote:

 Are you using an old version?

 - Mark

 http://about.me/markrmiller

 On Mar 6, 2014, at 11:50 AM, KNitin nitin.t...@gmail.com wrote:

  Hi
 
  When restarting a node in solrcloud, i run into scenarios where both the
  replicas for a shard get into recovering state and never come up
 causing
  the error No servers hosting this shard. To fix this, I either unload
 one
  core or restart one of the nodes again so that one of them becomes the
  leader.
 
  Is there a way to force leader election for a shard for solrcloud? Is
  there a way to break ties automatically (without restarting nodes) to
 make
  a node as the leader for the shard?
 
 
  Thanks
  Nitin

Solr Cloud Cores, Zookeepers and Zk Data

[KNitin]

Hi

 I am trying to understand the flow between zk and SolrCloud nodes during
writes and restarts.

*Writes*:

 When an indexing job runs , it looks like the leader for every shard  is
identified from zk and the write requests goes to the leader and then
eventually data flows to replicas.

Question: How frequently does zk and the nodes need to communicate during
the write operation? Could this cause a few shards/cores to get flaky (go
to down/recovering state at times if its unable to talk to zk?)

*Restarts*:

The size of data in my zk is around 60Mb and the zk cfg is as follows

tickTime=2000

dataDir=/mnt/data/zookeeper

clientPort=2181

initLimit=10

syncLimit=15

maxClientCnxns=100

maxSessionTimeout=12

autopurge.snapRetainCount=3

autopurge.purgeInterval=1

When I restart a node containing say 100 shards, it takes about 20-25
minutes to fully boot up and i find that Solr is *not* bottlenecked on
cpu,mem or any other system parameter (There are no auto warm or first
searcher queries running).  A few shards load faster than the others.

Question: Is the above setting good enough to stream 60Mb of data in total
from zk node everytime i restart a node? Is there anything I can do to
identify if zk is the bottleneck while booting up nodes


Thanks
Nitin

Re: Solr Cloud Cores, Zookeepers and Zk Data

[KNitin]

I should also mention that the watch count is in the order of 400-500 but
the maxClientConnections is 100. Not sure if this has to do with the issue
but just putting it out there


On Wed, Mar 5, 2014 at 11:37 AM, KNitin nitin.t...@gmail.com wrote:

 Hi

  I am trying to understand the flow between zk and SolrCloud nodes during
 writes and restarts.

 *Writes*:

  When an indexing job runs , it looks like the leader for every shard  is
 identified from zk and the write requests goes to the leader and then
 eventually data flows to replicas.

 Question: How frequently does zk and the nodes need to communicate during
 the write operation? Could this cause a few shards/cores to get flaky (go
 to down/recovering state at times if its unable to talk to zk?)

 *Restarts*:

 The size of data in my zk is around 60Mb and the zk cfg is as follows

 tickTime=2000

 dataDir=/mnt/data/zookeeper

 clientPort=2181

 initLimit=10

 syncLimit=15

 maxClientCnxns=100

 maxSessionTimeout=12

 autopurge.snapRetainCount=3

 autopurge.purgeInterval=1

 When I restart a node containing say 100 shards, it takes about 20-25
 minutes to fully boot up and i find that Solr is *not* bottlenecked on
 cpu,mem or any other system parameter (There are no auto warm or first
 searcher queries running).  A few shards load faster than the others.

 Question: Is the above setting good enough to stream 60Mb of data in total
 from zk node everytime i restart a node? Is there anything I can do to
 identify if zk is the bottleneck while booting up nodes


 Thanks
 Nitin

Re: SolrCloud Startup

[KNitin]

I did the following as you suggested. I have a lib dir under /mnt/solr/
(this is the solr.solr.home dir) and moved all my jars in it. I do not have
anySharedLib or lib references in my solr or solrconfig. xml file

The jars are not getting loaded for a few custom analyzers I have in the
schema.

Should I specify anywhere to use /mnt/solr/lib/ as the lib path to use
anywhere?

- Nitin


On Mon, Mar 3, 2014 at 3:06 PM, KNitin nitin.t...@gmail.com wrote:

 Thanks, Shawn.  Right now my solr.solr.home is not being passed from the
 java runtime

 Lets say /mnt/solr/ is my solr root. I can add all jars to /mnt/solr/lib/
 and use -Dsolr.solr.home=/mnt/solr/  , that should do it right?

 Thanks
 Nitin


 On Mon, Mar 3, 2014 at 2:44 PM, Shawn Heisey s...@elyograg.org wrote:

 On 3/3/2014 3:30 PM, KNitin wrote:

 A quick ping on this. To give more stats, I have 100's of collections on
 every node. The time it takes for one collection to boot up
 /loadonStartup
 is around 10-20 seconds (and sometimes even 1 minute). I do not have any
 query auto warming etc. On a per collection basis I load a bunch of
 libraries (for custom analyzer plugins) to compute the classpath. That
 might be a reason for the high boot up time

My solrconfig.xml entry is as follows

lib dir=/mnt/solr/lib/ regex=.*\.jar /

   Every core that boots up seems to be loading all jars over and over
 again.
 Is there a way to ask solr to load all jars only once?


 Three steps:

 1) Get rid of all your lib directives in solrconfig.xml entirely.
 2) Copy all the extra jars that you need into ${solr.solr.home}/lib.
 3) Remove any sharedLib parameter from your solr.xml file.

 Step 3 is required because you are on 4.3.1 (or later if you have already
 upgraded).

 The final comment on the following issue summarizes issues that I ran
 into while migrating this approach from 4.2.1 to later releases:

 https://issues.apache.org/jira/browse/SOLR-4852

 Thanks,
 Shawn

Re: SolrCloud Startup

[KNitin]

Thanks a lot, Shawn! I was missing an ICU jar as a part of my original
setup. I then copied the analysis jars into solr/lib and removed all
reference in solrconfig.xml and it worked like a charm

The permgen space also seems to have reduced significantly

Thanks
Nitin


On Tue, Mar 4, 2014 at 2:41 PM, Shawn Heisey s...@elyograg.org wrote:

 On 3/4/2014 3:09 PM, KNitin wrote:

 I did the following as you suggested. I have a lib dir under /mnt/solr/
 (this is the solr.solr.home dir) and moved all my jars in it. I do not
 have
 anySharedLib or lib references in my solr or solrconfig. xml file

 The jars are not getting loaded for a few custom analyzers I have in the
 schema.

 Should I specify anywhere to use /mnt/solr/lib/ as the lib path to use
 anywhere?


 The solr home is where solr.xml lives.  So if /mnt/solr is that location,
 then that would be where you want solr home to point.  Generally your core
 directories are also in solr.home, but if you've customized the locations,
 that may not be true.

 In 4.3 and later, the lib directory under the solr home is automatically
 added to the classpath.  Also in that version, if you *do* explicitly
 include it, it won't work right -- which is what prompted me to file
 SOLR-4852.  The symptoms were that the jars would get loaded (twice,
 actually), but the classes were not actually available.

 Thanks,
 Shawn

Re: Solr Heap, MMaps and Garbage Collection

[KNitin]

Regarding PermGen: Yes we have a bunch of custom jars loaded in solrcloud
(containing custom parsing, analyzers). But I haven't specifically enabled
any string interning. Does solr intern all strings in a collection by
default?

I agree with doc and Filter Query Cache. Query Result cache hits are
practically 0 for the large collection since our queries are tail by nature


Thanks
Nitin


On Mon, Mar 3, 2014 at 5:01 AM, Michael Sokolov 
msoko...@safaribooksonline.com wrote:

 On 3/3/2014 1:54 AM, KNitin wrote:

 3. 2.8 Gb - Perm Gen (I am guessing this is because of interned strings)

 As others have pointed out, this is really unusual for Solr.  We often see
 high permgen in our app servers due to dynamic class loading that the
 framework performs; maybe you are somehow loading lots of new Solr plugins,
 or otherwise creating lots of classes?  Of course if you have a plugin or
 something that does a lot of string interning, that could also be an
 explanation.

 -Mike

Re: Solr Heap, MMaps and Garbage Collection

[KNitin]

Is there a way to dump the contents of permgen and look at which classes
are occupying the most memory in that?

- Nitin


On Mon, Mar 3, 2014 at 11:19 AM, KNitin nitin.t...@gmail.com wrote:

 Regarding PermGen: Yes we have a bunch of custom jars loaded in solrcloud
 (containing custom parsing, analyzers). But I haven't specifically enabled
 any string interning. Does solr intern all strings in a collection by
 default?

 I agree with doc and Filter Query Cache. Query Result cache hits are
 practically 0 for the large collection since our queries are tail by nature


 Thanks
 Nitin


 On Mon, Mar 3, 2014 at 5:01 AM, Michael Sokolov 
 msoko...@safaribooksonline.com wrote:

 On 3/3/2014 1:54 AM, KNitin wrote:

 3. 2.8 Gb - Perm Gen (I am guessing this is because of interned strings)

 As others have pointed out, this is really unusual for Solr.  We often
 see high permgen in our app servers due to dynamic class loading that the
 framework performs; maybe you are somehow loading lots of new Solr plugins,
 or otherwise creating lots of classes?  Of course if you have a plugin or
 something that does a lot of string interning, that could also be an
 explanation.

 -Mike

Re: SolrCloud Startup

[KNitin]

A quick ping on this. To give more stats, I have 100's of collections on
every node. The time it takes for one collection to boot up /loadonStartup
is around 10-20 seconds (and sometimes even 1 minute). I do not have any
query auto warming etc. On a per collection basis I load a bunch of
libraries (for custom analyzer plugins) to compute the classpath. That
might be a reason for the high boot up time

  My solrconfig.xml entry is as follows

  lib dir=/mnt/solr/lib/ regex=.*\.jar /

 Every core that boots up seems to be loading all jars over and over again.
Is there a way to ask solr to load all jars only once?

Thanks
- Nitin


On Wed, Feb 26, 2014 at 3:06 PM, KNitin nitin.t...@gmail.com wrote:

 Thanks, Shawn. I will try to upgrade solr soon

 Reg firstSearcher: I think it does nothing now. I have configured to use
 ExternalFileLoader but there the external file has no contents. Most of the
 queries hitting the collection are expensive and tail queries. What will be
 your recommendation to warm the first Searcher/new Searcher?

 Thanks
 Nitin


 On Tue, Feb 25, 2014 at 4:12 PM, Shawn Heisey s...@elyograg.org wrote:

 On 2/25/2014 4:30 PM, KNitin wrote:

 Jeff :  Thanks. I have tried reload before but it is not reliable
 (atleast
 in 4.3.1). A few cores get initialized and few dont (show as just
 recovering or down) and hence had to move away from it. Is it a known
 issue
 in 4.3.1?


 With Solr 4.3.1, you are running into this bug with reloads under
 SolrCloud:

 https://issues.apache.org/jira/browse/SOLR-4805

 The only way to recover from this bug is to restart Solr.The bug is fixed
 in 4.4.0 and later.


  Shawn,Otis,Erick

   Yes I have reviewed the page before and have given 1/4 of my mem to JVM
 and the rest to RAM/Os Cache. (15 Gb heap and 45 G to rest. Totally 60G
 machine). I have also reviewed the tlog file and they are in the order of
 KB (4-10 or 30). I have SSD and the reads are hardly noticable (in the
 order of 100Kb during that time frame). I have also disabled swap on all
 machines

 Regarding firstSearcher, It is currently set to externalFileLoader. What
 is
 the use of first searcher? I havent played around with it


 I don't think it's a good idea to have extensive warming queries.  I do
 exactly one query in firstSearcher and newSearcher: a query for all
 documents with zero rows, sorted on our most common sort field.  This is
 designed purely to preload the sort data into the FieldCache.

 Thanks,
 Shawn

Re: SolrCloud Startup

[KNitin]

Thanks, Shawn.  Right now my solr.solr.home is not being passed from the
java runtime

Lets say /mnt/solr/ is my solr root. I can add all jars to /mnt/solr/lib/
and use -Dsolr.solr.home=/mnt/solr/  , that should do it right?

Thanks
Nitin


On Mon, Mar 3, 2014 at 2:44 PM, Shawn Heisey s...@elyograg.org wrote:

 On 3/3/2014 3:30 PM, KNitin wrote:

 A quick ping on this. To give more stats, I have 100's of collections on
 every node. The time it takes for one collection to boot up /loadonStartup
 is around 10-20 seconds (and sometimes even 1 minute). I do not have any
 query auto warming etc. On a per collection basis I load a bunch of
 libraries (for custom analyzer plugins) to compute the classpath. That
 might be a reason for the high boot up time

My solrconfig.xml entry is as follows

lib dir=/mnt/solr/lib/ regex=.*\.jar /

   Every core that boots up seems to be loading all jars over and over
 again.
 Is there a way to ask solr to load all jars only once?


 Three steps:

 1) Get rid of all your lib directives in solrconfig.xml entirely.
 2) Copy all the extra jars that you need into ${solr.solr.home}/lib.
 3) Remove any sharedLib parameter from your solr.xml file.

 Step 3 is required because you are on 4.3.1 (or later if you have already
 upgraded).

 The final comment on the following issue summarizes issues that I ran into
 while migrating this approach from 4.2.1 to later releases:

 https://issues.apache.org/jira/browse/SOLR-4852

 Thanks,
 Shawn

Solr Heap, MMaps and Garbage Collection

[KNitin]

Hi

I have very large index for a few collections and when they are being
queried, i see the Old gen space close to 100% Usage all the time. The
system becomes extremely slow due to GC activity right after that and it
gets into this cycle very often

I have given solr close to 30G of heap in a 65 GB ram machine and rest is
given to RAm. I have a lot of hits in filter,query result and document
caches and the size of all the caches is around 512 entries per
collection.Are all the caches used by solr on or off heap ?


Given this scenario where GC is the primary bottleneck what is a good
recommended memory settings for solr? Should i increase the heap memory
(that will only postpone the problem before the heap becomes full again
after a while) ? Will memory maps help at all in this scenario?


Kindly advise on the best practices
Thanks
Nitin

Re: Solr Heap, MMaps and Garbage Collection

[KNitin]

Thanks, Walter

Hit rate on the document caches is close to 70-80% and the filter caches
are a 100% hit (since most of our queries filter on the same fields but
have a different q parameter). Query result cache is not of great
importance to me since the hit rate their is almost negligible.

Does it mean i need to increase the size of my filter and document cache
for large indices?

The split up of my 25Gb heap usage is split as follows

1. 19 GB - Old Gen (100% pool utilization)
2.  3 Gb - New Gen (50% pool utilization)
3. 2.8 Gb - Perm Gen (I am guessing this is because of interned strings)
4. Survivor space is in the order of 300-400 MB and is almost always 100%
full.(Is this a major issue?)

We are also currently using Parallel GC collector but planning to move to
CMS for lesser stop-the-world gc times. If i increase the filter cache and
document cache entry sizes, they would also go to the Old gen right?

A very naive question: How does increasing young gen going to help if we
know that solr is already pushing major caches and other objects to old gen
because of their nature? My young gen pool utilization is still well under
50%


Thanks
Nitin


On Sun, Mar 2, 2014 at 9:31 PM, Walter Underwood wun...@wunderwood.orgwrote:

 An LRU cache will always fill up the old generation. Old objects are
 ejected, and those are usually in the old generation.

 Increasing the heap size will not eliminate this. It will make major, stop
 the world collections longer.

 Increase the new generation size until the rate of old gen increase slows
 down. Then choose a total heap size to control the frequency (and duration)
 of major collections.

 We run with the new generation at about 25% of the heap, so 8GB total and
 a 2GB newgen.

 A 512 entry cache is very small for query results or docs. We run with 10K
 or more entries for those. The filter cache size depends on your usage. We
 have only a handful of different filter queries, so a tiny cache is fine.

 What is your hit rate on the caches?

 wunder

 On Mar 2, 2014, at 7:42 PM, KNitin nitin.t...@gmail.com wrote:

  Hi
 
  I have very large index for a few collections and when they are being
  queried, i see the Old gen space close to 100% Usage all the time. The
  system becomes extremely slow due to GC activity right after that and it
  gets into this cycle very often
 
  I have given solr close to 30G of heap in a 65 GB ram machine and rest is
  given to RAm. I have a lot of hits in filter,query result and document
  caches and the size of all the caches is around 512 entries per
  collection.Are all the caches used by solr on or off heap ?
 
 
  Given this scenario where GC is the primary bottleneck what is a good
  recommended memory settings for solr? Should i increase the heap memory
  (that will only postpone the problem before the heap becomes full again
  after a while) ? Will memory maps help at all in this scenario?
 
 
  Kindly advise on the best practices
  Thanks
  Nitin

Perm Gen issues in SolrCloud

[KNitin]

Hi

 I am seeing the Perm Gen usage increase as i keep adding more collections.
What kind of strings get interned in solr? (Only schema , fields,
collection metadata or the data itself?)

Will Permgen space (atleast interned strings) increase proportional to the
size of the data in the collections or with the # of collections themselves?


I have temporarily increased the size of PermGen to deal with this but
would love to understand what goes on behind the scenes

Thanks
Nitin

Re: Perm Gen issues in SolrCloud

[KNitin]

Hi Furkan

 I have read that before but I haven't added any new classes or changed
anything with my setup. I just created more collections in solr. How will
that increase perm gen space ? Doesn't solr intern strings at all ?
Interned strings also go to the perm gen space right?

- Nitin


On Fri, Feb 28, 2014 at 3:11 PM, Furkan KAMACI furkankam...@gmail.comwrote:

 Hi;

 Jack has an answer for a PermGen usages:

 PermGen memory has to do with number of classes loaded, rather than
 documents.

 Here are a couple of pages that help explain Java PermGen issues. The
 bottom
 line is that you can increase the PermGen space, or enable unloading of
 classes, or at least trace class loading to see why the problem occurs.


 http://stackoverflow.com/questions/88235/how-to-deal-with-java-lang-outofmemoryerror-
 permgen-space-error

 http://www.brokenbuild.com/blog/2006/08/04/java-jvm-gc-permgen
 -and-memory-options/
 

 You can see the conversation from here:
 http://search-lucene.com/m/iMaR11lgj3Q1/permgensubj=PermGen+OOM+Error

 Thanks;
 Furkan KAMACI


 2014-02-28 21:37 GMT+02:00 KNitin nitin.t...@gmail.com:

  Hi
 
   I am seeing the Perm Gen usage increase as i keep adding more
 collections.
  What kind of strings get interned in solr? (Only schema , fields,
  collection metadata or the data itself?)
 
  Will Permgen space (atleast interned strings) increase proportional to
 the
  size of the data in the collections or with the # of collections
  themselves?
 
 
  I have temporarily increased the size of PermGen to deal with this but
  would love to understand what goes on behind the scenes
 
  Thanks
  Nitin
 

Tracing Solr Query Execution and Performance

[KNitin]

Hi there

  I have a few very expensive queries (atleast thats what the QTime tells
me) that is causing high CPU problems on a few nodes. Is there a way where
I can trace or do an explain on the solr query to see where it spends
more time? More like profiling on a per sub query basis?

I have tried using debug=timing as a part of the query and it gives me
stage level details (parsing, highlighting) but I need much more insights
into where a query is spending time on


Any help is much appreciated

Thanks
Nitin

Re: SolrCloud Startup

[KNitin]

Thanks, Shawn. I will try to upgrade solr soon

Reg firstSearcher: I think it does nothing now. I have configured to use
ExternalFileLoader but there the external file has no contents. Most of the
queries hitting the collection are expensive and tail queries. What will be
your recommendation to warm the first Searcher/new Searcher?

Thanks
Nitin


On Tue, Feb 25, 2014 at 4:12 PM, Shawn Heisey s...@elyograg.org wrote:

 On 2/25/2014 4:30 PM, KNitin wrote:

 Jeff :  Thanks. I have tried reload before but it is not reliable (atleast
 in 4.3.1). A few cores get initialized and few dont (show as just
 recovering or down) and hence had to move away from it. Is it a known
 issue
 in 4.3.1?


 With Solr 4.3.1, you are running into this bug with reloads under
 SolrCloud:

 https://issues.apache.org/jira/browse/SOLR-4805

 The only way to recover from this bug is to restart Solr.The bug is fixed
 in 4.4.0 and later.


  Shawn,Otis,Erick

   Yes I have reviewed the page before and have given 1/4 of my mem to JVM
 and the rest to RAM/Os Cache. (15 Gb heap and 45 G to rest. Totally 60G
 machine). I have also reviewed the tlog file and they are in the order of
 KB (4-10 or 30). I have SSD and the reads are hardly noticable (in the
 order of 100Kb during that time frame). I have also disabled swap on all
 machines

 Regarding firstSearcher, It is currently set to externalFileLoader. What
 is
 the use of first searcher? I havent played around with it


 I don't think it's a good idea to have extensive warming queries.  I do
 exactly one query in firstSearcher and newSearcher: a query for all
 documents with zero rows, sorted on our most common sort field.  This is
 designed purely to preload the sort data into the FieldCache.

 Thanks,
 Shawn

Re: Tracing Solr Query Execution and Performance

[KNitin]

Thanks, Jack. I will file a jira then. What are the generic ways to
improve/tune a solr query if we know its expensive? Does the analysis page
help with this at all?


On Wed, Feb 26, 2014 at 3:39 PM, Jack Krupansky j...@basetechnology.comwrote:

 I don't recall seeing anything related to passing the debug/debugQuery
 parameters on for inter-node shard queries and then add that to the
 aggregated response (if debug/debugQuery was specified.) Sounds worth a
 Jira.

 -- Jack Krupansky

 -Original Message- From: KNitin
 Sent: Wednesday, February 26, 2014 5:25 PM
 To: solr-user@lucene.apache.org
 Subject: Tracing Solr Query Execution and Performance


 Hi there

  I have a few very expensive queries (atleast thats what the QTime tells
 me) that is causing high CPU problems on a few nodes. Is there a way where
 I can trace or do an explain on the solr query to see where it spends
 more time? More like profiling on a per sub query basis?

 I have tried using debug=timing as a part of the query and it gives me
 stage level details (parsing, highlighting) but I need much more insights
 into where a query is spending time on


 Any help is much appreciated

 Thanks
 Nitin

Re: SolrCloud Startup

[KNitin]

Jeff :  Thanks. I have tried reload before but it is not reliable (atleast
in 4.3.1). A few cores get initialized and few dont (show as just
recovering or down) and hence had to move away from it. Is it a known issue
in 4.3.1?

Shawn,Otis,Erick

 Yes I have reviewed the page before and have given 1/4 of my mem to JVM
and the rest to RAM/Os Cache. (15 Gb heap and 45 G to rest. Totally 60G
machine). I have also reviewed the tlog file and they are in the order of
KB (4-10 or 30). I have SSD and the reads are hardly noticable (in the
order of 100Kb during that time frame). I have also disabled swap on all
machines

Regarding firstSearcher, It is currently set to externalFileLoader. What is
the use of first searcher? I havent played around with it

Thanks
Nitin





On Mon, Feb 24, 2014 at 7:58 PM, Erick Erickson erickerick...@gmail.comwrote:

 What is your firstSearcher set to in solrconfig.xml? If you're
 doing something really crazy there that might be an issue.

 But I think Otis' suggestion is a lot more probable. What
 are your autocommits configured to?

 Best,
 Erick


 On Mon, Feb 24, 2014 at 7:41 PM, Shawn Heisey s...@elyograg.org wrote:

   Hi
  
I have a 4 node solrcloud cluster with more than 50 collections with 4
   shards each. Everytime I want to make a schema change, I upload configs
  to
   zookeeper and then restart all nodes. However the restart of every node
  is
   very slow and takes about 20-30 minutes per node.
  
   Is it recommended to make loadOnStartup=false and allow solrcloud to
 lazy
   load? Is there a way to make schema changes without restarting
 solrcloud?
 
  I'm on my phone so getting a Url for you is hard. Search the wiki for
  SolrPerformanceProblems. There's a section there on slow startup.
 
  If that's not it, it's probably not enough RAM for the OS disk cache.
 That
  is also discussed on that wiki page.
 
  Thanks,
  Shawn
 
 
 
 

Re: SolrCloud Startup

[KNitin]

Erick: My autocommit is set to trigger every 30 seconds with
openSearcher=false. The autocommit for soft commits are disabled


On Tue, Feb 25, 2014 at 3:30 PM, KNitin nitin.t...@gmail.com wrote:

 Jeff :  Thanks. I have tried reload before but it is not reliable (atleast
 in 4.3.1). A few cores get initialized and few dont (show as just
 recovering or down) and hence had to move away from it. Is it a known issue
 in 4.3.1?

 Shawn,Otis,Erick

  Yes I have reviewed the page before and have given 1/4 of my mem to JVM
 and the rest to RAM/Os Cache. (15 Gb heap and 45 G to rest. Totally 60G
 machine). I have also reviewed the tlog file and they are in the order of
 KB (4-10 or 30). I have SSD and the reads are hardly noticable (in the
 order of 100Kb during that time frame). I have also disabled swap on all
 machines

 Regarding firstSearcher, It is currently set to externalFileLoader. What
 is the use of first searcher? I havent played around with it

 Thanks
 Nitin





 On Mon, Feb 24, 2014 at 7:58 PM, Erick Erickson 
 erickerick...@gmail.comwrote:

 What is your firstSearcher set to in solrconfig.xml? If you're
 doing something really crazy there that might be an issue.

 But I think Otis' suggestion is a lot more probable. What
 are your autocommits configured to?

 Best,
 Erick


 On Mon, Feb 24, 2014 at 7:41 PM, Shawn Heisey s...@elyograg.org wrote:

   Hi
  
I have a 4 node solrcloud cluster with more than 50 collections with
 4
   shards each. Everytime I want to make a schema change, I upload
 configs
  to
   zookeeper and then restart all nodes. However the restart of every
 node
  is
   very slow and takes about 20-30 minutes per node.
  
   Is it recommended to make loadOnStartup=false and allow solrcloud to
 lazy
   load? Is there a way to make schema changes without restarting
 solrcloud?
 
  I'm on my phone so getting a Url for you is hard. Search the wiki for
  SolrPerformanceProblems. There's a section there on slow startup.
 
  If that's not it, it's probably not enough RAM for the OS disk cache.
 That
  is also discussed on that wiki page.
 
  Thanks,
  Shawn
 
 
 
 

SolrCloud Startup

[KNitin]

Hi

 I have a 4 node solrcloud cluster with more than 50 collections with 4
shards each. Everytime I want to make a schema change, I upload configs to
zookeeper and then restart all nodes. However the restart of every node is
very slow and takes about 20-30 minutes per node.

Is it recommended to make loadOnStartup=false and allow solrcloud to lazy
load? Is there a way to make schema changes without restarting solrcloud?


Thanks

Re: Solr Segments, Segment Merges,Optimize

[KNitin]

Commit Parameters: Server does an auto commit every 30 seconds with
open_searcher=false. The pipeline does a hard commit only at the very end
of its run

The high CPU issue I am seeing is only during the reads and not during the
writes. Right now I see a direct corelation between latencies and # of
segments atleast for a few large collections. Will post back if the theory
is invalidated

Thanks
- Nitin


On Sat, Feb 22, 2014 at 10:01 PM, Erick Erickson erickerick...@gmail.comwrote:

 Well, it's always possible. I wouldn't expect the search time/CPU
 utilization to increase with # segments, within reasonable limits.
 At some point, the important parts of the index get read into memory
 and the number of segments is pretty irrelevant. You do mention
 that you have a heavy ingestion pipeline, which leads me to wonder
 whether you're committing too often, what are your commit
 parameters?

 For % deleted docs, I'm really talking about deletedDocs/numDocs.

 I suppose the interesting question is whether the CPU utilization you're
 seeing is _always_ correlated with # segments or are you seeing certain
 machines always having the high CPU utilization. I suppose you could
 issue a commit and see what difference that made.

 I rather doubt that the # of segments is the underlying issue, but that's
 nothing but a SWAG...

 Best,
 Erick




 On Sat, Feb 22, 2014 at 6:16 PM, KNitin nitin.t...@gmail.com wrote:

  Thanks, Erick.
 
  *2 There are, but you'll have to dig. *
 
  Any pointers on where to get started?
 
 
 
  *3 Well, I'd ask a counter-question. Are you seeing
  unacceptableperformance? If not, why worry? :)*
 
  When you mean % do you refer to deleted_docs/NumDocs or
  deleted_docs/Max_docs ? To answer your question, yes i see some of our
  shards taking 3x more time and 3x more cpu than other shards for the same
  queries and same number of hits (all shards have exact same number of
 docs
  but i see a few shards having more deleted documents than the rest).
 
  My understanding is that the Search time /CPU would increase with # of
  segments ?  The core of my issue is that few nodes are running with
  extremely high CPU (90+)  and rest are running under 30% CPU and the only
  difference between both is the  # of segments in the shards on the
  machines. The nodes running hot have shards with 30 segments and the ones
  running with lesser CPU contain 20 segments and much lesser deleted
  documents.
 
  Is it possible that a difference of 10 segments could impact CPU /Search
  time?
 
  Thanks
  - Nitin
 
 
  On Sat, Feb 22, 2014 at 4:36 PM, Erick Erickson erickerick...@gmail.com
  wrote:
 
   1 It Depends. Soft commits will not add a new segment. Hard commits
   with openSearcher=true or false _will_ create a new segment.
   2 There are, but you'll have to dig.
   3 Well, I'd ask a counter-question. Are you seeing unacceptable
   performance? If not, why worry? :)
  
   A better answer is that 24-28 segments is not at all unusual.
  
   By and large, don't bother with optimize/force merge. What I would do
 is
   look at the admin screen and note the percentage of deleted documents.
   If it's above some arbitrary number (I typically use 15-20%) and
 _stays_
   there, consider optimizing.
  
   However! There is a parameter you can explicitly set in solrconfig.xml
   (sorry, which one escapes me now) that increases the weight of the %
   deleted documents when the merge policy decides which segments
   to merge. Upping this number will have the effect of more aggressively
   merging segments with a greater % of deleted docs. But these are
 already
   pretty heavily weighted for merging already...
  
  
   Best,
   Erick
  
  
   On Sat, Feb 22, 2014 at 1:23 PM, KNitin nitin.t...@gmail.com wrote:
  
Hi
   
  I have the following questions
   
   
   1. I have a job that runs for 3-4 hours continuously committing
 data
   to
   a collection with auto commit of 30 seconds. Does it mean that
 every
   30
   seconds I would get a new solr segment ?
   2. My current segment merge policy is set to 10. Will merger
 always
   continue running in the background to reduce the segments ? Is
  there a
way
   to see metrics regarding segment merging from solr (mbeans or any
   other
   way)?
   3. A few of my collections are very large with around 24-28
 segments
   per
   shard and around 16 shards. Is it bad to have this many segments
  for a
   shard for a collection? Is it a good practice to optimize the
 index
   very
   often or just rely on segment merges alone?
   
   
   
Thanks for the help in advance
Nitin
   
  
 

Re: Solr Segments, Segment Merges,Optimize

[KNitin]

I should also mention that apart from committing, the pipeline also does a
bunch of deletes for stale documents (based on a custom version field). The
# of deletes can be very significant causing the % of deleted documents to
be easily 40-50% of the index itself

Thanks
KNitin


On Sun, Feb 23, 2014 at 12:02 AM, KNitin nitin.t...@gmail.com wrote:

 Commit Parameters: Server does an auto commit every 30 seconds with
 open_searcher=false. The pipeline does a hard commit only at the very end
 of its run

 The high CPU issue I am seeing is only during the reads and not during the
 writes. Right now I see a direct corelation between latencies and # of
 segments atleast for a few large collections. Will post back if the theory
 is invalidated

 Thanks
 - Nitin


 On Sat, Feb 22, 2014 at 10:01 PM, Erick Erickson 
 erickerick...@gmail.comwrote:

 Well, it's always possible. I wouldn't expect the search time/CPU
 utilization to increase with # segments, within reasonable limits.
 At some point, the important parts of the index get read into memory
 and the number of segments is pretty irrelevant. You do mention
 that you have a heavy ingestion pipeline, which leads me to wonder
 whether you're committing too often, what are your commit
 parameters?

 For % deleted docs, I'm really talking about deletedDocs/numDocs.

 I suppose the interesting question is whether the CPU utilization you're
 seeing is _always_ correlated with # segments or are you seeing certain
 machines always having the high CPU utilization. I suppose you could
 issue a commit and see what difference that made.

 I rather doubt that the # of segments is the underlying issue, but that's
 nothing but a SWAG...

 Best,
 Erick




 On Sat, Feb 22, 2014 at 6:16 PM, KNitin nitin.t...@gmail.com wrote:

  Thanks, Erick.
 
  *2 There are, but you'll have to dig. *
 
  Any pointers on where to get started?
 
 
 
  *3 Well, I'd ask a counter-question. Are you seeing
  unacceptableperformance? If not, why worry? :)*
 
  When you mean % do you refer to deleted_docs/NumDocs or
  deleted_docs/Max_docs ? To answer your question, yes i see some of our
  shards taking 3x more time and 3x more cpu than other shards for the
 same
  queries and same number of hits (all shards have exact same number of
 docs
  but i see a few shards having more deleted documents than the rest).
 
  My understanding is that the Search time /CPU would increase with # of
  segments ?  The core of my issue is that few nodes are running with
  extremely high CPU (90+)  and rest are running under 30% CPU and the
 only
  difference between both is the  # of segments in the shards on the
  machines. The nodes running hot have shards with 30 segments and the
 ones
  running with lesser CPU contain 20 segments and much lesser deleted
  documents.
 
  Is it possible that a difference of 10 segments could impact CPU /Search
  time?
 
  Thanks
  - Nitin
 
 
  On Sat, Feb 22, 2014 at 4:36 PM, Erick Erickson 
 erickerick...@gmail.com
  wrote:
 
   1 It Depends. Soft commits will not add a new segment. Hard commits
   with openSearcher=true or false _will_ create a new segment.
   2 There are, but you'll have to dig.
   3 Well, I'd ask a counter-question. Are you seeing unacceptable
   performance? If not, why worry? :)
  
   A better answer is that 24-28 segments is not at all unusual.
  
   By and large, don't bother with optimize/force merge. What I would do
 is
   look at the admin screen and note the percentage of deleted documents.
   If it's above some arbitrary number (I typically use 15-20%) and
 _stays_
   there, consider optimizing.
  
   However! There is a parameter you can explicitly set in solrconfig.xml
   (sorry, which one escapes me now) that increases the weight of the %
   deleted documents when the merge policy decides which segments
   to merge. Upping this number will have the effect of more aggressively
   merging segments with a greater % of deleted docs. But these are
 already
   pretty heavily weighted for merging already...
  
  
   Best,
   Erick
  
  
   On Sat, Feb 22, 2014 at 1:23 PM, KNitin nitin.t...@gmail.com wrote:
  
Hi
   
  I have the following questions
   
   
   1. I have a job that runs for 3-4 hours continuously committing
 data
   to
   a collection with auto commit of 30 seconds. Does it mean that
 every
   30
   seconds I would get a new solr segment ?
   2. My current segment merge policy is set to 10. Will merger
 always
   continue running in the background to reduce the segments ? Is
  there a
way
   to see metrics regarding segment merging from solr (mbeans or any
   other
   way)?
   3. A few of my collections are very large with around 24-28
 segments
   per
   shard and around 16 shards. Is it bad to have this many segments
  for a
   shard for a collection? Is it a good practice to optimize the
 index
   very
   often or just rely on segment merges alone?
   
   
   
Thanks

Re: Tweaking Solr Query Result Cache

[KNitin]

Thanks, Erick. Turned off the query cache and sharded more aggressively
helped bring down the latencies


On Thu, Feb 20, 2014 at 5:07 PM, Erick Erickson erickerick...@gmail.comwrote:

 What you _do_ want to do is add replicas so you distribute the CPU
 load across a bunch of machines.

 The QueryResultCache isn't very useful unless you have multiple queries
 that
 1 reference the _exact_ same query, q, fq, sorting and all
 2 don't page very far.

 This cache really only holds the document (internal Lucene) IDs for a
 window
 of hits. So say your window (configured in solrconfig.xml) is set to 50.
 For each
 of the query keys, 50 IDs are stored. Next time that exact query comes in,
 and
 _assuming_ start+rows  50, you'll get the IDs from the cache and not much
 action occurs. The design intent here is to satisfy a few pages of results.

 If you mean by tail queries that there is very little repetition of
 queries, then
 why bother with a cache at all? If the hit ratio is going towards 0 it's
 not doing
 you enough good to matter.


 FWIW,
 Erick


 On Thu, Feb 20, 2014 at 1:58 PM, KNitin nitin.t...@gmail.com wrote:

  Hello
 
I have a 4 node cluster running Solr cloud 4.3.1. I have a few large
  collections sharded 8 ways across all the 4 nodes (with 2 shards per
 node).
  The size of the shard for the large collections is around 600-700Mb
  containing around 250K+ documents.
 
  Currently the size of the query cache is around 512. We have a few jobs
  that run tail queries on these collections. The hit ratio of the cache
  drops to 0 when running these queries and also at the same time CPU
 spikes.
  The latencies are in the order of seconds in the above case. I verified
 GC
  behavior is normal (not killing cpu)
 
  The following are my questions
 
 
 1. Is it a good practice to vary the Query Result Cache size based on
 the size of the collection (large collections have large cache)?
 2. If most of your queries are tail queries, what is a good way to
 make
 your cache usage effective (higher hits)
 3. If lets say all your queries miss the cache, it is an OK behavior
 if
 your CPU spikes (to 90+%)
 4. Is there a recommended shard size (# of doc, size ) to use. A few
 of
 my collections are 100-200 Mb and the large ones are in teh order of
  800-1Gb
 
  Thanks a lot in advance
  Nitin
 

Solr Segments, Segment Merges,Optimize

[KNitin]

Hi

  I have the following questions


   1. I have a job that runs for 3-4 hours continuously committing data to
   a collection with auto commit of 30 seconds. Does it mean that every 30
   seconds I would get a new solr segment ?
   2. My current segment merge policy is set to 10. Will merger always
   continue running in the background to reduce the segments ? Is there a way
   to see metrics regarding segment merging from solr (mbeans or any other
   way)?
   3. A few of my collections are very large with around 24-28 segments per
   shard and around 16 shards. Is it bad to have this many segments for a
   shard for a collection? Is it a good practice to optimize the index very
   often or just rely on segment merges alone?



Thanks for the help in advance
Nitin

Re: Solr Segments, Segment Merges,Optimize

[KNitin]

Thanks, Erick.

*2 There are, but you'll have to dig. *

Any pointers on where to get started?



*3 Well, I'd ask a counter-question. Are you seeing
unacceptableperformance? If not, why worry? :)*

When you mean % do you refer to deleted_docs/NumDocs or
deleted_docs/Max_docs ? To answer your question, yes i see some of our
shards taking 3x more time and 3x more cpu than other shards for the same
queries and same number of hits (all shards have exact same number of docs
but i see a few shards having more deleted documents than the rest).

My understanding is that the Search time /CPU would increase with # of
segments ?  The core of my issue is that few nodes are running with
extremely high CPU (90+)  and rest are running under 30% CPU and the only
difference between both is the  # of segments in the shards on the
machines. The nodes running hot have shards with 30 segments and the ones
running with lesser CPU contain 20 segments and much lesser deleted
documents.

Is it possible that a difference of 10 segments could impact CPU /Search
time?

Thanks
- Nitin


On Sat, Feb 22, 2014 at 4:36 PM, Erick Erickson erickerick...@gmail.comwrote:

 1 It Depends. Soft commits will not add a new segment. Hard commits
 with openSearcher=true or false _will_ create a new segment.
 2 There are, but you'll have to dig.
 3 Well, I'd ask a counter-question. Are you seeing unacceptable
 performance? If not, why worry? :)

 A better answer is that 24-28 segments is not at all unusual.

 By and large, don't bother with optimize/force merge. What I would do is
 look at the admin screen and note the percentage of deleted documents.
 If it's above some arbitrary number (I typically use 15-20%) and _stays_
 there, consider optimizing.

 However! There is a parameter you can explicitly set in solrconfig.xml
 (sorry, which one escapes me now) that increases the weight of the %
 deleted documents when the merge policy decides which segments
 to merge. Upping this number will have the effect of more aggressively
 merging segments with a greater % of deleted docs. But these are already
 pretty heavily weighted for merging already...


 Best,
 Erick


 On Sat, Feb 22, 2014 at 1:23 PM, KNitin nitin.t...@gmail.com wrote:

  Hi
 
I have the following questions
 
 
 1. I have a job that runs for 3-4 hours continuously committing data
 to
 a collection with auto commit of 30 seconds. Does it mean that every
 30
 seconds I would get a new solr segment ?
 2. My current segment merge policy is set to 10. Will merger always
 continue running in the background to reduce the segments ? Is there a
  way
 to see metrics regarding segment merging from solr (mbeans or any
 other
 way)?
 3. A few of my collections are very large with around 24-28 segments
 per
 shard and around 16 shards. Is it bad to have this many segments for a
 shard for a collection? Is it a good practice to optimize the index
 very
 often or just rely on segment merges alone?
 
 
 
  Thanks for the help in advance
  Nitin
 

CloudServer 4.2.1 and SolrCloud 4.3.1

[KNitin]

Hi

 I have a question on CloudServer client for solrcloud. How does
CloudServer route requests to solr?  Does it use round robin internally or
does it take into account any other parameter for the node (example how
many replicas it has, etc) ?


Thanks
Nitin

Re: CloudServer 4.2.1 and SolrCloud 4.3.1

[KNitin]

Thanks, Shawn.


On Thu, Feb 20, 2014 at 11:29 AM, Shawn Heisey s...@elyograg.org wrote:

 On 2/20/2014 12:09 PM, KNitin wrote:

   I have a question on CloudServer client for solrcloud. How does
 CloudServer route requests to solr?  Does it use round robin internally or
 does it take into account any other parameter for the node (example how
 many replicas it has, etc) ?


 Mixing SolrJ and Solr versions when you are using CloudSolrServer and
 SolrCloud is a bad idea right now.  SolrCloud is evolving *VERY* quickly.
  At some point in the future there will be more stability between versions,
 but right now, success is unlikely unless they are the same version.

 By default, CloudSolrServer in version 4.2.1 will send updates to shard
 leaders (in a round-robin fashion). For queries, it uses a full
 round-robin.  I *think* it will limit the round-robin to only nodes that
 are hosting that collection, but I'm not sure about that part.

 http://lucene.apache.org/solr/4_2_1/solr-solrj/org/apache/
 solr/client/solrj/impl/CloudSolrServer.html#CloudSolrServer%28java.lang.
 String,%20org.apache.solr.client.solrj.impl.LBHttpSolrServer,%20boolean%29

 Newer versions of CloudSolrServer (4.5.x if I remember correctly, and
 4.6.x for sure) can route update requests directly to the leader of the
 correct shard.  It does require the same version of Solr.

 Thanks,
 Shawn

Tweaking Solr Query Result Cache

[KNitin]

Hello

  I have a 4 node cluster running Solr cloud 4.3.1. I have a few large
collections sharded 8 ways across all the 4 nodes (with 2 shards per node).
The size of the shard for the large collections is around 600-700Mb
containing around 250K+ documents.

Currently the size of the query cache is around 512. We have a few jobs
that run tail queries on these collections. The hit ratio of the cache
drops to 0 when running these queries and also at the same time CPU spikes.
The latencies are in the order of seconds in the above case. I verified GC
behavior is normal (not killing cpu)

The following are my questions


   1. Is it a good practice to vary the Query Result Cache size based on
   the size of the collection (large collections have large cache)?
   2. If most of your queries are tail queries, what is a good way to make
   your cache usage effective (higher hits)
   3. If lets say all your queries miss the cache, it is an OK behavior if
   your CPU spikes (to 90+%)
   4. Is there a recommended shard size (# of doc, size ) to use. A few of
   my collections are 100-200 Mb and the large ones are in teh order of 800-1Gb

Thanks a lot in advance
Nitin