Re: Modify data-conf.xml on the runtime

[Shawn Heisey]

On 4/11/2018 9:21 AM, rameshkjes wrote:
> I am doing configuration of solr with the xml and pdf dataset, it works
> perfect. But, I want to modify few things: 
> Such as, we can see below, "baseDir" and "filePrefix" is being defined
> manually. I want this to be defined on the runtime.

The way I would do this is to have DIH grab that information from URL
parameters on the request to start the import.

So your URL would look like this:

?command=full-import=somePath=someOtherPath

And your config would look like this:


  

Re: Atomic update with condition

[Shawn Heisey]

On 4/11/2018 10:52 AM, SOLR4189 wrote:
> How can I change field value by specific condition in indexing? 
>
> Indexed Doc in SOLR: { id:1, foo:A }
> Indexing Doc into SOLR: { id:1, foo: B }
>
> foo is single value field.
>
> Let's say I want to replace value of foo from A to B, if A > B, else do
> nothing.

To the best of my knowledge, Solr can't do that without custom code.

You would need to have your indexing software ask Solr for the current
information and compare it to decide whether to send the indexing request.

The closest thing I can think of to this that Solr *does* do is the
increment capability in the Atomic Update feature.  And it's only
semi-related, not precisely what you're asking for.  I think that atomic
update is the only time that Solr looks at the existing record before
indexing a replacement.

https://lucene.apache.org/solr/guide/7_3/updating-parts-of-documents.html#atomic-updates

Thanks,
Shawn

Re: Solr OOM Crashes / JVM tuning advice

[Shawn Heisey]

On 4/11/2018 9:23 AM, Adam Harrison-Fuller wrote:
> In addition, here is the GC log leading up to the crash.
>
> https://www.dropbox.com/s/sq09d6hbss9b5ov/solr_gc_log_20180410_1009.zip?dl=0

I pulled that log into the http://gceasy.io website. This is a REALLY
nice way to look at GC logs.  I do still use gcviewer for in-depth
analysis, but for quick examination and locating trouble spots, that
website is top-notch.

Based on what I saw, I don't think you're running out of heap.  It's
more likely that it's too many file handles or limits on the number of
processes/threads that can be started.  Most operating systems limit
both of these to 1024 by default.

If you're on a non-windows OS (which I think is probably likely, given
that you have said Solr is crashing), change to the logs directory and
type this command:

grep OutOfMemoryError `ls -1tr solr.log*`

If it finds anything, it should give you the most recent error last.

Thanks,
Shawn

Re: Solr OOM Crashes / JVM tuning advice

[Kevin Risden]

I'm going to share how I've debugged a similar OOM crash and solving it had
nothing to do with increasing heap.

https://risdenk.github.io/2017/12/18/ambari-infra-solr-ranger.html

This is specifically for Apache Ranger and how to fix it but you can treat
it just like any application using Solr.

There were a few things that caused issues "out of the blue":

   - Document TTL
  - The documents getting deleted after some time would trigger OOM
  (due to caches taking up too much heap)
   - Extra query load
  - caches again taking up too much memory
   - Extra inserts
  - too many commits refreshing caches and again going OOM

Many of these can be reduced by using docvalues for fields that you
typically sort/filter on.

Kevin Risden

On Wed, Apr 11, 2018 at 6:01 PM, Deepak Goel  wrote:

> A few observations:
>
> 1. The Old Gen Heap on 9th April is about 6GB occupied which then runs up
> to 9+GB on 10th April (It steadily increases throughout the day)
> 2. The Old Gen GC is never able to reclaim any free memory
>
>
>
> Deepak
> "Please stop cruelty to Animals, help by becoming a Vegan"
> +91 73500 12833
> deic...@gmail.com
>
> Facebook: https://www.facebook.com/deicool
> LinkedIn: www.linkedin.com/in/deicool
>
> "Plant a Tree, Go Green"
>
> On Wed, Apr 11, 2018 at 8:53 PM, Adam Harrison-Fuller <
> aharrison-ful...@mintel.com> wrote:
>
> > In addition, here is the GC log leading up to the crash.
> >
> > https://www.dropbox.com/s/sq09d6hbss9b5ov/solr_gc_log_
> > 20180410_1009.zip?dl=0
> >
> > Thanks!
> >
> > Adam
> >
> > On 11 April 2018 at 16:18, Adam Harrison-Fuller <
> > aharrison-ful...@mintel.com
> > > wrote:
> >
> > > Thanks for the advice so far.
> > >
> > > The directoryFactory is set to ${solr.directoryFactory:solr.
> > NRTCachingDirectoryFactory}.
> > >
> > >
> > > The servers workload is predominantly queries with updates taking place
> > > once a day.  It seems the servers are more likely to go down whilst the
> > > servers are indexing but not exclusively so.
> > >
> > > I'm having issues locating the actual out of memory exception.  I can
> > tell
> > > that it has ran out of memory as its called the oom_killer script which
> > as
> > > left a log file in the logs directory.  I cannot find the actual
> > exception
> > > in the solr.log or our solr_gc.log, any suggestions?
> > >
> > > Cheers,
> > > Adam
> > >
> > >
> > > On 11 April 2018 at 15:49, Walter Underwood 
> > wrote:
> > >
> > >> For readability, I’d use -Xmx12G instead of
> -XX:MaxHeapSize=12884901888.
> > >> Also, I always use a start size the same as the max size, since
> servers
> > >> will eventually grow to the max size. So:
> > >>
> > >> -Xmx12G -Xms12G
> > >>
> > >> wunder
> > >> Walter Underwood
> > >> wun...@wunderwood.org
> > >> http://observer.wunderwood.org/  (my blog)
> > >>
> > >> > On Apr 11, 2018, at 6:29 AM, Sujay Bawaskar <
> sujaybawas...@gmail.com>
> > >> wrote:
> > >> >
> > >> > What is directory factory defined in solrconfig.xml? Your JVM heap
> > >> should
> > >> > be tuned up with respect to that.
> > >> > How solr is being use,  is it more updates and less query or less
> > >> updates
> > >> > more queries?
> > >> > What is OOM error? Is it frequent GC or Error 12?
> > >> >
> > >> > On Wed, Apr 11, 2018 at 6:05 PM, Adam Harrison-Fuller <
> > >> > aharrison-ful...@mintel.com> wrote:
> > >> >
> > >> >> Hey Jesus,
> > >> >>
> > >> >> Thanks for the suggestions.  The Solr nodes have 4 CPUs assigned to
> > >> them.
> > >> >>
> > >> >> Cheers!
> > >> >> Adam
> > >> >>
> > >> >> On 11 April 2018 at 11:22, Jesus Olivan 
> > >> wrote:
> > >> >>
> > >> >>> Hi Adam,
> > >> >>>
> > >> >>> IMHO you could try increasing heap to 20 Gb (with 46 Gb of
> physical
> > >> RAM,
> > >> >>> your JVM can afford more RAM without threading penalties due to
> > >> outside
> > >> >>> heap RAM lacks.
> > >> >>>
> > >> >>> Another good one would be to increase
> -XX:CMSInitiatingOccupancyFrac
> > >> tion
> > >> >>> =50
> > >> >>> to 75. I think that CMS collector works better when Old generation
> > >> space
> > >> >> is
> > >> >>> more populated.
> > >> >>>
> > >> >>> I usually use to set Survivor spaces to lesser size. If you want
> to
> > >> try
> > >> >>> SurvivorRatio to 6, i think performance would be improved.
> > >> >>>
> > >> >>> Another good practice for me would be to set an static NewSize
> > instead
> > >> >>> of -XX:NewRatio=3.
> > >> >>> You could try to set -XX:NewSize=7000m and -XX:MaxNewSize=7000Mb
> > (one
> > >> >> third
> > >> >>> of total heap space is recommended).
> > >> >>>
> > >> >>> Finally, my best results after a deep JVM I+D related to Solr,
> came
> > >> >>> removing ScavengeBeforeRemark flag and applying this new one: +
> > >> >>> ParGCCardsPerStrideChunk.
> > >> >>>
> > >> >>> However, It would be a good one to set ParallelGCThreads and
> > >> >>> *ConcGCThreads *to their optimal value, and we need you system CPU
> > >> number
> 

Re: Solr OOM Crashes / JVM tuning advice

[Deepak Goel]

A few observations:

1. The Old Gen Heap on 9th April is about 6GB occupied which then runs up
to 9+GB on 10th April (It steadily increases throughout the day)
2. The Old Gen GC is never able to reclaim any free memory



Deepak
"Please stop cruelty to Animals, help by becoming a Vegan"
+91 73500 12833
deic...@gmail.com

Facebook: https://www.facebook.com/deicool
LinkedIn: www.linkedin.com/in/deicool

"Plant a Tree, Go Green"

On Wed, Apr 11, 2018 at 8:53 PM, Adam Harrison-Fuller <
aharrison-ful...@mintel.com> wrote:

> In addition, here is the GC log leading up to the crash.
>
> https://www.dropbox.com/s/sq09d6hbss9b5ov/solr_gc_log_
> 20180410_1009.zip?dl=0
>
> Thanks!
>
> Adam
>
> On 11 April 2018 at 16:18, Adam Harrison-Fuller <
> aharrison-ful...@mintel.com
> > wrote:
>
> > Thanks for the advice so far.
> >
> > The directoryFactory is set to ${solr.directoryFactory:solr.
> NRTCachingDirectoryFactory}.
> >
> >
> > The servers workload is predominantly queries with updates taking place
> > once a day.  It seems the servers are more likely to go down whilst the
> > servers are indexing but not exclusively so.
> >
> > I'm having issues locating the actual out of memory exception.  I can
> tell
> > that it has ran out of memory as its called the oom_killer script which
> as
> > left a log file in the logs directory.  I cannot find the actual
> exception
> > in the solr.log or our solr_gc.log, any suggestions?
> >
> > Cheers,
> > Adam
> >
> >
> > On 11 April 2018 at 15:49, Walter Underwood 
> wrote:
> >
> >> For readability, I’d use -Xmx12G instead of -XX:MaxHeapSize=12884901888.
> >> Also, I always use a start size the same as the max size, since servers
> >> will eventually grow to the max size. So:
> >>
> >> -Xmx12G -Xms12G
> >>
> >> wunder
> >> Walter Underwood
> >> wun...@wunderwood.org
> >> http://observer.wunderwood.org/  (my blog)
> >>
> >> > On Apr 11, 2018, at 6:29 AM, Sujay Bawaskar 
> >> wrote:
> >> >
> >> > What is directory factory defined in solrconfig.xml? Your JVM heap
> >> should
> >> > be tuned up with respect to that.
> >> > How solr is being use,  is it more updates and less query or less
> >> updates
> >> > more queries?
> >> > What is OOM error? Is it frequent GC or Error 12?
> >> >
> >> > On Wed, Apr 11, 2018 at 6:05 PM, Adam Harrison-Fuller <
> >> > aharrison-ful...@mintel.com> wrote:
> >> >
> >> >> Hey Jesus,
> >> >>
> >> >> Thanks for the suggestions.  The Solr nodes have 4 CPUs assigned to
> >> them.
> >> >>
> >> >> Cheers!
> >> >> Adam
> >> >>
> >> >> On 11 April 2018 at 11:22, Jesus Olivan 
> >> wrote:
> >> >>
> >> >>> Hi Adam,
> >> >>>
> >> >>> IMHO you could try increasing heap to 20 Gb (with 46 Gb of physical
> >> RAM,
> >> >>> your JVM can afford more RAM without threading penalties due to
> >> outside
> >> >>> heap RAM lacks.
> >> >>>
> >> >>> Another good one would be to increase -XX:CMSInitiatingOccupancyFrac
> >> tion
> >> >>> =50
> >> >>> to 75. I think that CMS collector works better when Old generation
> >> space
> >> >> is
> >> >>> more populated.
> >> >>>
> >> >>> I usually use to set Survivor spaces to lesser size. If you want to
> >> try
> >> >>> SurvivorRatio to 6, i think performance would be improved.
> >> >>>
> >> >>> Another good practice for me would be to set an static NewSize
> instead
> >> >>> of -XX:NewRatio=3.
> >> >>> You could try to set -XX:NewSize=7000m and -XX:MaxNewSize=7000Mb
> (one
> >> >> third
> >> >>> of total heap space is recommended).
> >> >>>
> >> >>> Finally, my best results after a deep JVM I+D related to Solr, came
> >> >>> removing ScavengeBeforeRemark flag and applying this new one: +
> >> >>> ParGCCardsPerStrideChunk.
> >> >>>
> >> >>> However, It would be a good one to set ParallelGCThreads and
> >> >>> *ConcGCThreads *to their optimal value, and we need you system CPU
> >> number
> >> >>> to know it. Can you provide this data, please?
> >> >>>
> >> >>> Regards
> >> >>>
> >> >>>
> >> >>> 2018-04-11 12:01 GMT+02:00 Adam Harrison-Fuller <
> >> >>> aharrison-ful...@mintel.com
> >>  :
> >> >>>
> >>  Hey all,
> >> 
> >>  I was wondering if I could get some JVM/GC tuning advice to resolve
> >> an
> >>  issue that we are experiencing.
> >> 
> >>  Full disclaimer, I am in no way a JVM/Solr expert so any advice you
> >> can
> >>  render would be greatly appreciated.
> >> 
> >>  Our Solr cloud nodes are having issues throwing OOM exceptions
> under
> >> >>> load.
> >>  This issue has only started manifesting itself over the last few
> >> months
> >>  during which time the only change I can discern is an increase in
> >> index
> >>  size.  They are running Solr 5.5.2 on OpenJDK version "1.8.0_101".
> >> The
> >>  index is currently 58G and the server has 46G of physical RAM and
> >> runs
> >>  nothing other than the Solr node.
> >> 
> >>  The JVM is invoked with the following JVM options:
> >>  

Re: Using Solr to search website and external Oracle ServiceCloud

[Emir Arnautović]

Hi,
You have several options:
1. keep date as is and implement federated search logic with two queries - one 
to Solr and other to Oracle. PRO: no need to change data flow. CON: Results are 
not comparable when it comes to scores and you will probably need to present it 
as separate groups
2. import data from Oracle to Solr and align it so it has the same structure. 
PRO: ability to run a single query and have mixed results based on relevancy. 
With mixing content, you can have better stats and have more relevant results.  
CON: need to change data flow or sync Oracle-Solr
3. import data from Oracle to Solr but keep FAQ as a separate collection. PRO:  
a single search engine and no need to align knowledge base content with site 
content. CON: the same issues as with 1. and 2.

HTH,
Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/



> On 10 Apr 2018, at 21:00, jclaros  wrote:
> 
> Hello,
> I have a drupal website that uses SOLR.
> When a user searches our website, I would like to now return results from 2
> sources:
> (1) our website
> (2) our external Oracle ServiceCloud knowledge base
> 
> Does anyone have any suggestions on how to do this?
> 
> 
> 
> 
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Nested streaming expression with differing "on" fields

[Sarvothaman Madhavan]

Hello solr users,

Is it possible to perform streaming expression of the following type:


intersect(
search (collection_3, fl=“field_1,field_2",sort="field_2 
asc",qt="/export",q=*:* ),
intersect( 
search (collection_1, fl=“field_1,field_2",sort=“field_1 
asc",qt="/export",q=*:*),
search (collection_2, fl="field_1,field_2",sort=“field_1 
asc",qt="/export",q=*:*),
on=“field_1"
), 
on=“field_2"
)

Currently I get a “Invalid ReducerStream - substream comparator (sort) must be 
a superset of this stream's comparator.” 

Logically, collection_1 and collection_2 are linked via field_1 but the 
collection_3 links to everything else via field_2. The other point to note here 
is that this expression is build dynamically based on inputs to the program. 

Any workaround that you guys can point to will be useful. 

Thanks,
Sarvo

Re: Solr OOM Crashes / JVM tuning advice

[Joe Obernberger]

Just as a side note, when Solr goes OOM and kills itself, and if you're 
running HDFS, you are guaranteed to have write.lock files left over.  If 
you're running lots of shards/replicas, you may have many files that you 
need to go into HDFS and delete before restarting.


-Joe


On 4/11/2018 10:46 AM, Shawn Heisey wrote:

On 4/11/2018 4:01 AM, Adam Harrison-Fuller wrote:

I was wondering if I could get some JVM/GC tuning advice to resolve an
issue that we are experiencing.

Full disclaimer, I am in no way a JVM/Solr expert so any advice you can
render would be greatly appreciated.

Our Solr cloud nodes are having issues throwing OOM exceptions under load.
This issue has only started manifesting itself over the last few months
during which time the only change I can discern is an increase in index
size.  They are running Solr 5.5.2 on OpenJDK version "1.8.0_101".  The
index is currently 58G and the server has 46G of physical RAM and runs
nothing other than the Solr node.

The advice I see about tuning your garbage collection won't help you.
GC tuning can do absolutely nothing about OutOfMemoryError problems.
Better tuning might *delay* the OOM, but it can't prevent it.

You need to figure out exactly what resource is running out.  Hopefully
one of the solr logfiles will have the actual OutOfMemoryError exception
information.  It might not be the heap.

Once you know what resource is running out and causing the OOM, then we
can look deeper.

A side note:  The OOM is not *technically* causing a crash, even though
that might be the visible behavior.  When Solr is started on a
non-windows system with the included scripts, it runs with a parameter
that calls a script on OOM. That script *very intentionally* kills
Solr.  This is done because program operation when OOM hits is
unpredictable, and there's a decent chance that if it keeps running,
your index will get corrupted.  That could happen anyway, but with quick
action to kill the program, it's less likely.


The JVM is invoked with the following JVM options:
-XX:CMSInitiatingOccupancyFraction=50 -XX:CMSMaxAbortablePrecleanTime=6000
-XX:+CMSParallelRemarkEnabled -XX:+CMSScavengeBeforeRemark
-XX:ConcGCThreads=4 -XX:InitialHeapSize=12884901888 -XX:+ManagementServer
-XX:MaxHeapSize=12884901888 -XX:MaxTenuringThreshold=8
-XX:NewRatio=3 -XX:OldPLABSize=16
-XX:OnOutOfMemoryError=/opt/solr/bin/oom_solr.sh 3 /data/gnpd/solr/logs
-XX:ParallelGCThreads=4
-XX:+ParallelRefProcEnabled -XX:PretenureSizeThreshold=67108864
-XX:+PrintGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps
-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC
-XX:+PrintTenuringDistribution -XX:SurvivorRatio=4
-XX:TargetSurvivorRatio=90
-XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers
-XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC

Solr 5.5.2 includes GC tuning options in its default configuration.
Unless you'd like to switch to G1, you might want to let Solr's start
script handle that for you instead of overriding the options.  The
defaults are substantially similar to what you have defined.


I have imported the GC logs into GCViewer and attached a link to a
screenshot showing the lead up to a OOM crash.  Interestingly the young
generation space is almost empty before the repeated GC's and subsequent
crash.
https://imgur.com/a/Wtlez

Can you share the actual GC logfile?  You'll need to use a file sharing
site to do that, attachments almost never work on the mailing list.

The info in the summary to the right of the graph seems to support your
contention that there is plenty of heap, so the OutOfMemoryError is
probably not related to heap memory.  You're going to have to look at
your logfiles to see what the root cause is.

Thanks,
Shawn


---
This email has been checked for viruses by AVG.
http://www.avg.com

Re: Modify data-conf.xml on the runtime

[Alexandre Rafalovitch]

I believe you can use variable substitution like
${dih.request.paramname} and then pass 'paramname=value' in your HTTP
request.

Regards,
   Alex.

On 11 April 2018 at 11:25, rameshkjes  wrote:
> Hi,
>
> I am doing configuration of solr with the xml and pdf dataset, it works
> perfect. But, I want to modify few things:
> Such as, we can see below, "baseDir" and "filePrefix" is being defined
> manually. I want this to be defined on the runtime. Consider that I have GUI
> and user is specifying the directories for datasets. So, I have separate
> java file that does this job. But, I don't know how to use data of that
> class to configure "baseDir" and "filePrefix". Could anyone help me that how
> can i access that data from java class and fill in the data-conf.xml.
>
> These are few fields of my data-conf.xml file.
>
>  
> baseDir="C:/Users/kuma_rm/github/ML/dataset/XmlMetaData/"
> fileName=".*xml" rootEntity="false" datasource="null"
> 
> transformer="pathos.utils.transformer.FilePathTransformer">
>
>
>  filePrefix="C:/Users/kuma_rm/github/ML/dataset/PdfMetaData/"
> fileSuffix=".pdf" oldFileSuffix=".xml" srcColName="file"/>
>
>
>
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Atomic update with condition

[SOLR4189]

Hi all,

How can I change field value by specific condition in indexing? 

Indexed Doc in SOLR: { id:1, foo:A }
Indexing Doc into SOLR: { id:1, foo: B }

foo is single value field.

Let's say I want to replace value of foo from A to B, if A > B, else do
nothing.

Thank you.




--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Decision on Number of shards and collection

[SOLR4189]

I advise you to read the book Solr in Action. To answer your question you
need to take account server resources that you have (CPU, RAM and disk),
take account index size and take account average size single doc.



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Modify data-conf.xml on the runtime

[rameshkjes]

Hi, 

I am doing configuration of solr with the xml and pdf dataset, it works
perfect. But, I want to modify few things: 
Such as, we can see below, "baseDir" and "filePrefix" is being defined
manually. I want this to be defined on the runtime. Consider that I have GUI
and user is specifying the directories for datasets. So, I have separate
java file that does this job. But, I don't know how to use data of that
class to configure "baseDir" and "filePrefix". Could anyone help me that how
can i access that data from java class and fill in the data-conf.xml.

These are few fields of my data-conf.xml file. 

 

 
 



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Modify data-conf.xml on the runtime

[rameshkjes]

Hi, 

I am doing configuration of solr with the xml and pdf dataset, it works
perfect. But, I want to modify few things: 
Such as, we can see below, "baseDir" and "filePrefix" is being defined
manually. I want this to be defined on the runtime. Consider that I have GUI
and user is specifying the directories for datasets. So, I have separate
java file that does this job. But, I don't know how to use data of that
class to configure "baseDir" and "filePrefix". Could anyone help me that how
can i access that data from java class and fill in the data-conf.xml.

These are few fields of my data-conf.xml file. 

 

 
 



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Decision on Number of shards and collection

[Emir Arnautović]

Hi,
Only you can tell what are acceptable query latency (I can tell you ideal - it 
is 0 :)
Usually you start test with a single shard and start adding documents to it and 
measure query latency. When you start being close to max allowed latency, you 
have your shard size. Then you try to estimate number of documents in index 
now/future and you divide that number by shard size to get the number of 
shards. You then test to see what is the number of shards you can have per 
node. You multiple number of shards by number of replicas you plan to have and 
divide by number of shards per node to estimate number of nodes.
If you are not happy with numbers, you change some assumption, e.g. node type, 
and redo tests.

HTH,
Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/



> On 11 Apr 2018, at 15:30, neotorand  wrote:
> 
> Hi Emir,
> Thanks a lot for your reply.
> so when i design a solr eco system i should start with some rough guess on
> shards and increase the number of shards to make performance better.what is
> the accepted/ideal Response Time.There should be a trade off between
> Response time and the number of shards as data keeps growing.
> 
> I agree we split our index when response time increases.So what could be
> that response time threshold or query Latency?
> 
> Thanks again!
> 
> 
> Regards
> priyadarshi
> 
> 
> 
> 
> 
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: replication

[Erick Erickson]

bq: are you simply flagging the fact that we wouldn't direct the queries to A
v. B v. C since SolrCloud will make the decisions itself as to which part
of the distro gets hit for the operation

Yep. SolrCloud takes care of it all itself. I should also add that there are
about a zillion metrics now available in Solr that you can use to make the
best use of hardware, including things like CPU usage, I/O, GC etc. SolrCloud
doesn't _yet_ make use of these but will in future. The current software LB
does a pretty simple round-robin distribution.

Best,
Erick

On Wed, Apr 11, 2018 at 5:57 AM, John Blythe  wrote:
> thanks, erick. great info.
>
> although you can't (yet) direct queries to one or the other. So just making
>> them all NRT and forgetting about it is reasonable.
>
>
> are you simply flagging the fact that we wouldn't direct the queries to A
> v. B v. C since SolrCloud will make the decisions itself as to which part
> of the distro gets hit for the operation? if not, can you expound on this a
> bit more?
>
> The very nature of merging is such that you will _always_ get large merges
>> until you have 5G segments (by default)
>
>
> bummer
>
> Quite possible, but you have to route things yourself. But in that case
>> you're limited to one machine to handle all your NRT traffic. I skimmed
>> your post so don't know whether your NRT traffic load is high enough to
>> worry about.
>
>
> ok. i think we'll take a two-pronged approach. for the immediate purposes
> of trying to solve an issue we've begun encountering we will begin
> thoroughtesting the load between various operations in the master-slave
> setup we've set up. pending the results, we can roll forward w a temporary
> patch in which all end-user touch points route through the primary box for
> read/write while large scale operations/processing we do in the background
> will point to the ELB the slaves are sitting behind. we'll also begin
> setting up a simple solrcloud instance to toy with per your suggestion
> above. inb4 tons more questions on my part :)
>
> thanks!
>
> --
> John Blythe
>
> On Tue, Apr 10, 2018 at 11:14 AM, Erick Erickson 
> wrote:
>
>> bq: should we try to bite the solrcloud bullet and be done w it
>>
>> that's what I'd do. As of 7.0 there are different "flavors", TLOG,
>> PULL and NRT so that's also a possibility, although you can't (yet)
>> direct queries to one or the other. So just making them all NRT and
>> forgetting about it is reasonable.
>>
>> bq:  is there some more config work we could put in place to avoid ...
>> commit issue and the ultra large merge dangers
>>
>> No. The very nature of merging is such that you will _always_ get
>> large merges until you have 5G segments (by default). The max segment
>> size (outside "optimize/forceMerge/expungeDeletes" which you shouldn't
>> do) is 5G so the steady-state worst-case segment pull is limited to
>> that.
>>
>> bq: maybe for our initial need we use Master for writing and user
>> access in NRT events, but slaves for the heavier backend
>>
>> Quite possible, but you have to route things yourself. But in that
>> case you're limited to one machine to handle all your NRT traffic. I
>> skimmed your post so don't know whether your NRT traffic load is high
>> enough to worry about.
>>
>> The very first thing I'd do is set up a simple SolrCloud setup and
>> give it a spin. Unless your indexing load is quite heavy, the added
>> work the NRT replicas have in SolrCloud isn't a problem so worrying
>> about that is premature optimization unless you have a heavy load.
>>
>> Best,
>> Erick
>>
>> On Mon, Apr 9, 2018 at 4:36 PM, John Blythe  wrote:
>> > Thanks a bunch for the thorough reply, Shawn.
>> >
>> > Phew. We’d chosen to go w Master-slave replication instead of SolrCloud
>> per
>> > the sudden need we had encountered and the desire to avoid the nuances
>> and
>> > changes related to moving to SolrCloud. But so much for this being a more
>> > straightforward solution, huh?
>> >
>> > Few questions:
>> > - should we try to bite the solrcloud bullet and be done w it?
>> > - is there some more config work we could put in place to avoid the soft
>> > commit issue and the ultra large merge dangers, keeping the replications
>> > happening quickly?
>> > - maybe for our initial need we use Master for writing and user access in
>> > NRT events, but slaves for the heavier backend processing. Thoughts?
>> > - anyone do consulting on this that would be interested in chatting?
>> >
>> > Thanks again!
>> >
>> > On Mon, Apr 9, 2018 at 18:18 Shawn Heisey  wrote:
>> >
>> >> On 4/9/2018 12:15 PM, John Blythe wrote:
>> >> > we're starting to dive into master/slave replication architecture.
>> we'll
>> >> > have 1 master w 4 slaves behind it. our app is NRT. if user performs
>> an
>> >> > action in section A's data they may choose to jump to section B which
>> >> will
>> >> > be dependent on having the updates 

Re: Decision on Number of shards and collection

[Erick Erickson]

50M is a ballpark number I use as a place to _start_ getting a handle
on capacity. It's useful solely to answer the "is it bigger than a breadbox
and smaller than a house" question. It's totally meaningless without
testing.

Say I'm talking to a client and we have no data. Some are scared
that their 10M docs will require lots of hardware. Saying "I usualy
expect to see 50M docs on a node" gives them
some confidence that it's not going to require a massive hardware
investment and they can go forward with a PoC.

OTOH I have other clients saying "We have 100B documents" and
I have to say "You could be talking 200 nodes" which gives them
incentive to do a PoC to get a hard number.

I do recommend you keep adding (perhaps synthetic) docs to your
node until it tips over. Finding your installation falls over at, say, 50M
docs means you need to start taking action beforehand. OTOH if you
load 150M docs on it and still function OK you can breathe a lot
easier...

Best,
Erick



On Wed, Apr 11, 2018 at 8:55 AM, Abhi Basu <9000r...@gmail.com> wrote:
>  *The BKM I have read so far (trying to find source) says 50 million
> docs/shard performs well. I have found this in my recent tests as well. But
> of course it depends on index structure, etc.*
>
> On Wed, Apr 11, 2018 at 10:37 AM, Shawn Heisey  wrote:
>
>> On 4/11/2018 4:15 AM, neotorand wrote:
>> > I believe heterogeneous data can be indexed to same collection and i can
>> > have multiple shards for the index to be partitioned.So whats the need
>> of a
>> > second collection?. yes when collection size grows i should look for more
>> > collection.what exactly that size is? what KPI drives the decision of
>> having
>> > more collection?Any pointers or links for best practice.
>>
>> There are no hard rules.  Many factors affect these decisions.
>>
>> https://lucidworks.com/2012/07/23/sizing-hardware-in-the-
>> abstract-why-we-dont-have-a-definitive-answer/
>>
>> Creating multiple collections should be done when there is a logical or
>> business reason for keeping different sets of data separate from each
>> other.  If there's never any need for people to query all the data at
>> once, then it might make sense to use separate collections.  Or you
>> might want to put them together just for convenience, and use data in
>> the index to filter the results to only the information that the user is
>> allowed to access.
>>
>> > when should i go for multiple shards?
>> > yes when shard size grows.Right? whats the size and how do i benchmark.
>>
>> Some indexes function really well with 300 million documents or more per
>> shard.  Other indexes struggle with less than a million per shard.  It's
>> impossible to give you any specific number.  It depends on a bunch of
>> factors.
>>
>> If query rate is very high, then you want to keep the shard count low.
>> Using one shard might not be possible due to index size, but it should
>> be as low as you can make it.  You're also going to want to have a lot
>> of replicas to handle the load.
>>
>> If query rate is extremely low, then sharding the index can actually
>> *improve* performance, because there will be idle CPU capacity that can
>> be used for the subqueries.
>>
>> Thanks,
>> Shawn
>>
>>
>
>
> --
> Abhi Basu

Re: Decision on Number of shards and collection

[Abhi Basu]

 *The BKM I have read so far (trying to find source) says 50 million
docs/shard performs well. I have found this in my recent tests as well. But
of course it depends on index structure, etc.*

On Wed, Apr 11, 2018 at 10:37 AM, Shawn Heisey  wrote:

> On 4/11/2018 4:15 AM, neotorand wrote:
> > I believe heterogeneous data can be indexed to same collection and i can
> > have multiple shards for the index to be partitioned.So whats the need
> of a
> > second collection?. yes when collection size grows i should look for more
> > collection.what exactly that size is? what KPI drives the decision of
> having
> > more collection?Any pointers or links for best practice.
>
> There are no hard rules.  Many factors affect these decisions.
>
> https://lucidworks.com/2012/07/23/sizing-hardware-in-the-
> abstract-why-we-dont-have-a-definitive-answer/
>
> Creating multiple collections should be done when there is a logical or
> business reason for keeping different sets of data separate from each
> other.  If there's never any need for people to query all the data at
> once, then it might make sense to use separate collections.  Or you
> might want to put them together just for convenience, and use data in
> the index to filter the results to only the information that the user is
> allowed to access.
>
> > when should i go for multiple shards?
> > yes when shard size grows.Right? whats the size and how do i benchmark.
>
> Some indexes function really well with 300 million documents or more per
> shard.  Other indexes struggle with less than a million per shard.  It's
> impossible to give you any specific number.  It depends on a bunch of
> factors.
>
> If query rate is very high, then you want to keep the shard count low.
> Using one shard might not be possible due to index size, but it should
> be as low as you can make it.  You're also going to want to have a lot
> of replicas to handle the load.
>
> If query rate is extremely low, then sharding the index can actually
> *improve* performance, because there will be idle CPU capacity that can
> be used for the subqueries.
>
> Thanks,
> Shawn
>
>


-- 
Abhi Basu

Re: in-place updates

[Emir Arnautović]

Hi Hendrik,
Documentation clearly states conditions when in-place updates are possible: 
https://lucene.apache.org/solr/guide/6_6/updating-parts-of-documents.html#UpdatingPartsofDocuments-In-PlaceUpdates
 

The first one mentions “numeric docValues”.

HTH,
Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/



> On 11 Apr 2018, at 07:34, Hendrik Haddorp  wrote:
> 
> Hi,
> 
> in 
> http://lucene.472066.n3.nabble.com/In-Place-Updates-not-working-as-expected-tp4375621p4380035.html
>  some restrictions on the supported fields are given. I could however not 
> find if in-place updates are supported for are field types or if they only 
> work for say numeric fields.
> 
> thanks,
> Hendrik

Re: Decision on Number of shards and collection

[Shawn Heisey]

On 4/11/2018 4:15 AM, neotorand wrote:
> I believe heterogeneous data can be indexed to same collection and i can
> have multiple shards for the index to be partitioned.So whats the need of a
> second collection?. yes when collection size grows i should look for more
> collection.what exactly that size is? what KPI drives the decision of having
> more collection?Any pointers or links for best practice.

There are no hard rules.  Many factors affect these decisions.

https://lucidworks.com/2012/07/23/sizing-hardware-in-the-abstract-why-we-dont-have-a-definitive-answer/

Creating multiple collections should be done when there is a logical or
business reason for keeping different sets of data separate from each
other.  If there's never any need for people to query all the data at
once, then it might make sense to use separate collections.  Or you
might want to put them together just for convenience, and use data in
the index to filter the results to only the information that the user is
allowed to access.

> when should i go for multiple shards?
> yes when shard size grows.Right? whats the size and how do i benchmark.

Some indexes function really well with 300 million documents or more per
shard.  Other indexes struggle with less than a million per shard.  It's
impossible to give you any specific number.  It depends on a bunch of
factors.

If query rate is very high, then you want to keep the shard count low. 
Using one shard might not be possible due to index size, but it should
be as low as you can make it.  You're also going to want to have a lot
of replicas to handle the load.

If query rate is extremely low, then sharding the index can actually
*improve* performance, because there will be idle CPU capacity that can
be used for the subqueries.

Thanks,
Shawn

Re: Solr OOM Crashes / JVM tuning advice

[Walter Underwood]

One other note on the JVM options, even though those aren’t the cause of the 
problem.

Don’t run four GC threads when you have four processors. That can use 100% of 
CPU just doing GC.

With four processors, I’d run one thread.

wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/  (my blog)

> On Apr 11, 2018, at 7:49 AM, Walter Underwood  wrote:
> 
> For readability, I’d use -Xmx12G instead of -XX:MaxHeapSize=12884901888. 
> Also, I always use a start size the same as the max size, since servers will 
> eventually grow to the max size. So:
> 
> -Xmx12G -Xms12G
> 
> wunder
> Walter Underwood
> wun...@wunderwood.org
> http://observer.wunderwood.org/  (my blog)
> 
>> On Apr 11, 2018, at 6:29 AM, Sujay Bawaskar  wrote:
>> 
>> What is directory factory defined in solrconfig.xml? Your JVM heap should
>> be tuned up with respect to that.
>> How solr is being use,  is it more updates and less query or less updates
>> more queries?
>> What is OOM error? Is it frequent GC or Error 12?
>> 
>> On Wed, Apr 11, 2018 at 6:05 PM, Adam Harrison-Fuller <
>> aharrison-ful...@mintel.com> wrote:
>> 
>>> Hey Jesus,
>>> 
>>> Thanks for the suggestions.  The Solr nodes have 4 CPUs assigned to them.
>>> 
>>> Cheers!
>>> Adam
>>> 
>>> On 11 April 2018 at 11:22, Jesus Olivan  wrote:
>>> 
 Hi Adam,
 
 IMHO you could try increasing heap to 20 Gb (with 46 Gb of physical RAM,
 your JVM can afford more RAM without threading penalties due to outside
 heap RAM lacks.
 
 Another good one would be to increase -XX:CMSInitiatingOccupancyFraction
 =50
 to 75. I think that CMS collector works better when Old generation space
>>> is
 more populated.
 
 I usually use to set Survivor spaces to lesser size. If you want to try
 SurvivorRatio to 6, i think performance would be improved.
 
 Another good practice for me would be to set an static NewSize instead
 of -XX:NewRatio=3.
 You could try to set -XX:NewSize=7000m and -XX:MaxNewSize=7000Mb (one
>>> third
 of total heap space is recommended).
 
 Finally, my best results after a deep JVM I+D related to Solr, came
 removing ScavengeBeforeRemark flag and applying this new one: +
 ParGCCardsPerStrideChunk.
 
 However, It would be a good one to set ParallelGCThreads and
 *ConcGCThreads *to their optimal value, and we need you system CPU number
 to know it. Can you provide this data, please?
 
 Regards
 
 
 2018-04-11 12:01 GMT+02:00 Adam Harrison-Fuller <
 aharrison-ful...@mintel.com
> :
 
> Hey all,
> 
> I was wondering if I could get some JVM/GC tuning advice to resolve an
> issue that we are experiencing.
> 
> Full disclaimer, I am in no way a JVM/Solr expert so any advice you can
> render would be greatly appreciated.
> 
> Our Solr cloud nodes are having issues throwing OOM exceptions under
 load.
> This issue has only started manifesting itself over the last few months
> during which time the only change I can discern is an increase in index
> size.  They are running Solr 5.5.2 on OpenJDK version "1.8.0_101".  The
> index is currently 58G and the server has 46G of physical RAM and runs
> nothing other than the Solr node.
> 
> The JVM is invoked with the following JVM options:
> -XX:CMSInitiatingOccupancyFraction=50 -XX:CMSMaxAbortablePrecleanTime=
 6000
> -XX:+CMSParallelRemarkEnabled -XX:+CMSScavengeBeforeRemark
> -XX:ConcGCThreads=4 -XX:InitialHeapSize=12884901888
 -XX:+ManagementServer
> -XX:MaxHeapSize=12884901888 -XX:MaxTenuringThreshold=8
> -XX:NewRatio=3 -XX:OldPLABSize=16
> -XX:OnOutOfMemoryError=/opt/solr/bin/oom_solr.sh 3
> /data/gnpd/solr/logs
> -XX:ParallelGCThreads=4
> -XX:+ParallelRefProcEnabled -XX:PretenureSizeThreshold=67108864
> -XX:+PrintGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps
> -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC
> -XX:+PrintTenuringDistribution -XX:SurvivorRatio=4
> -XX:TargetSurvivorRatio=90
> -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers
> -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
> 
> These values were decided upon serveral years by a colleague based upon
> some suggestions from this mailing group with an index size ~25G.
> 
> I have imported the GC logs into GCViewer and attached a link to a
> screenshot showing the lead up to a OOM crash.  Interestingly the young
> generation space is almost empty before the repeated GC's and
>>> subsequent
> crash.
> https://imgur.com/a/Wtlez
> 
> I was considering slowly increasing the amount of heap available to the
 JVM
> slowly until the crashes, any other suggestions?  I'm looking at trying
 to
> get the nodes stable 

Re: Indexing fails with partially done

[Emir Arnautović]

Hi Neo,
My DIH knowledge is a bit rusty, but I think that in best case, depending on 
your queries you might be able to use delta update to “resume” indexing, but it 
is likely that you cannot do that.

Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/



> On 11 Apr 2018, at 15:34, neotorand  wrote:
> 
> Thanks Emir
> with context to DIH do we have any Resume mechanism?
> 
> Regards
> Neo
> 
> 
> 
> 
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Solr OOM Crashes / JVM tuning advice

[Adam Harrison-Fuller]

In addition, here is the GC log leading up to the crash.

https://www.dropbox.com/s/sq09d6hbss9b5ov/solr_gc_log_20180410_1009.zip?dl=0

Thanks!

Adam

On 11 April 2018 at 16:18, Adam Harrison-Fuller  wrote:

> Thanks for the advice so far.
>
> The directoryFactory is set to 
> ${solr.directoryFactory:solr.NRTCachingDirectoryFactory}.
>
>
> The servers workload is predominantly queries with updates taking place
> once a day.  It seems the servers are more likely to go down whilst the
> servers are indexing but not exclusively so.
>
> I'm having issues locating the actual out of memory exception.  I can tell
> that it has ran out of memory as its called the oom_killer script which as
> left a log file in the logs directory.  I cannot find the actual exception
> in the solr.log or our solr_gc.log, any suggestions?
>
> Cheers,
> Adam
>
>
> On 11 April 2018 at 15:49, Walter Underwood  wrote:
>
>> For readability, I’d use -Xmx12G instead of -XX:MaxHeapSize=12884901888.
>> Also, I always use a start size the same as the max size, since servers
>> will eventually grow to the max size. So:
>>
>> -Xmx12G -Xms12G
>>
>> wunder
>> Walter Underwood
>> wun...@wunderwood.org
>> http://observer.wunderwood.org/  (my blog)
>>
>> > On Apr 11, 2018, at 6:29 AM, Sujay Bawaskar 
>> wrote:
>> >
>> > What is directory factory defined in solrconfig.xml? Your JVM heap
>> should
>> > be tuned up with respect to that.
>> > How solr is being use,  is it more updates and less query or less
>> updates
>> > more queries?
>> > What is OOM error? Is it frequent GC or Error 12?
>> >
>> > On Wed, Apr 11, 2018 at 6:05 PM, Adam Harrison-Fuller <
>> > aharrison-ful...@mintel.com> wrote:
>> >
>> >> Hey Jesus,
>> >>
>> >> Thanks for the suggestions.  The Solr nodes have 4 CPUs assigned to
>> them.
>> >>
>> >> Cheers!
>> >> Adam
>> >>
>> >> On 11 April 2018 at 11:22, Jesus Olivan 
>> wrote:
>> >>
>> >>> Hi Adam,
>> >>>
>> >>> IMHO you could try increasing heap to 20 Gb (with 46 Gb of physical
>> RAM,
>> >>> your JVM can afford more RAM without threading penalties due to
>> outside
>> >>> heap RAM lacks.
>> >>>
>> >>> Another good one would be to increase -XX:CMSInitiatingOccupancyFrac
>> tion
>> >>> =50
>> >>> to 75. I think that CMS collector works better when Old generation
>> space
>> >> is
>> >>> more populated.
>> >>>
>> >>> I usually use to set Survivor spaces to lesser size. If you want to
>> try
>> >>> SurvivorRatio to 6, i think performance would be improved.
>> >>>
>> >>> Another good practice for me would be to set an static NewSize instead
>> >>> of -XX:NewRatio=3.
>> >>> You could try to set -XX:NewSize=7000m and -XX:MaxNewSize=7000Mb (one
>> >> third
>> >>> of total heap space is recommended).
>> >>>
>> >>> Finally, my best results after a deep JVM I+D related to Solr, came
>> >>> removing ScavengeBeforeRemark flag and applying this new one: +
>> >>> ParGCCardsPerStrideChunk.
>> >>>
>> >>> However, It would be a good one to set ParallelGCThreads and
>> >>> *ConcGCThreads *to their optimal value, and we need you system CPU
>> number
>> >>> to know it. Can you provide this data, please?
>> >>>
>> >>> Regards
>> >>>
>> >>>
>> >>> 2018-04-11 12:01 GMT+02:00 Adam Harrison-Fuller <
>> >>> aharrison-ful...@mintel.com
>>  :
>> >>>
>>  Hey all,
>> 
>>  I was wondering if I could get some JVM/GC tuning advice to resolve
>> an
>>  issue that we are experiencing.
>> 
>>  Full disclaimer, I am in no way a JVM/Solr expert so any advice you
>> can
>>  render would be greatly appreciated.
>> 
>>  Our Solr cloud nodes are having issues throwing OOM exceptions under
>> >>> load.
>>  This issue has only started manifesting itself over the last few
>> months
>>  during which time the only change I can discern is an increase in
>> index
>>  size.  They are running Solr 5.5.2 on OpenJDK version "1.8.0_101".
>> The
>>  index is currently 58G and the server has 46G of physical RAM and
>> runs
>>  nothing other than the Solr node.
>> 
>>  The JVM is invoked with the following JVM options:
>>  -XX:CMSInitiatingOccupancyFraction=50 -XX:CMSMaxAbortablePrecleanTim
>> e=
>> >>> 6000
>>  -XX:+CMSParallelRemarkEnabled -XX:+CMSScavengeBeforeRemark
>>  -XX:ConcGCThreads=4 -XX:InitialHeapSize=12884901888
>> >>> -XX:+ManagementServer
>>  -XX:MaxHeapSize=12884901888 -XX:MaxTenuringThreshold=8
>>  -XX:NewRatio=3 -XX:OldPLABSize=16
>>  -XX:OnOutOfMemoryError=/opt/solr/bin/oom_solr.sh 3
>>  /data/gnpd/solr/logs
>>  -XX:ParallelGCThreads=4
>>  -XX:+ParallelRefProcEnabled -XX:PretenureSizeThreshold=67108864
>>  -XX:+PrintGC -XX:+PrintGCApplicationStoppedTime
>> -XX:+PrintGCDateStamps
>>  -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC
>>  -XX:+PrintTenuringDistribution -XX:SurvivorRatio=4
>>  -XX:TargetSurvivorRatio=90
>>  

Re: Solr OOM Crashes / JVM tuning advice

[Adam Harrison-Fuller]

Thanks for the advice so far.

The directoryFactory is set to
${solr.directoryFactory:solr.NRTCachingDirectoryFactory}.

The servers workload is predominantly queries with updates taking place
once a day.  It seems the servers are more likely to go down whilst the
servers are indexing but not exclusively so.

I'm having issues locating the actual out of memory exception.  I can tell
that it has ran out of memory as its called the oom_killer script which as
left a log file in the logs directory.  I cannot find the actual exception
in the solr.log or our solr_gc.log, any suggestions?

Cheers,
Adam


On 11 April 2018 at 15:49, Walter Underwood  wrote:

> For readability, I’d use -Xmx12G instead of -XX:MaxHeapSize=12884901888.
> Also, I always use a start size the same as the max size, since servers
> will eventually grow to the max size. So:
>
> -Xmx12G -Xms12G
>
> wunder
> Walter Underwood
> wun...@wunderwood.org
> http://observer.wunderwood.org/  (my blog)
>
> > On Apr 11, 2018, at 6:29 AM, Sujay Bawaskar 
> wrote:
> >
> > What is directory factory defined in solrconfig.xml? Your JVM heap should
> > be tuned up with respect to that.
> > How solr is being use,  is it more updates and less query or less updates
> > more queries?
> > What is OOM error? Is it frequent GC or Error 12?
> >
> > On Wed, Apr 11, 2018 at 6:05 PM, Adam Harrison-Fuller <
> > aharrison-ful...@mintel.com> wrote:
> >
> >> Hey Jesus,
> >>
> >> Thanks for the suggestions.  The Solr nodes have 4 CPUs assigned to
> them.
> >>
> >> Cheers!
> >> Adam
> >>
> >> On 11 April 2018 at 11:22, Jesus Olivan  wrote:
> >>
> >>> Hi Adam,
> >>>
> >>> IMHO you could try increasing heap to 20 Gb (with 46 Gb of physical
> RAM,
> >>> your JVM can afford more RAM without threading penalties due to outside
> >>> heap RAM lacks.
> >>>
> >>> Another good one would be to increase -XX:
> CMSInitiatingOccupancyFraction
> >>> =50
> >>> to 75. I think that CMS collector works better when Old generation
> space
> >> is
> >>> more populated.
> >>>
> >>> I usually use to set Survivor spaces to lesser size. If you want to try
> >>> SurvivorRatio to 6, i think performance would be improved.
> >>>
> >>> Another good practice for me would be to set an static NewSize instead
> >>> of -XX:NewRatio=3.
> >>> You could try to set -XX:NewSize=7000m and -XX:MaxNewSize=7000Mb (one
> >> third
> >>> of total heap space is recommended).
> >>>
> >>> Finally, my best results after a deep JVM I+D related to Solr, came
> >>> removing ScavengeBeforeRemark flag and applying this new one: +
> >>> ParGCCardsPerStrideChunk.
> >>>
> >>> However, It would be a good one to set ParallelGCThreads and
> >>> *ConcGCThreads *to their optimal value, and we need you system CPU
> number
> >>> to know it. Can you provide this data, please?
> >>>
> >>> Regards
> >>>
> >>>
> >>> 2018-04-11 12:01 GMT+02:00 Adam Harrison-Fuller <
> >>> aharrison-ful...@mintel.com
>  :
> >>>
>  Hey all,
> 
>  I was wondering if I could get some JVM/GC tuning advice to resolve an
>  issue that we are experiencing.
> 
>  Full disclaimer, I am in no way a JVM/Solr expert so any advice you
> can
>  render would be greatly appreciated.
> 
>  Our Solr cloud nodes are having issues throwing OOM exceptions under
> >>> load.
>  This issue has only started manifesting itself over the last few
> months
>  during which time the only change I can discern is an increase in
> index
>  size.  They are running Solr 5.5.2 on OpenJDK version "1.8.0_101".
> The
>  index is currently 58G and the server has 46G of physical RAM and runs
>  nothing other than the Solr node.
> 
>  The JVM is invoked with the following JVM options:
>  -XX:CMSInitiatingOccupancyFraction=50 -XX:
> CMSMaxAbortablePrecleanTime=
> >>> 6000
>  -XX:+CMSParallelRemarkEnabled -XX:+CMSScavengeBeforeRemark
>  -XX:ConcGCThreads=4 -XX:InitialHeapSize=12884901888
> >>> -XX:+ManagementServer
>  -XX:MaxHeapSize=12884901888 -XX:MaxTenuringThreshold=8
>  -XX:NewRatio=3 -XX:OldPLABSize=16
>  -XX:OnOutOfMemoryError=/opt/solr/bin/oom_solr.sh 3
>  /data/gnpd/solr/logs
>  -XX:ParallelGCThreads=4
>  -XX:+ParallelRefProcEnabled -XX:PretenureSizeThreshold=67108864
>  -XX:+PrintGC -XX:+PrintGCApplicationStoppedTime
> -XX:+PrintGCDateStamps
>  -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC
>  -XX:+PrintTenuringDistribution -XX:SurvivorRatio=4
>  -XX:TargetSurvivorRatio=90
>  -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers
>  -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
> 
>  These values were decided upon serveral years by a colleague based
> upon
>  some suggestions from this mailing group with an index size ~25G.
> 
>  I have imported the GC logs into GCViewer and attached a link to a
>  screenshot showing the lead up 

Re: Confusing error when creating a new core with TLS, service enabled

[Shawn Heisey]

On 4/11/2018 8:29 AM, Christopher Schultz wrote:
>> Unless you run Solr in cloud mode (which means using zookeeper), the
>> server cannot create the core directories itself. When running in
>> standalone mode, the core directory is created by the bin/solr program
>> doing the "create" -- which was running as root.
> That is ... surprising.[1]

I'm aware of that.  I know it as "the principle of least surprise". :) 
And I'm in complete agreement that we need to do better.

Given your role in Tomcat, I'm sure you know all about technical debt,
and how much of a struggle it is to pay that debt in order to make
improvements.  Decisions made years ago can cripple current efforts.

> It also means that one cannot remote-admin a Solr server. :(

If you take the plunge into SolrCloud, then you can.

> The way the installer and server work together is very unfortunate.
> bin/solr knows the euid of the server and, if running under root/sudo
> could easily mkdir/chown without crapping itself. Having installed a
> "service" using the Solr installer practically requires you to run
> bin/solr using sudo, and then it doesn't work. Is there a JIRA ticket
> already in existence where I can leave a comment?

I think this issue might be relevant for you:

https://issues.apache.org/jira/browse/SOLR-9590

I created it a while back.  Some of what I came up with will probably
change with more understanding, but the general notions are still about
the same.

All user switching is done in the init script, not in bin/solr.  We
never know whether bin/solr is being called by a user directly or as
part of service startup.  It's the init script that knows the user, not
bin/solr.  That probably needs to change.  I don't know if that would
happen as part of SOLR-9590 or in its own issue.

Thanks,
Shawn

Re: Indexing fails with partially done

[Shawn Heisey]

On 4/11/2018 6:46 AM, neotorand wrote:
> with Solrcloud What happens if indexing is partially completed and ensemble 
> goes down.What are the ways to Resume.In one of the scenario i am using 3 ZK 
> Node in ensemble.Lets say i am indexing 5 million data and i have partially 
> indexed the data and ZK ensemble goes down. What should be the best approach 
> for handling such scenario 

If you lose quorum on your ZK ensemble, SolrCloud will go read-only, and
indexing will not be possible.  Solr will remember the last clusterstate
that the ZK database held and will attempt to keep serving queries using
that information.  Whether or not that succeeds will depend on whether
your Solr servers stay operational.

If you restart a Solr server while ZK has no quorum, then I'm not sure
what happens.  It probably doesn't start correctly and may need another
restart after ZK is restored.

Thanks,
Shawn

Re: Solr OOM Crashes / JVM tuning advice

[Walter Underwood]

For readability, I’d use -Xmx12G instead of -XX:MaxHeapSize=12884901888. Also, 
I always use a start size the same as the max size, since servers will 
eventually grow to the max size. So:

-Xmx12G -Xms12G

wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/  (my blog)

> On Apr 11, 2018, at 6:29 AM, Sujay Bawaskar  wrote:
> 
> What is directory factory defined in solrconfig.xml? Your JVM heap should
> be tuned up with respect to that.
> How solr is being use,  is it more updates and less query or less updates
> more queries?
> What is OOM error? Is it frequent GC or Error 12?
> 
> On Wed, Apr 11, 2018 at 6:05 PM, Adam Harrison-Fuller <
> aharrison-ful...@mintel.com> wrote:
> 
>> Hey Jesus,
>> 
>> Thanks for the suggestions.  The Solr nodes have 4 CPUs assigned to them.
>> 
>> Cheers!
>> Adam
>> 
>> On 11 April 2018 at 11:22, Jesus Olivan  wrote:
>> 
>>> Hi Adam,
>>> 
>>> IMHO you could try increasing heap to 20 Gb (with 46 Gb of physical RAM,
>>> your JVM can afford more RAM without threading penalties due to outside
>>> heap RAM lacks.
>>> 
>>> Another good one would be to increase -XX:CMSInitiatingOccupancyFraction
>>> =50
>>> to 75. I think that CMS collector works better when Old generation space
>> is
>>> more populated.
>>> 
>>> I usually use to set Survivor spaces to lesser size. If you want to try
>>> SurvivorRatio to 6, i think performance would be improved.
>>> 
>>> Another good practice for me would be to set an static NewSize instead
>>> of -XX:NewRatio=3.
>>> You could try to set -XX:NewSize=7000m and -XX:MaxNewSize=7000Mb (one
>> third
>>> of total heap space is recommended).
>>> 
>>> Finally, my best results after a deep JVM I+D related to Solr, came
>>> removing ScavengeBeforeRemark flag and applying this new one: +
>>> ParGCCardsPerStrideChunk.
>>> 
>>> However, It would be a good one to set ParallelGCThreads and
>>> *ConcGCThreads *to their optimal value, and we need you system CPU number
>>> to know it. Can you provide this data, please?
>>> 
>>> Regards
>>> 
>>> 
>>> 2018-04-11 12:01 GMT+02:00 Adam Harrison-Fuller <
>>> aharrison-ful...@mintel.com
 :
>>> 
 Hey all,
 
 I was wondering if I could get some JVM/GC tuning advice to resolve an
 issue that we are experiencing.
 
 Full disclaimer, I am in no way a JVM/Solr expert so any advice you can
 render would be greatly appreciated.
 
 Our Solr cloud nodes are having issues throwing OOM exceptions under
>>> load.
 This issue has only started manifesting itself over the last few months
 during which time the only change I can discern is an increase in index
 size.  They are running Solr 5.5.2 on OpenJDK version "1.8.0_101".  The
 index is currently 58G and the server has 46G of physical RAM and runs
 nothing other than the Solr node.
 
 The JVM is invoked with the following JVM options:
 -XX:CMSInitiatingOccupancyFraction=50 -XX:CMSMaxAbortablePrecleanTime=
>>> 6000
 -XX:+CMSParallelRemarkEnabled -XX:+CMSScavengeBeforeRemark
 -XX:ConcGCThreads=4 -XX:InitialHeapSize=12884901888
>>> -XX:+ManagementServer
 -XX:MaxHeapSize=12884901888 -XX:MaxTenuringThreshold=8
 -XX:NewRatio=3 -XX:OldPLABSize=16
 -XX:OnOutOfMemoryError=/opt/solr/bin/oom_solr.sh 3
 /data/gnpd/solr/logs
 -XX:ParallelGCThreads=4
 -XX:+ParallelRefProcEnabled -XX:PretenureSizeThreshold=67108864
 -XX:+PrintGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps
 -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC
 -XX:+PrintTenuringDistribution -XX:SurvivorRatio=4
 -XX:TargetSurvivorRatio=90
 -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers
 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
 
 These values were decided upon serveral years by a colleague based upon
 some suggestions from this mailing group with an index size ~25G.
 
 I have imported the GC logs into GCViewer and attached a link to a
 screenshot showing the lead up to a OOM crash.  Interestingly the young
 generation space is almost empty before the repeated GC's and
>> subsequent
 crash.
 https://imgur.com/a/Wtlez
 
 I was considering slowly increasing the amount of heap available to the
>>> JVM
 slowly until the crashes, any other suggestions?  I'm looking at trying
>>> to
 get the nodes stable without having issues with the GC taking forever
>> to
 run.
 
 Additional information can be provided on request.
 
 Cheers!
 Adam
 
 --
 
 Mintel Group Ltd | 11 Pilgrim Street | London | EC4V 6RN
 Registered in
 England: Number 1475918. | VAT Number: GB 232 9342 72
 
 Contact details for
 our other offices can be found at http://www.mintel.com/office-
>> locations
 .
 
 This email and any attachments
 may 

Re: Solr OOM Crashes / JVM tuning advice

[Shawn Heisey]

On 4/11/2018 4:01 AM, Adam Harrison-Fuller wrote:
> I was wondering if I could get some JVM/GC tuning advice to resolve an
> issue that we are experiencing.
>
> Full disclaimer, I am in no way a JVM/Solr expert so any advice you can
> render would be greatly appreciated.
>
> Our Solr cloud nodes are having issues throwing OOM exceptions under load.
> This issue has only started manifesting itself over the last few months
> during which time the only change I can discern is an increase in index
> size.  They are running Solr 5.5.2 on OpenJDK version "1.8.0_101".  The
> index is currently 58G and the server has 46G of physical RAM and runs
> nothing other than the Solr node.

The advice I see about tuning your garbage collection won't help you. 
GC tuning can do absolutely nothing about OutOfMemoryError problems. 
Better tuning might *delay* the OOM, but it can't prevent it.

You need to figure out exactly what resource is running out.  Hopefully
one of the solr logfiles will have the actual OutOfMemoryError exception
information.  It might not be the heap.

Once you know what resource is running out and causing the OOM, then we
can look deeper.

A side note:  The OOM is not *technically* causing a crash, even though
that might be the visible behavior.  When Solr is started on a
non-windows system with the included scripts, it runs with a parameter
that calls a script on OOM. That script *very intentionally* kills
Solr.  This is done because program operation when OOM hits is
unpredictable, and there's a decent chance that if it keeps running,
your index will get corrupted.  That could happen anyway, but with quick
action to kill the program, it's less likely.

> The JVM is invoked with the following JVM options:
> -XX:CMSInitiatingOccupancyFraction=50 -XX:CMSMaxAbortablePrecleanTime=6000
> -XX:+CMSParallelRemarkEnabled -XX:+CMSScavengeBeforeRemark
> -XX:ConcGCThreads=4 -XX:InitialHeapSize=12884901888 -XX:+ManagementServer
> -XX:MaxHeapSize=12884901888 -XX:MaxTenuringThreshold=8
> -XX:NewRatio=3 -XX:OldPLABSize=16
> -XX:OnOutOfMemoryError=/opt/solr/bin/oom_solr.sh 3 /data/gnpd/solr/logs
> -XX:ParallelGCThreads=4
> -XX:+ParallelRefProcEnabled -XX:PretenureSizeThreshold=67108864
> -XX:+PrintGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps
> -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC
> -XX:+PrintTenuringDistribution -XX:SurvivorRatio=4
> -XX:TargetSurvivorRatio=90
> -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers
> -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC

Solr 5.5.2 includes GC tuning options in its default configuration. 
Unless you'd like to switch to G1, you might want to let Solr's start
script handle that for you instead of overriding the options.  The
defaults are substantially similar to what you have defined.

> I have imported the GC logs into GCViewer and attached a link to a
> screenshot showing the lead up to a OOM crash.  Interestingly the young
> generation space is almost empty before the repeated GC's and subsequent
> crash.
> https://imgur.com/a/Wtlez

Can you share the actual GC logfile?  You'll need to use a file sharing
site to do that, attachments almost never work on the mailing list.

The info in the summary to the right of the graph seems to support your
contention that there is plenty of heap, so the OutOfMemoryError is
probably not related to heap memory.  You're going to have to look at
your logfiles to see what the root cause is.

Thanks,
Shawn

RealTimeGetComponent performance

[Blackknight]

Hello guys!
I want to write an update processor which will use an old doc to compare
some fields with a new doc.
I saw that in solr code guys used RealTimeGetComponent for getting old doc
by index ID. How much it will influent on solr performance? In my case I
allways add docs with atomic upates and I know that DistribUpdateProcessor
uses RealTimeGetComponent too, may I send the finded doc in
DistribUpdateProcessor between other update processors? Or
RealTimeGetComponent is doing his work fine and it will not heart the solr
performance?
In my case I will use RealTimeGetComponent 2 times (ub DistribU.P. and on my
new U.P.)



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Confusing error when creating a new core with TLS, service enabled

[Christopher Schultz]

Shawn,

On 4/10/18 10:16 AM, Shawn Heisey wrote:
> On 4/10/2018 7:32 AM, Christopher Schultz wrote:
>>> What happened is that the new core directory was created as root,
>>> owned by root.
>> Was it? If my server is running as solr, how can it create directories
>> as root?
> 
> Unless you run Solr in cloud mode (which means using zookeeper), the
> server cannot create the core directories itself. When running in
> standalone mode, the core directory is created by the bin/solr program
> doing the "create" -- which was running as root.

That is ... surprising.[1]

> I know that because
> you needed the "-force" option.  So the core directory and its "conf"
> subdirectory (with the config) are created by the script, then Solr is
> asked (using the CoreAdmin API via http) to add that core.  It can't,
> because the new directory was created by root, and Solr can't write the
> core.properties file that defines the core for Solr.

Okay, then that makes sense. I'll try running bin/solr as "solr" via
sudo instead of merely as root. I was under the mistaken impression that
the server kept its own files in order.

It also means that one cannot remote-admin a Solr server. :(

> When running Solr in cloud mode, the configs are in zookeeper, so the
> create command on the script doesn't have to make the core directory in
> order for Solr to find the configuration.  It can simply upload the
> config to zookeeper and then tell Solr to create the collection, and
> Solr will do so, locating the configuration in ZooKeeper.

Good to know, though I'm not at the stage where I'm using ZK.

> You might be wondering why Solr can't create the core directories itself
> using the CoreAdmin API except in cloud mode.  This is because the
> CoreAdmin API is *OLD* and its functionality has not really changed
> since it was created.  Historically, it was only designed to add a core
> that had already been created.

*snapping sounds from inside brain*

> We probably need to "fix" this ... but
> it has never been a priority.  There are bigger problems and features to
> work on.  Cloud mode is much newer, and although the Collections API
> does utilize the CoreAdmin API behind the scenes, the user typically
> doesn't use CoreAdmin directly in cloud mode.
> 
>> The client may be running as root, but the server is running as 'solr'.
>> And the error occurs on the server, not the client. So, what's really
>> going on, here?
> 
> I hope I've explained that clearly above.

You have. Running bin/solr as user 'solr' was able to create the core.

The way the installer and server work together is very unfortunate.
bin/solr knows the euid of the server and, if running under root/sudo
could easily mkdir/chown without crapping itself. Having installed a
"service" using the Solr installer practically requires you to run
bin/solr using sudo, and then it doesn't work. Is there a JIRA ticket
already in existence where I can leave a comment?

Thanks,
-chris

[1] https://en.wikipedia.org/wiki/Principle_of_least_astonishment

Re: Indexing fails with partially done

[neotorand]

Thanks Emir
with context to DIH do we have any Resume mechanism?

Regards
Neo




--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Decision on Number of shards and collection

[neotorand]

Hi Emir,
Thanks a lot for your reply.
so when i design a solr eco system i should start with some rough guess on
shards and increase the number of shards to make performance better.what is
the accepted/ideal Response Time.There should be a trade off between
Response time and the number of shards as data keeps growing.

I agree we split our index when response time increases.So what could be
that response time threshold or query Latency?

Thanks again!


Regards
priyadarshi





--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Solr OOM Crashes / JVM tuning advice

[Sujay Bawaskar]

What is directory factory defined in solrconfig.xml? Your JVM heap should
be tuned up with respect to that.
How solr is being use,  is it more updates and less query or less updates
more queries?
What is OOM error? Is it frequent GC or Error 12?

On Wed, Apr 11, 2018 at 6:05 PM, Adam Harrison-Fuller <
aharrison-ful...@mintel.com> wrote:

> Hey Jesus,
>
> Thanks for the suggestions.  The Solr nodes have 4 CPUs assigned to them.
>
> Cheers!
> Adam
>
> On 11 April 2018 at 11:22, Jesus Olivan  wrote:
>
> > Hi Adam,
> >
> > IMHO you could try increasing heap to 20 Gb (with 46 Gb of physical RAM,
> > your JVM can afford more RAM without threading penalties due to outside
> > heap RAM lacks.
> >
> > Another good one would be to increase -XX:CMSInitiatingOccupancyFraction
> > =50
> > to 75. I think that CMS collector works better when Old generation space
> is
> > more populated.
> >
> > I usually use to set Survivor spaces to lesser size. If you want to try
> > SurvivorRatio to 6, i think performance would be improved.
> >
> > Another good practice for me would be to set an static NewSize instead
> > of -XX:NewRatio=3.
> > You could try to set -XX:NewSize=7000m and -XX:MaxNewSize=7000Mb (one
> third
> > of total heap space is recommended).
> >
> > Finally, my best results after a deep JVM I+D related to Solr, came
> > removing ScavengeBeforeRemark flag and applying this new one: +
> > ParGCCardsPerStrideChunk.
> >
> > However, It would be a good one to set ParallelGCThreads and
> > *ConcGCThreads *to their optimal value, and we need you system CPU number
> > to know it. Can you provide this data, please?
> >
> > Regards
> >
> >
> > 2018-04-11 12:01 GMT+02:00 Adam Harrison-Fuller <
> > aharrison-ful...@mintel.com
> > >:
> >
> > > Hey all,
> > >
> > > I was wondering if I could get some JVM/GC tuning advice to resolve an
> > > issue that we are experiencing.
> > >
> > > Full disclaimer, I am in no way a JVM/Solr expert so any advice you can
> > > render would be greatly appreciated.
> > >
> > > Our Solr cloud nodes are having issues throwing OOM exceptions under
> > load.
> > > This issue has only started manifesting itself over the last few months
> > > during which time the only change I can discern is an increase in index
> > > size.  They are running Solr 5.5.2 on OpenJDK version "1.8.0_101".  The
> > > index is currently 58G and the server has 46G of physical RAM and runs
> > > nothing other than the Solr node.
> > >
> > > The JVM is invoked with the following JVM options:
> > > -XX:CMSInitiatingOccupancyFraction=50 -XX:CMSMaxAbortablePrecleanTime=
> > 6000
> > > -XX:+CMSParallelRemarkEnabled -XX:+CMSScavengeBeforeRemark
> > > -XX:ConcGCThreads=4 -XX:InitialHeapSize=12884901888
> > -XX:+ManagementServer
> > > -XX:MaxHeapSize=12884901888 -XX:MaxTenuringThreshold=8
> > > -XX:NewRatio=3 -XX:OldPLABSize=16
> > > -XX:OnOutOfMemoryError=/opt/solr/bin/oom_solr.sh 3
> > > /data/gnpd/solr/logs
> > > -XX:ParallelGCThreads=4
> > > -XX:+ParallelRefProcEnabled -XX:PretenureSizeThreshold=67108864
> > > -XX:+PrintGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps
> > > -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC
> > > -XX:+PrintTenuringDistribution -XX:SurvivorRatio=4
> > > -XX:TargetSurvivorRatio=90
> > > -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers
> > > -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
> > >
> > > These values were decided upon serveral years by a colleague based upon
> > > some suggestions from this mailing group with an index size ~25G.
> > >
> > > I have imported the GC logs into GCViewer and attached a link to a
> > > screenshot showing the lead up to a OOM crash.  Interestingly the young
> > > generation space is almost empty before the repeated GC's and
> subsequent
> > > crash.
> > > https://imgur.com/a/Wtlez
> > >
> > > I was considering slowly increasing the amount of heap available to the
> > JVM
> > > slowly until the crashes, any other suggestions?  I'm looking at trying
> > to
> > > get the nodes stable without having issues with the GC taking forever
> to
> > > run.
> > >
> > > Additional information can be provided on request.
> > >
> > > Cheers!
> > > Adam
> > >
> > > --
> > >
> > > Mintel Group Ltd | 11 Pilgrim Street | London | EC4V 6RN
> > > Registered in
> > > England: Number 1475918. | VAT Number: GB 232 9342 72
> > >
> > > Contact details for
> > > our other offices can be found at http://www.mintel.com/office-
> locations
> > > .
> > >
> > > This email and any attachments
> > > may include content that is confidential, privileged
> > > or otherwise
> > > protected under applicable law. Unauthorised disclosure, copying,
> > > distribution
> > > or use of the contents is prohibited and may be unlawful. If
> > > you have received this email in error,
> > > including without appropriate
> > > authorisation, then please reply to the sender about the error
> 

Re: Decision on Number of shards and collection

[Emir Arnautović]

Hi Neo,
Shard size determines query latency, so you split your index when queries 
become too slow. Distributed search comes with some overhead, so oversharding 
is not the way to go either. There is no hard rule what are the best numbers, 
but here  are some thought how to approach this: 
http://www.od-bits.com/2018/01/solrelasticsearch-capacity-planning.html 


HTH,
Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/



> On 11 Apr 2018, at 12:15, neotorand  wrote:
> 
> Hi Team
> First of all i take this opportunity to thank you all for creating a
> beautiful place where people can explore ,learn and debate.
> 
> I have been on my knees for couple of days to decide on this.
> 
> When i am creating a solr cloud eco system i need to decide on number of
> shards and collection.
> What are the best practices for taking this decisions.
> 
> I believe heterogeneous data can be indexed to same collection and i can
> have multiple shards for the index to be partitioned.So whats the need of a
> second collection?. yes when collection size grows i should look for more
> collection.what exactly that size is? what KPI drives the decision of having
> more collection?Any pointers or links for best practice.
> 
> when should i go for multiple shards?
> yes when shard size grows.Right? whats the size and how do i benchmark.
> 
> I am sorry for my question if its already asked but googled all the ecospace
> quora,stackoverflow,lucid
> 
> Regards
> Neo
> 
> 
> 
> 
> 
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Solr OOM Crashes / JVM tuning advice

[Emir Arnautović]

Hi Adam,
From Solr’s point of view, you should probably check your caches, mostly 
filterCache, fieldCache and fieldValueCache.

HTH,
Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/



> On 11 Apr 2018, at 14:35, Adam Harrison-Fuller  
> wrote:
> 
> Hey Jesus,
> 
> Thanks for the suggestions.  The Solr nodes have 4 CPUs assigned to them.
> 
> Cheers!
> Adam
> 
> On 11 April 2018 at 11:22, Jesus Olivan  wrote:
> 
>> Hi Adam,
>> 
>> IMHO you could try increasing heap to 20 Gb (with 46 Gb of physical RAM,
>> your JVM can afford more RAM without threading penalties due to outside
>> heap RAM lacks.
>> 
>> Another good one would be to increase -XX:CMSInitiatingOccupancyFraction
>> =50
>> to 75. I think that CMS collector works better when Old generation space is
>> more populated.
>> 
>> I usually use to set Survivor spaces to lesser size. If you want to try
>> SurvivorRatio to 6, i think performance would be improved.
>> 
>> Another good practice for me would be to set an static NewSize instead
>> of -XX:NewRatio=3.
>> You could try to set -XX:NewSize=7000m and -XX:MaxNewSize=7000Mb (one third
>> of total heap space is recommended).
>> 
>> Finally, my best results after a deep JVM I+D related to Solr, came
>> removing ScavengeBeforeRemark flag and applying this new one: +
>> ParGCCardsPerStrideChunk.
>> 
>> However, It would be a good one to set ParallelGCThreads and
>> *ConcGCThreads *to their optimal value, and we need you system CPU number
>> to know it. Can you provide this data, please?
>> 
>> Regards
>> 
>> 
>> 2018-04-11 12:01 GMT+02:00 Adam Harrison-Fuller <
>> aharrison-ful...@mintel.com
>>> :
>> 
>>> Hey all,
>>> 
>>> I was wondering if I could get some JVM/GC tuning advice to resolve an
>>> issue that we are experiencing.
>>> 
>>> Full disclaimer, I am in no way a JVM/Solr expert so any advice you can
>>> render would be greatly appreciated.
>>> 
>>> Our Solr cloud nodes are having issues throwing OOM exceptions under
>> load.
>>> This issue has only started manifesting itself over the last few months
>>> during which time the only change I can discern is an increase in index
>>> size.  They are running Solr 5.5.2 on OpenJDK version "1.8.0_101".  The
>>> index is currently 58G and the server has 46G of physical RAM and runs
>>> nothing other than the Solr node.
>>> 
>>> The JVM is invoked with the following JVM options:
>>> -XX:CMSInitiatingOccupancyFraction=50 -XX:CMSMaxAbortablePrecleanTime=
>> 6000
>>> -XX:+CMSParallelRemarkEnabled -XX:+CMSScavengeBeforeRemark
>>> -XX:ConcGCThreads=4 -XX:InitialHeapSize=12884901888
>> -XX:+ManagementServer
>>> -XX:MaxHeapSize=12884901888 -XX:MaxTenuringThreshold=8
>>> -XX:NewRatio=3 -XX:OldPLABSize=16
>>> -XX:OnOutOfMemoryError=/opt/solr/bin/oom_solr.sh 3
>>> /data/gnpd/solr/logs
>>> -XX:ParallelGCThreads=4
>>> -XX:+ParallelRefProcEnabled -XX:PretenureSizeThreshold=67108864
>>> -XX:+PrintGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps
>>> -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC
>>> -XX:+PrintTenuringDistribution -XX:SurvivorRatio=4
>>> -XX:TargetSurvivorRatio=90
>>> -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers
>>> -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
>>> 
>>> These values were decided upon serveral years by a colleague based upon
>>> some suggestions from this mailing group with an index size ~25G.
>>> 
>>> I have imported the GC logs into GCViewer and attached a link to a
>>> screenshot showing the lead up to a OOM crash.  Interestingly the young
>>> generation space is almost empty before the repeated GC's and subsequent
>>> crash.
>>> https://imgur.com/a/Wtlez
>>> 
>>> I was considering slowly increasing the amount of heap available to the
>> JVM
>>> slowly until the crashes, any other suggestions?  I'm looking at trying
>> to
>>> get the nodes stable without having issues with the GC taking forever to
>>> run.
>>> 
>>> Additional information can be provided on request.
>>> 
>>> Cheers!
>>> Adam
>>> 
>>> --
>>> 
>>> Mintel Group Ltd | 11 Pilgrim Street | London | EC4V 6RN
>>> Registered in
>>> England: Number 1475918. | VAT Number: GB 232 9342 72
>>> 
>>> Contact details for
>>> our other offices can be found at http://www.mintel.com/office-locations
>>> .
>>> 
>>> This email and any attachments
>>> may include content that is confidential, privileged
>>> or otherwise
>>> protected under applicable law. Unauthorised disclosure, copying,
>>> distribution
>>> or use of the contents is prohibited and may be unlawful. If
>>> you have received this email in error,
>>> including without appropriate
>>> authorisation, then please reply to the sender about the error
>>> and delete
>>> this email and any attachments.
>>> 
>>> 
>> 
> 
> -- 
> 
> Mintel Group Ltd | 11 Pilgrim Street | London | EC4V 6RN

Re: Indexing fails with partially done

[Emir Arnautović]

Hi,
First of all, the purpose of having 3 ZK in ensemble is to minimise the chances 
of loosing quorum. With 3 ZK, you can loose one and still have operable ZK 
ensemble. You should monitor it and alert if you loose one ZK and react to 
alert before you loose the second one. If you are wondering how to monitor it, 
you can check out Sematext’s performance monitor: https://sematext.com/spm/ 

If you loose the second ZK, then you either ignored the alert or you are really 
unlucky person or you have something terribly wrong in your setup.

Anyway, if that happens, there is nothing in Solr that keeps track what is 
indexed and what is not. You need to track that on your client side. It can be 
loosely tracking by some timestamp or strictly tracked by acknowledging every 
indexed document. In some cases it might be the simplest if you do full 
reindexing.

HTH,
Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/



> On 11 Apr 2018, at 14:46, neotorand  wrote:
> 
> with Solrcloud What happens if indexing is partially completed and ensemble 
> goes down.What are the ways to Resume.In one of the scenario i am using 3 ZK 
> Node in ensemble.Lets say i am indexing 5 million data and i have partially 
> indexed the data and ZK ensemble goes down. What should be the best approach 
> for handling such scenario 
> 
> Regards 
> Neo 
> 
> 
> 
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: replication

[John Blythe]

thanks, erick. great info.

although you can't (yet) direct queries to one or the other. So just making
> them all NRT and forgetting about it is reasonable.


are you simply flagging the fact that we wouldn't direct the queries to A
v. B v. C since SolrCloud will make the decisions itself as to which part
of the distro gets hit for the operation? if not, can you expound on this a
bit more?

The very nature of merging is such that you will _always_ get large merges
> until you have 5G segments (by default)


bummer

Quite possible, but you have to route things yourself. But in that case
> you're limited to one machine to handle all your NRT traffic. I skimmed
> your post so don't know whether your NRT traffic load is high enough to
> worry about.


ok. i think we'll take a two-pronged approach. for the immediate purposes
of trying to solve an issue we've begun encountering we will begin
thoroughtesting the load between various operations in the master-slave
setup we've set up. pending the results, we can roll forward w a temporary
patch in which all end-user touch points route through the primary box for
read/write while large scale operations/processing we do in the background
will point to the ELB the slaves are sitting behind. we'll also begin
setting up a simple solrcloud instance to toy with per your suggestion
above. inb4 tons more questions on my part :)

thanks!

--
John Blythe

On Tue, Apr 10, 2018 at 11:14 AM, Erick Erickson 
wrote:

> bq: should we try to bite the solrcloud bullet and be done w it
>
> that's what I'd do. As of 7.0 there are different "flavors", TLOG,
> PULL and NRT so that's also a possibility, although you can't (yet)
> direct queries to one or the other. So just making them all NRT and
> forgetting about it is reasonable.
>
> bq:  is there some more config work we could put in place to avoid ...
> commit issue and the ultra large merge dangers
>
> No. The very nature of merging is such that you will _always_ get
> large merges until you have 5G segments (by default). The max segment
> size (outside "optimize/forceMerge/expungeDeletes" which you shouldn't
> do) is 5G so the steady-state worst-case segment pull is limited to
> that.
>
> bq: maybe for our initial need we use Master for writing and user
> access in NRT events, but slaves for the heavier backend
>
> Quite possible, but you have to route things yourself. But in that
> case you're limited to one machine to handle all your NRT traffic. I
> skimmed your post so don't know whether your NRT traffic load is high
> enough to worry about.
>
> The very first thing I'd do is set up a simple SolrCloud setup and
> give it a spin. Unless your indexing load is quite heavy, the added
> work the NRT replicas have in SolrCloud isn't a problem so worrying
> about that is premature optimization unless you have a heavy load.
>
> Best,
> Erick
>
> On Mon, Apr 9, 2018 at 4:36 PM, John Blythe  wrote:
> > Thanks a bunch for the thorough reply, Shawn.
> >
> > Phew. We’d chosen to go w Master-slave replication instead of SolrCloud
> per
> > the sudden need we had encountered and the desire to avoid the nuances
> and
> > changes related to moving to SolrCloud. But so much for this being a more
> > straightforward solution, huh?
> >
> > Few questions:
> > - should we try to bite the solrcloud bullet and be done w it?
> > - is there some more config work we could put in place to avoid the soft
> > commit issue and the ultra large merge dangers, keeping the replications
> > happening quickly?
> > - maybe for our initial need we use Master for writing and user access in
> > NRT events, but slaves for the heavier backend processing. Thoughts?
> > - anyone do consulting on this that would be interested in chatting?
> >
> > Thanks again!
> >
> > On Mon, Apr 9, 2018 at 18:18 Shawn Heisey  wrote:
> >
> >> On 4/9/2018 12:15 PM, John Blythe wrote:
> >> > we're starting to dive into master/slave replication architecture.
> we'll
> >> > have 1 master w 4 slaves behind it. our app is NRT. if user performs
> an
> >> > action in section A's data they may choose to jump to section B which
> >> will
> >> > be dependent on having the updates from their action in section A. as
> >> such,
> >> > we're thinking that the replication time should be set to 1-2s (the
> >> chances
> >> > of them arriving at section B quickly enough to catch the 2s gap is
> >> highly
> >> > unlikely at best).
> >>
> >> Once you start talking about master-slave replication, my assumption is
> >> that you're not running SolrCloud.  You would NOT want to try and mix
> >> SolrCloud with replication.  The features do not play well together.
> >> SolrCloud with NRT replicas (this is the only replica type that exists
> >> in 6.x and earlier) may be a better option than master-slave
> replication.
> >>
> >> > since the replicas will simply be looking for new files it seems like
> >> this
> >> > would be a lightweight 

Indexing fails with partially done

[neotorand]

with Solrcloud What happens if indexing is partially completed and ensemble 
goes down.What are the ways to Resume.In one of the scenario i am using 3 ZK 
Node in ensemble.Lets say i am indexing 5 million data and i have partially 
indexed the data and ZK ensemble goes down. What should be the best approach 
for handling such scenario 

Regards 
Neo 



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Solr OOM Crashes / JVM tuning advice

[Adam Harrison-Fuller]

Hey Jesus,

Thanks for the suggestions.  The Solr nodes have 4 CPUs assigned to them.

Cheers!
Adam

On 11 April 2018 at 11:22, Jesus Olivan  wrote:

> Hi Adam,
>
> IMHO you could try increasing heap to 20 Gb (with 46 Gb of physical RAM,
> your JVM can afford more RAM without threading penalties due to outside
> heap RAM lacks.
>
> Another good one would be to increase -XX:CMSInitiatingOccupancyFraction
> =50
> to 75. I think that CMS collector works better when Old generation space is
> more populated.
>
> I usually use to set Survivor spaces to lesser size. If you want to try
> SurvivorRatio to 6, i think performance would be improved.
>
> Another good practice for me would be to set an static NewSize instead
> of -XX:NewRatio=3.
> You could try to set -XX:NewSize=7000m and -XX:MaxNewSize=7000Mb (one third
> of total heap space is recommended).
>
> Finally, my best results after a deep JVM I+D related to Solr, came
> removing ScavengeBeforeRemark flag and applying this new one: +
> ParGCCardsPerStrideChunk.
>
> However, It would be a good one to set ParallelGCThreads and
> *ConcGCThreads *to their optimal value, and we need you system CPU number
> to know it. Can you provide this data, please?
>
> Regards
>
>
> 2018-04-11 12:01 GMT+02:00 Adam Harrison-Fuller <
> aharrison-ful...@mintel.com
> >:
>
> > Hey all,
> >
> > I was wondering if I could get some JVM/GC tuning advice to resolve an
> > issue that we are experiencing.
> >
> > Full disclaimer, I am in no way a JVM/Solr expert so any advice you can
> > render would be greatly appreciated.
> >
> > Our Solr cloud nodes are having issues throwing OOM exceptions under
> load.
> > This issue has only started manifesting itself over the last few months
> > during which time the only change I can discern is an increase in index
> > size.  They are running Solr 5.5.2 on OpenJDK version "1.8.0_101".  The
> > index is currently 58G and the server has 46G of physical RAM and runs
> > nothing other than the Solr node.
> >
> > The JVM is invoked with the following JVM options:
> > -XX:CMSInitiatingOccupancyFraction=50 -XX:CMSMaxAbortablePrecleanTime=
> 6000
> > -XX:+CMSParallelRemarkEnabled -XX:+CMSScavengeBeforeRemark
> > -XX:ConcGCThreads=4 -XX:InitialHeapSize=12884901888
> -XX:+ManagementServer
> > -XX:MaxHeapSize=12884901888 -XX:MaxTenuringThreshold=8
> > -XX:NewRatio=3 -XX:OldPLABSize=16
> > -XX:OnOutOfMemoryError=/opt/solr/bin/oom_solr.sh 3
> > /data/gnpd/solr/logs
> > -XX:ParallelGCThreads=4
> > -XX:+ParallelRefProcEnabled -XX:PretenureSizeThreshold=67108864
> > -XX:+PrintGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps
> > -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC
> > -XX:+PrintTenuringDistribution -XX:SurvivorRatio=4
> > -XX:TargetSurvivorRatio=90
> > -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers
> > -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
> >
> > These values were decided upon serveral years by a colleague based upon
> > some suggestions from this mailing group with an index size ~25G.
> >
> > I have imported the GC logs into GCViewer and attached a link to a
> > screenshot showing the lead up to a OOM crash.  Interestingly the young
> > generation space is almost empty before the repeated GC's and subsequent
> > crash.
> > https://imgur.com/a/Wtlez
> >
> > I was considering slowly increasing the amount of heap available to the
> JVM
> > slowly until the crashes, any other suggestions?  I'm looking at trying
> to
> > get the nodes stable without having issues with the GC taking forever to
> > run.
> >
> > Additional information can be provided on request.
> >
> > Cheers!
> > Adam
> >
> > --
> >
> > Mintel Group Ltd | 11 Pilgrim Street | London | EC4V 6RN
> > Registered in
> > England: Number 1475918. | VAT Number: GB 232 9342 72
> >
> > Contact details for
> > our other offices can be found at http://www.mintel.com/office-locations
> > .
> >
> > This email and any attachments
> > may include content that is confidential, privileged
> > or otherwise
> > protected under applicable law. Unauthorised disclosure, copying,
> > distribution
> > or use of the contents is prohibited and may be unlawful. If
> > you have received this email in error,
> > including without appropriate
> > authorisation, then please reply to the sender about the error
> > and delete
> > this email and any attachments.
> >
> >
>

-- 

Mintel Group Ltd | 11 Pilgrim Street | London | EC4V 6RN
Registered in 
England: Number 1475918. | VAT Number: GB 232 9342 72

Contact details for 
our other offices can be found at http://www.mintel.com/office-locations 
.

This email and any attachments 
may include content that is confidential, privileged 
or otherwise 
protected under applicable law. Unauthorised disclosure, copying, 
distribution 
or use of the contents is prohibited and may be unlawful. If 
you have 

Re: Solr OOM Crashes / JVM tuning advice

[Jesus Olivan]

Hi Adam,

IMHO you could try increasing heap to 20 Gb (with 46 Gb of physical RAM,
your JVM can afford more RAM without threading penalties due to outside
heap RAM lacks.

Another good one would be to increase -XX:CMSInitiatingOccupancyFraction=50
to 75. I think that CMS collector works better when Old generation space is
more populated.

I usually use to set Survivor spaces to lesser size. If you want to try
SurvivorRatio to 6, i think performance would be improved.

Another good practice for me would be to set an static NewSize instead
of -XX:NewRatio=3.
You could try to set -XX:NewSize=7000m and -XX:MaxNewSize=7000Mb (one third
of total heap space is recommended).

Finally, my best results after a deep JVM I+D related to Solr, came
removing ScavengeBeforeRemark flag and applying this new one: +
ParGCCardsPerStrideChunk.

However, It would be a good one to set ParallelGCThreads and
*ConcGCThreads *to their optimal value, and we need you system CPU number
to know it. Can you provide this data, please?

Regards


2018-04-11 12:01 GMT+02:00 Adam Harrison-Fuller :

> Hey all,
>
> I was wondering if I could get some JVM/GC tuning advice to resolve an
> issue that we are experiencing.
>
> Full disclaimer, I am in no way a JVM/Solr expert so any advice you can
> render would be greatly appreciated.
>
> Our Solr cloud nodes are having issues throwing OOM exceptions under load.
> This issue has only started manifesting itself over the last few months
> during which time the only change I can discern is an increase in index
> size.  They are running Solr 5.5.2 on OpenJDK version "1.8.0_101".  The
> index is currently 58G and the server has 46G of physical RAM and runs
> nothing other than the Solr node.
>
> The JVM is invoked with the following JVM options:
> -XX:CMSInitiatingOccupancyFraction=50 -XX:CMSMaxAbortablePrecleanTime=6000
> -XX:+CMSParallelRemarkEnabled -XX:+CMSScavengeBeforeRemark
> -XX:ConcGCThreads=4 -XX:InitialHeapSize=12884901888 -XX:+ManagementServer
> -XX:MaxHeapSize=12884901888 -XX:MaxTenuringThreshold=8
> -XX:NewRatio=3 -XX:OldPLABSize=16
> -XX:OnOutOfMemoryError=/opt/solr/bin/oom_solr.sh 3
> /data/gnpd/solr/logs
> -XX:ParallelGCThreads=4
> -XX:+ParallelRefProcEnabled -XX:PretenureSizeThreshold=67108864
> -XX:+PrintGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps
> -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC
> -XX:+PrintTenuringDistribution -XX:SurvivorRatio=4
> -XX:TargetSurvivorRatio=90
> -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers
> -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
>
> These values were decided upon serveral years by a colleague based upon
> some suggestions from this mailing group with an index size ~25G.
>
> I have imported the GC logs into GCViewer and attached a link to a
> screenshot showing the lead up to a OOM crash.  Interestingly the young
> generation space is almost empty before the repeated GC's and subsequent
> crash.
> https://imgur.com/a/Wtlez
>
> I was considering slowly increasing the amount of heap available to the JVM
> slowly until the crashes, any other suggestions?  I'm looking at trying to
> get the nodes stable without having issues with the GC taking forever to
> run.
>
> Additional information can be provided on request.
>
> Cheers!
> Adam
>
> --
>
> Mintel Group Ltd | 11 Pilgrim Street | London | EC4V 6RN
> Registered in
> England: Number 1475918. | VAT Number: GB 232 9342 72
>
> Contact details for
> our other offices can be found at http://www.mintel.com/office-locations
> .
>
> This email and any attachments
> may include content that is confidential, privileged
> or otherwise
> protected under applicable law. Unauthorised disclosure, copying,
> distribution
> or use of the contents is prohibited and may be unlawful. If
> you have received this email in error,
> including without appropriate
> authorisation, then please reply to the sender about the error
> and delete
> this email and any attachments.
>
>

Re: Default Index config

[mganeshs]

Hi Shawn, 

We found following  link

  
where its mentioned like in 6.2.1 it's faster where as in 6.6 its slower.
Keep this, we too tried with 6.2.1 in our performance environment and we
found that CPU usage came down to 60 to 70% where as in 6.5.1 it was always
more than 95% 

Settings are same and data size and indexing speed remains same. Pls check
the  JVM snapshot

  
when we index using 6.2.1


Following is the  snapshot

 
taken with 6.5.1

Is there any reason why such a huge difference with CPU usage patterns
between 6.2.1 and 6.5.1 ? 

Can we do something in 6.5.1 to make it as 6.2.1? Because we don't want to
downgrade to 6.2.1 from 6.5.1. 

Let us know your thoughts on this.

Thanks and Regards,





--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Decision on Number of shards and collection

[neotorand]

Hi Team
First of all i take this opportunity to thank you all for creating a
beautiful place where people can explore ,learn and debate.

I have been on my knees for couple of days to decide on this.

When i am creating a solr cloud eco system i need to decide on number of
shards and collection.
What are the best practices for taking this decisions.

I believe heterogeneous data can be indexed to same collection and i can
have multiple shards for the index to be partitioned.So whats the need of a
second collection?. yes when collection size grows i should look for more
collection.what exactly that size is? what KPI drives the decision of having
more collection?Any pointers or links for best practice.

when should i go for multiple shards?
yes when shard size grows.Right? whats the size and how do i benchmark.

I am sorry for my question if its already asked but googled all the ecospace
quora,stackoverflow,lucid

Regards
Neo





--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Solr OOM Crashes / JVM tuning advice

[Adam Harrison-Fuller]

Hey all,

I was wondering if I could get some JVM/GC tuning advice to resolve an
issue that we are experiencing.

Full disclaimer, I am in no way a JVM/Solr expert so any advice you can
render would be greatly appreciated.

Our Solr cloud nodes are having issues throwing OOM exceptions under load.
This issue has only started manifesting itself over the last few months
during which time the only change I can discern is an increase in index
size.  They are running Solr 5.5.2 on OpenJDK version "1.8.0_101".  The
index is currently 58G and the server has 46G of physical RAM and runs
nothing other than the Solr node.

The JVM is invoked with the following JVM options:
-XX:CMSInitiatingOccupancyFraction=50 -XX:CMSMaxAbortablePrecleanTime=6000
-XX:+CMSParallelRemarkEnabled -XX:+CMSScavengeBeforeRemark
-XX:ConcGCThreads=4 -XX:InitialHeapSize=12884901888 -XX:+ManagementServer
-XX:MaxHeapSize=12884901888 -XX:MaxTenuringThreshold=8
-XX:NewRatio=3 -XX:OldPLABSize=16
-XX:OnOutOfMemoryError=/opt/solr/bin/oom_solr.sh 3 /data/gnpd/solr/logs
-XX:ParallelGCThreads=4
-XX:+ParallelRefProcEnabled -XX:PretenureSizeThreshold=67108864
-XX:+PrintGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps
-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC
-XX:+PrintTenuringDistribution -XX:SurvivorRatio=4
-XX:TargetSurvivorRatio=90
-XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers
-XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC

These values were decided upon serveral years by a colleague based upon
some suggestions from this mailing group with an index size ~25G.

I have imported the GC logs into GCViewer and attached a link to a
screenshot showing the lead up to a OOM crash.  Interestingly the young
generation space is almost empty before the repeated GC's and subsequent
crash.
https://imgur.com/a/Wtlez

I was considering slowly increasing the amount of heap available to the JVM
slowly until the crashes, any other suggestions?  I'm looking at trying to
get the nodes stable without having issues with the GC taking forever to
run.

Additional information can be provided on request.

Cheers!
Adam

-- 

Mintel Group Ltd | 11 Pilgrim Street | London | EC4V 6RN
Registered in 
England: Number 1475918. | VAT Number: GB 232 9342 72

Contact details for 
our other offices can be found at http://www.mintel.com/office-locations 
.

This email and any attachments 
may include content that is confidential, privileged 
or otherwise 
protected under applicable law. Unauthorised disclosure, copying, 
distribution 
or use of the contents is prohibited and may be unlawful. If 
you have received this email in error,
including without appropriate 
authorisation, then please reply to the sender about the error 
and delete 
this email and any attachments.

Re: Ignore Field from indexing

[Emir Arnautović]

Hi,
You have two options when it comes to updating:
1. Send complete document with the same id that will replace existing document.
2. Use atomic updates to send changes, but not that fields need to be stored: 
https://lucene.apache.org/solr/guide/6_6/updating-parts-of-documents.html#UpdatingPartsofDocuments-AtomicUpdates
 


Solr (Lucene) does not support actual updating of indexed document, but only 
flagging existing document as deleted and inserting a new document with same 
id. It needs to know all values of this new document and it can come from a 
client (option 1) or it can use existing document to read some values (option 
2).

HTH,
Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/



> On 10 Apr 2018, at 12:30, swap  wrote:
> 
> Hi
> 
> I have document indexed. Email-Id is unique key in document. On updating I
> need to ignore few field if its already exists. Please let me know if
> something more required.
> 
> 
> 
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html