Re: check softCommit , autocommit and hard commit count

[Puppy Linux Distros]

Hello,

Thanks Shawn. Can you provide command to find the total number of
autocommits in the solr.log?

On Thu, Nov 30, 2017 at 7:20 PM, Shawn Heisey  wrote:

> On 11/30/2017 4:36 AM, Puppy Linux Distros wrote:
>
>> I am trying to calculate the total number of softCommit , autocommit and
>> hard commit from the solr logs. Can you please check whether the below
>> commands are correct ?
>>
>> Let me know how to find the total softcommit, hardcommit and autocommit
>> from the logs.
>>
>>
>> *1. totalcommit=`cat $solrlogfile | grep "start commit" | wc -l`*
>>
>> *totalcommit =  **41906*
>>
>>
>> *2. totalsoftcommit=`cat $solrlogfile | grep "start commit" | grep
>> "softCommit=true" | wc -l`*
>>
>> *totalsoftcommit =  **921*
>>
>
> These look reasonable ... but be aware that the default logging config
> will roll the solr.log file to a new empty file when it reaches 4
> megabytes, which doesn't really take that long on a busy server, so if
> you're only looking at "solr.log" you may have an incomplete picture.  I
> personally change the roll size limit to 4 gigabytes so solr.log covers a
> lot more time.
>
> Solr restarts will *also* roll/archive logfiles, so you probably can't
> just look through every file in the logs directory that starts with
> "solr.log" -- it may be difficult to figure out exactly which files apply
> to the current running instance.  It might turn out that I'm completely
> wrong in that statement -- I haven't confirmed exactly what a Solr restart
> actually does with the logfiles.
>
> *3. totalhardcommits=`cat $solrlogfile | grep "start commit" | grep
>> "softCommit=false" | grep "openSearcher=true" | wc -l`*
>>
>> *totalhardcommits=  **40982*
>>
>
> If you have configured autoCommit in solrconfig.xml and have set
> openSearcher to false in that config, then there will be hard commits that
> *don't* open a new searcher, so the "openSearcher=true" part will not catch
> those commits.  Example configs in recent versions have autoCommit set up
> this way, and this recommended config for *everybody*.  The default
> autoCommit interval in the example configs is 15 seconds, which I think is
> a little too aggressive, but this kind of commit is typically very fast, so
> I've never seen that config cause problems.
>
> The example configs do not have autoSoftCommit configured.  If users want
> to automatically do commits for visibility, we recommend that they use
> autoSoftCommit.
>
> *4.  totalautocommit=`cat $solrlogfile | grep "realtime" | wc -l`*
>>
>> *totalautocommit= 3*
>>
>
> These aren't autoCommits.  They are new searchers for the realtime get
> handler, which is capable of accessing documents that haven't been
> committed yet.  In addition to the index on disk, it searches the
> transaction logs.  Opening a new realtime searcher should be very fast, and
> they happen without any configuration. I'm not sure why you're only seeing
> this happen three times here. Presumably in a log where there are 4
> total commits, you are doing a fair amount of indexing, so I would have
> expected a new realtime searcher to have been created much more frequently,
> even if there were no commits done at all.
>
> Maybe the realtime get handler can use the standard searcher, and only
> opens a new realtime searcher in cases where new documents have been
> indexed but there hasn't been a recent commit that opens a new searcher.
> If that's the case, then I have no idea how long it would wait before
> firing up a new realtime searcher.  I wouldn't expect that to be very long
> ... so if your indexing/committing cycles are normally very fast, maybe
> Solr doesn't feel it's necessary to open realtime searchers very often.
>
> Thanks,
> Shawn
>
>


-- 
Regards,

Vivek CV

Re: merge metrics not showing up in Jconsole

[suresh pendap]

I see only these metrics in my Jconsole window

[image: Inline image 1]

On Sun, Dec 3, 2017 at 4:19 PM, suresh pendap 
wrote:

> Hi,
> I am using Solr version 6.6.0 and am following the document
> https://lucene.apache.org/solr/guide/6_6/metrics-reporting.html#index-
> merge-metrics to enable Index merge metrics.
>
> I have added the below config to my solrconfig.xml file
>
>
>   
> 
>   524288
>   true
>   true
> 
> ...
>   
>
>
>
> Then I restarted the solr servers.  I do see the INDEX merge metrics when
> I use the REST API
> http://localhost:8983/solr/admin/metrics?wt=json&=core
>
> However I do not see the INDEX category and the merge metrics in the
> Jconsole window.
>
> Is this a known issue?
>
> Do I need to configure something else to be able to expose the Index Merge
> metrics as JMX mbeans?
>
>
> Thanks
> Suresh
>
>

merge metrics not showing up in Jconsole

[suresh pendap]

Hi,
I am using Solr version 6.6.0 and am following the document
https://lucene.apache.org/solr/guide/6_6/metrics-reporting.html#index-merge-metrics
to enable Index merge metrics.

I have added the below config to my solrconfig.xml file


  

  524288
  true
  true

...
  



Then I restarted the solr servers.  I do see the INDEX merge metrics when I
use the REST API
http://localhost:8983/solr/admin/metrics?wt=json&=core

However I do not see the INDEX category and the merge metrics in the
Jconsole window.

Is this a known issue?

Do I need to configure something else to be able to expose the Index Merge
metrics as JMX mbeans?


Thanks
Suresh

Solr merge related metrics not showing up in jconsole

[suresh pendap]

Hello,

I am using Solr version 6.6 and I am following the document to get the
Segment merge related metrics
https://lucene.apache.org/solr/guide/6_6/metrics-reporting.html#index-merge-metrics

I added the configuration to expose the merge related metrics to my
solrconfig.xml file as below



  524288
  true
  true

...
  

Re: JVM GC Issue

[Shawn Heisey]

On 12/2/2017 6:59 PM, S G wrote:

I am a bit curious on the docValues implementation.
I understand that docValues do not use JVM memory and
they make use of OS cache - that is why they are more performant.

But to return any response from the docValues, the values in the
docValues' column-oriented-structures would need to be brought
into the JVM's memory. And that will then increase the pressure
on the JVM's memory anyways. So how do docValues actually
help from memory perspective?


What I'm writing below is my understanding of docValues.  If it turns 
out that I've gotten any of it wrong, that is MY error, not Solr's.


When there are no docValues, Solr must do something called "uninverting 
the index" in order to satisfy certain operations -- primarily faceting, 
grouping, and sorting.


A Lucene index is an inverted index.  This means that it is a big list 
of terms, and then each of those entries is a second list that describes 
which fields and documents have the term, as well as some other 
information like positions.  Uninverting the index is pretty efficient, 
but it does take time.  The uninverted index structure is a list of all 
terms for a specific field.  Then there's a second phase -- the info in 
the uninverted field is read and processed for the query operation, 
which will use heap.  I do not know if there are additional phases. 
There might be.


In case you don't know, in the Lucene index, docValues data on disk 
consists of every entry in the index for one field, written sequentially 
in an uncompressed format.


This means that for those query types, docValues is *exactly* what Solr 
needs for the first phase.  And instead of generating it into heap 
memory and then reading it, Solr can just read the data right off the 
disk (which the OS caches, so it might be REALLY fast and use OS memory) 
in order to handle second and later phases.  This is faster than 
building an uninverted field, and consumes no heap memory.


As I mentioned, the uninverted data is built from indexed terms.  The 
contents of docValue data is the same as a stored field -- the original 
indexed data.  Because docValues cannot be added to fields using 
solr.TextField, the only type that undergoes text analysis, there's no 
possibility of a difference between an uninverted field and docValues.


Thanks,
Shawn

Dataimporter status

[Mahmoud Almokadem]

We're facing an issue related to the dataimporter status on new Admin UI
(7.0.1).

Calling to the API
http://solrip/solr/collection/dataimport?_=1512314812090=status=on=json


returns different status despite the importer is running
The messages are swapped between the following when refreshing the page:
{
  "responseHeader":{
"status":0,
"QTime":0},
  "initArgs":[
"defaults",[
  "config","data-config-online-live-pervoice.xml"]],
  "command":"status",
  "status":"idle",
  "importResponse":"",
  "statusMessages":{}}

===
{
  "responseHeader":{
"status":0,
"QTime":0},
  "initArgs":[
"defaults",[
  "config","data-config-online-live-pervoice.xml"]],
  "command":"status",
  "status":"idle",
  "importResponse":"",
  "statusMessages":{
"Total Requests made to DataSource":"2",
"Total Rows Fetched":"715",
"Total Documents Processed":"679",
"Total Documents Skipped":"0",
"Full Dump Started":"2017-12-03 18:22:31",
"":"Indexing completed. Added/Updated: 679 documents. Deleted 0
documents.",
"Committed":"2017-12-03 18:22:32",
"Total Documents Failed":"36",
"Time taken":"0:0:54.638",
"Full Import failed":"2017-12-03 18:22:32"}}


The old Admin UI was working well.

Is that a bug on the new Admin UI?

Thanks,
Mahmoud

Re: JVM GC Issue

[Yonik Seeley]

On Sat, Dec 2, 2017 at 8:59 PM, S G  wrote:
> I am a bit curious on the docValues implementation.
> I understand that docValues do not use JVM memory and
> they make use of OS cache - that is why they are more performant.
>
> But to return any response from the docValues, the values in the
> docValues' column-oriented-structures would need to be brought
> into the JVM's memory. And that will then increase the pressure
> on the JVM's memory anyways. So how do docValues actually
> help from memory perspective?

docValues are not more performant than the on-heap fieldCache once a
fieldCache entry has been populated.
docValues do help with memory in a number of ways:
1) off-heap memory use degrades much more gracefully... heap memory
will just cause an OOM exception when heap size is exceeded
2) off-heap memory (memory used by OS cache) can be dynamically shared
with other processes running on the system (the OS allocates as
needed)
3) easier to configure (as opposed to max-heap size) since the OS will
just automatically use all free memory to cache relevant parts of
memory mapped disk files
4) off-heap memory does not need to be garbage collected (helps with
people that have huge GC pauses)

-Yonik

Log page auto refresh

[Mahmoud Almokadem]

Hello,

I've an issue related to the Log page on the new Admin UI (7.0.1), When I
expand an item, it collapsed again after short time.

This behavior is different than the old Admin UI.

Thanks,
Mahmoud