Re: SolrCloud indexing triggers merges and timeouts

[Rahul Goswami]

Upon further investigation on this issue, I see the below log lines during
the indexing process:

2019-06-06 22:24:56.203 INFO  (qtp1169794610-5652)
[c:UM_IndexServer_MailArchiv_Spelle_66AC8340-4734-438A-9D1D-A84B659B1623
s:shard22 r:core_node87
x:UM_IndexServer_MailArchiv_Spelle_66AC8340-4734-438A-9D1D-A84B659B1623_shard22_replica_n84]
org.apache.solr.update.LoggingInfoStream [FP][qtp1169794610-5652]: trigger
flush: activeBytes=352402600 deleteBytes=279 vs limit=104857600
2019-06-06 22:24:56.203 INFO  (qtp1169794610-5652)
[c:UM_IndexServer_MailArchiv_Spelle_66AC8340-4734-438A-9D1D-A84B659B1623
s:shard22 r:core_node87
x:UM_IndexServer_MailArchiv_Spelle_66AC8340-4734-438A-9D1D-A84B659B1623_shard22_replica_n84]
org.apache.solr.update.LoggingInfoStream [FP][qtp1169794610-5652]: thread
state has 352402600 bytes; docInRAM=1
2019-06-06 22:24:56.204 INFO  (qtp1169794610-5652)
[c:UM_IndexServer_MailArchiv_Spelle_66AC8340-4734-438A-9D1D-A84B659B1623
s:shard22 r:core_node87
x:UM_IndexServer_MailArchiv_Spelle_66AC8340-4734-438A-9D1D-A84B659B1623_shard22_replica_n84]
org.apache.solr.update.LoggingInfoStream [FP][qtp1169794610-5652]: 1 in-use
non-flushing threads states
2019-06-06 22:24:56.204 INFO  (qtp1169794610-5652)
[c:UM_IndexServer_MailArchiv_Spelle_66AC8340-4734-438A-9D1D-A84B659B1623
s:shard22 r:core_node87

I have the below questions:
1) The log line which says "thread state has 352402600 bytes; docInRAM=1 ",
does it mean that the buffer was flushed to disk with only one huge
document ?
2) If yes, does this flush create a segment with just one document ?
3) Heap dump analysis shows large (>350 MB) instances of
DocumentWritersPerThread. Does one instance of this class correspond to one
document?


Help is much appreciated.

Thanks,
Rahul


On Fri, Jul 5, 2019 at 2:11 AM Rahul Goswami  wrote:

> Shawn,Erick,
> Thank you for the explanation. The merge scheduler params make sense now.
>
> Thanks,
> Rahul
>
> On Wed, Jul 3, 2019 at 11:30 AM Erick Erickson 
> wrote:
>
>> Two more tidbits to add to Shawn’s explanation:
>>
>> There are heuristics built in to ConcurrentMergeScheduler.
>> From the Javadocs:
>> * If it's an SSD,
>> *  {@code maxThreadCount} is set to {@code max(1, min(4,
>> cpuCoreCount/2))},
>> *  otherwise 1.  Note that detection only currently works on
>> *  Linux; other platforms will assume the index is not on an SSD.
>>
>> Second, TieredMergePolicy (the default) merges in “tiers” that
>> are of similar size. So you can have multiple merges going on
>> at the same time on disjoint sets of segments.
>>
>> Best,
>> Erick
>>
>> > On Jul 3, 2019, at 7:54 AM, Shawn Heisey  wrote:
>> >
>> > On 7/2/2019 10:53 PM, Rahul Goswami wrote:
>> >> Hi Shawn,
>> >> Thank you for the detailed suggestions. Although, I would like to
>> >> understand the maxMergeCount and maxThreadCount params better. The
>> >> documentation
>> >> <
>> https://lucene.apache.org/solr/guide/7_3/indexconfig-in-solrconfig.html#mergescheduler
>> >
>> >> mentions
>> >> that
>> >> maxMergeCount : The maximum number of simultaneous merges that are
>> allowed.
>> >> maxThreadCount : The maximum number of simultaneous merge threads that
>> >> should be running at once
>> >> Since one thread can only do 1 merge at any given point of time, how
>> does
>> >> maxMergeCount being greater than maxThreadCount help anyway? I am
>> having
>> >> difficulty wrapping my head around this, and would appreciate if you
>> could
>> >> help clear it for me.
>> >
>> > The maxMergeCount setting controls the number of merges that can be
>> *scheduled* at the same time.  As soon as that number of merges is reached,
>> the indexing thread(s) will be paused until the number of merges in the
>> schedule drops below this number.  This ensures that no more merges will be
>> scheduled.
>> >
>> > By setting maxMergeCount higher than the number of merges that are
>> expected in the schedule, you can ensure that indexing will never be
>> paused.  It would require very atypical merge policy settings for the
>> number of scheduled merges to ever reach six.  On my own indexing, I
>> reached three scheduled merges quite frequently.  The default setting for
>> maxMergeCount is three.
>> >
>> > The maxThreadCount setting controls how many of the scheduled merges
>> will be simultaneously executed. With index data on standard spinning
>> disks, you do not want to increase this number beyond 1, or you will have a
>> performance problem due to thrashing disk heads.  If your data is on SSD,
>> you can make it larger than 1.
>> >
>> > Thanks,
>> > Shawn
>>
>>

Re: SolrCloud indexing triggers merges and timeouts

[Rahul Goswami]

Shawn,Erick,
Thank you for the explanation. The merge scheduler params make sense now.

Thanks,
Rahul

On Wed, Jul 3, 2019 at 11:30 AM Erick Erickson 
wrote:

> Two more tidbits to add to Shawn’s explanation:
>
> There are heuristics built in to ConcurrentMergeScheduler.
> From the Javadocs:
> * If it's an SSD,
> *  {@code maxThreadCount} is set to {@code max(1, min(4, cpuCoreCount/2))},
> *  otherwise 1.  Note that detection only currently works on
> *  Linux; other platforms will assume the index is not on an SSD.
>
> Second, TieredMergePolicy (the default) merges in “tiers” that
> are of similar size. So you can have multiple merges going on
> at the same time on disjoint sets of segments.
>
> Best,
> Erick
>
> > On Jul 3, 2019, at 7:54 AM, Shawn Heisey  wrote:
> >
> > On 7/2/2019 10:53 PM, Rahul Goswami wrote:
> >> Hi Shawn,
> >> Thank you for the detailed suggestions. Although, I would like to
> >> understand the maxMergeCount and maxThreadCount params better. The
> >> documentation
> >> <
> https://lucene.apache.org/solr/guide/7_3/indexconfig-in-solrconfig.html#mergescheduler
> >
> >> mentions
> >> that
> >> maxMergeCount : The maximum number of simultaneous merges that are
> allowed.
> >> maxThreadCount : The maximum number of simultaneous merge threads that
> >> should be running at once
> >> Since one thread can only do 1 merge at any given point of time, how
> does
> >> maxMergeCount being greater than maxThreadCount help anyway? I am having
> >> difficulty wrapping my head around this, and would appreciate if you
> could
> >> help clear it for me.
> >
> > The maxMergeCount setting controls the number of merges that can be
> *scheduled* at the same time.  As soon as that number of merges is reached,
> the indexing thread(s) will be paused until the number of merges in the
> schedule drops below this number.  This ensures that no more merges will be
> scheduled.
> >
> > By setting maxMergeCount higher than the number of merges that are
> expected in the schedule, you can ensure that indexing will never be
> paused.  It would require very atypical merge policy settings for the
> number of scheduled merges to ever reach six.  On my own indexing, I
> reached three scheduled merges quite frequently.  The default setting for
> maxMergeCount is three.
> >
> > The maxThreadCount setting controls how many of the scheduled merges
> will be simultaneously executed. With index data on standard spinning
> disks, you do not want to increase this number beyond 1, or you will have a
> performance problem due to thrashing disk heads.  If your data is on SSD,
> you can make it larger than 1.
> >
> > Thanks,
> > Shawn
>
>

Re: SolrCloud indexing triggers merges and timeouts

[Erick Erickson]

Two more tidbits to add to Shawn’s explanation:

There are heuristics built in to ConcurrentMergeScheduler.
From the Javadocs:
* If it's an SSD,
*  {@code maxThreadCount} is set to {@code max(1, min(4, cpuCoreCount/2))},
*  otherwise 1.  Note that detection only currently works on
*  Linux; other platforms will assume the index is not on an SSD.

Second, TieredMergePolicy (the default) merges in “tiers” that
are of similar size. So you can have multiple merges going on
at the same time on disjoint sets of segments.

Best,
Erick

> On Jul 3, 2019, at 7:54 AM, Shawn Heisey  wrote:
> 
> On 7/2/2019 10:53 PM, Rahul Goswami wrote:
>> Hi Shawn,
>> Thank you for the detailed suggestions. Although, I would like to
>> understand the maxMergeCount and maxThreadCount params better. The
>> documentation
>> 
>> mentions
>> that
>> maxMergeCount : The maximum number of simultaneous merges that are allowed.
>> maxThreadCount : The maximum number of simultaneous merge threads that
>> should be running at once
>> Since one thread can only do 1 merge at any given point of time, how does
>> maxMergeCount being greater than maxThreadCount help anyway? I am having
>> difficulty wrapping my head around this, and would appreciate if you could
>> help clear it for me.
> 
> The maxMergeCount setting controls the number of merges that can be 
> *scheduled* at the same time.  As soon as that number of merges is reached, 
> the indexing thread(s) will be paused until the number of merges in the 
> schedule drops below this number.  This ensures that no more merges will be 
> scheduled.
> 
> By setting maxMergeCount higher than the number of merges that are expected 
> in the schedule, you can ensure that indexing will never be paused.  It would 
> require very atypical merge policy settings for the number of scheduled 
> merges to ever reach six.  On my own indexing, I reached three scheduled 
> merges quite frequently.  The default setting for maxMergeCount is three.
> 
> The maxThreadCount setting controls how many of the scheduled merges will be 
> simultaneously executed. With index data on standard spinning disks, you do 
> not want to increase this number beyond 1, or you will have a performance 
> problem due to thrashing disk heads.  If your data is on SSD, you can make it 
> larger than 1.
> 
> Thanks,
> Shawn

Re: SolrCloud indexing triggers merges and timeouts

[Shawn Heisey]

On 7/2/2019 10:53 PM, Rahul Goswami wrote:

Hi Shawn,

Thank you for the detailed suggestions. Although, I would like to
understand the maxMergeCount and maxThreadCount params better. The
documentation

mentions
that

maxMergeCount : The maximum number of simultaneous merges that are allowed.
maxThreadCount : The maximum number of simultaneous merge threads that
should be running at once

Since one thread can only do 1 merge at any given point of time, how does
maxMergeCount being greater than maxThreadCount help anyway? I am having
difficulty wrapping my head around this, and would appreciate if you could
help clear it for me.


The maxMergeCount setting controls the number of merges that can be 
*scheduled* at the same time.  As soon as that number of merges is 
reached, the indexing thread(s) will be paused until the number of 
merges in the schedule drops below this number.  This ensures that no 
more merges will be scheduled.


By setting maxMergeCount higher than the number of merges that are 
expected in the schedule, you can ensure that indexing will never be 
paused.  It would require very atypical merge policy settings for the 
number of scheduled merges to ever reach six.  On my own indexing, I 
reached three scheduled merges quite frequently.  The default setting 
for maxMergeCount is three.


The maxThreadCount setting controls how many of the scheduled merges 
will be simultaneously executed.  With index data on standard spinning 
disks, you do not want to increase this number beyond 1, or you will 
have a performance problem due to thrashing disk heads.  If your data is 
on SSD, you can make it larger than 1.


Thanks,
Shawn

Re: SolrCloud indexing triggers merges and timeouts

[Rahul Goswami]

Hi Shawn,

Thank you for the detailed suggestions. Although, I would like to
understand the maxMergeCount and maxThreadCount params better. The
documentation

mentions
that

maxMergeCount : The maximum number of simultaneous merges that are allowed.
maxThreadCount : The maximum number of simultaneous merge threads that
should be running at once

Since one thread can only do 1 merge at any given point of time, how does
maxMergeCount being greater than maxThreadCount help anyway? I am having
difficulty wrapping my head around this, and would appreciate if you could
help clear it for me.

Thanks,
Rahul

On Thu, Jun 13, 2019 at 7:33 AM Shawn Heisey  wrote:

> On 6/6/2019 9:00 AM, Rahul Goswami wrote:
> > *OP Reply* : Total 48 GB per node... I couldn't see another software
> using
> > a lot of memory.
> > I am honestly not sure about the reason for change of directory factory
> to
> > SimpleFSDirectoryFactory. But I was told that with mmap at one point we
> > started to see the shared memory usage on Windows go up significantly,
> > intermittently freezing the system.
> > Could the choice of DirectoryFactory here be a factor for the long
> > updates/frequent merges?
>
> With about 24GB of RAM to cache 1.4TB of index data, you're never going
> to have good performance.  Any query you do is probably going to read
> more than 24GB of data from the index, which means that it cannot come
> from memory, some of it must come from disk, which is incredibly slow
> compared to memory.
>
> MMap is more efficient than "simple" filesystem access.  I do not know
> if you would see markedly better performance, but getting rid of the
> DirectoryFactory config and letting Solr choose its default might help.
>
> > How many total documents (maxDoc, not numDoc) are in that 1.4 TB of
> > space?
> > *OP Reply:* Also, there are nearly 12.8 million total docs (maxDoc, NOT
> > numDoc) in that 1.4 TB space
>
> Unless you're doing faceting or grouping on fields with extremely high
> cardinality, which I find to be rarely useful except for data mining,
> 24GB of heap for 12.8 million docs seems very excessive.  I was
> expecting this number to be something like 500 million or more ... that
> small document count must mean each document is HUGE.  Can you take
> steps to reduce the index size, perhaps by setting stored, indexed,
> and/or docValues to "false" on some of your fields, and having your
> application go to the system of record for full details on each
> document?  You will have to reindex after making changes like that.
>
> >> Can you share the GC log that Solr writes?
> > *OP Reply:*  Please find the GC logs and thread dumps at this location
> > https://drive.google.com/open?id=1slsYkAcsH7OH-7Pma91k6t5T72-tIPlw
>
> The larger GC log was unrecognized by both gcviwer and gceasy.io ... the
> smaller log shows heap usage about 10GB, but it only covers 10 minutes,
> so it's not really conclusive for diagnosis.  The first thing I can
> suggest to try is to reduce the heap size to 12GB ... but I do not know
> if that's actually going to work.  Indexing might require more memory.
> The idea here is to make more memory available to the OS disk cache ...
> with your index size, you're probably going to need to add memory to the
> system (not the heap).
>
> > Another observation is that the CPU usage reaches around 70% (through
> > manual monitoring) when the indexing starts and the merges are observed.
> It
> > is well below 50% otherwise.
>
> Indexing will increase load, and that increase is often very
> significant.  Adding memory to the system is your best bet for better
> performance.  I'd want 1TB of memory for a 1.4TB index ... but I know
> that memory sizes that high are extremely expensive, and for most
> servers, not even possible.  512GB or 256GB is more attainable, and
> would have better performance than 48GB.
>
> > Also, should something be altered with the mergeScheduler setting ?
> > "mergeScheduler":{
> >  "class":"org.apache.lucene.index.ConcurrentMergeScheduler",
> >  "maxMergeCount":2,
> >  "maxThreadCount":2},
>
> Do not configure maxThreadCount beyond 1 unless your data is on SSD.  It
> will slow things down a lot due to the fact that standard disks must
> move the disk head to read/write from different locations, and head
> moves take time.  SSD can do I/O from any location without pauses, so
> more threads would probably help performance rather than hurt it.
>
> Increase maxMergeCount to 6 -- at 2, large merges will probably stop
> indexing entirely.  With a larger number, Solr can keep indexing even
> when there's a huge segment merge happening.
>
> Thanks,
> Shawn
>

Re: SolrCloud indexing triggers merges and timeouts

[Shawn Heisey]

On 6/6/2019 9:00 AM, Rahul Goswami wrote:

*OP Reply* : Total 48 GB per node... I couldn't see another software using
a lot of memory.
I am honestly not sure about the reason for change of directory factory to
SimpleFSDirectoryFactory. But I was told that with mmap at one point we
started to see the shared memory usage on Windows go up significantly,
intermittently freezing the system.
Could the choice of DirectoryFactory here be a factor for the long
updates/frequent merges?


With about 24GB of RAM to cache 1.4TB of index data, you're never going 
to have good performance.  Any query you do is probably going to read 
more than 24GB of data from the index, which means that it cannot come 
from memory, some of it must come from disk, which is incredibly slow 
compared to memory.


MMap is more efficient than "simple" filesystem access.  I do not know 
if you would see markedly better performance, but getting rid of the 
DirectoryFactory config and letting Solr choose its default might help.



How many total documents (maxDoc, not numDoc) are in that 1.4 TB of
space?
*OP Reply:* Also, there are nearly 12.8 million total docs (maxDoc, NOT
numDoc) in that 1.4 TB space


Unless you're doing faceting or grouping on fields with extremely high 
cardinality, which I find to be rarely useful except for data mining, 
24GB of heap for 12.8 million docs seems very excessive.  I was 
expecting this number to be something like 500 million or more ... that 
small document count must mean each document is HUGE.  Can you take 
steps to reduce the index size, perhaps by setting stored, indexed, 
and/or docValues to "false" on some of your fields, and having your 
application go to the system of record for full details on each 
document?  You will have to reindex after making changes like that.



Can you share the GC log that Solr writes?

*OP Reply:*  Please find the GC logs and thread dumps at this location
https://drive.google.com/open?id=1slsYkAcsH7OH-7Pma91k6t5T72-tIPlw


The larger GC log was unrecognized by both gcviwer and gceasy.io ... the 
smaller log shows heap usage about 10GB, but it only covers 10 minutes, 
so it's not really conclusive for diagnosis.  The first thing I can 
suggest to try is to reduce the heap size to 12GB ... but I do not know 
if that's actually going to work.  Indexing might require more memory. 
The idea here is to make more memory available to the OS disk cache ... 
with your index size, you're probably going to need to add memory to the 
system (not the heap).



Another observation is that the CPU usage reaches around 70% (through
manual monitoring) when the indexing starts and the merges are observed. It
is well below 50% otherwise.


Indexing will increase load, and that increase is often very 
significant.  Adding memory to the system is your best bet for better 
performance.  I'd want 1TB of memory for a 1.4TB index ... but I know 
that memory sizes that high are extremely expensive, and for most 
servers, not even possible.  512GB or 256GB is more attainable, and 
would have better performance than 48GB.



Also, should something be altered with the mergeScheduler setting ?
"mergeScheduler":{
 "class":"org.apache.lucene.index.ConcurrentMergeScheduler",
 "maxMergeCount":2,
 "maxThreadCount":2},


Do not configure maxThreadCount beyond 1 unless your data is on SSD.  It 
will slow things down a lot due to the fact that standard disks must 
move the disk head to read/write from different locations, and head 
moves take time.  SSD can do I/O from any location without pauses, so 
more threads would probably help performance rather than hurt it.


Increase maxMergeCount to 6 -- at 2, large merges will probably stop 
indexing entirely.  With a larger number, Solr can keep indexing even 
when there's a huge segment merge happening.


Thanks,
Shawn

Re: SolrCloud indexing triggers merges and timeouts

[Rahul Goswami]

Updating the thread with further findings:

So turns out that the nodes hosting Solr are VMs with Virtual disks.
Additionally, a Windows system process (the infamous PID 4) is hogging a
lot of disk. This is indicated by disk reponse times in excess of 100 ms
and a disk drive queue length of 5 which would be considered very high. The
indexing is running in two parallel threads each sending a batch of 50 docs
per request. I would like to believe this is not too high (?). The docs are
not too heavy either, only containing metadata fields.So the disk IO seems
to be the bottleneck at this point, causing commits and merges to take more
time than they should. This is causing update routing to the leader replica
to take more than 10 mins, resulting into read time outs, and eventually
failed updates.
I could not find anything alarming in the GC logs I shared earlier.

Will update the thread with more findings as I have them and the attempted
solutions. At this point I am considering increasing the Jetty timeout and
increasing the distribUpdateConnTimeout to a higher value to let the
indexing proceed slowly but successfully. In the meantime, would greatly
appreciate any other ideas/measures.

Thanks,
Rahul


On Thu, Jun 6, 2019 at 11:00 AM Rahul Goswami  wrote:

> Thank you for your responses. Please find additional details about the
> setup below:
>
> We are using Solr 7.2.1
>
> > I have a solrcloud setup on Windows server with below config:
> > 3 nodes,
> > 24 shards with replication factor 2
> > Each node hosts 16 cores.
>
> 16 CPU cores, or 16 Solr cores?  The info may not be all that useful
> either way, but just in case, it should be clarified.
>
> *OP Reply:* 16 Solr cores (i.e. replicas)
>
> > Index size is 1.4 TB per node
> > Xms 8 GB , Xmx 24 GB
> > Directory factory used is SimpleFSDirectoryFactory
>
> How much total memory in the server?  Is there other software using
> significant levels of memory?
>
> *OP Reply* : Total 48 GB per node... I couldn't see another software
> using a lot of memory.
> I am honestly not sure about the reason for change of directory factory to
> SimpleFSDirectoryFactory. But I was told that with mmap at one point we
> started to see the shared memory usage on Windows go up significantly,
> intermittently freezing the system.
> Could the choice of DirectoryFactory here be a factor for the long
> updates/frequent merges?
>
> > How many total documents (maxDoc, not numDoc) are in that 1.4 TB of
> space?
> *OP Reply:* Also, there are nearly 12.8 million total docs (maxDoc, NOT
> numDoc) in that 1.4 TB space
>
> > Can you share the GC log that Solr writes?
> *OP Reply:*  Please find the GC logs and thread dumps at this location
> https://drive.google.com/open?id=1slsYkAcsH7OH-7Pma91k6t5T72-tIPlw
>
> Another observation is that the CPU usage reaches around 70% (through
> manual monitoring) when the indexing starts and the merges are observed. It
> is well below 50% otherwise.
>
> Also, should something be altered with the mergeScheduler setting ?
> "mergeScheduler":{
> "class":"org.apache.lucene.index.ConcurrentMergeScheduler",
> "maxMergeCount":2,
> "maxThreadCount":2},
>
> Thanks,
> Rahul
>
>
> On Wed, Jun 5, 2019 at 4:24 PM Shawn Heisey  wrote:
>
>> On 6/5/2019 9:39 AM, Rahul Goswami wrote:
>> > I have a solrcloud setup on Windows server with below config:
>> > 3 nodes,
>> > 24 shards with replication factor 2
>> > Each node hosts 16 cores.
>>
>> 16 CPU cores, or 16 Solr cores?  The info may not be all that useful
>> either way, but just in case, it should be clarified.
>>
>> > Index size is 1.4 TB per node
>> > Xms 8 GB , Xmx 24 GB
>> > Directory factory used is SimpleFSDirectoryFactory
>>
>> How much total memory in the server?  Is there other software using
>> significant levels of memory?
>>
>> Why did you opt to change the DirectoryFactory away from Solr's default?
>>   The default is chosen with care ... any other choice will probably
>> result in lower performance.  The default in recent versions of Solr is
>> NRTCachingDirectoryFactory, which uses MMap for file access.
>>
>> http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
>>
>> The screenshot described here might become useful for more in-depth
>> troubleshooting:
>>
>>
>> https://wiki.apache.org/solr/SolrPerformanceProblems#Process_listing_on_Windows
>>
>> How many total documents (maxDoc, not numDoc) are in that 1.4 TB of space?
>>
>> > The cloud is all nice and green for the most part. Only when we start
>> > indexing, within a few seconds, I start seeing Read timeouts and socket
>> > write errors and replica recoveries thereafter. We are indexing in 2
>> > parallel threads in batches of 50 docs per update request. After
>> examining
>> > the thread dump, I see segment merges happening. My understanding is
>> that
>> > this is the cause, and the timeouts and recoveries are the symptoms. Is
>> my
>> > understanding correct? If yes, what steps could I take to help the
>> > 

Re: SolrCloud indexing triggers merges and timeouts

[Rahul Goswami]

Thank you for your responses. Please find additional details about the
setup below:

We are using Solr 7.2.1

> I have a solrcloud setup on Windows server with below config:
> 3 nodes,
> 24 shards with replication factor 2
> Each node hosts 16 cores.

16 CPU cores, or 16 Solr cores?  The info may not be all that useful
either way, but just in case, it should be clarified.

*OP Reply:* 16 Solr cores (i.e. replicas)

> Index size is 1.4 TB per node
> Xms 8 GB , Xmx 24 GB
> Directory factory used is SimpleFSDirectoryFactory

How much total memory in the server?  Is there other software using
significant levels of memory?

*OP Reply* : Total 48 GB per node... I couldn't see another software using
a lot of memory.
I am honestly not sure about the reason for change of directory factory to
SimpleFSDirectoryFactory. But I was told that with mmap at one point we
started to see the shared memory usage on Windows go up significantly,
intermittently freezing the system.
Could the choice of DirectoryFactory here be a factor for the long
updates/frequent merges?

> How many total documents (maxDoc, not numDoc) are in that 1.4 TB of
space?
*OP Reply:* Also, there are nearly 12.8 million total docs (maxDoc, NOT
numDoc) in that 1.4 TB space

> Can you share the GC log that Solr writes?
*OP Reply:*  Please find the GC logs and thread dumps at this location
https://drive.google.com/open?id=1slsYkAcsH7OH-7Pma91k6t5T72-tIPlw

Another observation is that the CPU usage reaches around 70% (through
manual monitoring) when the indexing starts and the merges are observed. It
is well below 50% otherwise.

Also, should something be altered with the mergeScheduler setting ?
"mergeScheduler":{
"class":"org.apache.lucene.index.ConcurrentMergeScheduler",
"maxMergeCount":2,
"maxThreadCount":2},

Thanks,
Rahul


On Wed, Jun 5, 2019 at 4:24 PM Shawn Heisey  wrote:

> On 6/5/2019 9:39 AM, Rahul Goswami wrote:
> > I have a solrcloud setup on Windows server with below config:
> > 3 nodes,
> > 24 shards with replication factor 2
> > Each node hosts 16 cores.
>
> 16 CPU cores, or 16 Solr cores?  The info may not be all that useful
> either way, but just in case, it should be clarified.
>
> > Index size is 1.4 TB per node
> > Xms 8 GB , Xmx 24 GB
> > Directory factory used is SimpleFSDirectoryFactory
>
> How much total memory in the server?  Is there other software using
> significant levels of memory?
>
> Why did you opt to change the DirectoryFactory away from Solr's default?
>   The default is chosen with care ... any other choice will probably
> result in lower performance.  The default in recent versions of Solr is
> NRTCachingDirectoryFactory, which uses MMap for file access.
>
> http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
>
> The screenshot described here might become useful for more in-depth
> troubleshooting:
>
>
> https://wiki.apache.org/solr/SolrPerformanceProblems#Process_listing_on_Windows
>
> How many total documents (maxDoc, not numDoc) are in that 1.4 TB of space?
>
> > The cloud is all nice and green for the most part. Only when we start
> > indexing, within a few seconds, I start seeing Read timeouts and socket
> > write errors and replica recoveries thereafter. We are indexing in 2
> > parallel threads in batches of 50 docs per update request. After
> examining
> > the thread dump, I see segment merges happening. My understanding is that
> > this is the cause, and the timeouts and recoveries are the symptoms. Is
> my
> > understanding correct? If yes, what steps could I take to help the
> > situation. I do see that the difference between "Num Docs" and "Max Docs"
> > is about 20%.
>
> Segment merges are a completely normal part of Lucene's internal
> operation.  They should never cause problems like you have described.
>
> My best guess is that a 24GB heap is too small.  Or possibly WAY too
> large, although with the index size you have mentioned, that seems
> unlikely.
>
> Can you share the GC log that Solr writes?  The problem should occur
> during the timeframe covered by the log, and the log should be as large
> as possible.  You'll need to use a file sharing site -- attaching it to
> an email is not going to work.
>
> What version of Solr?
>
> Thanks,
> Shawn
>

Re: SolrCloud indexing triggers merges and timeouts

[Shawn Heisey]

On 6/5/2019 9:39 AM, Rahul Goswami wrote:

I have a solrcloud setup on Windows server with below config:
3 nodes,
24 shards with replication factor 2
Each node hosts 16 cores.


16 CPU cores, or 16 Solr cores?  The info may not be all that useful 
either way, but just in case, it should be clarified.



Index size is 1.4 TB per node
Xms 8 GB , Xmx 24 GB
Directory factory used is SimpleFSDirectoryFactory


How much total memory in the server?  Is there other software using 
significant levels of memory?


Why did you opt to change the DirectoryFactory away from Solr's default? 
 The default is chosen with care ... any other choice will probably 
result in lower performance.  The default in recent versions of Solr is 
NRTCachingDirectoryFactory, which uses MMap for file access.


http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html

The screenshot described here might become useful for more in-depth 
troubleshooting:


https://wiki.apache.org/solr/SolrPerformanceProblems#Process_listing_on_Windows

How many total documents (maxDoc, not numDoc) are in that 1.4 TB of space?


The cloud is all nice and green for the most part. Only when we start
indexing, within a few seconds, I start seeing Read timeouts and socket
write errors and replica recoveries thereafter. We are indexing in 2
parallel threads in batches of 50 docs per update request. After examining
the thread dump, I see segment merges happening. My understanding is that
this is the cause, and the timeouts and recoveries are the symptoms. Is my
understanding correct? If yes, what steps could I take to help the
situation. I do see that the difference between "Num Docs" and "Max Docs"
is about 20%.


Segment merges are a completely normal part of Lucene's internal 
operation.  They should never cause problems like you have described.


My best guess is that a 24GB heap is too small.  Or possibly WAY too 
large, although with the index size you have mentioned, that seems unlikely.


Can you share the GC log that Solr writes?  The problem should occur 
during the timeframe covered by the log, and the log should be as large 
as possible.  You'll need to use a file sharing site -- attaching it to 
an email is not going to work.


What version of Solr?

Thanks,
Shawn

Re: SolrCloud indexing triggers merges and timeouts

[Walter Underwood]

Yes, set Xmx and Xms the same.

We run an 8 GB heap for all our clusters. Unless you are doing some really
memory-intensive stuff like faceting, 8 GB should be fine.

wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/  (my blog)

> On Jun 5, 2019, at 1:05 PM, Gus Heck  wrote:
> 
> Probably not a solution, but so.ething I notice off the bat... generally
> you want Xmx and Xms set to the same value so the jvm doesn't have to spend
> time asking for more and more memory, and also reduce the chance that the
> memory is not available by the time solr needs it.
> 
> On Wed, Jun 5, 2019, 11:39 AM Rahul Goswami  wrote:
> 
>> Hello,
>> I have a solrcloud setup on Windows server with below config:
>> 3 nodes,
>> 24 shards with replication factor 2
>> Each node hosts 16 cores.
>> 
>> Index size is 1.4 TB per node
>> Xms 8 GB , Xmx 24 GB
>> Directory factory used is SimpleFSDirectoryFactory
>> 
>> The cloud is all nice and green for the most part. Only when we start
>> indexing, within a few seconds, I start seeing Read timeouts and socket
>> write errors and replica recoveries thereafter. We are indexing in 2
>> parallel threads in batches of 50 docs per update request. After examining
>> the thread dump, I see segment merges happening. My understanding is that
>> this is the cause, and the timeouts and recoveries are the symptoms. Is my
>> understanding correct? If yes, what steps could I take to help the
>> situation. I do see that the difference between "Num Docs" and "Max Docs"
>> is about 20%.
>> 
>> Would appreciate your help.
>> 
>> Thanks,
>> Rahul
>> 

Re: SolrCloud indexing triggers merges and timeouts

[Gus Heck]

Probably not a solution, but so.ething I notice off the bat... generally
you want Xmx and Xms set to the same value so the jvm doesn't have to spend
time asking for more and more memory, and also reduce the chance that the
memory is not available by the time solr needs it.

On Wed, Jun 5, 2019, 11:39 AM Rahul Goswami  wrote:

> Hello,
> I have a solrcloud setup on Windows server with below config:
> 3 nodes,
> 24 shards with replication factor 2
> Each node hosts 16 cores.
>
> Index size is 1.4 TB per node
> Xms 8 GB , Xmx 24 GB
> Directory factory used is SimpleFSDirectoryFactory
>
> The cloud is all nice and green for the most part. Only when we start
> indexing, within a few seconds, I start seeing Read timeouts and socket
> write errors and replica recoveries thereafter. We are indexing in 2
> parallel threads in batches of 50 docs per update request. After examining
> the thread dump, I see segment merges happening. My understanding is that
> this is the cause, and the timeouts and recoveries are the symptoms. Is my
> understanding correct? If yes, what steps could I take to help the
> situation. I do see that the difference between "Num Docs" and "Max Docs"
> is about 20%.
>
> Would appreciate your help.
>
> Thanks,
> Rahul
>

Re: SolrCloud indexing

[Erick Erickson]

BTW, the effort you put into creating a process to totally replace
your index won't be wasted. I pretty much guarantee that you'll have
to re-index everything upon occasion if for no other reason than the
requirements will change. "Oh, you know that field we said never would
be used for phrase searches? We changed our mind". "We decided
to support 14 new use cases." ;)...

On Sun, Apr 15, 2018 at 10:32 PM, Moshe Recanati | KMS <mos...@kmslh.com> wrote:
> Hi Shawn,
> Thank you.
> I just need to run full indexing due to massive changes in the document.
>
> Regards,
> Moshe Recanati
> CTO
> Mobile  + 972-52-6194481
> Skype:  recanati
>
> More at:  www.kmslh.com | LinkedIn | FB
>
>
> -Original Message-
> From: Shawn Heisey <apa...@elyograg.org>
> Sent: Sunday, April 15, 2018 8:23 PM
> To: solr-user@lucene.apache.org
> Subject: Re: SolrCloud indexing
>
> On 4/15/2018 1:22 AM, Moshe Recanati | KMS wrote:
>>
>> We’re using SolrCloud as part of our product solution for High
>> Availability.
>>
>> During upgrade of a version we need to run full index build on our
>> Solr data.
>>
>
> What are you upgrading?  If it's Solr, you should pause/stop indexing while 
> you do the upgrade.  You'll have to stop Solr processes to upgrade them, and 
> even if you're using an external load balancer, it takes a little bit of time 
> for failover to occur.
>
> It would be up to your indexing software to handle errors in that situation.  
> There is nothing that Solr can do about that.  If your indexing software 
> correctly detects and handles errors, then you might be able to restart Solr 
> instances without a problem.
>
>> I would like to know if as part of SolrCloud we can manage it and make
>> sure that items are available during the index so only once specific
>> item is indexed it’s changing with no affect on end-user.
>>
>
> I can't decipher exactly what you're asking here.
>
> Thanks,
> Shawn
>

RE: SolrCloud indexing

[Moshe Recanati | KMS]

Hi Shawn,
Thank you.
I just need to run full indexing due to massive changes in the document.

Regards,
Moshe Recanati
CTO
Mobile  + 972-52-6194481
Skype    :  recanati

More at:  www.kmslh.com | LinkedIn | FB


-Original Message-
From: Shawn Heisey <apa...@elyograg.org> 
Sent: Sunday, April 15, 2018 8:23 PM
To: solr-user@lucene.apache.org
Subject: Re: SolrCloud indexing

On 4/15/2018 1:22 AM, Moshe Recanati | KMS wrote:
>
> We’re using SolrCloud as part of our product solution for High 
> Availability.
>
> During upgrade of a version we need to run full index build on our 
> Solr data.
>

What are you upgrading?  If it's Solr, you should pause/stop indexing while you 
do the upgrade.  You'll have to stop Solr processes to upgrade them, and even 
if you're using an external load balancer, it takes a little bit of time for 
failover to occur.

It would be up to your indexing software to handle errors in that situation.  
There is nothing that Solr can do about that.  If your indexing software 
correctly detects and handles errors, then you might be able to restart Solr 
instances without a problem.

> I would like to know if as part of SolrCloud we can manage it and make 
> sure that items are available during the index so only once specific 
> item is indexed it’s changing with no affect on end-user.
>

I can't decipher exactly what you're asking here.

Thanks,
Shawn

RE: SolrCloud indexing

[Moshe Recanati | KMS]

Hi Erick,
Thank you very much.
I'll check it out.

Regards,
Moshe Recanati
CTO
Mobile  + 972-52-6194481
Skype    :  recanati

More at:  www.kmslh.com | LinkedIn | FB


-Original Message-
From: Erick Erickson <erickerick...@gmail.com> 
Sent: Monday, April 16, 2018 7:01 AM
To: solr-user <solr-user@lucene.apache.org>
Subject: Re: SolrCloud indexing

I think you're saying you want to prove out the upgrade in some kind of test 
setup then switch live traffic. What's commonly used for that is collection 
aliasing. You just create a new collection and populate it and check it out. 
When you're satisfied that it's doing what you want, use the collections API 
CREATEALIAS command to seamlessly switch.
Here's the sequence:
old_collection is active
create new_collection, index to it and make sure it is doing what you want 
CREATEALIAS pointing old_collection at new_colletion.

The advantage here is that you have as long as you want to verify your new 
collection is fine. You can also switch back if you need to.

The disadvantage is that you effectively need extra hardware. You can mitigate 
this somewhat by bringing up your new collection with a limited number of 
replicas and after switching you delete replicas from the old collection and 
add them to the new one. Be very very careful here though that your collection 
commands
(ADDREPLICA/DELETEREPLICA) don't actually operate on the new collection!

Best,
Erick

On Sun, Apr 15, 2018 at 10:22 AM, Shawn Heisey <apa...@elyograg.org> wrote:
> On 4/15/2018 1:22 AM, Moshe Recanati | KMS wrote:
>>
>>
>> We’re using SolrCloud as part of our product solution for High 
>> Availability.
>>
>> During upgrade of a version we need to run full index build on our 
>> Solr data.
>>
>
> What are you upgrading?  If it's Solr, you should pause/stop indexing 
> while you do the upgrade.  You'll have to stop Solr processes to 
> upgrade them, and even if you're using an external load balancer, it 
> takes a little bit of time for failover to occur.
>
> It would be up to your indexing software to handle errors in that situation.
> There is nothing that Solr can do about that.  If your indexing 
> software correctly detects and handles errors, then you might be able 
> to restart Solr instances without a problem.
>
>> I would like to know if as part of SolrCloud we can manage it and 
>> make sure that items are available during the index so only once 
>> specific item is indexed it’s changing with no affect on end-user.
>>
>
> I can't decipher exactly what you're asking here.
>
> Thanks,
> Shawn
>

Re: SolrCloud indexing

[Erick Erickson]

I think you're saying you want to prove out the upgrade in some kind
of test setup then switch live traffic. What's commonly used for that
is collection aliasing. You just create a new collection and populate
it and check it out. When you're satisfied that it's doing what you
want, use the collections API CREATEALIAS command to seamlessly
switch.
Here's the sequence:
old_collection is active
create new_collection, index to it and make sure it is doing what you want
CREATEALIAS pointing old_collection at new_colletion.

The advantage here is that you have as long as you want to verify your
new collection is fine. You can also switch back if you need to.

The disadvantage is that you effectively need extra hardware. You can
mitigate this somewhat by bringing up your new collection with a
limited number of replicas and after switching you delete replicas
from the old collection and add them to the new one. Be very very
careful here though that your collection commands
(ADDREPLICA/DELETEREPLICA) don't actually operate on the new
collection!

Best,
Erick

On Sun, Apr 15, 2018 at 10:22 AM, Shawn Heisey  wrote:
> On 4/15/2018 1:22 AM, Moshe Recanati | KMS wrote:
>>
>>
>> We’re using SolrCloud as part of our product solution for High
>> Availability.
>>
>> During upgrade of a version we need to run full index build on our Solr
>> data.
>>
>
> What are you upgrading?  If it's Solr, you should pause/stop indexing while
> you do the upgrade.  You'll have to stop Solr processes to upgrade them, and
> even if you're using an external load balancer, it takes a little bit of
> time for failover to occur.
>
> It would be up to your indexing software to handle errors in that situation.
> There is nothing that Solr can do about that.  If your indexing software
> correctly detects and handles errors, then you might be able to restart Solr
> instances without a problem.
>
>> I would like to know if as part of SolrCloud we can manage it and make
>> sure that items are available during the index so only once specific item is
>> indexed it’s changing with no affect on end-user.
>>
>
> I can't decipher exactly what you're asking here.
>
> Thanks,
> Shawn
>

Re: SolrCloud indexing

[Shawn Heisey]

On 4/15/2018 1:22 AM, Moshe Recanati | KMS wrote:


We’re using SolrCloud as part of our product solution for High 
Availability.


During upgrade of a version we need to run full index build on our 
Solr data.




What are you upgrading?  If it's Solr, you should pause/stop indexing 
while you do the upgrade.  You'll have to stop Solr processes to upgrade 
them, and even if you're using an external load balancer, it takes a 
little bit of time for failover to occur.


It would be up to your indexing software to handle errors in that 
situation.  There is nothing that Solr can do about that.  If your 
indexing software correctly detects and handles errors, then you might 
be able to restart Solr instances without a problem.


I would like to know if as part of SolrCloud we can manage it and make 
sure that items are available during the index so only once specific 
item is indexed it’s changing with no affect on end-user.




I can't decipher exactly what you're asking here.

Thanks,
Shawn

Re: SolrCloud indexing -- 2 collections, 2 indexes, sharing the same nodes possible?

[Susheel Kumar]

1) As regards naming of the shards: Is using the same naming for the shards
o.k. in this constellation? I.e. does it create trouble to have e.g.
"Shard001", "Shard002", etc. in collection1 and "Shard001", "Shard002",
etc. as well in collection2?
>> The default naming convention for shards would be
"_shard#_replica#".  So complete name will be different
like coll1_shard1_replica1 and coll2_shard1_replica1

2) Performance: In my current single collection setup, I have 2 shards per
node. After creating the second collection, there will be 4 shards per
node. Do I have to edit the RAM per node value (raise the -m parameter when
starting the node)? In my case, I am quite sure that the collections will
never be queried simultaneously. So will the "running but idle" collection
slow me down?
>> Its up to you how you setup JVM.  You can have one JVM instance running
on port assume 8080 and have multiple shards/collections or you can setup
two JVM/solr instances on a node running on different ports like 8080 and
8081 etc. I would suggest to start and test with one JVM and setup multiple
collections until run into performance bottleneck and then split into JVM
with different heaps etc.



On Wed, Aug 30, 2017 at 12:42 PM, Johannes Knaus <kn...@mpdl.mpg.de> wrote:

> Thank you, Susheel, for the quick response.
>
> So, that means that when I create a new collection, it shards will be
> newly created at each node, right?
> Thus, if I have two collections with
> numShards=38,
> maxShardsPerNode=2 and
> replicationFactor=2
> on my 38 nodes, then this would result in each node "hosting" 4 shards
> (two from each collection).
>
> If this is correct, I have two follow up questions:
>
> 1) As regards naming of the shards: Is using the same naming for the
> shards o.k. in this constellation? I.e. does it create trouble to have e.g.
> "Shard001", "Shard002", etc. in collection1 and "Shard001", "Shard002",
> etc. as well in collection2?
>
> 2) Performance: In my current single collection setup, I have 2 shards per
> node. After creating the second collection, there will be 4 shards per
> node. Do I have to edit the RAM per node value (raise the -m parameter when
> starting the node)? In my case, I am quite sure that the collections will
> never be queried simultaneously. So will the "running but idle" collection
> slow me down?
>
> Johannes
>
> -Ursprüngliche Nachricht-
> Von: Susheel Kumar [mailto:susheel2...@gmail.com]
> Gesendet: Mittwoch, 30. August 2017 17:36
> An: solr-user@lucene.apache.org
> Betreff: Re: SolrCloud indexing -- 2 collections, 2 indexes, sharing the
> same nodes possible?
>
> Yes, absolutely.  You can create as many as collections you need (like you
> would create table in relational world).
>
> On Wed, Aug 30, 2017 at 10:13 AM, Johannes Knaus <kn...@mpdl.mpg.de>
> wrote:
>
> > I have a working SolrCloud-Setup with 38 nodes with a collection
> > spanning over these nodes with 2 shards per node and replication
> > factor 2 and a router field.
> >
> > Now I got some new data for indexing which has the same structure and
> > size as my existing index in the described collection.
> > However, although it has the same structure the new data to be indexed
> > should not be mixed with the old data.
> >
> > Do I have create another 38 new nodes and a new collection and index
> > the new data or is there a better / more efficient way I could use the
> > existing nodes?
> > Is it possible that the 2 collections could share the 38 nodes without
> > the indexes being mixed?
> >
> > Thanks for your help.
> >
> > Johannes
> >
>

Re: SolrCloud indexing -- 2 collections, 2 indexes, sharing the same nodes possible?

[Johannes Knaus]

Thank you, Susheel, for the quick response.

So, that means that when I create a new collection, it shards will be newly 
created at each node, right?
Thus, if I have two collections with 
numShards=38, 
maxShardsPerNode=2 and 
replicationFactor=2 
on my 38 nodes, then this would result in each node "hosting" 4 shards (two 
from each collection).

If this is correct, I have two follow up questions:

1) As regards naming of the shards: Is using the same naming for the shards 
o.k. in this constellation? I.e. does it create trouble to have e.g. 
"Shard001", "Shard002", etc. in collection1 and "Shard001", "Shard002", etc. as 
well in collection2?

2) Performance: In my current single collection setup, I have 2 shards per 
node. After creating the second collection, there will be 4 shards per node. Do 
I have to edit the RAM per node value (raise the -m parameter when starting the 
node)? In my case, I am quite sure that the collections will never be queried 
simultaneously. So will the "running but idle" collection slow me down?

Johannes

-Ursprüngliche Nachricht-
Von: Susheel Kumar [mailto:susheel2...@gmail.com] 
Gesendet: Mittwoch, 30. August 2017 17:36
An: solr-user@lucene.apache.org
Betreff: Re: SolrCloud indexing -- 2 collections, 2 indexes, sharing the same 
nodes possible?

Yes, absolutely.  You can create as many as collections you need (like you 
would create table in relational world).

On Wed, Aug 30, 2017 at 10:13 AM, Johannes Knaus <kn...@mpdl.mpg.de> wrote:

> I have a working SolrCloud-Setup with 38 nodes with a collection 
> spanning over these nodes with 2 shards per node and replication 
> factor 2 and a router field.
>
> Now I got some new data for indexing which has the same structure and 
> size as my existing index in the described collection.
> However, although it has the same structure the new data to be indexed 
> should not be mixed with the old data.
>
> Do I have create another 38 new nodes and a new collection and index 
> the new data or is there a better / more efficient way I could use the 
> existing nodes?
> Is it possible that the 2 collections could share the 38 nodes without 
> the indexes being mixed?
>
> Thanks for your help.
>
> Johannes
>

Re: SolrCloud indexing -- 2 collections, 2 indexes, sharing the same nodes possible?

[Susheel Kumar]

Yes, absolutely.  You can create as many as collections you need (like you
would create table in relational world).

On Wed, Aug 30, 2017 at 10:13 AM, Johannes Knaus  wrote:

> I have a working SolrCloud-Setup with 38 nodes with a collection spanning
> over these nodes with 2 shards per node and replication factor 2 and a
> router field.
>
> Now I got some new data for indexing which has the same structure and size
> as my existing index in the described collection.
> However, although it has the same structure the new data to be indexed
> should not be mixed with the old data.
>
> Do I have create another 38 new nodes and a new collection and index the
> new data or is there a better / more efficient way I could use the existing
> nodes?
> Is it possible that the 2 collections could share the 38 nodes without the
> indexes being mixed?
>
> Thanks for your help.
>
> Johannes
>

Re: SolrCloud indexing

[Bill Au]

Thanks for the reply.

Actually in our case we want the timestamp to be populated locally on each
node in the SolrCloud cluster.  We want to see if there is any delay in the
document being distributed within the cluster.  Just want to confirm that
the timestamp can be use for that purpose.

Bill

On Sat, May 9, 2015 at 11:37 PM, Shawn Heisey apa...@elyograg.org wrote:

 On 5/9/2015 8:41 PM, Bill Au wrote:
  Is the behavior of document being indexed independently on each node in a
  SolrCloud cluster new in 5.x or is that true in 4.x also?
 
  If the document is indexed independently on each node, then if I query
 the
  document from each node directly, a timestamp could hold different values
  since the document is indexed independently, right?
 
  field name=timestamp type=date indexed=true stored=true
  default=NOW /

 SolrCloud has had that behavior from day one, when it was released in
 version 4.0.  You are correct that it can result in a different
 timestamp on each replica if the default comes from schema.xml.

 I am pretty sure that the solution for this problem is to set up an
 update processor chain that includes TimestampUpdateProcessorFactory to
 populate the timestamp field before the document is distributed to each
 replica.

 https://cwiki.apache.org/confluence/display/solr/Update+Request+Processors

 Thanks,
 Shawn

Re: SolrCloud indexing

[Shawn Heisey]

On 5/9/2015 8:41 PM, Bill Au wrote:
 Is the behavior of document being indexed independently on each node in a
 SolrCloud cluster new in 5.x or is that true in 4.x also?
 
 If the document is indexed independently on each node, then if I query the
 document from each node directly, a timestamp could hold different values
 since the document is indexed independently, right?
 
 field name=timestamp type=date indexed=true stored=true
 default=NOW /

SolrCloud has had that behavior from day one, when it was released in
version 4.0.  You are correct that it can result in a different
timestamp on each replica if the default comes from schema.xml.

I am pretty sure that the solution for this problem is to set up an
update processor chain that includes TimestampUpdateProcessorFactory to
populate the timestamp field before the document is distributed to each
replica.

https://cwiki.apache.org/confluence/display/solr/Update+Request+Processors

Thanks,
Shawn

Re: SolrCloud indexing

[Bill Au]

Is the behavior of document being indexed independently on each node in a
SolrCloud cluster new in 5.x or is that true in 4.x also?

If the document is indexed independently on each node, then if I query the
document from each node directly, a timestamp could hold different values
since the document is indexed independently, right?

field name=timestamp type=date indexed=true stored=true
default=NOW /

Bill

On Fri, May 8, 2015 at 6:39 PM, Vincenzo D'Amore v.dam...@gmail.com wrote:

 I have just added a comment to the CWiki.
 Thanks again for your prompt answer Erick.

 Best,
 Vincenzo

 On Fri, May 8, 2015 at 12:39 AM, Erick Erickson erickerick...@gmail.com
 wrote:

  bq: ...forwards the index notation to itself and any replicas...
 
  That's just odd phrasing.
 
  All that means is that the document sent through the indexing process
  on the leader and all followers for a shard and
  is indexed independently on each.
 
  This is as opposed to the old master/slave situation where the master
  indexed the doc, but the slave got the indexed
  version as part of a segment when it replicated.
 
  Could you add a comment to the CWiki calling the phrasing out? It
  really is a bit mysterious.
 
  Best,
  Erick
 
  On Thu, May 7, 2015 at 2:18 PM, Vincenzo D'Amore v.dam...@gmail.com
  wrote:
   Thanks Shawn.
  
   Just to make the picture more clear, I'm trying to understand why a 3
  node
   solrcloud cluster and a old style solr server take same time to index
  same
   documents.
  
   But in the wiki is written:
  
   If the machine is a leader, SolrCloud determines which shard the
 document
   should go to, forwards the document the leader for that shard, indexes
  the
   document for this shard, and *forwards the index notation to itself
 and
   any replicas*.
  
  
  
 
 https://cwiki.apache.org/confluence/display/solr/Shards+and+Indexing+Data+in+SolrCloud
  
  
   Could you please explain what does it mean forwards the index
 notation
  ?
  
   On the other hand, on solrcloud I have 3 shards and 2 replicas for each
   shard. So, every node is indexing all the documents and this explains
 why
   solrcloud consumes same time compared to an old-style solr server.
  
  
  
   On Thu, May 7, 2015 at 3:08 PM, Shawn Heisey apa...@elyograg.org
  wrote:
  
   On 5/7/2015 3:04 AM, Vincenzo D'Amore wrote:
Thanks Erick. I'm not sure I got your answer.
   
I try to recap, when the raw document has to be indexed, it will be
forwarded to shard leader. Shard leader indexes the document for
 that
shard, and then forwards the indexed document to any replicas.
   
I want just be sure that when the raw document is forwarded from the
   leader
to the replicas it will be indexed only one time on the shard
 leader.
   From
what I understand replicas do not indexes, only the leader indexes.
  
   The document is indexed by all replicas.  There is no way to forward
 the
   indexed document, it can only forward the source document ... so each
   replica must index it independently.
  
   The old-style master-slave replication (which existed long before
   SolrCloud) copies the finished Lucene segments, so only the master
   actually does indexing.
  
   SolrCloud doesn't have a master, only multiple replicas, one of which
 is
   elected leader, and replication only comes into the picture if
 there's a
   serious problem and Solr determines that it can't use the transaction
   log to recover the index.
  
   Thanks,
   Shawn
  
  
  
  
   --
   Vincenzo D'Amore
   email: v.dam...@gmail.com
   skype: free.dev
   mobile: +39 349 8513251
 



 --
 Vincenzo D'Amore
 email: v.dam...@gmail.com
 skype: free.dev
 mobile: +39 349 8513251

Re: SolrCloud indexing

[Vincenzo D'Amore]

I have just added a comment to the CWiki.
Thanks again for your prompt answer Erick.

Best,
Vincenzo

On Fri, May 8, 2015 at 12:39 AM, Erick Erickson erickerick...@gmail.com
wrote:

 bq: ...forwards the index notation to itself and any replicas...

 That's just odd phrasing.

 All that means is that the document sent through the indexing process
 on the leader and all followers for a shard and
 is indexed independently on each.

 This is as opposed to the old master/slave situation where the master
 indexed the doc, but the slave got the indexed
 version as part of a segment when it replicated.

 Could you add a comment to the CWiki calling the phrasing out? It
 really is a bit mysterious.

 Best,
 Erick

 On Thu, May 7, 2015 at 2:18 PM, Vincenzo D'Amore v.dam...@gmail.com
 wrote:
  Thanks Shawn.
 
  Just to make the picture more clear, I'm trying to understand why a 3
 node
  solrcloud cluster and a old style solr server take same time to index
 same
  documents.
 
  But in the wiki is written:
 
  If the machine is a leader, SolrCloud determines which shard the document
  should go to, forwards the document the leader for that shard, indexes
 the
  document for this shard, and *forwards the index notation to itself and
  any replicas*.
 
 
 
 https://cwiki.apache.org/confluence/display/solr/Shards+and+Indexing+Data+in+SolrCloud
 
 
  Could you please explain what does it mean forwards the index notation
 ?
 
  On the other hand, on solrcloud I have 3 shards and 2 replicas for each
  shard. So, every node is indexing all the documents and this explains why
  solrcloud consumes same time compared to an old-style solr server.
 
 
 
  On Thu, May 7, 2015 at 3:08 PM, Shawn Heisey apa...@elyograg.org
 wrote:
 
  On 5/7/2015 3:04 AM, Vincenzo D'Amore wrote:
   Thanks Erick. I'm not sure I got your answer.
  
   I try to recap, when the raw document has to be indexed, it will be
   forwarded to shard leader. Shard leader indexes the document for that
   shard, and then forwards the indexed document to any replicas.
  
   I want just be sure that when the raw document is forwarded from the
  leader
   to the replicas it will be indexed only one time on the shard leader.
  From
   what I understand replicas do not indexes, only the leader indexes.
 
  The document is indexed by all replicas.  There is no way to forward the
  indexed document, it can only forward the source document ... so each
  replica must index it independently.
 
  The old-style master-slave replication (which existed long before
  SolrCloud) copies the finished Lucene segments, so only the master
  actually does indexing.
 
  SolrCloud doesn't have a master, only multiple replicas, one of which is
  elected leader, and replication only comes into the picture if there's a
  serious problem and Solr determines that it can't use the transaction
  log to recover the index.
 
  Thanks,
  Shawn
 
 
 
 
  --
  Vincenzo D'Amore
  email: v.dam...@gmail.com
  skype: free.dev
  mobile: +39 349 8513251




-- 
Vincenzo D'Amore
email: v.dam...@gmail.com
skype: free.dev
mobile: +39 349 8513251

Re: SolrCloud indexing

[Vincenzo D'Amore]

Thanks Shawn.

Just to make the picture more clear, I'm trying to understand why a 3 node
solrcloud cluster and a old style solr server take same time to index same
documents.

But in the wiki is written:

If the machine is a leader, SolrCloud determines which shard the document
 should go to, forwards the document the leader for that shard, indexes the
 document for this shard, and *forwards the index notation to itself and
 any replicas*.


https://cwiki.apache.org/confluence/display/solr/Shards+and+Indexing+Data+in+SolrCloud


Could you please explain what does it mean forwards the index notation ?

On the other hand, on solrcloud I have 3 shards and 2 replicas for each
shard. So, every node is indexing all the documents and this explains why
solrcloud consumes same time compared to an old-style solr server.



On Thu, May 7, 2015 at 3:08 PM, Shawn Heisey apa...@elyograg.org wrote:

 On 5/7/2015 3:04 AM, Vincenzo D'Amore wrote:
  Thanks Erick. I'm not sure I got your answer.
 
  I try to recap, when the raw document has to be indexed, it will be
  forwarded to shard leader. Shard leader indexes the document for that
  shard, and then forwards the indexed document to any replicas.
 
  I want just be sure that when the raw document is forwarded from the
 leader
  to the replicas it will be indexed only one time on the shard leader.
 From
  what I understand replicas do not indexes, only the leader indexes.

 The document is indexed by all replicas.  There is no way to forward the
 indexed document, it can only forward the source document ... so each
 replica must index it independently.

 The old-style master-slave replication (which existed long before
 SolrCloud) copies the finished Lucene segments, so only the master
 actually does indexing.

 SolrCloud doesn't have a master, only multiple replicas, one of which is
 elected leader, and replication only comes into the picture if there's a
 serious problem and Solr determines that it can't use the transaction
 log to recover the index.

 Thanks,
 Shawn




-- 
Vincenzo D'Amore
email: v.dam...@gmail.com
skype: free.dev
mobile: +39 349 8513251

Re: SolrCloud indexing

[Vincenzo D'Amore]

Thanks Erick. I'm not sure I got your answer.

I try to recap, when the raw document has to be indexed, it will be
forwarded to shard leader. Shard leader indexes the document for that
shard, and then forwards the indexed document to any replicas.

I want just be sure that when the raw document is forwarded from the leader
to the replicas it will be indexed only one time on the shard leader. From
what I understand replicas do not indexes, only the leader indexes.

Best regards,
Vincenzo


On Wed, May 6, 2015 at 3:07 AM, Erick Erickson erickerick...@gmail.com
wrote:

 bq: Does it mean that all the indexing is done by the leaders in one node?

 no. The raw document is forwarded from the leader to the replica and
 it's indexed on all the nodes. The leader has a little bit of extra
 work to do routing the docs, but that's it. Shouldn't be a problem
 with 3 shards.

 bq: If so, how do I distribute the indexing (the shard leaders) across
 nodes?

 You don't really need to bother I don't think, especially if you don't
 see significantly higher CPU utilization on the leader. If you
 absolutely MUST distribute leadership, see the Collections API and the
 REBALANCELEADERS and BALANCESHARDUNIQUE (Solr 5.1 only) but frankly I
 wouldn't worry about it unless and until you had demonstrated need.

 Best,
 Erick

 On Tue, May 5, 2015 at 6:28 AM, Vincenzo D'Amore v.dam...@gmail.com
 wrote:
  Hi all,
 
  I have 3 nodes and there are 3 shards but looking at solrcloud admin I
 see
  that all the leaders are on the same node.
 
  If I understood well looking at  solr documentation
  
 https://cwiki.apache.org/confluence/display/solr/Shards+and+Indexing+Data+in+SolrCloud
 
  :
 
  When a document is sent to a machine for indexing, the system first
  determines if the machine is a replica or a leader.
  If the machine is a replica, the document is forwarded to the leader for
  processing.
  If the machine is a leader, SolrCloud determines which shard the
 document
  should go to, forwards the document the leader for that shard, indexes
 the
  document for this shard, and forwards the index notation to itself and
 any
  replicas.
 
 
  So I have 3 nodes, with 3 shards and 2 replicas of each shard.
 
 
 http://picpaste.com/pics/Screen_Shot_2015-05-05_at_15.19.54-Xp8uztpt.1430832218.png
 
  Does it mean that all the indexing is done by the leaders in one node? If
  so, how do I distribute the indexing (the shard leaders) across nodes?
 
 
  --
  Vincenzo D'Amore
  email: v.dam...@gmail.com
  skype: free.dev
  mobile: +39 349 8513251




-- 
Vincenzo D'Amore
email: v.dam...@gmail.com
skype: free.dev
mobile: +39 349 8513251

Re: SolrCloud indexing

[Erick Erickson]

bq: ...forwards the index notation to itself and any replicas...

That's just odd phrasing.

All that means is that the document sent through the indexing process
on the leader and all followers for a shard and
is indexed independently on each.

This is as opposed to the old master/slave situation where the master
indexed the doc, but the slave got the indexed
version as part of a segment when it replicated.

Could you add a comment to the CWiki calling the phrasing out? It
really is a bit mysterious.

Best,
Erick

On Thu, May 7, 2015 at 2:18 PM, Vincenzo D'Amore v.dam...@gmail.com wrote:
 Thanks Shawn.

 Just to make the picture more clear, I'm trying to understand why a 3 node
 solrcloud cluster and a old style solr server take same time to index same
 documents.

 But in the wiki is written:

 If the machine is a leader, SolrCloud determines which shard the document
 should go to, forwards the document the leader for that shard, indexes the
 document for this shard, and *forwards the index notation to itself and
 any replicas*.


 https://cwiki.apache.org/confluence/display/solr/Shards+and+Indexing+Data+in+SolrCloud


 Could you please explain what does it mean forwards the index notation ?

 On the other hand, on solrcloud I have 3 shards and 2 replicas for each
 shard. So, every node is indexing all the documents and this explains why
 solrcloud consumes same time compared to an old-style solr server.



 On Thu, May 7, 2015 at 3:08 PM, Shawn Heisey apa...@elyograg.org wrote:

 On 5/7/2015 3:04 AM, Vincenzo D'Amore wrote:
  Thanks Erick. I'm not sure I got your answer.
 
  I try to recap, when the raw document has to be indexed, it will be
  forwarded to shard leader. Shard leader indexes the document for that
  shard, and then forwards the indexed document to any replicas.
 
  I want just be sure that when the raw document is forwarded from the
 leader
  to the replicas it will be indexed only one time on the shard leader.
 From
  what I understand replicas do not indexes, only the leader indexes.

 The document is indexed by all replicas.  There is no way to forward the
 indexed document, it can only forward the source document ... so each
 replica must index it independently.

 The old-style master-slave replication (which existed long before
 SolrCloud) copies the finished Lucene segments, so only the master
 actually does indexing.

 SolrCloud doesn't have a master, only multiple replicas, one of which is
 elected leader, and replication only comes into the picture if there's a
 serious problem and Solr determines that it can't use the transaction
 log to recover the index.

 Thanks,
 Shawn




 --
 Vincenzo D'Amore
 email: v.dam...@gmail.com
 skype: free.dev
 mobile: +39 349 8513251

Re: SolrCloud indexing

[Shawn Heisey]

On 5/7/2015 3:04 AM, Vincenzo D'Amore wrote:
 Thanks Erick. I'm not sure I got your answer.
 
 I try to recap, when the raw document has to be indexed, it will be
 forwarded to shard leader. Shard leader indexes the document for that
 shard, and then forwards the indexed document to any replicas.
 
 I want just be sure that when the raw document is forwarded from the leader
 to the replicas it will be indexed only one time on the shard leader. From
 what I understand replicas do not indexes, only the leader indexes.

The document is indexed by all replicas.  There is no way to forward the
indexed document, it can only forward the source document ... so each
replica must index it independently.

The old-style master-slave replication (which existed long before
SolrCloud) copies the finished Lucene segments, so only the master
actually does indexing.

SolrCloud doesn't have a master, only multiple replicas, one of which is
elected leader, and replication only comes into the picture if there's a
serious problem and Solr determines that it can't use the transaction
log to recover the index.

Thanks,
Shawn

Re: SolrCloud indexing

[Erick Erickson]

bq: Does it mean that all the indexing is done by the leaders in one node?

no. The raw document is forwarded from the leader to the replica and
it's indexed on all the nodes. The leader has a little bit of extra
work to do routing the docs, but that's it. Shouldn't be a problem
with 3 shards.

bq: If so, how do I distribute the indexing (the shard leaders) across nodes?

You don't really need to bother I don't think, especially if you don't
see significantly higher CPU utilization on the leader. If you
absolutely MUST distribute leadership, see the Collections API and the
REBALANCELEADERS and BALANCESHARDUNIQUE (Solr 5.1 only) but frankly I
wouldn't worry about it unless and until you had demonstrated need.

Best,
Erick

On Tue, May 5, 2015 at 6:28 AM, Vincenzo D'Amore v.dam...@gmail.com wrote:
 Hi all,

 I have 3 nodes and there are 3 shards but looking at solrcloud admin I see
 that all the leaders are on the same node.

 If I understood well looking at  solr documentation
 https://cwiki.apache.org/confluence/display/solr/Shards+and+Indexing+Data+in+SolrCloud
 :

 When a document is sent to a machine for indexing, the system first
 determines if the machine is a replica or a leader.
 If the machine is a replica, the document is forwarded to the leader for
 processing.
 If the machine is a leader, SolrCloud determines which shard the document
 should go to, forwards the document the leader for that shard, indexes the
 document for this shard, and forwards the index notation to itself and any
 replicas.


 So I have 3 nodes, with 3 shards and 2 replicas of each shard.

 http://picpaste.com/pics/Screen_Shot_2015-05-05_at_15.19.54-Xp8uztpt.1430832218.png

 Does it mean that all the indexing is done by the leaders in one node? If
 so, how do I distribute the indexing (the shard leaders) across nodes?


 --
 Vincenzo D'Amore
 email: v.dam...@gmail.com
 skype: free.dev
 mobile: +39 349 8513251

Re: solrcloud indexing completed event

[Giovanni Bricconi]

Thank you Erick,


Fortunately I can modify the data feeding process to start my post-indexing
tasks.




2014-06-30 22:13 GMT+02:00 Erick Erickson erickerick...@gmail.com:

 The paradigm is different. In SolrCloud when a client sends an indexing
 request to any node in the system, when the response comes back all the
 nodes (leaders, followers, etc) have _all_ received the update and
 processed it. So you don't have to care in the same way.

 As far as different segments, versions, and all that this is entirely
 expected.
 Considering the above. Packet-leader. leader-follower. Each of them is
 independently indexing the documents, there is no replication. So, since
 the two servers started at different times, things like the autocommit
 interval
 can kick in at different times and the indexes diverge in terms of segment
 counts, version numbers, whatever. They'll return the same _documents_,
 but

 FWIW,
 Erick

 On Mon, Jun 30, 2014 at 7:55 AM, Giovanni Bricconi
 giovanni.bricc...@banzai.it wrote:
  Hello
 
  I have one application that queries solr; when the index version changes
  this application has to redo some tasks.
 
  Since I have more than one solr server, I would like to start these tasks
  when all solr nodes are synchronized.
 
  With master/slave configuration the application simply watched
  http://myhost:8080/solr/admin/cores?action=STATUScore=0bis
  on each solr node and checked that the commit time msec was equal. When
 the
  time changes and becomes equal on all the nodes the replication is
 complete
  and it is safe to restart the tasks.
 
  Now I would like to switch to a solrcloud configuration, splitting the
 core
  0bis in 3 shards, with 2 replicas for each shard.
 
  After refeeding the collection I tried the same approach calling
 
 
 http://myhost:8080/solr/admin/cores?action=STATUScore=0bis_shard3_replica2
 
  for each core of the collection, but with suprise I have found that on
 the
  same stripe the version of the index, the number of segments and even the
  commit time msec was different!!
 
  I was thinking that it was possible to check some parameter on each
  stripe's core to check that everithing was up to date, but this does not
  seem to be true.
 
  Is it possible somehow to capture the commit done on every core of the
  collection event?
 
  Thank you
 
  Giovanni

Re: solrcloud indexing completed event

[Erick Erickson]

The paradigm is different. In SolrCloud when a client sends an indexing
request to any node in the system, when the response comes back all the
nodes (leaders, followers, etc) have _all_ received the update and
processed it. So you don't have to care in the same way.

As far as different segments, versions, and all that this is entirely expected.
Considering the above. Packet-leader. leader-follower. Each of them is
independently indexing the documents, there is no replication. So, since
the two servers started at different times, things like the autocommit interval
can kick in at different times and the indexes diverge in terms of segment
counts, version numbers, whatever. They'll return the same _documents_,
but

FWIW,
Erick

On Mon, Jun 30, 2014 at 7:55 AM, Giovanni Bricconi
giovanni.bricc...@banzai.it wrote:
 Hello

 I have one application that queries solr; when the index version changes
 this application has to redo some tasks.

 Since I have more than one solr server, I would like to start these tasks
 when all solr nodes are synchronized.

 With master/slave configuration the application simply watched
 http://myhost:8080/solr/admin/cores?action=STATUScore=0bis
 on each solr node and checked that the commit time msec was equal. When the
 time changes and becomes equal on all the nodes the replication is complete
 and it is safe to restart the tasks.

 Now I would like to switch to a solrcloud configuration, splitting the core
 0bis in 3 shards, with 2 replicas for each shard.

 After refeeding the collection I tried the same approach calling

 http://myhost:8080/solr/admin/cores?action=STATUScore=0bis_shard3_replica2

 for each core of the collection, but with suprise I have found that on the
 same stripe the version of the index, the number of segments and even the
 commit time msec was different!!

 I was thinking that it was possible to check some parameter on each
 stripe's core to check that everithing was up to date, but this does not
 seem to be true.

 Is it possible somehow to capture the commit done on every core of the
 collection event?

 Thank you

 Giovanni

RE: SolrCloud Indexing question

[Kalyan Kuram]

Thank you so much for the suggestion, Is the same recommended for querying too 
i found it very slow when i do query using clousolrserver
Kalyan

 Date: Tue, 6 Aug 2013 13:25:37 -0600
 From: s...@elyograg.org
 To: solr-user@lucene.apache.org
 Subject: Re: SolrCloud Indexing question
 
 On 8/6/2013 12:55 PM, Kalyan Kuram wrote:
  Hi AllI need suggestion on how to send indexing commands to 2 different 
  solr server,Basically i want to mirror my index,here is the scenarioi have 
  2 cluster,
  each cluster has one master and 2 slaves with external zookeeper in the 
  fronti need suggestion on what solr api class i should use to send indexing 
  commands to 2 masters,will LBHttpSolrServer do the indexing or is this only 
  used for querying
  If there is a better approach please suggest
  Kalyan  
 
 If you're using zookeeper, then your index is SolrCloud, and you don't 
 have masters and slaves.  The traditional master/slave replication model 
 does not apply to SolrCloud.
 
 With SolrCloud, there is no need to have two independent clusters.  If a 
 server dies, the other servers in the cloud will keep the cluster 
 operational.  When you bring the dead server back with the proper 
 config, it will automatically be synchronized with the cluster.
 
 For a Java program with SolrJ, use a CloudSolrServer object for each 
 cluster.  The constructor for CloudSolrServer accepts the same zkHost 
 parameter that you give to each Solr server when starting in SolrCloud 
 mode.  You cannot index to independent clusters at the same time through 
 one object - if they truly are independent SolrCloud installs, you have 
 to manage updates to both of them independently.
 
 Thanks,
 Shawn
 
  

Re: SolrCloud Indexing question

[Shawn Heisey]

On 8/6/2013 12:55 PM, Kalyan Kuram wrote:

Hi AllI need suggestion on how to send indexing commands to 2 different solr 
server,Basically i want to mirror my index,here is the scenarioi have 2 cluster,
each cluster has one master and 2 slaves with external zookeeper in the fronti 
need suggestion on what solr api class i should use to send indexing commands 
to 2 masters,will LBHttpSolrServer do the indexing or is this only used for 
querying
If there is a better approach please suggest
Kalyan  


If you're using zookeeper, then your index is SolrCloud, and you don't 
have masters and slaves.  The traditional master/slave replication model 
does not apply to SolrCloud.


With SolrCloud, there is no need to have two independent clusters.  If a 
server dies, the other servers in the cloud will keep the cluster 
operational.  When you bring the dead server back with the proper 
config, it will automatically be synchronized with the cluster.


For a Java program with SolrJ, use a CloudSolrServer object for each 
cluster.  The constructor for CloudSolrServer accepts the same zkHost 
parameter that you give to each Solr server when starting in SolrCloud 
mode.  You cannot index to independent clusters at the same time through 
one object - if they truly are independent SolrCloud installs, you have 
to manage updates to both of them independently.


Thanks,
Shawn

RE: SolrCloud indexing blocks if node is recovering

[Markus Jelsma]

https://issues.apache.org/jira/browse/SOLR-4038
Still trying to gather the logs
 
 
-Original message-
 From:Mark Miller markrmil...@gmail.com
 Sent: Sat 03-Nov-2012 14:17
 To: Markus Jelsma markus.jel...@openindex.io
 Cc: solr-user@lucene.apache.org
 Subject: Re: SolrCloud indexing blocks if node is recovering
 
 The OOM machine and any surrounding if possible (eg especially the leader of 
 the shard).
 
 Not sure what I'm looking for yet, so the more info the better.
 
 - Mark
 
 On Nov 3, 2012, at 5:23 AM, Markus Jelsma markus.jel...@openindex.io wrote:
 
  Hi - yes, i should be able to make sense out of them next monday. I assume 
  you're not too interested in the OOM machine but all surrounding nodes that 
  blocked instead? 
  
  
  
  -Original message-
  From:Mark Miller markrmil...@gmail.com
  Sent: Sat 03-Nov-2012 03:14
  To: solr-user@lucene.apache.org
  Subject: Re: SolrCloud indexing blocks if node is recovering
  
  Doesn't sound right. Still have the logs?
  
  - Mark
  
  On Fri, Nov 2, 2012 at 9:45 AM, Markus Jelsma
  markus.jel...@openindex.io wrote:
  Hi,
  
  We just tested indexing some million docs from Hadoop to a 10 node 2 rep 
  SolrCloud cluster with this week's trunk. One of the nodes gave an OOM 
  but indexing continued without interruption. When i restarted the node 
  indexing stopped completely, the node tried to recover - which was 
  unsuccessful. I restarted the node again but that wasn't very helpful 
  either. Finally i decided to stop the node completely and see what 
  happens - indexing resumed.
  
  Why or how won't the other nodes accept incoming documents when one node 
  behaves really bad? The dying node wasn't the node we were sending 
  documents to and we are not using CloudSolrServer yet (see other thread). 
  Is this known behavior? Is it a bug?
  
  Thanks,
  Markus
  
  
  
  -- 
  - Mark
  
 
 

RE: SolrCloud indexing blocks if node is recovering

[Markus Jelsma]

Hi - yes, i should be able to make sense out of them next monday. I assume 
you're not too interested in the OOM machine but all surrounding nodes that 
blocked instead? 
 


-Original message-
 From:Mark Miller markrmil...@gmail.com
 Sent: Sat 03-Nov-2012 03:14
 To: solr-user@lucene.apache.org
 Subject: Re: SolrCloud indexing blocks if node is recovering
 
 Doesn't sound right. Still have the logs?
 
 - Mark
 
 On Fri, Nov 2, 2012 at 9:45 AM, Markus Jelsma
 markus.jel...@openindex.io wrote:
  Hi,
 
  We just tested indexing some million docs from Hadoop to a 10 node 2 rep 
  SolrCloud cluster with this week's trunk. One of the nodes gave an OOM but 
  indexing continued without interruption. When i restarted the node indexing 
  stopped completely, the node tried to recover - which was unsuccessful. I 
  restarted the node again but that wasn't very helpful either. Finally i 
  decided to stop the node completely and see what happens - indexing resumed.
 
  Why or how won't the other nodes accept incoming documents when one node 
  behaves really bad? The dying node wasn't the node we were sending 
  documents to and we are not using CloudSolrServer yet (see other thread). 
  Is this known behavior? Is it a bug?
 
  Thanks,
  Markus
 
 
 
 -- 
 - Mark
 

Re: SolrCloud indexing blocks if node is recovering

[Mark Miller]

The OOM machine and any surrounding if possible (eg especially the leader of 
the shard).

Not sure what I'm looking for yet, so the more info the better.

- Mark

On Nov 3, 2012, at 5:23 AM, Markus Jelsma markus.jel...@openindex.io wrote:

 Hi - yes, i should be able to make sense out of them next monday. I assume 
 you're not too interested in the OOM machine but all surrounding nodes that 
 blocked instead? 
 
 
 
 -Original message-
 From:Mark Miller markrmil...@gmail.com
 Sent: Sat 03-Nov-2012 03:14
 To: solr-user@lucene.apache.org
 Subject: Re: SolrCloud indexing blocks if node is recovering
 
 Doesn't sound right. Still have the logs?
 
 - Mark
 
 On Fri, Nov 2, 2012 at 9:45 AM, Markus Jelsma
 markus.jel...@openindex.io wrote:
 Hi,
 
 We just tested indexing some million docs from Hadoop to a 10 node 2 rep 
 SolrCloud cluster with this week's trunk. One of the nodes gave an OOM but 
 indexing continued without interruption. When i restarted the node indexing 
 stopped completely, the node tried to recover - which was unsuccessful. I 
 restarted the node again but that wasn't very helpful either. Finally i 
 decided to stop the node completely and see what happens - indexing resumed.
 
 Why or how won't the other nodes accept incoming documents when one node 
 behaves really bad? The dying node wasn't the node we were sending 
 documents to and we are not using CloudSolrServer yet (see other thread). 
 Is this known behavior? Is it a bug?
 
 Thanks,
 Markus
 
 
 
 -- 
 - Mark
 

Re: SolrCloud indexing blocks if node is recovering

[Mark Miller]

Doesn't sound right. Still have the logs?

- Mark

On Fri, Nov 2, 2012 at 9:45 AM, Markus Jelsma
markus.jel...@openindex.io wrote:
 Hi,

 We just tested indexing some million docs from Hadoop to a 10 node 2 rep 
 SolrCloud cluster with this week's trunk. One of the nodes gave an OOM but 
 indexing continued without interruption. When i restarted the node indexing 
 stopped completely, the node tried to recover - which was unsuccessful. I 
 restarted the node again but that wasn't very helpful either. Finally i 
 decided to stop the node completely and see what happens - indexing resumed.

 Why or how won't the other nodes accept incoming documents when one node 
 behaves really bad? The dying node wasn't the node we were sending documents 
 to and we are not using CloudSolrServer yet (see other thread). Is this known 
 behavior? Is it a bug?

 Thanks,
 Markus



-- 
- Mark

Re: SolrCloud indexing question

[Jamie Johnson]

my understanding is that you can send your updates/deletes to any
shard and they will be forwarded to the leader automatically.  That
being said your leader will always be the place where the index
happens and then distributed to the other replicas.

On Fri, Apr 20, 2012 at 7:54 AM, Darren Govoni dar...@ontrenet.com wrote:
 Hi,
  I just wanted to make sure I understand how distributed indexing works
 in solrcloud.

 Can I index locally at each shard to avoid throttling a central port? Or
 all the indexing has to go through a single shard leader?

 thanks

Re: SolrCloud indexing question

[Darren Govoni]

Gotcha.

Now does that mean if I have 5 threads all writing to a local shard,
will that shard piggyhop those index requests onto a SINGLE connection
to the leader? Or will they spawn 5 connections from the shard to the
leader? I really hope the formerthe latter won't scale well.

On Fri, 2012-04-20 at 10:28 -0400, Jamie Johnson wrote:
 my understanding is that you can send your updates/deletes to any
 shard and they will be forwarded to the leader automatically.  That
 being said your leader will always be the place where the index
 happens and then distributed to the other replicas.
 
 On Fri, Apr 20, 2012 at 7:54 AM, Darren Govoni dar...@ontrenet.com wrote:
  Hi,
   I just wanted to make sure I understand how distributed indexing works
  in solrcloud.
 
  Can I index locally at each shard to avoid throttling a central port? Or
  all the indexing has to go through a single shard leader?
 
  thanks
 
 
 

Re: SolrCloud indexing question

[Jamie Johnson]

I believe the SolrJ code round robins which server the request is sent
to and as such probably wouldn't send to the same server in your case,
but if you had an HttpSolrServer for instance and were pointing to
only one particular intsance my guess would be that would be 5
separate requests from the server you hit.  Especially since in all
likelihood those documents wouldn't be destined for the same shard as
the others (unless of course you only had 1 shard and you sent these
to the replica)

On Fri, Apr 20, 2012 at 3:02 PM, Darren Govoni dar...@ontrenet.com wrote:
 Gotcha.

 Now does that mean if I have 5 threads all writing to a local shard,
 will that shard piggyhop those index requests onto a SINGLE connection
 to the leader? Or will they spawn 5 connections from the shard to the
 leader? I really hope the formerthe latter won't scale well.

 On Fri, 2012-04-20 at 10:28 -0400, Jamie Johnson wrote:
 my understanding is that you can send your updates/deletes to any
 shard and they will be forwarded to the leader automatically.  That
 being said your leader will always be the place where the index
 happens and then distributed to the other replicas.

 On Fri, Apr 20, 2012 at 7:54 AM, Darren Govoni dar...@ontrenet.com wrote:
  Hi,
   I just wanted to make sure I understand how distributed indexing works
  in solrcloud.
 
  Can I index locally at each shard to avoid throttling a central port? Or
  all the indexing has to go through a single shard leader?
 
  thanks