Re: SolrCloud 4.8.1 - commit wait

[Erick Erickson]

Glad to hear it's solved! The suggester stuff is
way cool, but can surprise you!

Erick

On Thu, Dec 17, 2015 at 2:54 AM, Vincenzo D'Amore  wrote:
> Great!!! Great Erick! It was a buildOnCommit.
>
> Many thanks for your help.
>
>
>
> On Wed, Dec 16, 2015 at 6:30 PM, Erick Erickson 
> wrote:
>
>> Quick scan, but probably this:
>>  INFO
>>  o.a.solr.spelling.suggest.Suggester - build()
>>
>> The suggester build process can easily take many minutes, there's some
>> explanation here:
>> https://lucidworks.com/blog/2015/03/04/solr-suggester/
>>
>> the short form is that depending on how it's defined, it may have to
>> read _all_ the
>> documents in your entire corpus to build the suggester structures. And
>> you apparently
>> have buildOnCommit set to true.
>>
>> Note particularly the caveats there about the Solr version required so that
>> buildOnStartup=false is honored.
>>
>> Best,
>> Erick
>>
>> On Wed, Dec 16, 2015 at 2:34 AM, Vincenzo D'Amore 
>> wrote:
>> > Hi,
>> >
>> > an update. Hope you can help me.
>> >
>> > I have stopped all the other working collections, in order to have a
>> clean
>> > log file.
>> >
>> > at 11:01:16 an hard commit has been issued
>> >
>> > 2015-12-16 11:01:49,839 [http-bio-8080-exec-824] INFO
>> >  org.apache.solr.update.UpdateHandler - start
>> >
>> commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
>> >
>> > at 11:11:31,344 the commit has been completed.
>> >
>> > The commit was ended logging this line, I suppose 615021 is the wait time
>> > (roughly 10 minutes) :
>> >
>> > 2015-12-16 11:11:31,343 [http-bio-8080-exec-991] INFO
>> >  o.a.s.u.processor.LogUpdateProcessor - [catalogo_shard2_replica3]
>> > webapp=/solr path=/update
>> >
>> params={waitSearcher=true=true=false=javabin=2}
>> > {commit=} 0 615021
>> >
>> > During this 10 minutes, the server logged "only" thes lines, looking at
>> > them I don't see anything of strange:
>> >
>> > 2015-12-16 11:01:50,705 [http-bio-8080-exec-824] INFO
>> >  o.a.solr.search.SolrIndexSearcher - Opening
>> > Searcher@6d5c31e2[catalogo_shard1_replica2]
>> > main
>> > 2015-12-16 11:01:50,724 [http-bio-8080-exec-824] INFO
>> >  org.apache.solr.update.UpdateHandler - end_commit_flush
>> > 2015-12-16 11:02:20,722 [searcherExecutor-108-thread-1] INFO
>> >  o.a.solr.spelling.suggest.Suggester - build()
>> > 2015-12-16 11:02:21,846 [http-bio-8080-exec-824] INFO
>> >  o.a.s.u.processor.LogUpdateProcessor - [catalogo_shard1_replica2]
>> > webapp=/solr path=/update
>> >
>> params={update.distrib=FROMLEADER=true=true=true=false=
>> >
>> http://192.168.101.118:8080/solr/catalogo_shard2_replica3/_end_point=true=javabin=2=false
>> }
>> > {commit=} 0 32007
>> > 2015-12-16 11:05:47,162 [http-bio-8080-exec-1037] INFO
>> >  org.apache.solr.update.UpdateHandler - start
>> >
>> commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
>> > 2015-12-16 11:05:47,970 [http-bio-8080-exec-1037] INFO
>> >  o.a.solr.search.SolrIndexSearcher - Opening
>> > Searcher@4ede7ac5[catalogo_shard2_replica3]
>> > main
>> > 2015-12-16 11:05:47,989 [http-bio-8080-exec-1037] INFO
>> >  org.apache.solr.update.UpdateHandler - end_commit_flush
>> > 2015-12-16 11:06:03,063 [commitScheduler-115-thread-1] INFO
>> >  org.apache.solr.update.UpdateHandler - start
>> >
>> commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
>> > 2015-12-16 11:06:03,896 [commitScheduler-115-thread-1] INFO
>> >  o.a.solr.search.SolrIndexSearcher - Opening
>> > Searcher@2bf4fd3a[catalogo_shard3_replica1]
>> > realtime
>> > 2015-12-16 11:06:03,913 [commitScheduler-115-thread-1] INFO
>> >  org.apache.solr.update.UpdateHandler - end_commit_flush
>> > 2015-12-16 11:06:19,435 [searcherExecutor-111-thread-1] INFO
>> >  o.a.solr.spelling.suggest.Suggester - build()
>> > 2015-12-16 11:06:20,589 [http-bio-8080-exec-1037] INFO
>> >  o.a.s.u.processor.LogUpdateProcessor - [catalogo_shard2_replica3]
>> > webapp=/solr path=/update
>> >
>> params={update.distrib=FROMLEADER=true=true=true=false=
>> >
>> http://192.168.101.118:8080/solr/catalogo_shard2_replica3/_end_point=true=javabin=2=false
>> }
>> > {commit=} 0 33427
>> > 2015-12-16 11:08:07,076 [http-bio-8080-exec-1037] INFO
>> >  org.apache.solr.update.UpdateHandler - start
>> >
>> commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
>> > 2015-12-16 11:08:07,076 [http-bio-8080-exec-1037] INFO
>> >  org.apache.solr.update.UpdateHandler - No uncommitted changes. Skipping
>> > IW.commit.
>> > 2015-12-16 11:08:07,076 [http-bio-8080-exec-1037] INFO
>> >  o.a.solr.search.SolrIndexSearcher - Opening
>> > Searcher@75b2727f[catalogo_shard3_replica1]
>> > main
>> > 2015-12-16 11:08:07,084 [http-bio-8080-exec-1037] INFO
>> >  

Re: SolrCloud 4.8.1 - commit wait

[Vincenzo D'Amore]

Great!!! Great Erick! It was a buildOnCommit.

Many thanks for your help.



On Wed, Dec 16, 2015 at 6:30 PM, Erick Erickson 
wrote:

> Quick scan, but probably this:
>  INFO
>  o.a.solr.spelling.suggest.Suggester - build()
>
> The suggester build process can easily take many minutes, there's some
> explanation here:
> https://lucidworks.com/blog/2015/03/04/solr-suggester/
>
> the short form is that depending on how it's defined, it may have to
> read _all_ the
> documents in your entire corpus to build the suggester structures. And
> you apparently
> have buildOnCommit set to true.
>
> Note particularly the caveats there about the Solr version required so that
> buildOnStartup=false is honored.
>
> Best,
> Erick
>
> On Wed, Dec 16, 2015 at 2:34 AM, Vincenzo D'Amore 
> wrote:
> > Hi,
> >
> > an update. Hope you can help me.
> >
> > I have stopped all the other working collections, in order to have a
> clean
> > log file.
> >
> > at 11:01:16 an hard commit has been issued
> >
> > 2015-12-16 11:01:49,839 [http-bio-8080-exec-824] INFO
> >  org.apache.solr.update.UpdateHandler - start
> >
> commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
> >
> > at 11:11:31,344 the commit has been completed.
> >
> > The commit was ended logging this line, I suppose 615021 is the wait time
> > (roughly 10 minutes) :
> >
> > 2015-12-16 11:11:31,343 [http-bio-8080-exec-991] INFO
> >  o.a.s.u.processor.LogUpdateProcessor - [catalogo_shard2_replica3]
> > webapp=/solr path=/update
> >
> params={waitSearcher=true=true=false=javabin=2}
> > {commit=} 0 615021
> >
> > During this 10 minutes, the server logged "only" thes lines, looking at
> > them I don't see anything of strange:
> >
> > 2015-12-16 11:01:50,705 [http-bio-8080-exec-824] INFO
> >  o.a.solr.search.SolrIndexSearcher - Opening
> > Searcher@6d5c31e2[catalogo_shard1_replica2]
> > main
> > 2015-12-16 11:01:50,724 [http-bio-8080-exec-824] INFO
> >  org.apache.solr.update.UpdateHandler - end_commit_flush
> > 2015-12-16 11:02:20,722 [searcherExecutor-108-thread-1] INFO
> >  o.a.solr.spelling.suggest.Suggester - build()
> > 2015-12-16 11:02:21,846 [http-bio-8080-exec-824] INFO
> >  o.a.s.u.processor.LogUpdateProcessor - [catalogo_shard1_replica2]
> > webapp=/solr path=/update
> >
> params={update.distrib=FROMLEADER=true=true=true=false=
> >
> http://192.168.101.118:8080/solr/catalogo_shard2_replica3/_end_point=true=javabin=2=false
> }
> > {commit=} 0 32007
> > 2015-12-16 11:05:47,162 [http-bio-8080-exec-1037] INFO
> >  org.apache.solr.update.UpdateHandler - start
> >
> commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
> > 2015-12-16 11:05:47,970 [http-bio-8080-exec-1037] INFO
> >  o.a.solr.search.SolrIndexSearcher - Opening
> > Searcher@4ede7ac5[catalogo_shard2_replica3]
> > main
> > 2015-12-16 11:05:47,989 [http-bio-8080-exec-1037] INFO
> >  org.apache.solr.update.UpdateHandler - end_commit_flush
> > 2015-12-16 11:06:03,063 [commitScheduler-115-thread-1] INFO
> >  org.apache.solr.update.UpdateHandler - start
> >
> commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
> > 2015-12-16 11:06:03,896 [commitScheduler-115-thread-1] INFO
> >  o.a.solr.search.SolrIndexSearcher - Opening
> > Searcher@2bf4fd3a[catalogo_shard3_replica1]
> > realtime
> > 2015-12-16 11:06:03,913 [commitScheduler-115-thread-1] INFO
> >  org.apache.solr.update.UpdateHandler - end_commit_flush
> > 2015-12-16 11:06:19,435 [searcherExecutor-111-thread-1] INFO
> >  o.a.solr.spelling.suggest.Suggester - build()
> > 2015-12-16 11:06:20,589 [http-bio-8080-exec-1037] INFO
> >  o.a.s.u.processor.LogUpdateProcessor - [catalogo_shard2_replica3]
> > webapp=/solr path=/update
> >
> params={update.distrib=FROMLEADER=true=true=true=false=
> >
> http://192.168.101.118:8080/solr/catalogo_shard2_replica3/_end_point=true=javabin=2=false
> }
> > {commit=} 0 33427
> > 2015-12-16 11:08:07,076 [http-bio-8080-exec-1037] INFO
> >  org.apache.solr.update.UpdateHandler - start
> >
> commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
> > 2015-12-16 11:08:07,076 [http-bio-8080-exec-1037] INFO
> >  org.apache.solr.update.UpdateHandler - No uncommitted changes. Skipping
> > IW.commit.
> > 2015-12-16 11:08:07,076 [http-bio-8080-exec-1037] INFO
> >  o.a.solr.search.SolrIndexSearcher - Opening
> > Searcher@75b2727f[catalogo_shard3_replica1]
> > main
> > 2015-12-16 11:08:07,084 [http-bio-8080-exec-1037] INFO
> >  org.apache.solr.update.UpdateHandler - end_commit_flush
> > 2015-12-16 11:08:39,040 [searcherExecutor-114-thread-1] INFO
> >  o.a.solr.spelling.suggest.Suggester - build()
> > 2015-12-16 11:08:40,286 [http-bio-8080-exec-1037] INFO
> >  o.a.s.u.processor.LogUpdateProcessor - [catalogo_shard3_replica1]
> > webapp=/solr 

Re: SolrCloud 4.8.1 - commit wait

[Erick Erickson]

Quick scan, but probably this:
 INFO
 o.a.solr.spelling.suggest.Suggester - build()

The suggester build process can easily take many minutes, there's some
explanation here:
https://lucidworks.com/blog/2015/03/04/solr-suggester/

the short form is that depending on how it's defined, it may have to
read _all_ the
documents in your entire corpus to build the suggester structures. And
you apparently
have buildOnCommit set to true.

Note particularly the caveats there about the Solr version required so that
buildOnStartup=false is honored.

Best,
Erick

On Wed, Dec 16, 2015 at 2:34 AM, Vincenzo D'Amore  wrote:
> Hi,
>
> an update. Hope you can help me.
>
> I have stopped all the other working collections, in order to have a clean
> log file.
>
> at 11:01:16 an hard commit has been issued
>
> 2015-12-16 11:01:49,839 [http-bio-8080-exec-824] INFO
>  org.apache.solr.update.UpdateHandler - start
> commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
>
> at 11:11:31,344 the commit has been completed.
>
> The commit was ended logging this line, I suppose 615021 is the wait time
> (roughly 10 minutes) :
>
> 2015-12-16 11:11:31,343 [http-bio-8080-exec-991] INFO
>  o.a.s.u.processor.LogUpdateProcessor - [catalogo_shard2_replica3]
> webapp=/solr path=/update
> params={waitSearcher=true=true=false=javabin=2}
> {commit=} 0 615021
>
> During this 10 minutes, the server logged "only" thes lines, looking at
> them I don't see anything of strange:
>
> 2015-12-16 11:01:50,705 [http-bio-8080-exec-824] INFO
>  o.a.solr.search.SolrIndexSearcher - Opening
> Searcher@6d5c31e2[catalogo_shard1_replica2]
> main
> 2015-12-16 11:01:50,724 [http-bio-8080-exec-824] INFO
>  org.apache.solr.update.UpdateHandler - end_commit_flush
> 2015-12-16 11:02:20,722 [searcherExecutor-108-thread-1] INFO
>  o.a.solr.spelling.suggest.Suggester - build()
> 2015-12-16 11:02:21,846 [http-bio-8080-exec-824] INFO
>  o.a.s.u.processor.LogUpdateProcessor - [catalogo_shard1_replica2]
> webapp=/solr path=/update
> params={update.distrib=FROMLEADER=true=true=true=false=
> http://192.168.101.118:8080/solr/catalogo_shard2_replica3/_end_point=true=javabin=2=false}
> {commit=} 0 32007
> 2015-12-16 11:05:47,162 [http-bio-8080-exec-1037] INFO
>  org.apache.solr.update.UpdateHandler - start
> commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
> 2015-12-16 11:05:47,970 [http-bio-8080-exec-1037] INFO
>  o.a.solr.search.SolrIndexSearcher - Opening
> Searcher@4ede7ac5[catalogo_shard2_replica3]
> main
> 2015-12-16 11:05:47,989 [http-bio-8080-exec-1037] INFO
>  org.apache.solr.update.UpdateHandler - end_commit_flush
> 2015-12-16 11:06:03,063 [commitScheduler-115-thread-1] INFO
>  org.apache.solr.update.UpdateHandler - start
> commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
> 2015-12-16 11:06:03,896 [commitScheduler-115-thread-1] INFO
>  o.a.solr.search.SolrIndexSearcher - Opening
> Searcher@2bf4fd3a[catalogo_shard3_replica1]
> realtime
> 2015-12-16 11:06:03,913 [commitScheduler-115-thread-1] INFO
>  org.apache.solr.update.UpdateHandler - end_commit_flush
> 2015-12-16 11:06:19,435 [searcherExecutor-111-thread-1] INFO
>  o.a.solr.spelling.suggest.Suggester - build()
> 2015-12-16 11:06:20,589 [http-bio-8080-exec-1037] INFO
>  o.a.s.u.processor.LogUpdateProcessor - [catalogo_shard2_replica3]
> webapp=/solr path=/update
> params={update.distrib=FROMLEADER=true=true=true=false=
> http://192.168.101.118:8080/solr/catalogo_shard2_replica3/_end_point=true=javabin=2=false}
> {commit=} 0 33427
> 2015-12-16 11:08:07,076 [http-bio-8080-exec-1037] INFO
>  org.apache.solr.update.UpdateHandler - start
> commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
> 2015-12-16 11:08:07,076 [http-bio-8080-exec-1037] INFO
>  org.apache.solr.update.UpdateHandler - No uncommitted changes. Skipping
> IW.commit.
> 2015-12-16 11:08:07,076 [http-bio-8080-exec-1037] INFO
>  o.a.solr.search.SolrIndexSearcher - Opening
> Searcher@75b2727f[catalogo_shard3_replica1]
> main
> 2015-12-16 11:08:07,084 [http-bio-8080-exec-1037] INFO
>  org.apache.solr.update.UpdateHandler - end_commit_flush
> 2015-12-16 11:08:39,040 [searcherExecutor-114-thread-1] INFO
>  o.a.solr.spelling.suggest.Suggester - build()
> 2015-12-16 11:08:40,286 [http-bio-8080-exec-1037] INFO
>  o.a.s.u.processor.LogUpdateProcessor - [catalogo_shard3_replica1]
> webapp=/solr path=/update
> params={update.distrib=FROMLEADER=true=true=true=false=
> http://192.168.101.118:8080/solr/catalogo_shard2_replica3/_end_point=true=javabin=2=false}
> {commit=} 0 33211
>
> Could some component be the cause of this wait? Something like a suggester
> or a spellchecker cache?
> But if yes, I should see the activity in log file, isn't it?
>
> Best regards,
> Vincenzo
>
>
> On Sat, Dec 12, 

Re: SolrCloud 4.8.1 - commit wait

[Vincenzo D'Amore]

Hi,

an update. Hope you can help me.

I have stopped all the other working collections, in order to have a clean
log file.

at 11:01:16 an hard commit has been issued

2015-12-16 11:01:49,839 [http-bio-8080-exec-824] INFO
 org.apache.solr.update.UpdateHandler - start
commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}

at 11:11:31,344 the commit has been completed.

The commit was ended logging this line, I suppose 615021 is the wait time
(roughly 10 minutes) :

2015-12-16 11:11:31,343 [http-bio-8080-exec-991] INFO
 o.a.s.u.processor.LogUpdateProcessor - [catalogo_shard2_replica3]
webapp=/solr path=/update
params={waitSearcher=true=true=false=javabin=2}
{commit=} 0 615021

During this 10 minutes, the server logged "only" thes lines, looking at
them I don't see anything of strange:

2015-12-16 11:01:50,705 [http-bio-8080-exec-824] INFO
 o.a.solr.search.SolrIndexSearcher - Opening
Searcher@6d5c31e2[catalogo_shard1_replica2]
main
2015-12-16 11:01:50,724 [http-bio-8080-exec-824] INFO
 org.apache.solr.update.UpdateHandler - end_commit_flush
2015-12-16 11:02:20,722 [searcherExecutor-108-thread-1] INFO
 o.a.solr.spelling.suggest.Suggester - build()
2015-12-16 11:02:21,846 [http-bio-8080-exec-824] INFO
 o.a.s.u.processor.LogUpdateProcessor - [catalogo_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER=true=true=true=false=
http://192.168.101.118:8080/solr/catalogo_shard2_replica3/_end_point=true=javabin=2=false}
{commit=} 0 32007
2015-12-16 11:05:47,162 [http-bio-8080-exec-1037] INFO
 org.apache.solr.update.UpdateHandler - start
commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
2015-12-16 11:05:47,970 [http-bio-8080-exec-1037] INFO
 o.a.solr.search.SolrIndexSearcher - Opening
Searcher@4ede7ac5[catalogo_shard2_replica3]
main
2015-12-16 11:05:47,989 [http-bio-8080-exec-1037] INFO
 org.apache.solr.update.UpdateHandler - end_commit_flush
2015-12-16 11:06:03,063 [commitScheduler-115-thread-1] INFO
 org.apache.solr.update.UpdateHandler - start
commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
2015-12-16 11:06:03,896 [commitScheduler-115-thread-1] INFO
 o.a.solr.search.SolrIndexSearcher - Opening
Searcher@2bf4fd3a[catalogo_shard3_replica1]
realtime
2015-12-16 11:06:03,913 [commitScheduler-115-thread-1] INFO
 org.apache.solr.update.UpdateHandler - end_commit_flush
2015-12-16 11:06:19,435 [searcherExecutor-111-thread-1] INFO
 o.a.solr.spelling.suggest.Suggester - build()
2015-12-16 11:06:20,589 [http-bio-8080-exec-1037] INFO
 o.a.s.u.processor.LogUpdateProcessor - [catalogo_shard2_replica3]
webapp=/solr path=/update
params={update.distrib=FROMLEADER=true=true=true=false=
http://192.168.101.118:8080/solr/catalogo_shard2_replica3/_end_point=true=javabin=2=false}
{commit=} 0 33427
2015-12-16 11:08:07,076 [http-bio-8080-exec-1037] INFO
 org.apache.solr.update.UpdateHandler - start
commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
2015-12-16 11:08:07,076 [http-bio-8080-exec-1037] INFO
 org.apache.solr.update.UpdateHandler - No uncommitted changes. Skipping
IW.commit.
2015-12-16 11:08:07,076 [http-bio-8080-exec-1037] INFO
 o.a.solr.search.SolrIndexSearcher - Opening
Searcher@75b2727f[catalogo_shard3_replica1]
main
2015-12-16 11:08:07,084 [http-bio-8080-exec-1037] INFO
 org.apache.solr.update.UpdateHandler - end_commit_flush
2015-12-16 11:08:39,040 [searcherExecutor-114-thread-1] INFO
 o.a.solr.spelling.suggest.Suggester - build()
2015-12-16 11:08:40,286 [http-bio-8080-exec-1037] INFO
 o.a.s.u.processor.LogUpdateProcessor - [catalogo_shard3_replica1]
webapp=/solr path=/update
params={update.distrib=FROMLEADER=true=true=true=false=
http://192.168.101.118:8080/solr/catalogo_shard2_replica3/_end_point=true=javabin=2=false}
{commit=} 0 33211

Could some component be the cause of this wait? Something like a suggester
or a spellchecker cache?
But if yes, I should see the activity in log file, isn't it?

Best regards,
Vincenzo


On Sat, Dec 12, 2015 at 7:50 PM, Erick Erickson 
wrote:

> Autowarm times will only happen when the commit has openSearcher=true
> or on a soft commit. But maybe your log levels aren't at INFO for the right
> code...
>
> That said, your autowarm counts at 0 probably means that you're not seeing
> any autowarming really, so that might be a red herring. Your newSearcher
> event in solrconfig.xml will still be fired, but may be commented out.
>
> This is still something of a puzzle. With an index this size, your hard
> commits should never take more than a second or two unless you're
> in some very strange state. Stack traces would be in order if lengthening
> the commit interval doesn't work.
>
> Best,
> Erick
>
> On Fri, Dec 11, 2015 at 5:58 PM, Vincenzo D'Amore 
> wrote:
> > 

Re: SolrCloud 4.8.1 - commit wait

[Erick Erickson]

Autowarm times will only happen when the commit has openSearcher=true
or on a soft commit. But maybe your log levels aren't at INFO for the right
code...

That said, your autowarm counts at 0 probably means that you're not seeing
any autowarming really, so that might be a red herring. Your newSearcher
event in solrconfig.xml will still be fired, but may be commented out.

This is still something of a puzzle. With an index this size, your hard
commits should never take more than a second or two unless you're
in some very strange state. Stack traces would be in order if lengthening
the commit interval doesn't work.

Best,
Erick

On Fri, Dec 11, 2015 at 5:58 PM, Vincenzo D'Amore  wrote:
> Hi All,
>
> an update, I have switched logging from WARN to INFO for all except for
> those two:
>
> - org.apache.solr.core
> - org.apache.solr.handler.component.SpellCheckComponent
>
> Well, looking at log file I'm unable to find any autowarm log line, even
> after few updates and commits.
>
> Looking at solrconfig.xml I see most autowarmCount parameters are set to 0
>
>  autowarmCount="0" />
>  autowarmCount="0" />
>  autowarmCount="0" />
>  ="10" initialSize="0" autowarmCount="10" regenerator="solr.NoOpRegenerator"
> />
>
> Not sure what this means...
>
> On Sat, Dec 12, 2015 at 1:13 AM, Vincenzo D'Amore 
> wrote:
>
>> Thanks Erick, Mark,
>>
>> I'll raise maxTime asap.
>> Just to be sure understand, given that I have openSearcher=false, I
>> suppose it shouldn't trigger autowarming at least until a commit is
>> executed, shouldn't it?
>>
>> Anyway, I don't understand, given that maxTime is very aggressive, why
>> hard commit takes so long.
>>
>> Thanks again for your answers.
>> Vincenzo
>>
>>
>> On Fri, Dec 11, 2015 at 7:22 PM, Erick Erickson 
>> wrote:
>>
>>> First of all, your autocommit settings are _very_ aggressive. Committing
>>> every second is far to frequent IMO.
>>>
>>> As an aside, I generally prefer to omit the maxDocs as it's not all
>>> that predictable,
>>> but that's a personal preference and really doesn't bear on your problem..
>>>
>>> My _guess_ is that you are doing a lot of autowarming. The number of docs
>>> doesn't really matter if your autowarming is taking forever, your Solr
>>> logs
>>> should report the autowarm times at INFO level, have you checked those?
>>>
>>> The commit settings shouldn't be a problem in terms of your server dying,
>>> the indexing process flushes docs to the tlog independent of committing so
>>> upon restart they should be recovered. Here's a blog on the subject:
>>>
>>>
>>> https://lucidworks.com/blog/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
>>>
>>> Best,
>>> Erick
>>>
>>> On Fri, Dec 11, 2015 at 8:24 AM, Vincenzo D'Amore 
>>> wrote:
>>> > Hi all,
>>> >
>>> > I have a SolrCloud cluster with a collection (2.5M docs) with 3 shards
>>> and
>>> > 15 replicas.
>>> > There is a solrj application that feeds the collection, updating few
>>> > documents every hour, I don't understand why, at end of process, the
>>> hard
>>> > commit takes about 8/10 minutes.
>>> >
>>> > Even if there are only few hundreds of documents.
>>> >
>>> > This is the autocommit configuration:
>>> >
>>> > 
>>> > 1
>>> > 1000
>>> > false
>>> > 
>>> >
>>> > In your experience why hard commit takes so long even for so few
>>> documents?
>>> >
>>> > Now I'm changing the code to softcommit, calling commit (waitFlush =
>>> > false, waitSearcher
>>> > = false, softCommit = true);
>>> >
>>> > solrServer.commit(false, false, true);.
>>> >
>>> > I have configured NRTCachingDirectoryFactory, but I'm a little bit
>>> worried
>>> > if a server goes down (something like: kill -9, SolrCloud crashes, out
>>> of
>>> > memory, etc.), and if, using this strategy
>>> softcommit+NRTCachingDirectory,
>>> > SolrCloud instance could not recover a replica.
>>> >
>>> > Should I worry about this new configuration? I was thinking to take a
>>> > snapshot of everything every day, in order to recover immediately the
>>> > index. Could this be considered a best practice?
>>> >
>>> > Thanks in advance for your time,
>>> > Vincenzo
>>> >
>>> > --
>>> > Vincenzo D'Amore
>>> > email: v.dam...@gmail.com
>>> > skype: free.dev
>>> > mobile: +39 349 8513251
>>>
>>
>>
>>
>> --
>> Vincenzo D'Amore
>> email: v.dam...@gmail.com
>> skype: free.dev
>> mobile: +39 349 8513251
>>
>
>
>
> --
> Vincenzo D'Amore
> email: v.dam...@gmail.com
> skype: free.dev
> mobile: +39 349 8513251

SolrCloud 4.8.1 - commit wait

[Vincenzo D'Amore]

Hi all,

I have a SolrCloud cluster with a collection (2.5M docs) with 3 shards and
15 replicas.
There is a solrj application that feeds the collection, updating few
documents every hour, I don't understand why, at end of process, the hard
commit takes about 8/10 minutes.

Even if there are only few hundreds of documents.

This is the autocommit configuration:


1
1000
false


In your experience why hard commit takes so long even for so few documents?

Now I'm changing the code to softcommit, calling commit (waitFlush =
false, waitSearcher
= false, softCommit = true);

solrServer.commit(false, false, true);.

I have configured NRTCachingDirectoryFactory, but I'm a little bit worried
if a server goes down (something like: kill -9, SolrCloud crashes, out of
memory, etc.), and if, using this strategy softcommit+NRTCachingDirectory,
SolrCloud instance could not recover a replica.

Should I worry about this new configuration? I was thinking to take a
snapshot of everything every day, in order to recover immediately the
index. Could this be considered a best practice?

Thanks in advance for your time,
Vincenzo

-- 
Vincenzo D'Amore
email: v.dam...@gmail.com
skype: free.dev
mobile: +39 349 8513251

Re: SolrCloud 4.8.1 - commit wait

[Vincenzo D'Amore]

Thanks Erick, Mark,

I'll raise maxTime asap.
Just to be sure understand, given that I have openSearcher=false, I suppose
it shouldn't trigger autowarming at least until a commit is executed,
shouldn't it?

Anyway, I don't understand, given that maxTime is very aggressive, why hard
commit takes so long.

Thanks again for your answers.
Vincenzo


On Fri, Dec 11, 2015 at 7:22 PM, Erick Erickson 
wrote:

> First of all, your autocommit settings are _very_ aggressive. Committing
> every second is far to frequent IMO.
>
> As an aside, I generally prefer to omit the maxDocs as it's not all
> that predictable,
> but that's a personal preference and really doesn't bear on your problem..
>
> My _guess_ is that you are doing a lot of autowarming. The number of docs
> doesn't really matter if your autowarming is taking forever, your Solr logs
> should report the autowarm times at INFO level, have you checked those?
>
> The commit settings shouldn't be a problem in terms of your server dying,
> the indexing process flushes docs to the tlog independent of committing so
> upon restart they should be recovered. Here's a blog on the subject:
>
>
> https://lucidworks.com/blog/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
>
> Best,
> Erick
>
> On Fri, Dec 11, 2015 at 8:24 AM, Vincenzo D'Amore 
> wrote:
> > Hi all,
> >
> > I have a SolrCloud cluster with a collection (2.5M docs) with 3 shards
> and
> > 15 replicas.
> > There is a solrj application that feeds the collection, updating few
> > documents every hour, I don't understand why, at end of process, the hard
> > commit takes about 8/10 minutes.
> >
> > Even if there are only few hundreds of documents.
> >
> > This is the autocommit configuration:
> >
> > 
> > 1
> > 1000
> > false
> > 
> >
> > In your experience why hard commit takes so long even for so few
> documents?
> >
> > Now I'm changing the code to softcommit, calling commit (waitFlush =
> > false, waitSearcher
> > = false, softCommit = true);
> >
> > solrServer.commit(false, false, true);.
> >
> > I have configured NRTCachingDirectoryFactory, but I'm a little bit
> worried
> > if a server goes down (something like: kill -9, SolrCloud crashes, out of
> > memory, etc.), and if, using this strategy
> softcommit+NRTCachingDirectory,
> > SolrCloud instance could not recover a replica.
> >
> > Should I worry about this new configuration? I was thinking to take a
> > snapshot of everything every day, in order to recover immediately the
> > index. Could this be considered a best practice?
> >
> > Thanks in advance for your time,
> > Vincenzo
> >
> > --
> > Vincenzo D'Amore
> > email: v.dam...@gmail.com
> > skype: free.dev
> > mobile: +39 349 8513251
>



-- 
Vincenzo D'Amore
email: v.dam...@gmail.com
skype: free.dev
mobile: +39 349 8513251

Re: SolrCloud 4.8.1 - commit wait

[Mark Miller]

He has waitSearcher as false it looks, so all the time should be in the
commit. So that amount of time does sound odd.

I would certainly change those commit settings though. I would not use
maxDocs, that is an ugly way to control this. And one second is much too
aggressive as Erick says.

If you want to attempt that kind of visibility, you should use the
softAutoCommit. The regular autoCommit should be at least 15 or 20 seconds.

- Mark

On Fri, Dec 11, 2015 at 1:22 PM Erick Erickson 
wrote:

> First of all, your autocommit settings are _very_ aggressive. Committing
> every second is far to frequent IMO.
>
> As an aside, I generally prefer to omit the maxDocs as it's not all
> that predictable,
> but that's a personal preference and really doesn't bear on your problem..
>
> My _guess_ is that you are doing a lot of autowarming. The number of docs
> doesn't really matter if your autowarming is taking forever, your Solr logs
> should report the autowarm times at INFO level, have you checked those?
>
> The commit settings shouldn't be a problem in terms of your server dying,
> the indexing process flushes docs to the tlog independent of committing so
> upon restart they should be recovered. Here's a blog on the subject:
>
>
> https://lucidworks.com/blog/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
>
> Best,
> Erick
>
> On Fri, Dec 11, 2015 at 8:24 AM, Vincenzo D'Amore 
> wrote:
> > Hi all,
> >
> > I have a SolrCloud cluster with a collection (2.5M docs) with 3 shards
> and
> > 15 replicas.
> > There is a solrj application that feeds the collection, updating few
> > documents every hour, I don't understand why, at end of process, the hard
> > commit takes about 8/10 minutes.
> >
> > Even if there are only few hundreds of documents.
> >
> > This is the autocommit configuration:
> >
> > 
> > 1
> > 1000
> > false
> > 
> >
> > In your experience why hard commit takes so long even for so few
> documents?
> >
> > Now I'm changing the code to softcommit, calling commit (waitFlush =
> > false, waitSearcher
> > = false, softCommit = true);
> >
> > solrServer.commit(false, false, true);.
> >
> > I have configured NRTCachingDirectoryFactory, but I'm a little bit
> worried
> > if a server goes down (something like: kill -9, SolrCloud crashes, out of
> > memory, etc.), and if, using this strategy
> softcommit+NRTCachingDirectory,
> > SolrCloud instance could not recover a replica.
> >
> > Should I worry about this new configuration? I was thinking to take a
> > snapshot of everything every day, in order to recover immediately the
> > index. Could this be considered a best practice?
> >
> > Thanks in advance for your time,
> > Vincenzo
> >
> > --
> > Vincenzo D'Amore
> > email: v.dam...@gmail.com
> > skype: free.dev
> > mobile: +39 349 8513251
>
-- 
- Mark
about.me/markrmiller

Re: SolrCloud 4.8.1 - commit wait

[Vincenzo D'Amore]

Hi All,

an update, I have switched logging from WARN to INFO for all except for
those two:

- org.apache.solr.core
- org.apache.solr.handler.component.SpellCheckComponent

Well, looking at log file I'm unable to find any autowarm log line, even
after few updates and commits.

Looking at solrconfig.xml I see most autowarmCount parameters are set to 0






Not sure what this means...

On Sat, Dec 12, 2015 at 1:13 AM, Vincenzo D'Amore 
wrote:

> Thanks Erick, Mark,
>
> I'll raise maxTime asap.
> Just to be sure understand, given that I have openSearcher=false, I
> suppose it shouldn't trigger autowarming at least until a commit is
> executed, shouldn't it?
>
> Anyway, I don't understand, given that maxTime is very aggressive, why
> hard commit takes so long.
>
> Thanks again for your answers.
> Vincenzo
>
>
> On Fri, Dec 11, 2015 at 7:22 PM, Erick Erickson 
> wrote:
>
>> First of all, your autocommit settings are _very_ aggressive. Committing
>> every second is far to frequent IMO.
>>
>> As an aside, I generally prefer to omit the maxDocs as it's not all
>> that predictable,
>> but that's a personal preference and really doesn't bear on your problem..
>>
>> My _guess_ is that you are doing a lot of autowarming. The number of docs
>> doesn't really matter if your autowarming is taking forever, your Solr
>> logs
>> should report the autowarm times at INFO level, have you checked those?
>>
>> The commit settings shouldn't be a problem in terms of your server dying,
>> the indexing process flushes docs to the tlog independent of committing so
>> upon restart they should be recovered. Here's a blog on the subject:
>>
>>
>> https://lucidworks.com/blog/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
>>
>> Best,
>> Erick
>>
>> On Fri, Dec 11, 2015 at 8:24 AM, Vincenzo D'Amore 
>> wrote:
>> > Hi all,
>> >
>> > I have a SolrCloud cluster with a collection (2.5M docs) with 3 shards
>> and
>> > 15 replicas.
>> > There is a solrj application that feeds the collection, updating few
>> > documents every hour, I don't understand why, at end of process, the
>> hard
>> > commit takes about 8/10 minutes.
>> >
>> > Even if there are only few hundreds of documents.
>> >
>> > This is the autocommit configuration:
>> >
>> > 
>> > 1
>> > 1000
>> > false
>> > 
>> >
>> > In your experience why hard commit takes so long even for so few
>> documents?
>> >
>> > Now I'm changing the code to softcommit, calling commit (waitFlush =
>> > false, waitSearcher
>> > = false, softCommit = true);
>> >
>> > solrServer.commit(false, false, true);.
>> >
>> > I have configured NRTCachingDirectoryFactory, but I'm a little bit
>> worried
>> > if a server goes down (something like: kill -9, SolrCloud crashes, out
>> of
>> > memory, etc.), and if, using this strategy
>> softcommit+NRTCachingDirectory,
>> > SolrCloud instance could not recover a replica.
>> >
>> > Should I worry about this new configuration? I was thinking to take a
>> > snapshot of everything every day, in order to recover immediately the
>> > index. Could this be considered a best practice?
>> >
>> > Thanks in advance for your time,
>> > Vincenzo
>> >
>> > --
>> > Vincenzo D'Amore
>> > email: v.dam...@gmail.com
>> > skype: free.dev
>> > mobile: +39 349 8513251
>>
>
>
>
> --
> Vincenzo D'Amore
> email: v.dam...@gmail.com
> skype: free.dev
> mobile: +39 349 8513251
>



-- 
Vincenzo D'Amore
email: v.dam...@gmail.com
skype: free.dev
mobile: +39 349 8513251

Re: SolrCloud 4.8.1 - commit wait

[Erick Erickson]

First of all, your autocommit settings are _very_ aggressive. Committing
every second is far to frequent IMO.

As an aside, I generally prefer to omit the maxDocs as it's not all
that predictable,
but that's a personal preference and really doesn't bear on your problem..

My _guess_ is that you are doing a lot of autowarming. The number of docs
doesn't really matter if your autowarming is taking forever, your Solr logs
should report the autowarm times at INFO level, have you checked those?

The commit settings shouldn't be a problem in terms of your server dying,
the indexing process flushes docs to the tlog independent of committing so
upon restart they should be recovered. Here's a blog on the subject:

https://lucidworks.com/blog/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/

Best,
Erick

On Fri, Dec 11, 2015 at 8:24 AM, Vincenzo D'Amore  wrote:
> Hi all,
>
> I have a SolrCloud cluster with a collection (2.5M docs) with 3 shards and
> 15 replicas.
> There is a solrj application that feeds the collection, updating few
> documents every hour, I don't understand why, at end of process, the hard
> commit takes about 8/10 minutes.
>
> Even if there are only few hundreds of documents.
>
> This is the autocommit configuration:
>
> 
> 1
> 1000
> false
> 
>
> In your experience why hard commit takes so long even for so few documents?
>
> Now I'm changing the code to softcommit, calling commit (waitFlush =
> false, waitSearcher
> = false, softCommit = true);
>
> solrServer.commit(false, false, true);.
>
> I have configured NRTCachingDirectoryFactory, but I'm a little bit worried
> if a server goes down (something like: kill -9, SolrCloud crashes, out of
> memory, etc.), and if, using this strategy softcommit+NRTCachingDirectory,
> SolrCloud instance could not recover a replica.
>
> Should I worry about this new configuration? I was thinking to take a
> snapshot of everything every day, in order to recover immediately the
> index. Could this be considered a best practice?
>
> Thanks in advance for your time,
> Vincenzo
>
> --
> Vincenzo D'Amore
> email: v.dam...@gmail.com
> skype: free.dev
> mobile: +39 349 8513251