RE: Cannot set pollInterval in SolrCloud for PULL or TLOG replica

[Vadim Ivanov]

In my use case, I don't have bulk updates. 
I just have continuously heavy updates on most cores

But maybe you can try to set updateHandler.autoCommit.maxTime through config 
api before and after bulk updates.
updateHandler.autoCommit.maxTime
https://lucene.apache.org/solr/guide/7_6/config-api.html

-- 
Vadim


> -Original Message-
> From: Dmitry Vorotilin [mailto:d.voroti...@gmail.com]
> Sent: Wednesday, April 17, 2019 7:22 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Cannot set pollInterval in SolrCloud for PULL or TLOG replica
> 
> It looks like `/solr//replication?command=disablepoll` doesn't work
> in cloud mode so there's no way to change settings for interval as well as
> to say replicas to stop polling.
> My own conclusion: if you have bulk updates and commit with
> openSearcher=true only at the end PULL/TLOG replicas isn't your choice, the
> only option you have is NRT which burns CPU on all machines slowing down
> all select queries.
> 
> On Wed, Apr 17, 2019 at 3:25 PM Dmitry Vorotilin 
> wrote:
> 
> > Hi Vadim, thank you seems like we both had similar questions.
> > So I think that all confirms that it's not configurable for now. That's in
> > fact a pity because it only makes sense to use PULL/TLOG replicas in order
> > to save CPU and not reindex docs on every node but current situation with
> > reopening searcher every time ruins it all at least for bulk updates. The
> > only solution I see now is to use manual replication and trigger it on
> > every node after leader optimized index and this configuration was
> > available on master-salve legacy...
> >
> > On Tue, Apr 16, 2019 at 6:30 PM Vadim Ivanov <
> > vadim.iva...@spb.ntk-intourist.ru> wrote:
> >
> >> Hi, Dmitri
> >> There was discussion here a while ago...
> >>
> >> http://lucene.472066.n3.nabble.com/Soft-commit-and-new-replica-types-
> td4417253.html
> >> May be it helps you somehow.
> >>
> >> --
> >> Vadim
> >>
> >>
> >> > -Original Message-
> >> > From: Dmitry Vorotilin [mailto:d.voroti...@gmail.com]
> >> > Sent: Tuesday, April 16, 2019 9:41 AM
> >> > To: solr-user@lucene.apache.org
> >> > Subject: Cannot set pollInterval in SolrCloud for PULL or TLOG replica
> >> >
> >> > Hi everyone,
> >> >
> >> > We have SolrCloud cluster with 3 zk and 3 solr nodes. It's 1 shard only
> >> and
> >> > all replicas are PULL.
> >> > We have bulk updates so like once a day we reindex all cores (no soft
> >> > commits, only hard commit every 15s), do commit with
> openSearcher=true
> >> > and
> >> > all our indexes become available for search.
> >> >
> >> > The issue is that for PULL replication when leader reindexing starts it
> >> > downloads index every
> >> > hard commit / 2 seconds (o.a.s.h.ReplicationHandler Poll scheduled at an
> >> > interval of 7000ms) then puts index into proper directory and just
> >> reopens
> >> > searcher so that we see no changes on leader because there was no
> commit
> >> > with openSearcher=true yet and that index keeps growing on PULL
> >> replicas.
> >> >
> >> > Judging by this page
> >> >  >> > replication-in-solr>
> >> > there's no setting for pollInterval or when to start replication on
> >> slaves
> >> > in SolrCloud and the info is rather confusing because in cloud we still
> >> use
> >> > the same handlers which we cannot configure.
> >> >
> >> > We changed replication from NRT to PULL because we don't need
> realtime
> >> > and
> >> > burn CPU with bulk updates on every machine, but this constantly
> >> catching
> >> > up index on slaves isn't any better...
> >> >
> >> > Do you know any way to fix it?
> >>
> >>

Re: Cannot set pollInterval in SolrCloud for PULL or TLOG replica

[Dmitry Vorotilin]

It looks like `/solr//replication?command=disablepoll` doesn't work
in cloud mode so there's no way to change settings for interval as well as
to say replicas to stop polling.
My own conclusion: if you have bulk updates and commit with
openSearcher=true only at the end PULL/TLOG replicas isn't your choice, the
only option you have is NRT which burns CPU on all machines slowing down
all select queries.

On Wed, Apr 17, 2019 at 3:25 PM Dmitry Vorotilin 
wrote:

> Hi Vadim, thank you seems like we both had similar questions.
> So I think that all confirms that it's not configurable for now. That's in
> fact a pity because it only makes sense to use PULL/TLOG replicas in order
> to save CPU and not reindex docs on every node but current situation with
> reopening searcher every time ruins it all at least for bulk updates. The
> only solution I see now is to use manual replication and trigger it on
> every node after leader optimized index and this configuration was
> available on master-salve legacy...
>
> On Tue, Apr 16, 2019 at 6:30 PM Vadim Ivanov <
> vadim.iva...@spb.ntk-intourist.ru> wrote:
>
>> Hi, Dmitri
>> There was discussion here a while ago...
>>
>> http://lucene.472066.n3.nabble.com/Soft-commit-and-new-replica-types-td4417253.html
>> May be it helps you somehow.
>>
>> --
>> Vadim
>>
>>
>> > -Original Message-
>> > From: Dmitry Vorotilin [mailto:d.voroti...@gmail.com]
>> > Sent: Tuesday, April 16, 2019 9:41 AM
>> > To: solr-user@lucene.apache.org
>> > Subject: Cannot set pollInterval in SolrCloud for PULL or TLOG replica
>> >
>> > Hi everyone,
>> >
>> > We have SolrCloud cluster with 3 zk and 3 solr nodes. It's 1 shard only
>> and
>> > all replicas are PULL.
>> > We have bulk updates so like once a day we reindex all cores (no soft
>> > commits, only hard commit every 15s), do commit with openSearcher=true
>> > and
>> > all our indexes become available for search.
>> >
>> > The issue is that for PULL replication when leader reindexing starts it
>> > downloads index every
>> > hard commit / 2 seconds (o.a.s.h.ReplicationHandler Poll scheduled at an
>> > interval of 7000ms) then puts index into proper directory and just
>> reopens
>> > searcher so that we see no changes on leader because there was no commit
>> > with openSearcher=true yet and that index keeps growing on PULL
>> replicas.
>> >
>> > Judging by this page
>> > > > replication-in-solr>
>> > there's no setting for pollInterval or when to start replication on
>> slaves
>> > in SolrCloud and the info is rather confusing because in cloud we still
>> use
>> > the same handlers which we cannot configure.
>> >
>> > We changed replication from NRT to PULL because we don't need realtime
>> > and
>> > burn CPU with bulk updates on every machine, but this constantly
>> catching
>> > up index on slaves isn't any better...
>> >
>> > Do you know any way to fix it?
>>
>>

Re: Cannot set pollInterval in SolrCloud for PULL or TLOG replica

[Dmitry Vorotilin]

Hi Vadim, thank you seems like we both had similar questions.
So I think that all confirms that it's not configurable for now. That's in
fact a pity because it only makes sense to use PULL/TLOG replicas in order
to save CPU and not reindex docs on every node but current situation with
reopening searcher every time ruins it all at least for bulk updates. The
only solution I see now is to use manual replication and trigger it on
every node after leader optimized index and this configuration was
available on master-salve legacy...

On Tue, Apr 16, 2019 at 6:30 PM Vadim Ivanov <
vadim.iva...@spb.ntk-intourist.ru> wrote:

> Hi, Dmitri
> There was discussion here a while ago...
>
> http://lucene.472066.n3.nabble.com/Soft-commit-and-new-replica-types-td4417253.html
> May be it helps you somehow.
>
> --
> Vadim
>
>
> > -Original Message-
> > From: Dmitry Vorotilin [mailto:d.voroti...@gmail.com]
> > Sent: Tuesday, April 16, 2019 9:41 AM
> > To: solr-user@lucene.apache.org
> > Subject: Cannot set pollInterval in SolrCloud for PULL or TLOG replica
> >
> > Hi everyone,
> >
> > We have SolrCloud cluster with 3 zk and 3 solr nodes. It's 1 shard only
> and
> > all replicas are PULL.
> > We have bulk updates so like once a day we reindex all cores (no soft
> > commits, only hard commit every 15s), do commit with openSearcher=true
> > and
> > all our indexes become available for search.
> >
> > The issue is that for PULL replication when leader reindexing starts it
> > downloads index every
> > hard commit / 2 seconds (o.a.s.h.ReplicationHandler Poll scheduled at an
> > interval of 7000ms) then puts index into proper directory and just
> reopens
> > searcher so that we see no changes on leader because there was no commit
> > with openSearcher=true yet and that index keeps growing on PULL replicas.
> >
> > Judging by this page
> >  > replication-in-solr>
> > there's no setting for pollInterval or when to start replication on
> slaves
> > in SolrCloud and the info is rather confusing because in cloud we still
> use
> > the same handlers which we cannot configure.
> >
> > We changed replication from NRT to PULL because we don't need realtime
> > and
> > burn CPU with bulk updates on every machine, but this constantly catching
> > up index on slaves isn't any better...
> >
> > Do you know any way to fix it?
>
>

RE: Cannot set pollInterval in SolrCloud for PULL or TLOG replica

[Vadim Ivanov]

Hi, Dmitri
There was discussion here a while ago...
http://lucene.472066.n3.nabble.com/Soft-commit-and-new-replica-types-td4417253.html
May be it helps you somehow.

-- 
Vadim


> -Original Message-
> From: Dmitry Vorotilin [mailto:d.voroti...@gmail.com]
> Sent: Tuesday, April 16, 2019 9:41 AM
> To: solr-user@lucene.apache.org
> Subject: Cannot set pollInterval in SolrCloud for PULL or TLOG replica
> 
> Hi everyone,
> 
> We have SolrCloud cluster with 3 zk and 3 solr nodes. It's 1 shard only and
> all replicas are PULL.
> We have bulk updates so like once a day we reindex all cores (no soft
> commits, only hard commit every 15s), do commit with openSearcher=true
> and
> all our indexes become available for search.
> 
> The issue is that for PULL replication when leader reindexing starts it
> downloads index every
> hard commit / 2 seconds (o.a.s.h.ReplicationHandler Poll scheduled at an
> interval of 7000ms) then puts index into proper directory and just reopens
> searcher so that we see no changes on leader because there was no commit
> with openSearcher=true yet and that index keeps growing on PULL replicas.
> 
> Judging by this page
>  replication-in-solr>
> there's no setting for pollInterval or when to start replication on slaves
> in SolrCloud and the info is rather confusing because in cloud we still use
> the same handlers which we cannot configure.
> 
> We changed replication from NRT to PULL because we don't need realtime
> and
> burn CPU with bulk updates on every machine, but this constantly catching
> up index on slaves isn't any better...
> 
> Do you know any way to fix it?