Re: CDCR Invalid Number on deletes

[Amrit Sarkar]

Hi Chris,

Sorry I was off work for few days and didn't follow the conversation. The
link is directing me to
https://issues.apache.org/jira/projects/SOLR/issues/SOLR-12063. I think we
have fixed the issue stated by you in the jira, though the symptoms were
different than yours.

Amrit Sarkar
Search Engineer
Lucidworks, Inc.
415-589-9269
www.lucidworks.com
Twitter http://twitter.com/lucidworks
LinkedIn: https://www.linkedin.com/in/sarkaramrit2
Medium: https://medium.com/@sarkaramrit2

On Wed, Mar 21, 2018 at 1:17 AM, Chris Troullis 
wrote:

> Nevermind I found itthe link you posted links me to SOLR-12036 instead
> of SOLR-12063 for some reason.
>
> On Tue, Mar 20, 2018 at 1:51 PM, Chris Troullis 
> wrote:
>
> > Hey Amrit,
> >
> > Did you happen to see my last reply?  Is SOLR-12036 the correct JIRA?
> >
> > Thanks,
> >
> > Chris
> >
> > On Wed, Mar 7, 2018 at 1:52 PM, Chris Troullis 
> > wrote:
> >
> >> Hey Amrit, thanks for the reply!
> >>
> >> I checked out SOLR-12036, but it doesn't look like it has to do with
> >> CDCR, and the patch that is attached doesn't look CDCR related. Are you
> >> sure that's the correct JIRA number?
> >>
> >> Thanks,
> >>
> >> Chris
> >>
> >> On Wed, Mar 7, 2018 at 11:21 AM, Amrit Sarkar 
> >> wrote:
> >>
> >>> Hey Chris,
> >>>
> >>> I figured a separate issue while working on CDCR which may relate to
> your
> >>> problem. Please see jira: *SOLR-12063*
> >>> . This
> >>> is a
> >>> bug got introduced when we supported the bidirectional approach where
> an
> >>> extra flag in tlog entry for cdcr is added.
> >>>
> >>> This part of the code is messing up:
> >>> *UpdateLog.java.RecentUpdates::update()::*
> >>>
> >>> switch (oper) {
> >>>   case UpdateLog.ADD:
> >>>   case UpdateLog.UPDATE_INPLACE:
> >>>   case UpdateLog.DELETE:
> >>>   case UpdateLog.DELETE_BY_QUERY:
> >>> Update update = new Update();
> >>> update.log = oldLog;
> >>> update.pointer = reader.position();
> >>> update.version = version;
> >>>
> >>> if (oper == UpdateLog.UPDATE_INPLACE && entry.size() == 5) {
> >>>   update.previousVersion = (Long) entry.get(UpdateLog.PREV_VERSI
> >>> ON_IDX);
> >>> }
> >>> updatesForLog.add(update);
> >>> updates.put(version, update);
> >>>
> >>> if (oper == UpdateLog.DELETE_BY_QUERY) {
> >>>   deleteByQueryList.add(update);
> >>> } else if (oper == UpdateLog.DELETE) {
> >>>   deleteList.add(new DeleteUpdate(version,
> >>> (byte[])entry.get(entry.size()-1)));
> >>> }
> >>>
> >>> break;
> >>>
> >>>   case UpdateLog.COMMIT:
> >>> break;
> >>>   default:
> >>> throw new SolrException(SolrException.ErrorCode.SERVER_ERROR,
> >>> "Unknown Operation! " + oper);
> >>> }
> >>>
> >>> deleteList.add(new DeleteUpdate(version, (byte[])entry.get(entry.size()
> >>> -1)));
> >>>
> >>> is expecting the last entry to be the payload, but everywhere in the
> >>> project, *pos:[2] *is the index for the payload, while the last entry
> in
> >>> source code is *boolean* in / after Solr 7.2, denoting update is cdcr
> >>> forwarded or typical. UpdateLog.java.RecentUpdates is used to in cdcr
> >>> sync,
> >>> checkpoint operations and hence it is a legit bug, slipped the tests I
> >>> wrote.
> >>>
> >>> The immediate fix patch is uploaded and I am awaiting feedback on that.
> >>> Meanwhile if it is possible for you to apply the patch, build the jar
> and
> >>> try it out, please do and let us know.
> >>>
> >>> For, *SOLR-9394* , if
> >>> you
> >>> can comment on the JIRA and post the sample docs, solr logs, relevant
> >>> information, I can give it a thorough look.
> >>>
> >>> Amrit Sarkar
> >>> Search Engineer
> >>> Lucidworks, Inc.
> >>> 415-589-9269
> >>> www.lucidworks.com
> >>> Twitter http://twitter.com/lucidworks
> >>> LinkedIn: https://www.linkedin.com/in/sarkaramrit2
> >>> Medium: https://medium.com/@sarkaramrit2
> >>>
> >>> On Wed, Mar 7, 2018 at 1:35 AM, Chris Troullis 
> >>> wrote:
> >>>
> >>> > Hi all,
> >>> >
> >>> > We recently upgraded to Solr 7.2.0 as we saw that there were some
> CDCR
> >>> bug
> >>> > fixes and features added that would finally let us be able to make
> use
> >>> of
> >>> > it (bi-directional syncing was the big one). The first time we tried
> to
> >>> > implement we ran into all kinds of errors, but this time we were able
> >>> to
> >>> > get it mostly working.
> >>> >
> >>> > The issue we seem to be having now is that any time a document is
> >>> deleted
> >>> > via deleteById from a collection on the primary node, we are flooded
> >>> with
> >>> > "Invalid Number" errors followed by a random sequence of characters
> >>> when
> >>> > CDCR tries to sync the update to the backup site. This happens on all
> >>> of
> >>> > our collections where our id fields are defined as longs (some 

Re: CDCR Invalid Number on deletes

[Chris Troullis]

Nevermind I found itthe link you posted links me to SOLR-12036 instead
of SOLR-12063 for some reason.

On Tue, Mar 20, 2018 at 1:51 PM, Chris Troullis 
wrote:

> Hey Amrit,
>
> Did you happen to see my last reply?  Is SOLR-12036 the correct JIRA?
>
> Thanks,
>
> Chris
>
> On Wed, Mar 7, 2018 at 1:52 PM, Chris Troullis 
> wrote:
>
>> Hey Amrit, thanks for the reply!
>>
>> I checked out SOLR-12036, but it doesn't look like it has to do with
>> CDCR, and the patch that is attached doesn't look CDCR related. Are you
>> sure that's the correct JIRA number?
>>
>> Thanks,
>>
>> Chris
>>
>> On Wed, Mar 7, 2018 at 11:21 AM, Amrit Sarkar 
>> wrote:
>>
>>> Hey Chris,
>>>
>>> I figured a separate issue while working on CDCR which may relate to your
>>> problem. Please see jira: *SOLR-12063*
>>> . This
>>> is a
>>> bug got introduced when we supported the bidirectional approach where an
>>> extra flag in tlog entry for cdcr is added.
>>>
>>> This part of the code is messing up:
>>> *UpdateLog.java.RecentUpdates::update()::*
>>>
>>> switch (oper) {
>>>   case UpdateLog.ADD:
>>>   case UpdateLog.UPDATE_INPLACE:
>>>   case UpdateLog.DELETE:
>>>   case UpdateLog.DELETE_BY_QUERY:
>>> Update update = new Update();
>>> update.log = oldLog;
>>> update.pointer = reader.position();
>>> update.version = version;
>>>
>>> if (oper == UpdateLog.UPDATE_INPLACE && entry.size() == 5) {
>>>   update.previousVersion = (Long) entry.get(UpdateLog.PREV_VERSI
>>> ON_IDX);
>>> }
>>> updatesForLog.add(update);
>>> updates.put(version, update);
>>>
>>> if (oper == UpdateLog.DELETE_BY_QUERY) {
>>>   deleteByQueryList.add(update);
>>> } else if (oper == UpdateLog.DELETE) {
>>>   deleteList.add(new DeleteUpdate(version,
>>> (byte[])entry.get(entry.size()-1)));
>>> }
>>>
>>> break;
>>>
>>>   case UpdateLog.COMMIT:
>>> break;
>>>   default:
>>> throw new SolrException(SolrException.ErrorCode.SERVER_ERROR,
>>> "Unknown Operation! " + oper);
>>> }
>>>
>>> deleteList.add(new DeleteUpdate(version, (byte[])entry.get(entry.size()
>>> -1)));
>>>
>>> is expecting the last entry to be the payload, but everywhere in the
>>> project, *pos:[2] *is the index for the payload, while the last entry in
>>> source code is *boolean* in / after Solr 7.2, denoting update is cdcr
>>> forwarded or typical. UpdateLog.java.RecentUpdates is used to in cdcr
>>> sync,
>>> checkpoint operations and hence it is a legit bug, slipped the tests I
>>> wrote.
>>>
>>> The immediate fix patch is uploaded and I am awaiting feedback on that.
>>> Meanwhile if it is possible for you to apply the patch, build the jar and
>>> try it out, please do and let us know.
>>>
>>> For, *SOLR-9394* , if
>>> you
>>> can comment on the JIRA and post the sample docs, solr logs, relevant
>>> information, I can give it a thorough look.
>>>
>>> Amrit Sarkar
>>> Search Engineer
>>> Lucidworks, Inc.
>>> 415-589-9269
>>> www.lucidworks.com
>>> Twitter http://twitter.com/lucidworks
>>> LinkedIn: https://www.linkedin.com/in/sarkaramrit2
>>> Medium: https://medium.com/@sarkaramrit2
>>>
>>> On Wed, Mar 7, 2018 at 1:35 AM, Chris Troullis 
>>> wrote:
>>>
>>> > Hi all,
>>> >
>>> > We recently upgraded to Solr 7.2.0 as we saw that there were some CDCR
>>> bug
>>> > fixes and features added that would finally let us be able to make use
>>> of
>>> > it (bi-directional syncing was the big one). The first time we tried to
>>> > implement we ran into all kinds of errors, but this time we were able
>>> to
>>> > get it mostly working.
>>> >
>>> > The issue we seem to be having now is that any time a document is
>>> deleted
>>> > via deleteById from a collection on the primary node, we are flooded
>>> with
>>> > "Invalid Number" errors followed by a random sequence of characters
>>> when
>>> > CDCR tries to sync the update to the backup site. This happens on all
>>> of
>>> > our collections where our id fields are defined as longs (some of them
>>> the
>>> > ids are compound keys and are strings).
>>> >
>>> > Here's a sample exception:
>>> >
>>> > org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException:
>>> Error
>>> > from server at http://ip/solr/collection_shard1_replica_n1: Invalid
>>> > Number:  ]
>>> > -s
>>> > at
>>> > org.apache.solr.client.solrj.impl.CloudSolrClient.
>>> > directUpdate(CloudSolrClient.java:549)
>>> > at
>>> > org.apache.solr.client.solrj.impl.CloudSolrClient.
>>> > sendRequest(CloudSolrClient.java:1012)
>>> > at
>>> > org.apache.solr.client.solrj.impl.CloudSolrClient.
>>> > requestWithRetryOnStaleState(CloudSolrClient.java:883)
>>> > at
>>> > org.apache.solr.client.solrj.impl.CloudSolrClient.
>>> > requestWithRetryOnStaleState(CloudSolrClient.java:945)
>>> > at
>>> > 

Re: CDCR Invalid Number on deletes

[Chris Troullis]

Hey Amrit,

Did you happen to see my last reply?  Is SOLR-12036 the correct JIRA?

Thanks,

Chris

On Wed, Mar 7, 2018 at 1:52 PM, Chris Troullis  wrote:

> Hey Amrit, thanks for the reply!
>
> I checked out SOLR-12036, but it doesn't look like it has to do with CDCR,
> and the patch that is attached doesn't look CDCR related. Are you sure
> that's the correct JIRA number?
>
> Thanks,
>
> Chris
>
> On Wed, Mar 7, 2018 at 11:21 AM, Amrit Sarkar 
> wrote:
>
>> Hey Chris,
>>
>> I figured a separate issue while working on CDCR which may relate to your
>> problem. Please see jira: *SOLR-12063*
>> . This
>> is a
>> bug got introduced when we supported the bidirectional approach where an
>> extra flag in tlog entry for cdcr is added.
>>
>> This part of the code is messing up:
>> *UpdateLog.java.RecentUpdates::update()::*
>>
>> switch (oper) {
>>   case UpdateLog.ADD:
>>   case UpdateLog.UPDATE_INPLACE:
>>   case UpdateLog.DELETE:
>>   case UpdateLog.DELETE_BY_QUERY:
>> Update update = new Update();
>> update.log = oldLog;
>> update.pointer = reader.position();
>> update.version = version;
>>
>> if (oper == UpdateLog.UPDATE_INPLACE && entry.size() == 5) {
>>   update.previousVersion = (Long) entry.get(UpdateLog.PREV_VERSI
>> ON_IDX);
>> }
>> updatesForLog.add(update);
>> updates.put(version, update);
>>
>> if (oper == UpdateLog.DELETE_BY_QUERY) {
>>   deleteByQueryList.add(update);
>> } else if (oper == UpdateLog.DELETE) {
>>   deleteList.add(new DeleteUpdate(version,
>> (byte[])entry.get(entry.size()-1)));
>> }
>>
>> break;
>>
>>   case UpdateLog.COMMIT:
>> break;
>>   default:
>> throw new SolrException(SolrException.ErrorCode.SERVER_ERROR,
>> "Unknown Operation! " + oper);
>> }
>>
>> deleteList.add(new DeleteUpdate(version, (byte[])entry.get(entry.size()
>> -1)));
>>
>> is expecting the last entry to be the payload, but everywhere in the
>> project, *pos:[2] *is the index for the payload, while the last entry in
>> source code is *boolean* in / after Solr 7.2, denoting update is cdcr
>> forwarded or typical. UpdateLog.java.RecentUpdates is used to in cdcr
>> sync,
>> checkpoint operations and hence it is a legit bug, slipped the tests I
>> wrote.
>>
>> The immediate fix patch is uploaded and I am awaiting feedback on that.
>> Meanwhile if it is possible for you to apply the patch, build the jar and
>> try it out, please do and let us know.
>>
>> For, *SOLR-9394* , if
>> you
>> can comment on the JIRA and post the sample docs, solr logs, relevant
>> information, I can give it a thorough look.
>>
>> Amrit Sarkar
>> Search Engineer
>> Lucidworks, Inc.
>> 415-589-9269
>> www.lucidworks.com
>> Twitter http://twitter.com/lucidworks
>> LinkedIn: https://www.linkedin.com/in/sarkaramrit2
>> Medium: https://medium.com/@sarkaramrit2
>>
>> On Wed, Mar 7, 2018 at 1:35 AM, Chris Troullis 
>> wrote:
>>
>> > Hi all,
>> >
>> > We recently upgraded to Solr 7.2.0 as we saw that there were some CDCR
>> bug
>> > fixes and features added that would finally let us be able to make use
>> of
>> > it (bi-directional syncing was the big one). The first time we tried to
>> > implement we ran into all kinds of errors, but this time we were able to
>> > get it mostly working.
>> >
>> > The issue we seem to be having now is that any time a document is
>> deleted
>> > via deleteById from a collection on the primary node, we are flooded
>> with
>> > "Invalid Number" errors followed by a random sequence of characters when
>> > CDCR tries to sync the update to the backup site. This happens on all of
>> > our collections where our id fields are defined as longs (some of them
>> the
>> > ids are compound keys and are strings).
>> >
>> > Here's a sample exception:
>> >
>> > org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: Error
>> > from server at http://ip/solr/collection_shard1_replica_n1: Invalid
>> > Number:  ]
>> > -s
>> > at
>> > org.apache.solr.client.solrj.impl.CloudSolrClient.
>> > directUpdate(CloudSolrClient.java:549)
>> > at
>> > org.apache.solr.client.solrj.impl.CloudSolrClient.
>> > sendRequest(CloudSolrClient.java:1012)
>> > at
>> > org.apache.solr.client.solrj.impl.CloudSolrClient.
>> > requestWithRetryOnStaleState(CloudSolrClient.java:883)
>> > at
>> > org.apache.solr.client.solrj.impl.CloudSolrClient.
>> > requestWithRetryOnStaleState(CloudSolrClient.java:945)
>> > at
>> > org.apache.solr.client.solrj.impl.CloudSolrClient.
>> > requestWithRetryOnStaleState(CloudSolrClient.java:945)
>> > at
>> > org.apache.solr.client.solrj.impl.CloudSolrClient.
>> > requestWithRetryOnStaleState(CloudSolrClient.java:945)
>> > at
>> > org.apache.solr.client.solrj.impl.CloudSolrClient.
>> > 

Re: CDCR Invalid Number on deletes

[Chris Troullis]

Hey Amrit, thanks for the reply!

I checked out SOLR-12036, but it doesn't look like it has to do with CDCR,
and the patch that is attached doesn't look CDCR related. Are you sure
that's the correct JIRA number?

Thanks,

Chris

On Wed, Mar 7, 2018 at 11:21 AM, Amrit Sarkar 
wrote:

> Hey Chris,
>
> I figured a separate issue while working on CDCR which may relate to your
> problem. Please see jira: *SOLR-12063*
> . This is
> a
> bug got introduced when we supported the bidirectional approach where an
> extra flag in tlog entry for cdcr is added.
>
> This part of the code is messing up:
> *UpdateLog.java.RecentUpdates::update()::*
>
> switch (oper) {
>   case UpdateLog.ADD:
>   case UpdateLog.UPDATE_INPLACE:
>   case UpdateLog.DELETE:
>   case UpdateLog.DELETE_BY_QUERY:
> Update update = new Update();
> update.log = oldLog;
> update.pointer = reader.position();
> update.version = version;
>
> if (oper == UpdateLog.UPDATE_INPLACE && entry.size() == 5) {
>   update.previousVersion = (Long) entry.get(UpdateLog.PREV_
> VERSION_IDX);
> }
> updatesForLog.add(update);
> updates.put(version, update);
>
> if (oper == UpdateLog.DELETE_BY_QUERY) {
>   deleteByQueryList.add(update);
> } else if (oper == UpdateLog.DELETE) {
>   deleteList.add(new DeleteUpdate(version,
> (byte[])entry.get(entry.size()-1)));
> }
>
> break;
>
>   case UpdateLog.COMMIT:
> break;
>   default:
> throw new SolrException(SolrException.ErrorCode.SERVER_ERROR,
> "Unknown Operation! " + oper);
> }
>
> deleteList.add(new DeleteUpdate(version, (byte[])entry.get(entry.size()
> -1)));
>
> is expecting the last entry to be the payload, but everywhere in the
> project, *pos:[2] *is the index for the payload, while the last entry in
> source code is *boolean* in / after Solr 7.2, denoting update is cdcr
> forwarded or typical. UpdateLog.java.RecentUpdates is used to in cdcr sync,
> checkpoint operations and hence it is a legit bug, slipped the tests I
> wrote.
>
> The immediate fix patch is uploaded and I am awaiting feedback on that.
> Meanwhile if it is possible for you to apply the patch, build the jar and
> try it out, please do and let us know.
>
> For, *SOLR-9394* , if you
> can comment on the JIRA and post the sample docs, solr logs, relevant
> information, I can give it a thorough look.
>
> Amrit Sarkar
> Search Engineer
> Lucidworks, Inc.
> 415-589-9269
> www.lucidworks.com
> Twitter http://twitter.com/lucidworks
> LinkedIn: https://www.linkedin.com/in/sarkaramrit2
> Medium: https://medium.com/@sarkaramrit2
>
> On Wed, Mar 7, 2018 at 1:35 AM, Chris Troullis 
> wrote:
>
> > Hi all,
> >
> > We recently upgraded to Solr 7.2.0 as we saw that there were some CDCR
> bug
> > fixes and features added that would finally let us be able to make use of
> > it (bi-directional syncing was the big one). The first time we tried to
> > implement we ran into all kinds of errors, but this time we were able to
> > get it mostly working.
> >
> > The issue we seem to be having now is that any time a document is deleted
> > via deleteById from a collection on the primary node, we are flooded with
> > "Invalid Number" errors followed by a random sequence of characters when
> > CDCR tries to sync the update to the backup site. This happens on all of
> > our collections where our id fields are defined as longs (some of them
> the
> > ids are compound keys and are strings).
> >
> > Here's a sample exception:
> >
> > org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: Error
> > from server at http://ip/solr/collection_shard1_replica_n1: Invalid
> > Number:  ]
> > -s
> > at
> > org.apache.solr.client.solrj.impl.CloudSolrClient.
> > directUpdate(CloudSolrClient.java:549)
> > at
> > org.apache.solr.client.solrj.impl.CloudSolrClient.
> > sendRequest(CloudSolrClient.java:1012)
> > at
> > org.apache.solr.client.solrj.impl.CloudSolrClient.
> > requestWithRetryOnStaleState(CloudSolrClient.java:883)
> > at
> > org.apache.solr.client.solrj.impl.CloudSolrClient.
> > requestWithRetryOnStaleState(CloudSolrClient.java:945)
> > at
> > org.apache.solr.client.solrj.impl.CloudSolrClient.
> > requestWithRetryOnStaleState(CloudSolrClient.java:945)
> > at
> > org.apache.solr.client.solrj.impl.CloudSolrClient.
> > requestWithRetryOnStaleState(CloudSolrClient.java:945)
> > at
> > org.apache.solr.client.solrj.impl.CloudSolrClient.
> > requestWithRetryOnStaleState(CloudSolrClient.java:945)
> > at
> > org.apache.solr.client.solrj.impl.CloudSolrClient.
> > requestWithRetryOnStaleState(CloudSolrClient.java:945)
> > at
> > org.apache.solr.client.solrj.impl.CloudSolrClient.request(
> > CloudSolrClient.java:816)
> > at
> > 

Re: CDCR Invalid Number on deletes

[Amrit Sarkar]

Hey Chris,

I figured a separate issue while working on CDCR which may relate to your
problem. Please see jira: *SOLR-12063*
. This is a
bug got introduced when we supported the bidirectional approach where an
extra flag in tlog entry for cdcr is added.

This part of the code is messing up:
*UpdateLog.java.RecentUpdates::update()::*

switch (oper) {
  case UpdateLog.ADD:
  case UpdateLog.UPDATE_INPLACE:
  case UpdateLog.DELETE:
  case UpdateLog.DELETE_BY_QUERY:
Update update = new Update();
update.log = oldLog;
update.pointer = reader.position();
update.version = version;

if (oper == UpdateLog.UPDATE_INPLACE && entry.size() == 5) {
  update.previousVersion = (Long) entry.get(UpdateLog.PREV_VERSION_IDX);
}
updatesForLog.add(update);
updates.put(version, update);

if (oper == UpdateLog.DELETE_BY_QUERY) {
  deleteByQueryList.add(update);
} else if (oper == UpdateLog.DELETE) {
  deleteList.add(new DeleteUpdate(version,
(byte[])entry.get(entry.size()-1)));
}

break;

  case UpdateLog.COMMIT:
break;
  default:
throw new SolrException(SolrException.ErrorCode.SERVER_ERROR,
"Unknown Operation! " + oper);
}

deleteList.add(new DeleteUpdate(version, (byte[])entry.get(entry.size()-1)));

is expecting the last entry to be the payload, but everywhere in the
project, *pos:[2] *is the index for the payload, while the last entry in
source code is *boolean* in / after Solr 7.2, denoting update is cdcr
forwarded or typical. UpdateLog.java.RecentUpdates is used to in cdcr sync,
checkpoint operations and hence it is a legit bug, slipped the tests I
wrote.

The immediate fix patch is uploaded and I am awaiting feedback on that.
Meanwhile if it is possible for you to apply the patch, build the jar and
try it out, please do and let us know.

For, *SOLR-9394* , if you
can comment on the JIRA and post the sample docs, solr logs, relevant
information, I can give it a thorough look.

Amrit Sarkar
Search Engineer
Lucidworks, Inc.
415-589-9269
www.lucidworks.com
Twitter http://twitter.com/lucidworks
LinkedIn: https://www.linkedin.com/in/sarkaramrit2
Medium: https://medium.com/@sarkaramrit2

On Wed, Mar 7, 2018 at 1:35 AM, Chris Troullis  wrote:

> Hi all,
>
> We recently upgraded to Solr 7.2.0 as we saw that there were some CDCR bug
> fixes and features added that would finally let us be able to make use of
> it (bi-directional syncing was the big one). The first time we tried to
> implement we ran into all kinds of errors, but this time we were able to
> get it mostly working.
>
> The issue we seem to be having now is that any time a document is deleted
> via deleteById from a collection on the primary node, we are flooded with
> "Invalid Number" errors followed by a random sequence of characters when
> CDCR tries to sync the update to the backup site. This happens on all of
> our collections where our id fields are defined as longs (some of them the
> ids are compound keys and are strings).
>
> Here's a sample exception:
>
> org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: Error
> from server at http://ip/solr/collection_shard1_replica_n1: Invalid
> Number:  ]
> -s
> at
> org.apache.solr.client.solrj.impl.CloudSolrClient.
> directUpdate(CloudSolrClient.java:549)
> at
> org.apache.solr.client.solrj.impl.CloudSolrClient.
> sendRequest(CloudSolrClient.java:1012)
> at
> org.apache.solr.client.solrj.impl.CloudSolrClient.
> requestWithRetryOnStaleState(CloudSolrClient.java:883)
> at
> org.apache.solr.client.solrj.impl.CloudSolrClient.
> requestWithRetryOnStaleState(CloudSolrClient.java:945)
> at
> org.apache.solr.client.solrj.impl.CloudSolrClient.
> requestWithRetryOnStaleState(CloudSolrClient.java:945)
> at
> org.apache.solr.client.solrj.impl.CloudSolrClient.
> requestWithRetryOnStaleState(CloudSolrClient.java:945)
> at
> org.apache.solr.client.solrj.impl.CloudSolrClient.
> requestWithRetryOnStaleState(CloudSolrClient.java:945)
> at
> org.apache.solr.client.solrj.impl.CloudSolrClient.
> requestWithRetryOnStaleState(CloudSolrClient.java:945)
> at
> org.apache.solr.client.solrj.impl.CloudSolrClient.request(
> CloudSolrClient.java:816)
> at
> org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
> at
> org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
> at
> org.apache.solr.handler.CdcrReplicator.sendRequest(
> CdcrReplicator.java:140)
> at
> org.apache.solr.handler.CdcrReplicator.run(CdcrReplicator.java:104)
> at
> org.apache.solr.handler.CdcrReplicatorScheduler.lambda$null$0(
> CdcrReplicatorScheduler.java:81)
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.
> lambda$execute$0(ExecutorUtil.java:188)
> at
>