Re: number of documents exceed 2147483519

[Hongxu Ma]

I was trying "SPLITSHARD" in my test env and encountered a strange behavior:
I created a 1M doc collection and it contained 8 shards, then split shard1, 
after split:

  *   SPLITSHARD returned success.
  *   looks good
 *   shard1 (became inactive) -> shard1_0 and shard1_1
 *   range 8000-9fff -> 8000-8fff and 9000-9fff
  *   but
 *   the doc number is increasing: docnum of shard1_0 (70702) + docnum of 
shard1_1(67980) > docnum of shard1(124818)

I tested many times and this issue happened every time. Why?

Thanks.









From: Hongxu Ma 
Sent: Monday, March 16, 2020 16:46
To: solr-user@lucene.apache.org 
Subject: number of documents exceed 2147483519

Hi
I'm using solr-cloud (ver 6.6), got an error:
org.apache.solr.common.SolrException: Exception writing document id (null) to 
the index; possible analysis error: number of documents in the index cannot 
exceed 2147483519

After googled it, I know the number is exceed one solr shard limit.
The collection has 64 shards, so I think total limit is 20B*64=128B

My question is:
I don't want to recreate index (then split to more shards) and also don't want 
to delete docs.
Can I using the "SPLITSHARD" api to fix this issue?
https://lucene.apache.org/solr/guide/6_6/collections-api.html#CollectionsAPI-splitshard

After split each shard (now 128 shards), I think the total limit is increasing 
to 256B, right?

Thanks.


Collections API | Apache Solr Reference Guide 
6.6<https://lucene.apache.org/solr/guide/6_6/collections-api.html#CollectionsAPI-splitshard>
The Collections API is used to enable you to create, remove, or reload 
collections, but in the context of SolrCloud you can also use it to create 
collections with a specific number of shards and replicas.
lucene.apache.org

number of documents exceed 2147483519

[Hongxu Ma]

Hi
I'm using solr-cloud (ver 6.6), got an error:
org.apache.solr.common.SolrException: Exception writing document id (null) to 
the index; possible analysis error: number of documents in the index cannot 
exceed 2147483519

After googled it, I know the number is exceed one solr shard limit.
The collection has 64 shards, so I think total limit is 20B*64=128B

My question is:
I don't want to recreate index (then split to more shards) and also don't want 
to delete docs.
Can I using the "SPLITSHARD" api to fix this issue?
https://lucene.apache.org/solr/guide/6_6/collections-api.html#CollectionsAPI-splitshard

After split each shard (now 128 shards), I think the total limit is increasing 
to 256B, right?

Thanks.


Collections API | Apache Solr Reference Guide 
6.6
The Collections API is used to enable you to create, remove, or reload 
collections, but in the context of SolrCloud you can also use it to create 
collections with a specific number of shards and replicas.
lucene.apache.org