You can not store more than 2G docs per shard in "Lucene 4.x" codecs. This is a documented Lucene limit:
"Similarly, Lucene uses a Java int to refer to document numbers, and the index file format uses an Int32 on-disk to store document numbers. This is a limitation of both the index file format and the current implementation. Eventually these should be replaced with either UInt64 values, or better yet, VInt values which have no limit." https://lucene.apache.org/core/4_9_1/core/org/apache/lucene/codecs/lucene49/package-summary.html#Limitations Jörg On Tue, Oct 14, 2014 at 5:37 PM, Prasanth R <[email protected]> wrote: > Thanks for the reply. > > My scenario here is, > 1) No nested docs. > 2) I don't have any limit per shard.. > > I didn't know about internal limit of ES. > On Oct 14, 2014 8:23 PM, "Alexandre Rafalovitch" <[email protected]> > wrote: > >> On 14 October 2014 10:33, Prasanth R <[email protected]> wrote: >> > There is no upper limit... >> >> Well, then you must have an infinitely scalable architecture and a >> decision when the content starts getting shared. So, then the question >> is what is your individual shard allowed to grow to. Which is "how >> many documents - including nested - you are expecting to have in a >> single shard". >> >> Because, ElasticSearch has an internal limit and you just hit it. So, >> the question is whether it is intentional, unintentional or a result >> of a bug. >> >> Regards, >> Alex. >> >> Personal: http://www.outerthoughts.com/ and @arafalov >> Solr resources and newsletter: http://www.solr-start.com/ and @solrstart >> Solr popularizers community: https://www.linkedin.com/groups?gid=6713853 >> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "elasticsearch" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/elasticsearch/H74KAYmGtoc/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/CAEFAe-G8GsLZpeO9-b5_RGs%3D-tGBNp39QgeY2rywjRnOQfcfnw%40mail.gmail.com >> . >> For more options, visit https://groups.google.com/d/optout. >> > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/CAJLGCR1LRDnv9Nw71py%3DPFksSqLnXOG6x6-K0kUW1j26%3D6_pYA%40mail.gmail.com > <https://groups.google.com/d/msgid/elasticsearch/CAJLGCR1LRDnv9Nw71py%3DPFksSqLnXOG6x6-K0kUW1j26%3D6_pYA%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHFeV%3D6C%2BZWSs85yhZnzU%3D-Eycw4aPZUy4%2BDLYEL_0bXQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
