Hi, the total number of documents in an index of lucene is Integer.MAX_VALUE. So using a single lucene index to index billions documents is not a proper ways. You should consider using Solr Cloud or Elasticsearch to index your documents.
On 19:43, Fri, 12 Jun 2015 Gimantha Bandara <giman...@wso2.com> wrote: > Hi all, > > We are using Lucene 4.10.3 for indexing. Recently we changed our > implementation so that we give data batchwise to lucene to index. Earlier > we just query all the data from the data source and index all data at > once. It works well. But the number of entries can be up to billions. So > getting all the data entries from the data source causes OutOfMemory > sometimes. So we changed the implementation to So that Lucene indexes the > data batchwise. Now we are getting the following exception. Can anyone tell > me what that exception means? > > java.lang.ArrayIndexOutOfBoundsException: 147 > at > > org.apache.lucene.codecs.lucene41.Lucene41PostingsReader$BlockDocsEnum.advance(Lucene41PostingsReader.java:538) > at org.apache.lucene.search.TermScorer.advance(TermScorer.java:85) > at > > org.apache.lucene.search.ConjunctionScorer.doNext(ConjunctionScorer.java:82) > at > > org.apache.lucene.search.ConjunctionScorer.nextDoc(ConjunctionScorer.java:100) > at > org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:192) > at > org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:163) > at org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35) > at > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621) > at > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:309) > at > org.apache.lucene.facet.FacetsCollector.doSearch(FacetsCollector.java:294) > at > org.apache.lucene.facet.FacetsCollector.search(FacetsCollector.java:198) > > > -- > Gimantha Bandara > Software Engineer > WSO2. Inc : http://wso2.com > Mobile : +94714961919 >
