I guess, somewhere solr is referring the Integer datatype to define the maximum number of documents.
This is limiting to the integer range to 2147483647. I am not able to find out the root cause of this issue. Can someone shed some light over this issue to identify the root cause of it. Regards, Suresh.A From: Arumugam, Suresh [mailto:suresh.arumu...@emc.com] Sent: Wednesday, February 04, 2015 1:54 PM To: solr-user@lucene.apache.org Cc: Habeeb, Anwar Subject: Exception while loading 2 Billion + Documents in Solr 4.8.0 Hi All, We are trying to load 14+ Billion documents into Solr. But we are failing to load them into Solr. Solr version: 4.8.0 Analyzer used: ClassicTokenizer for index as well as query. Can someone help me in getting into the core of this issue? For 14+ Billion document load, we are loading 2Billion batches using the dataimport with single thread. First batch completed successfully & added 2 Billion documents Second batch, dataimport is showing as successful completion. But the no of documents is still 2 Billion with the following exception in the logs. Screenshot of Dataimport handler: [cid:image001.jpg@01D04083.905A6100] Exception: 2/4/2015, 1:09:06 PM ERROR SolrWriter Exception while solr commit. org.apache.solr.common.SolrException: Error opening new searcher at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1550) at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1662) at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1406) at org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:611) at org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:95) at org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(UpdateRequestProcessor.java:64) at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalCommit(DistributedUpdateProcessor.java:1458) at org.apache.solr.update.processor.DistributedUpdateProcessor.processCommit(DistributedUpdateProcessor.java:1435) at org.apache.solr.update.processor.LogUpdateProcessor.processCommit(LogUpdateProcessorFactory.java:157) at org.apache.solr.handler.dataimport.SolrWriter.commit(SolrWriter.java:108) at org.apache.solr.handler.dataimport.DocBuilder.finish(DocBuilder.java:309) at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:257) at org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:444) at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:485) at org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:464) Caused by: java.lang.IllegalArgumentException: Too many documents, composite IndexReaders cannot exceed 2147483647 at org.apache.lucene.index.BaseCompositeReader.<init>(BaseCompositeReader.java:77) at org.apache.lucene.index.DirectoryReader.<init>(DirectoryReader.java:369) at org.apache.lucene.index.StandardDirectoryReader.<init>(StandardDirectoryReader.java:43) at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:117) at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:385) at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:289) at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:274) at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:251) at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1461) Please let me know if we need more details on this. Thanks in advance. Regards, Suresh.A, EMC Corporation.
