Javier, Your minX is slightly greater than maxX, which is interpreted as a line that wraps nearly the entire globe. Is that what you intended?
If this is what you intended, then you got bitten by this unfixed bug: https://issues.apache.org/jira/browse/LUCENE-4550 As a work-around, you could split that horizontal line into two equal pieces and index them as separate values for the document. ~ David Javier Molina wrote > Hi, > > I have been struggling the past days trying to spot what was causing my > solr 4.0 to go Out Of Memory when doing a dataimport full-import > > After troubleshooting I found that the problem is not related with data > volume but instead with one particular record in my DB. > > The offending record has a location value of 147.33767500000002 -42.88601 > 147.337675 -42.88601 > > As you can see, the min and max latitude have the same value, that is the > rectangle is indeed a line. > > The location field uses the default definition for field type > location_rpt, > extract from schema.xml shown below. > > > <fieldType name="location_rpt" > class="solr.SpatialRecursivePrefixTreeFieldType"^M > geo="true" distErrPct="0.025" maxDistErr="0.000009" > units="degrees" > /> > > We have configured the delta settings for dataimport to take a particular > id from our db records, and what it is interesting is that if instead of > doing a full-import I issue a delta-import the operation succeed. Note > that > in both cases I am just importing the same particular record, in the case > of full-import I specify start and rows parameters, as seen in the log. > > My understanding is that defining a rectangle as a line it is still a > valid > area (so to speak) so the operation should succeed, nevertheless if those > values are not a valid there should be a validation in place in order to > prevent the OutOfMemoryError. > > > See log below for an example of the run leading to the Out Of Memory > Error. > > Thanks in advance for your feedback. > > Regards, > Javier > > > 21/01/2013 12:00:38 PM org.apache.solr.core.SolrCore execute > INFO: [coreDap] webapp=/solr path=/dataimport > params={optimize=false&clean=true&commit=true&start=1995&verbose=true&command=full-import&rows=1} > status=0 QTime=10 > 21/01/2013 12:00:38 PM org.apache.solr.handler.dataimport.DataImporter > doFullImport > INFO: Starting Full Import > 21/01/2013 12:00:38 PM > org.apache.solr.handler.dataimport.SimplePropertiesWriter > readIndexerProperties > INFO: Read dataimport.properties > 21/01/2013 12:00:38 PM org.apache.solr.core.SolrCore execute > INFO: [coreDap] webapp=/solr path=/dataimport params={command=status} > status=0 QTime=1 > 21/01/2013 12:00:38 PM org.apache.solr.core.SolrDeletionPolicy onInit > INFO: SolrDeletionPolicy.onInit: commits:num=1 > > commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/DMS_dev/r01/solr/coreDap/data/index > lockFactory=org.apache.lucene.store.NativeFSLockFactory@616181be; > maxCacheMB=48.0 > maxMergeSizeMB=4.0),segFN=segments_mq,generation=818,filenames=[_12y_Lucene40_0.tip, > _12w_Lucene40_0.prx, _12w_nrm.cfe, _12z_Lucene40_0.tim, > _12z_Lucene40_0.frq, _12z.si, _12y_Lucene40_0.prx, _12z_nrm.cfe, > _12w_Lucene40_0.frq, _12w.fdx, segments_mq, _12y_nrm.cfs, _12w.fnm, > _12w_Lucene40_0.tip, _12w.si, _12w_Lucene40_0.tim, _12w.fdt, _12z_nrm.cfs, > _12z.fnm, _12y.fdx, _12y.fnm, _12z.fdx, _12z_Lucene40_0.prx, _12y.fdt, > _12z.fdt, _12y_nrm.cfe, _12y.si, _12z_Lucene40_0.tip, _12y_Lucene40_0.frq, > _12w_nrm.cfs, _12y_Lucene40_0.tim] > 21/01/2013 12:00:38 PM org.apache.solr.core.SolrDeletionPolicy > updateCommits > INFO: newest commit = 818 > 21/01/2013 12:00:38 PM org.apache.solr.search.SolrIndexSearcher > <init> > INFO: Opening Searcher@70f4d063 realtime > 21/01/2013 12:00:38 PM org.apache.solr.handler.dataimport.JdbcDataSource$1 > call > INFO: Creating a connection for entity dataCollection with URL: > jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST= > myserver.mydomain.com)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME= > myserver.mydomain.com))) > 21/01/2013 12:00:39 PM org.apache.solr.handler.dataimport.JdbcDataSource$1 > call > INFO: Time taken for getConnection(): 40 > 21/01/2013 12:00:39 PM org.apache.solr.handler.dataimport.JdbcDataSource$1 > call > INFO: Creating a connection for entity topic with URL: jdbc:oracle:thin:@ > (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=myserver.mydomain.com > )(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=myserver.mydomain.com))) > 21/01/2013 12:00:39 PM org.apache.solr.handler.dataimport.JdbcDataSource$1 > call > INFO: Time taken for getConnection(): 47 > 21/01/2013 12:00:39 PM org.apache.solr.handler.dataimport.JdbcDataSource$1 > call > INFO: Creating a connection for entity usergroup with URL: > jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST= > myserver.mydomain.com)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME= > myserver.mydomain.com))) > 21/01/2013 12:00:39 PM org.apache.solr.handler.dataimport.JdbcDataSource$1 > call > INFO: Time taken for getConnection(): 34 > 21/01/2013 12:00:39 PM org.apache.solr.handler.dataimport.JdbcDataSource$1 > call > INFO: Creating a connection for entity usergroupDNs with URL: > jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST= > myserver.mydomain.com)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME= > myserver.mydomain.com))) > 21/01/2013 12:00:39 PM org.apache.solr.handler.dataimport.JdbcDataSource$1 > call > INFO: Time taken for getConnection(): 39 > 21/01/2013 12:00:39 PM org.apache.solr.handler.dataimport.JdbcDataSource$1 > call > INFO: Creating a connection for entity person with URL: jdbc:oracle:thin:@ > (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=myserver.mydomain.com > )(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=myserver.mydomain.com))) > 21/01/2013 12:00:39 PM org.apache.solr.handler.dataimport.JdbcDataSource$1 > call > INFO: Time taken for getConnection(): 33 > 21/01/2013 12:00:39 PM > org.apache.solr.schema.AbstractSpatialPrefixTreeFieldType > newSpatialStrategy > INFO: > location_rpt{class=org.apache.solr.schema.SpatialRecursivePrefixTreeFieldType,analyzer=org.apache.solr.schema.FieldType$DefaultAnalyzer,args={distErrPct=0.025, > geo=true, maxDistErr=0.000009, units=degrees}} strat: > RecursivePrefixTreeStrategy(prefixGridScanLevel:7,SPG:(GeohashPrefixTree(maxLevels:11,ctx:SpatialContext{geo=true, > calculator=Haversine, > worldBounds=Rect(minX=-180.0,maxX=180.0,minY=-90.0,maxY=90.0)}))) > maxLevels: 11 > 21/01/2013 12:00:40 PM org.apache.solr.core.SolrCore execute > INFO: [coreDap] webapp=/solr path=/dataimport params={command=status} > status=0 QTime=0 > 21/01/2013 12:00:53 PM org.apache.solr.update.DirectUpdateHandler2 commit > INFO: start > commit{flags=0,_version_=0,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false} > 21/01/2013 12:00:54 PM org.apache.solr.core.SolrDeletionPolicy onCommit > INFO: SolrDeletionPolicy.onCommit: commits:num=2 > > commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/DMS_dev/r01/solr/coreDap/data/index > lockFactory=org.apache.lucene.store.NativeFSLockFactory@616181be; > maxCacheMB=48.0 > maxMergeSizeMB=4.0),segFN=segments_mq,generation=818,filenames=[_12y_Lucene40_0.tip, > _12w_Lucene40_0.prx, _12w_nrm.cfe, _12z_Lucene40_0.tim, > _12z_Lucene40_0.frq, _12z.si, _12y_Lucene40_0.prx, _12z_nrm.cfe, > _12w_Lucene40_0.frq, _12w.fdx, segments_mq, _12y_nrm.cfs, _12w.fnm, > _12w_Lucene40_0.tip, _12w.si, _12w_Lucene40_0.tim, _12w.fdt, _12z_nrm.cfs, > _12z.fnm, _12y.fdx, _12y.fnm, _12z.fdx, _12z_Lucene40_0.prx, _12y.fdt, > _12z.fdt, _12y_nrm.cfe, _12y.si, _12z_Lucene40_0.tip, _12y_Lucene40_0.frq, > _12w_nrm.cfs, _12y_Lucene40_0.tim] > > commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/DMS_dev/r01/solr/coreDap/data/index > lockFactory=org.apache.lucene.store.NativeFSLockFactory@616181be; > maxCacheMB=48.0 > maxMergeSizeMB=4.0),segFN=segments_mr,generation=819,filenames=[segments_mr] > 21/01/2013 12:00:54 PM org.apache.solr.core.SolrDeletionPolicy > updateCommits > INFO: newest commit = 819 > 21/01/2013 12:00:54 PM org.apache.solr.update.DirectUpdateHandler2 commit > INFO: end_commit_flush > > 21/01/2013 12:05:11 PM org.apache.solr.update.processor.LogUpdateProcessor > finish > INFO: [coreDap] webapp=/solr path=/dataimport > params={optimize=false&clean=true&commit=true&start=1995&verbose=true&command=full-import&rows=1} > status=0 QTime=10 {deleteByQuery=*:* (-1424731709158457344)} 0 10 > 21/01/2013 12:05:11 PM org.apache.solr.common.SolrException log > SEVERE: Full Import failed:java.lang.RuntimeException: > java.lang.RuntimeException: > org.apache.solr.handler.dataimport.DataImportHandlerException: > java.lang.OutOfMemoryError: Java heap space > at > org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:273) > at > org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:382) > at > org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:448) > at > org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:429) > Caused by: java.lang.RuntimeException: > org.apache.solr.handler.dataimport.DataImportHandlerException: > java.lang.OutOfMemoryError: Java heap space > at > org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:413) > at > org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:326) > at > org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:234) > ... 3 more > Caused by: org.apache.solr.handler.dataimport.DataImportHandlerException: > java.lang.OutOfMemoryError: Java heap space > at > org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:542) > at > org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:411) > ... 5 more > Caused by: java.lang.OutOfMemoryError: Java heap space > at > com.spatial4j.core.context.SpatialContext.makeRectangle(SpatialContext.java:172) > at > com.spatial4j.core.io.GeohashUtils.decodeBoundary(GeohashUtils.java:150) > at > org.apache.lucene.spatial.prefix.tree.GeohashPrefixTree$GhCell.getShape(GeohashPrefixTree.java:142) > at > org.apache.lucene.spatial.prefix.tree.Node.getSubCells(Node.java:155) > at > org.apache.lucene.spatial.prefix.tree.SpatialPrefixTree.recursiveGetNodes(SpatialPrefixTree.java:157) > at > org.apache.lucene.spatial.prefix.tree.SpatialPrefixTree.recursiveGetNodes(SpatialPrefixTree.java:175) > at > org.apache.lucene.spatial.prefix.tree.SpatialPrefixTree.recursiveGetNodes(SpatialPrefixTree.java:175) > at > org.apache.lucene.spatial.prefix.tree.SpatialPrefixTree.recursiveGetNodes(SpatialPrefixTree.java:175) > at > org.apache.lucene.spatial.prefix.tree.SpatialPrefixTree.recursiveGetNodes(SpatialPrefixTree.java:175) > at > org.apache.lucene.spatial.prefix.tree.SpatialPrefixTree.recursiveGetNodes(SpatialPrefixTree.java:175) > at > org.apache.lucene.spatial.prefix.tree.SpatialPrefixTree.recursiveGetNodes(SpatialPrefixTree.java:175) > at > org.apache.lucene.spatial.prefix.tree.SpatialPrefixTree.recursiveGetNodes(SpatialPrefixTree.java:175) > at > org.apache.lucene.spatial.prefix.tree.SpatialPrefixTree.recursiveGetNodes(SpatialPrefixTree.java:175) > at > org.apache.lucene.spatial.prefix.tree.SpatialPrefixTree.recursiveGetNodes(SpatialPrefixTree.java:175) > at > org.apache.lucene.spatial.prefix.tree.SpatialPrefixTree.getNodes(SpatialPrefixTree.java:142) > at > org.apache.lucene.spatial.prefix.tree.GeohashPrefixTree.getNodes(GeohashPrefixTree.java:98) > at > org.apache.lucene.spatial.prefix.PrefixTreeStrategy.createIndexableFields(PrefixTreeStrategy.java:126) > at > org.apache.lucene.spatial.prefix.PrefixTreeStrategy.createIndexableFields(PrefixTreeStrategy.java:121) > at > org.apache.solr.schema.AbstractSpatialFieldType.createFields(AbstractSpatialFieldType.java:110) > at > org.apache.solr.schema.AbstractSpatialFieldType.createFields(AbstractSpatialFieldType.java:55) > at > org.apache.solr.update.DocumentBuilder.addField(DocumentBuilder.java:193) > at > org.apache.solr.update.DocumentBuilder.toDocument(DocumentBuilder.java:270) > at > org.apache.solr.update.AddUpdateCommand.getLuceneDocument(AddUpdateCommand.java:73) > at > org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:208) > at > org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:61) > at > org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:51) > at > org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:432) > at > org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:557) > at > org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:325) > at > org.apache.solr.update.processor.LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:100) > at > org.apache.solr.handler.dataimport.SolrWriter.upload(SolrWriter.java:70) > at > org.apache.solr.handler.dataimport.DataImportHandler$1.upload(DataImportHandler.java:233) > > 21/01/2013 12:05:11 PM org.apache.solr.update.DirectUpdateHandler2 > rollback > INFO: start rollback{flags=0,_version_=0} > 21/01/2013 12:05:11 PM org.apache.solr.update.DefaultSolrCoreState > newIndexWriter > INFO: Creating new IndexWriter... > 21/01/2013 12:05:11 PM org.apache.solr.update.DefaultSolrCoreState > newIndexWriter > INFO: Waiting until IndexWriter is unused... core=coreDap > 21/01/2013 12:05:11 PM org.apache.solr.update.DefaultSolrCoreState > newIndexWriter > INFO: Rollback old IndexWriter... core=coreDap > 21/01/2013 12:05:11 PM org.apache.solr.core.CachingDirectoryFactory get > INFO: return new directory for /DMS_dev/r01/solr/coreDap/data/index > forceNew:true > 21/01/2013 12:05:11 PM org.apache.solr.core.SolrDeletionPolicy onInit > INFO: SolrDeletionPolicy.onInit: commits:num=1 > > commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/DMS_dev/r01/solr/coreDap/data/index > lockFactory=org.apache.lucene.store.NativeFSLockFactory@7ab9fcb0; > maxCacheMB=48.0 > maxMergeSizeMB=4.0),segFN=segments_mr,generation=819,filenames=[segments_mr] > 21/01/2013 12:05:11 PM org.apache.solr.core.SolrDeletionPolicy > updateCommits > INFO: newest commit = 819 > 21/01/2013 12:05:11 PM org.apache.solr.update.DefaultSolrCoreState > newIndexWriter > INFO: New IndexWriter is ready to be used. > 21/01/2013 12:05:11 PM org.apache.solr.update.DirectUpdateHandler2 > rollback > INFO: end_rollback ----- Author: http://www.packtpub.com/apache-solr-3-enterprise-search-server/book -- View this message in context: http://lucene.472066.n3.nabble.com/Spatial-Dataimport-full-import-results-in-OutOfMemory-for-a-rectangle-defining-a-line-tp4034928p4035163.html Sent from the Solr - User mailing list archive at Nabble.com.
