Hi. I get OOE with Solr 1.3 Autowarm seem to be the villain in cojunction with FieldCache somehow. JVM args: -Xmx512m -Xms512m -Xss128k
Index size is ~4 Million docs, where I index text and store database primary keys. du /srv/solr/feedItem/data/index/ 1.7G /srv/solr/feedItem/data/index/ To ensure that the docs I index do not swell to much I only allow 5K per doc to over the wire i.e. I substring 0, 5000 on the field "content" I have removed "firstSearcher" and "newSearcher" since the queries I used before killed performance on reindexing the whole index. I will add them later again when I get into a delta update index state. Stacktrace. [06:25:53.122] [null] /update wt=xml&version=2.2 0 3165 [06:25:53.877] Error during auto-warming of key: [EMAIL PROTECTED]:java.lang.OutOfMemoryError: Java heap space [06:25:53.877] at org.apache.lucene.index.TermBuffer.toTerm(TermBuffer.java :104) [06:25:53.877] at org.apache.lucene.index.SegmentTermEnum.term( SegmentTermEnum.java:159) [06:25:53.877] at org.apache.lucene.index.SegmentMergeInfo.next( SegmentMergeInfo.java:66) [06:25:53.877] at org.apache.lucene.index.MultiTermEnum.next( MultiReader.java:315) [06:25:53.877] at org.apache.lucene.search.FieldCacheImpl$10.createValue( FieldCacheImpl.java:388) [06:25:53.877] at org.apache.lucene.search.FieldCacheImpl$Cache.get( FieldCacheImpl.java:72) [06:25:53.877] at org.apache.lucene.search.FieldCacheImpl.getStringIndex( FieldCacheImpl.java:350) [06:25:53.877] at org.apache.lucene.search.FieldSortedHitQueue.comparatorString( FieldSortedHitQueue.java:266) [06:25:53.877] at org.apache.lucene.search.FieldSortedHitQueue$1.createValue( FieldSortedHitQueue.java:182) [06:25:53.877] at org.apache.lucene.search.FieldCacheImpl$Cache.get( FieldCacheImpl.java:72) [06:25:53.877] at org.apache.lucene.search.FieldSortedHitQueue.getCachedComparator( FieldSortedHitQueue.java:155) [06:25:53.877] at org.apache.lucene.search.FieldSortedHitQueue.<init>( FieldSortedHitQueue.java:56) [06:25:53.877] at org.apache.solr.search.SolrIndexSearcher.getDocListNC( SolrIndexSearcher.java:862) [06:25:53.877] at org.apache.solr.search.SolrIndexSearcher.getDocListC( SolrIndexSearcher.java:808) [06:25:53.877] at org.apache.solr.search.SolrIndexSearcher.access$000( SolrIndexSearcher.java:56) [06:25:53.877] at org.apache.solr.search.SolrIndexSearcher$2.regenerateItem (SolrIndexSearcher.java:254) [06:25:53.877] at org.apache.solr.search.LRUCache.warm(LRUCache.java:192) [06:25:53.877] at org.apache.solr.search.SolrIndexSearcher.warm( SolrIndexSearcher.java:1393) [06:25:53.877] at org.apache.solr.core.SolrCore$2.call(SolrCore.java:702) [06:25:53.877] at java.util.concurrent.FutureTask$Sync.innerRun( FutureTask.java:269) [06:25:53.877] at java.util.concurrent.FutureTask.run(FutureTask.java:123) [06:25:53.877] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask( ThreadPoolExecutor.java:650) [06:25:53.877] at java.util.concurrent.ThreadPoolExecutor$Worker.run( ThreadPoolExecutor.java:675) [06:25:53.877] at java.lang.Thread.run(Thread.java:595) Help anyone? Attaching schema.xml and solrconfig.xml Kindly //Marcus Herou
<?xml version="1.0" encoding="UTF-8" ?> <!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> <!-- This is the Solr schema file. This file should be named "schema.xml" and should be in the conf directory under the solr home (i.e. ./solr/conf/schema.xml by default) or located where the classloader for the Solr webapp can find it. This example schema is the recommended starting point for users. It should be kept correct and concise, usable out-of-the-box. For more information, on how to customize this file, please see http://wiki.apache.org/solr/SchemaXml --> <schema name="example" version="1.1"> <types> <fieldType name="string" class="solr.StrField" sortMissingLast="true" omitNorms="true"/> <fieldType name="boolean" class="solr.BoolField" sortMissingLast="true" omitNorms="true"/> <fieldType name="integer" class="solr.IntField" omitNorms="true"/> <fieldType name="long" class="solr.LongField" omitNorms="true"/> <fieldType name="float" class="solr.FloatField" omitNorms="true"/> <fieldType name="double" class="solr.DoubleField" omitNorms="true"/> <fieldType name="sint" class="solr.SortableIntField" sortMissingLast="true" omitNorms="true"/> <fieldType name="slong" class="solr.SortableLongField" sortMissingLast="true" omitNorms="true"/> <fieldType name="sfloat" class="solr.SortableFloatField" sortMissingLast="true" omitNorms="true"/> <fieldType name="sdouble" class="solr.SortableDoubleField" sortMissingLast="true" omitNorms="true"/> <!-- The format for this date field is of the form 1995-12-31T23:59:59Z, and is a more restricted form of the canonical representation of dateTime http://www.w3.org/TR/xmlschema-2/#dateTime The trailing "Z" designates UTC time and is mandatory. Optional fractional seconds are allowed: 1995-12-31T23:59:59.999Z All other components are mandatory. Expressions can also be used to denote calculations that should be performed relative to "NOW" to determine the value, ie... NOW/HOUR ... Round to the start of the current hour NOW-1DAY ... Exactly 1 day prior to now NOW/DAY+6MONTHS+3DAYS ... 6 months and 3 days in the future from the start of the current day Consult the DateField javadocs for more information. --> <fieldType name="date" class="solr.DateField" sortMissingLast="true" omitNorms="true"/> <!-- A text field that only splits on whitespace for exact matching of words --> <fieldType name="text_ws" class="solr.TextField" positionIncrementGap="100"> <analyzer> <tokenizer class="solr.WhitespaceTokenizerFactory"/> </analyzer> </fieldType> <fieldType name="text" class="solr.TextField" positionIncrementGap="100"> <analyzer type="index"> <tokenizer class="solr.WhitespaceTokenizerFactory"/> <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt"/> <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="1" catenateNumbers="1" catenateAll="0"/> <filter class="solr.LowerCaseFilterFactory"/> </analyzer> <analyzer type="query"> <tokenizer class="solr.WhitespaceTokenizerFactory"/> <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt"/> <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="0" catenateNumbers="0" catenateAll="0"/> <filter class="solr.LowerCaseFilterFactory"/> </analyzer> </fieldType> <fieldtype name="ignored" stored="false" indexed="false" class="solr.StrField" /> </types> <fields> <field name="feedId" type="integer" indexed="true" stored="treu" required="true" /> <field name="feedItemId" type="long" indexed="true" stored="true" required="true" /> <field name="siteId" type="integer" indexed="true" stored="true" required="false" /> <field name="partnerType" type="integer" indexed="true" stored="false" required="true" /> <field name="uid" type="string" indexed="true" stored="false" required="true" /> <field name="link" type="string" indexed="true" stored="false" required="true" /> <field name="description" type="text" indexed="true" stored="false" required="true" /> <field name="title" type="text" indexed="true" stored="false" required="true" /> <field name="publishDate" type="date" indexed="true" stored="false" required="true" /> <field name="author" type="string" indexed="true" stored="false" required="false" /> <field name="keyWordId" type="integer" indexed="true" stored="false" required="false" multiValued="true"/> </fields> <uniqueKey>uid</uniqueKey> <!-- field for the QueryParser to use when an explicit fieldname is absent --> <defaultSearchField>description</defaultSearchField> <!-- SolrQueryParser configuration: defaultOperator="AND|OR" --> <solrQueryParser defaultOperator="OR"/> </schema>
<?xml version="1.0" encoding="UTF-8" ?> <!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> <config> <!-- Set this to 'false' if you want solr to continue working after it has encountered an severe configuration error. In a production environment, you may want solr to keep working even if one handler is mis-configured. You may also set this to false using by setting the system property: -Dsolr.abortOnConfigurationError=false --> <abortOnConfigurationError>${solr.abortOnConfigurationError:true}</abortOnConfigurationError> <!-- Used to specify an alternate directory to hold all index data other than the default ./data under the Solr home. If replication is in use, this should match the replication configuration. --> <!-- <dataDir>./solr/data</dataDir> --> <indexDefaults> <!-- Values here affect all index writers and act as a default unless overridden. --> <useCompoundFile>false</useCompoundFile> <mergeFactor>10</mergeFactor> <maxBufferedDocs>10000</maxBufferedDocs> <maxMergeDocs>2147483647</maxMergeDocs> <maxFieldLength>10000</maxFieldLength> <writeLockTimeout>1000</writeLockTimeout> <commitLockTimeout>10000</commitLockTimeout> </indexDefaults> <mainIndex> <!-- options specific to the main on-disk lucene index --> <useCompoundFile>false</useCompoundFile> <mergeFactor>25</mergeFactor> <maxBufferedDocs>10000</maxBufferedDocs> <maxMergeDocs>2147483647</maxMergeDocs> <maxFieldLength>10000</maxFieldLength> <!-- If true, unlock any held write or commit locks on startup. This defeats the locking mechanism that allows multiple processes to safely access a lucene index, and should be used with care. --> <!--<writeLockTimeout>1000</writeLockTimeout> <commitLockTimeout>10000</commitLockTimeout>--> <unlockOnStartup>true</unlockOnStartup> </mainIndex> <!-- the default high-performance update handler --> <updateHandler class="solr.DirectUpdateHandler2"> <!-- A prefix of "solr." for class names is an alias that causes solr to search appropriate packages, including org.apache.solr.(search|update|request|core|analysis) --> <!-- autocommit pending docs if certain criteria are met --> <autoCommit> <maxDocs>1000</maxDocs> <maxTime>300000</maxTime> </autoCommit> <!-- The RunExecutableListener executes an external command. exe - the name of the executable to run dir - dir to use as the current working directory. default="." wait - the calling thread waits until the executable returns. default="true" args - the arguments to pass to the program. default=nothing env - environment variables to set. default=nothing --> <!-- A postCommit event is fired after every commit or optimize command <listener event="postCommit" class="solr.RunExecutableListener"> <str name="exe">snapshooter</str> <str name="dir">solr/bin</str> <bool name="wait">true</bool> <arr name="args"> <str>arg1</str> <str>arg2</str> </arr> <arr name="env"> <str>MYVAR=val1</str> </arr> </listener> --> <!-- A postOptimize event is fired only after every optimize command, useful in conjunction with index distribution to only distribute optimized indicies <listener event="postOptimize" class="solr.RunExecutableListener"> <str name="exe">snapshooter</str> <str name="dir">solr/bin</str> <bool name="wait">true</bool> </listener> --> </updateHandler> <query> <!-- Maximum number of clauses in a boolean query... can affect range or prefix queries that expand to big boolean queries. An exception is thrown if exceeded. --> <maxBooleanClauses>1024</maxBooleanClauses> <!-- Cache used by SolrIndexSearcher for filters (DocSets), unordered sets of *all* documents that match a query. When a new searcher is opened, its caches may be prepopulated or "autowarmed" using data from caches in the old searcher. autowarmCount is the number of items to prepopulate. For LRUCache, the autowarmed items will be the most recently accessed items. Parameters: class - the SolrCache implementation (currently only LRUCache) size - the maximum number of entries in the cache initialSize - the initial capacity (number of entries) of the cache. (seel java.util.HashMap) autowarmCount - the number of entries to prepopulate from and old cache. --> <filterCache class="solr.LRUCache" size="512" initialSize="512" autowarmCount="256"/> <!-- queryResultCache caches results of searches - ordered lists of document ids (DocList) based on a query, a sort, and the range of documents requested. --> <queryResultCache class="solr.LRUCache" size="512" initialSize="512" autowarmCount="256"/> <!-- documentCache caches Lucene Document objects (the stored fields for each document). Since Lucene internal document ids are transient, this cache will not be autowarmed. --> <documentCache class="solr.LRUCache" size="512" initialSize="512" autowarmCount="0"/> <!-- If true, stored fields that are not requested will be loaded lazily. This can result in a significant speed improvement if the usual case is to not load all stored fields, especially if the skipped fields are large compressed text fields. --> <enableLazyFieldLoading>true</enableLazyFieldLoading> <!-- Example of a generic cache. These caches may be accessed by name through SolrIndexSearcher.getCache(),cacheLookup(), and cacheInsert(). The purpose is to enable easy caching of user/application level data. The regenerator argument should be specified as an implementation of solr.search.CacheRegenerator if autowarming is desired. --> <!-- <cache name="myUserCache" class="solr.LRUCache" size="4096" initialSize="1024" autowarmCount="1024" regenerator="org.mycompany.mypackage.MyRegenerator" /> --> <!-- An optimization that attempts to use a filter to satisfy a search. If the requested sort does not include score, then the filterCache will be checked for a filter matching the query. If found, the filter will be used as the source of document ids, and then the sort will be applied to that. <useFilterForSortedQuery>true</useFilterForSortedQuery> --> <!-- An optimization for use with the queryResultCache. When a search is requested, a superset of the requested number of document ids are collected. For example, if a search for a particular query requests matching documents 10 through 19, and queryWindowSize is 50, then documents 0 through 50 will be collected and cached. Any further requests in that range can be satisfied via the cache. --> <queryResultWindowSize>10</queryResultWindowSize> <!-- This entry enables an int hash representation for filters (DocSets) when the number of items in the set is less than maxSize. For smaller sets, this representation is more memory efficient, more efficient to iterate over, and faster to take intersections. --> <HashDocSet maxSize="3000" loadFactor="0.75"/> <!-- boolToFilterOptimizer converts boolean clauses with zero boost into cached filters if the number of docs selected by the clause exceeds the threshold (represented as a fraction of the total index) --> <boolTofilterOptimizer enabled="true" cacheSize="32" threshold=".05"/> <!-- a newSearcher event is fired whenever a new searcher is being prepared and there is a current searcher handling requests (aka registered). --> <!-- QuerySenderListener takes an array of NamedList and executes a local query request for each NamedList in sequence. --> <!--<listener event="newSearcher" class="solr.QuerySenderListener"> <arr name="queries"> <lst> <str name="q">(title:(apple^4) OR description:(apple));publishDate asc</str> <str name="start">0</str> <str name="rows">15</str> </lst> </arr> </listener> <listener event="firstSearcher" class="solr.QuerySenderListener"> <arr name="queries"> <lst> <str name="q">(title:(apple^4) OR description:(apple));publishDate asc</str> <str name="start">0</str> <str name="rows">15</str> </lst> </arr> </listener>--> <!-- If a search request comes in and there is no current registered searcher, then immediately register the still warming searcher and use it. If "false" then all requests will block until the first searcher is done warming. --> <useColdSearcher>false</useColdSearcher> <!-- Maximum number of searchers that may be warming in the background concurrently. An error is returned if this limit is exceeded. Recommend 1-2 for read-only slaves, higher for masters w/o cache warming. --> <maxWarmingSearchers>4</maxWarmingSearchers> </query> <!-- Let the dispatch filter handler /select?qt=XXX handleSelect=true will use consistent error handling for /select and /update handleSelect=false will use solr1.1 style error formatting --> <requestDispatcher handleSelect="true" > <!--Make sure your system has some authentication before enabling remote streaming! --> <requestParsers enableRemoteStreaming="false" multipartUploadLimitInKB="2048" /> </requestDispatcher> <!-- requestHandler plugins... incoming queries will be dispatched to the correct handler based on the qt (query type) param matching the name of registered handlers. The "standard" request handler is the default and will be used if qt is not specified in the request. --> <requestHandler name="standard" class="solr.StandardRequestHandler"> <!-- default values for query parameters --> <lst name="defaults"> <str name="echoParams">explicit</str> <!-- <int name="rows">10</int> <str name="fl">*</str> <str name="version">2.1</str> --> </lst> </requestHandler> <!-- DisMaxRequestHandler allows easy searching across multiple fields for simple user-entered phrases. see http://wiki.apache.org/solr/DisMaxRequestHandler --> <requestHandler name="dismax" class="solr.DisMaxRequestHandler" > <lst name="defaults"> <str name="echoParams">explicit</str> <float name="tie">0.01</float> <str name="qf"> text^0.5 features^1.0 name^1.2 sku^1.5 id^10.0 manu^1.1 cat^1.4 </str> <str name="pf"> text^0.2 features^1.1 name^1.5 manu^1.4 manu_exact^1.9 </str> <str name="bf"> ord(poplarity)^0.5 recip(rord(price),1,1000,1000)^0.3 </str> <str name="fl"> id,name,price,score </str> <str name="mm"> 2<-1 5<-2 6<90% </str> <int name="ps">100</int> <str name="q.alt">*:*</str> </lst> </requestHandler> <!-- Note how you can register the same handler multiple times with different names (and different init parameters) --> <requestHandler name="partitioned" class="solr.DisMaxRequestHandler" > <lst name="defaults"> <str name="echoParams">explicit</str> <str name="qf">text^0.5 features^1.0 name^1.2 sku^1.5 id^10.0</str> <str name="mm">2<-1 5<-2 6<90%</str> <!-- This is an example of using Date Math to specify a constantly moving date range in a config... --> <str name="bq">incubationdate_dt:[* TO NOW/DAY-1MONTH]^2.2</str> </lst> <!-- In addition to defaults, "appends" params can be specified to identify values which should be appended to the list of multi-val params from the query (or the existing "defaults"). In this example, the param "fq=instock:true" will be appended to any query time fq params the user may specify, as a mechanism for partitioning the index, independent of any user selected filtering that may also be desired (perhaps as a result of faceted searching). NOTE: there is *absolutely* nothing a client can do to prevent these "appends" values from being used, so don't use this mechanism unless you are sure you always want it. --> <lst name="appends"> <str name="fq">inStock:true</str> </lst> <!-- "invariants" are a way of letting the Solr maintainer lock down the options available to Solr clients. Any params values specified here are used regardless of what values may be specified in either the query, the "defaults", or the "appends" params. In this example, the facet.field and facet.query params are fixed, limiting the facets clients can use. Faceting is not turned on by default - but if the client does specify facet=true in the request, these are the only facets they will be able to see counts for; regardless of what other facet.field or facet.query params they may specify. NOTE: there is *absolutely* nothing a client can do to prevent these "invariants" values from being used, so don't use this mechanism unless you are sure you always want it. --> <lst name="invariants"> <str name="facet.field">cat</str> <str name="facet.field">manu_exact</str> <str name="facet.query">price:[* TO 500]</str> <str name="facet.query">price:[500 TO *]</str> </lst> </requestHandler> <requestHandler name="instock" class="solr.DisMaxRequestHandler" > <!-- for legacy reasons, DisMaxRequestHandler will assume all init params are "defaults" if you don't explicitly specify any defaults. --> <str name="fq"> inStock:true </str> <str name="qf"> text^0.5 features^1.0 name^1.2 sku^1.5 id^10.0 manu^1.1 cat^1.4 </str> <str name="mm"> 2<-1 5<-2 6<90% </str> </requestHandler> <!-- SpellCheckerRequestHandler takes in a word (or several words) as the value of the "q" parameter and returns a list of alternative spelling suggestions. If invoked with a ...&cmd=rebuild, it will rebuild the spellchecker index. --> <requestHandler name="spellchecker" class="solr.SpellCheckerRequestHandler" startup="lazy"> <!-- default values for query parameters --> <lst name="defaults"> <int name="suggestionCount">1</int> <float name="accuracy">0.5</float> </lst> <!-- Main init params for handler --> <!-- The directory where your SpellChecker Index should live. --> <!-- May be absolute, or relative to the Solr "dataDir" directory. --> <!-- If this option is not specified, a RAM directory will be used --> <str name="spellcheckerIndexDir">spell</str> <!-- the field in your schema that you want to be able to build --> <!-- your spell index on. This should be a field that uses a very --> <!-- simple FieldType without a lot of Analysis (ie: string) --> <str name="termSourceField">word</str> </requestHandler> <!-- Update request handler. Note: Since solr1.1 requestHandlers requires a valid content type header if posted in the body. For example, curl now requires: -H 'Content-type:text/xml; charset=utf-8' The response format differs from solr1.1 formatting and returns a standard error code. To enable solr1.1 behavior, remove the /update handler or change its path --> <requestHandler name="/update" class="solr.XmlUpdateRequestHandler" /> <!-- CSV update handler, loaded on demand --> <requestHandler name="/update/csv" class="solr.CSVRequestHandler" startup="lazy" /> <!-- Admin Handlers. TODO? There could be a single handler that loads them all... --> <requestHandler name="/admin/luke" class="org.apache.solr.handler.admin.LukeRequestHandler" /> <requestHandler name="/admin/system" class="org.apache.solr.handler.admin.SystemInfoHandler" /> <requestHandler name="/admin/plugins" class="org.apache.solr.handler.admin.PluginInfoHandler" /> <requestHandler name="/admin/threads" class="org.apache.solr.handler.admin.ThreadDumpHandler" /> <requestHandler name="/admin/properties" class="org.apache.solr.handler.admin.PropertiesRequestHandler" /> <!-- Echo the request contents back to the client --> <requestHandler name="/debug/dump" class="solr.DumpRequestHandler" > <lst name="defaults"> <str name="echoParams">explicit</str> <!-- for all params (including the default etc) use: 'all' --> <str name="echoHandler">true</str> </lst> </requestHandler> <!-- queryResponseWriter plugins... query responses will be written using the writer specified by the 'wt' request parameter matching the name of a registered writer. The "standard" writer is the default and will be used if 'wt' is not specified in the request. XMLResponseWriter will be used if nothing is specified here. The json, python, and ruby writers are also available by default.--> <queryResponseWriter name="standard" class="org.apache.solr.request.XMLResponseWriter"/> <queryResponseWriter name="json" class="org.apache.solr.request.JSONResponseWriter"/> <!--<queryResponseWriter name="python" class="org.apache.solr.request.PythonResponseWriter"/> <queryResponseWriter name="ruby" class="org.apache.solr.request.RubyResponseWriter"/> <queryResponseWriter name="custom" class="com.example.MyResponseWriter"/> --> <!-- XSLT response writer transforms the XML output by any xslt file found in Solr's conf/xslt directory. Changes to xslt files are checked for every xsltCacheLifetimeSeconds. --> <queryResponseWriter name="xslt" class="org.apache.solr.request.XSLTResponseWriter"> <int name="xsltCacheLifetimeSeconds">5</int> </queryResponseWriter> <!-- config for the admin interface --> <admin> <defaultQuery>solr</defaultQuery> <gettableFiles>solrconfig.xml schema.xml admin-extra.html</gettableFiles> <!-- pingQuery should be "URLish" ... & separated key=val pairs ... but there shouldn't be any URL escaping of the values --> <pingQuery> qt=standard&q=solrpingquery </pingQuery> <!-- configure a healthcheck file for servers behind a loadbalancer <healthcheck type="file">server-enabled</healthcheck> --> </admin> </config>