Thanks for your help, Mark. We can start by posting our SOLR config files, although I'm not sure if that will be helpful (we don't see much in there regarding boosts). See attached. How SOLR actually configures and interfaces with Lucene is a bit of an unknown to us, so I'm not sure we can get down to the raw Lucene configuration and interaction.
That being said, in addition to the SOLR config files, see our attached XML which we post to SOLR to add the document to the index. How do you know that boost should affect fieldNorm linearly? Is there some code you can point us to? We looked through the Lucene source for a while, but it was kind of hard to track this down. One note: we're on an old version of Lucene - a nightly build between 2.0.0and 2.1.0. Mike On 1/30/08, Mark Miller <[EMAIL PROTECTED]> wrote: > > I would say you def misconfigured something. Doubling your doc boost > will double your fieldNorm approximately (I think the precision isn't > perfect). > > I don't know what your doing wrong in such a small test, but your > fieldNorm should *not* be exploding like that. > > Can you post some code? > > - Mark > > Mike Grafton wrote: > > Hello folks, > > > > We're trying to use Lucene's scoring to do a fairly basic thing: give a > > document (in this case, we index "articles") a boost based on an integer > > value that we know at index-time. We want the document boost to affect > the > > final document score linearly. > > > > We thought that assigning a document boost based on this value would do > the > > trick, but the behavior we're seeing doesn't match what we expect given > the > > online documentation. In fact, we see that a linear increase in > document > > boost yields an exponential increase in the 'fieldNorm' component of the > > score for each term of the query that matched the document. Here's a > > small table of values that relate the document boost we pass in to the > > fieldNorm contribution returned by Lucene: > > > > boost fieldNorm > > 1.0 0.3125 = (5/16, 2^-1.678) > > 2.0 20.0 = (2^4.3219, 2^1 * 10) > > 3.0 256.0 = (2^8) > > 4.0 1280.0 = (2^10.3219, 2^7 * 10) > > 5.0 5120.0 = (2^12.3219, 2^9 * 10) > > 6.0 16384.0 = (2^14.0) > > 7.0 40960.0 = (2^15.3219, 2^12 * 10) > > 8.0 81920.0 = (2^16.3219, 2^13 * 10) > > 10.0 327680.0 = (2^18.3219, 2^15 * 10) > > > > This example is using a query with two terms against a document that > > contains those terms and a few others, in one searchable field. > > > > Is this the way document boost is supposed to work? Or have we > > misconfigured something? If we cannot use document boost to affect > scoring > > linearly, is there some other technique we can use? > > > > By the way, we're using SOLR to access Lucene. We can give more > information > > if necessary, such as our SOLR schema.xml, if folks think that would > help > > explain things. Let us know what other information we can provide. > > > > Thanks, > > Mike > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > >
<?xml version="1.0" ?> <!-- The Solr schema file. This file should be named "schema.xml" and should be in the conf directory under the solr home (i.e. ./solr/conf/schema.xml by default) or located where the classloader for the Solr webapp can find it. For more information, on how to customize this file, please see... http://wiki.apache.org/solr/SchemaXml --> <schema name="example" version="1.1"> <types> <!-- field type definitions. The "name" attribute is just a label to be used by field definitions. The "class" attribute and any other attributes determine the real behavior of the fieldtype. --> <!-- The StringField type is not analyzed, but indexed/stored verbatim --> <fieldtype name="string" class="solr.StrField" sortMissingLast="true"/> <!-- boolean type: "true" or "false" --> <fieldtype name="boolean" class="solr.BoolField" sortMissingLast="true"/> <!-- The optional sortMissingLast and sortMissingFirst attributes are currently supported on types that are sorted internally as a strings. - If sortMissingLast="true" then a sort on this field will cause documents without the field to come after documents with the field, regardless of the requested sort order (asc or desc). - If sortMissingFirst="true" then a sort on this field will cause documents without the field to come before documents with the field, regardless of the requested sort order. - If sortMissingLast="false" and sortMissingFirst="false" (the default), then default lucene sorting will be used which places docs without the field first in an ascending sort and last in a descending sort. --> <!-- numeric field types that store and index the text value verbatim (and hence don't support range queries since the lexicographic ordering isn't equal to the numeric ordering) --> <fieldtype name="integer" class="solr.IntField"/> <fieldtype name="long" class="solr.LongField"/> <fieldtype name="float" class="solr.FloatField"/> <fieldtype name="double" class="solr.DoubleField"/> <!-- Numeric field types that manipulate the value into a string value that isn't human readable in it's internal form, but with a lexicographic ordering the same as the numeric ordering so that range queries correctly work. --> <fieldtype name="sint" class="solr.SortableIntField" sortMissingLast="true"/> <fieldtype name="slong" class="solr.SortableLongField" sortMissingLast="true"/> <fieldtype name="sfloat" class="solr.SortableFloatField" sortMissingLast="true"/> <fieldtype name="sdouble" class="solr.SortableDoubleField" sortMissingLast="true"/> <!-- The format for this date field is of the form 1995-12-31T23:59:59Z, and is a more restricted form of the canonical representation of dateTime http://www.w3.org/TR/xmlschema-2/#dateTime The trailing "Z" designates UTC time and is mandatory. Optional fractional seconds are allowed: 1995-12-31T23:59:59.999Z All other components are mandatory. --> <fieldtype name="date" class="solr.DateField" sortMissingLast="true"/> <!-- solr.TextField allows the specification of custom text analyzers specified as a tokenizer and a list of token filters. Different analyzers may be specified for indexing and querying. The optional positionIncrementGap puts space between multiple fields of this type on the same document, with the purpose of preventing false phrase matching across fields. For more info on customizing your analyzer chain, please see... http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters --> <!-- Standard analyzer commonly used by Lucene developers --> <!-- Standard analyzer commonly used by Lucene developers --> <fieldtype name="text_lu" class="solr.TextField" positionIncrementGap="100"> <analyzer> <tokenizer class="solr.StandardTokenizerFactory"/> <filter class="solr.StandardFilterFactory"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.StopFilterFactory"/> <filter class="solr.EnglishPorterFilterFactory"/> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer> </fieldtype> <!-- One could also specify an existing Analyzer implementation in Java via the class attribute on the analyzer element: <fieldtype name="text_lu" class="solr.TextField"> <analyzer class="org.apache.lucene.analysis.snowball.SnowballAnalyzer"/> </fieldType> --> <!-- A text field that only splits on whitespace for more exact matching --> <fieldtype name="text_ws" class="solr.TextField" positionIncrementGap="100"> <analyzer> <tokenizer class="solr.WhitespaceTokenizerFactory"/> </analyzer> </fieldtype> <!-- A text field that uses WordDelimiterFilter to enable splitting and matching of words on case-change, alpha numeric boundaries, and non-alphanumeric chars so that a query of "wifi" or "wi fi" could match a document containing "Wi-Fi". Synonyms and stopwords are customized by external files, and stemming is enabled Duplicate tokens at the same position (which may result from Stemmed Synonyms or WordDelim parts) are removed. --> <fieldtype name="text" class="solr.TextField" positionIncrementGap="100"> <analyzer type="index"> <tokenizer class="solr.WhitespaceTokenizerFactory"/> <!-- in this example, we will only use synonyms at query time <filter class="solr.SynonymFilterFactory" synonyms="index_synonyms.txt" ignoreCase="true" expand="false"/> --> <filter class="solr.StopFilterFactory" ignoreCase="true"/> <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="1" catenateNumbers="1" catenateAll="0"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.EnglishPorterFilterFactory" protected="protwords.txt"/> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer> <analyzer type="query"> <tokenizer class="solr.WhitespaceTokenizerFactory"/> <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true"/> <filter class="solr.StopFilterFactory" ignoreCase="true"/> <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="0" catenateNumbers="0" catenateAll="0"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.EnglishPorterFilterFactory" protected="protwords.txt"/> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer> </fieldtype> <!-- Less flexible matching, but less false matches. Probably not ideal for product names but may be good for SKUs. Can insert dashes in the wrong place and still match. --> <fieldtype name="textTight" class="solr.TextField" positionIncrementGap="100" > <analyzer> <tokenizer class="solr.WhitespaceTokenizerFactory"/> <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="false"/> <filter class="solr.StopFilterFactory" ignoreCase="true"/> <filter class="solr.WordDelimiterFilterFactory" generateWordParts="0" generateNumberParts="0" catenateWords="1" catenateNumbers="1" catenateAll="0"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.EnglishPorterFilterFactory" protected="protwords.txt"/> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer> </fieldtype> </types> <fields> <!-- Valid attributes for fields: name: mandatory - the name for the field type: mandatory - the name of a previously defined type from the <types> section indexed: true if this field should be indexed (searchable) stored: true if this field should be retrievable multiValued: true if this field may contain multiple values per document omitNorms: (expert) set to true to omit the norms associated with this field (this disables length normalization and index-time boosting for the field) --> <field name="id" type="string" indexed="false" stored="true"/> <field name="class" type="text" indexed="true" stored="true"/> <field name="unique_id" type="string" indexed="true" stored="true"/> <field name="body" type="text" indexed="true" stored="false"/> <field name="title" type="text" indexed="true" stored="false"/> <field name="messages_string" type="text" indexed="true" stored="false"/> <field name="participants_string" type="text" indexed="true" stored="false"/> <field name="tags_string" type="text" indexed="true" stored="false"/> <field name="site_id" type="integer" indexed="true" stored="false"/> <field name="site_group_id" type="integer" indexed="true" stored="false"/> <!-- catchall field, containing all other searchable text fields (implemented via copyField further on in this schema --> <field name="text" type="text" indexed="true" stored="false" multiValued="true"/> <!-- non-tokenized version of manufacturer to make it easier to sort or group results by name. copied from "name" via copyField --> <!-- Dynamic field definitions. If a field name is not found, dynamicFields will be used if the name matches any of the patterns. RESTRICTION: the glob-like pattern in the name attribute must have a "*" only at the start or the end. EXAMPLE: name="*_i" will match any field ending in _i (like myid_i, z_i) Longer patterns will be matched first. if equal size patterns both match, the first appearing in the schema will be used. --> <dynamicField name="*_i" type="sint" indexed="true" stored="true"/> <dynamicField name="*_s" type="string" indexed="true" stored="true"/> <dynamicField name="*_l" type="slong" indexed="true" stored="true"/> <dynamicField name="*_t" type="text" indexed="true" stored="true"/> <dynamicField name="*_b" type="boolean" indexed="true" stored="true"/> <dynamicField name="*_f" type="sfloat" indexed="true" stored="true"/> <dynamicField name="*_d" type="sdouble" indexed="true" stored="true"/> <dynamicField name="*_dt" type="date" indexed="true" stored="true"/> </fields> <!-- field to use to determine and enforce document uniqueness. --> <uniqueKey>unique_id</uniqueKey> <!-- field for the QueryParser to use when an explicit fieldname is absent --> <defaultSearchField>text</defaultSearchField> <!-- SolrQueryParser configuration: defaultOperator="AND|OR" --> <solrQueryParser defaultOperator="OR"/> <!-- copyField commands copy one field to another at the time a document is added to the index. It's used either to index the same field different ways, or to add multiple fields to the same field for easier/faster searching. --> <!--<copyField source="cat" dest="text"/>--> <copyField source="body" dest="text"/> <copyField source="title" dest="text"/> <copyField source="messages_string" dest="text"/> <copyField source="participants_string" dest="text"/> <copyField source="tags_string" dest="text"/> <!-- Similarity is the scoring routine for each document vs a query. A custom similarity may be specified here, but the default is fine for most applications. --> <!-- <similarity class="org.apache.lucene.search.DefaultSimilarity"/> --> </schema>
<?xml version="1.0" ?> <config> <!-- Used to specify an alternate directory to hold all index data other than the default ./data under the Solr home. If replication is in use, this should match the replication configuration. --> <dataDir>../../../solrdata/development</dataDir> <indexDefaults> <!-- Values here affect all index writers and act as a default unless overridden. --> <useCompoundFile>false</useCompoundFile> <mergeFactor>10</mergeFactor> <maxBufferedDocs>1000</maxBufferedDocs> <maxMergeDocs>2147483647</maxMergeDocs> <maxFieldLength>10000</maxFieldLength> <writeLockTimeout>1000</writeLockTimeout> <commitLockTimeout>10000</commitLockTimeout> </indexDefaults> <mainIndex> <!-- options specific to the main on-disk lucene index --> <useCompoundFile>false</useCompoundFile> <mergeFactor>10</mergeFactor> <maxBufferedDocs>1000</maxBufferedDocs> <maxMergeDocs>2147483647</maxMergeDocs> <maxFieldLength>10000</maxFieldLength> <!-- If true, unlock any held write or commit locks on startup. This defeats the locking mechanism that allows multiple processes to safely access a lucene index, and should be used with care. --> <unlockOnStartup>false</unlockOnStartup> </mainIndex> <!-- the default high-performance update handler --> <updateHandler class="solr.DirectUpdateHandler2"> <!-- The RunExecutableListener executes an external command. exe - the name of the executable to run dir - dir to use as the current working directory. default="." wait - the calling thread waits until the executable returns. default="true" args - the arguments to pass to the program. default=nothing env - environment variables to set. default=nothing --> <!-- A postCommit event is fired after every commit or optimize command <listener event="postCommit" class="solr.RunExecutableListener"> <str name="exe">snapshooter</str> <str name="dir">solr/bin</str> <bool name="wait">true</bool> <arr name="args"> <str>arg1</str> <str>arg2</str> </arr> <arr name="env"> <str>MYVAR=val1</str> </arr> </listener> --> <!-- A postOptimize event is fired only after every optimize command, useful in conjunction with index distribution to only distribute optimized indicies <listener event="postOptimize" class="solr.RunExecutableListener"> <str name="exe">snapshooter</str> <str name="dir">solr/bin</str> <bool name="wait">true</bool> </listener> --> </updateHandler> <query> <!-- Maximum number of clauses in a boolean query... can affect range or prefix queries that expand to big boolean queries. An exception is thrown if exceeded. --> <maxBooleanClauses>1024</maxBooleanClauses> <!-- Cache used by SolrIndexSearcher for filters (DocSets), unordered sets of *all* documents that match a query. When a new searcher is opened, its caches may be prepopulated or "autowarmed" using data from caches in the old searcher. autowarmCount is the number of items to prepopulate. For LRUCache, the autowarmed items will be the most recently accessed items. Parameters: class - the SolrCache implementation (currently only LRUCache) size - the maximum number of entries in the cache initialSize - the initial capacity (number of entries) of the cache. (seel java.util.HashMap) autowarmCount - the number of entries to prepopulate from and old cache. --> <filterCache class="solr.LRUCache" size="512" initialSize="512" autowarmCount="256"/> <!-- queryResultCache caches results of searches - ordered lists of document ids (DocList) based on a query, a sort, and the range of documents requested. --> <queryResultCache class="solr.LRUCache" size="512" initialSize="512" autowarmCount="256"/> <!-- documentCache caches Lucene Document objects (the stored fields for each document). Since Lucene internal document ids are transient, this cache will not be autowarmed. --> <documentCache class="solr.LRUCache" size="512" initialSize="512" autowarmCount="0"/> <!-- Example of a generic cache. These caches may be accessed by name through SolrIndexSearcher.getCache(),cacheLookup(), and cacheInsert(). The purpose is to enable easy caching of user/application level data. The regenerator argument should be specified as an implementation of solr.search.CacheRegenerator if autowarming is desired. --> <!-- <cache name="myUserCache" class="solr.LRUCache" size="4096" initialSize="1024" autowarmCount="1024" regenerator="org.mycompany.mypackage.MyRegenerator" /> --> <!-- An optimization that attempts to use a filter to satisfy a search. If the requested sort does not include score, then the filterCache will be checked for a filter matching the query. If found, the filter will be used as the source of document ids, and then the sort will be applied to that. --> <useFilterForSortedQuery>true</useFilterForSortedQuery> <!-- An optimization for use with the queryResultCache. When a search is requested, a superset of the requested number of document ids are collected. For example, if a search for a particular query requests matching documents 10 through 19, and queryWindowSize is 50, then documents 0 through 50 will be collected and cached. Any further requests in that range can be satisfied via the cache. --> <queryResultWindowSize>10</queryResultWindowSize> <!-- This entry enables an int hash representation for filters (DocSets) when the number of items in the set is less than maxSize. For smaller sets, this representation is more memory efficient, more efficient to iterate over, and faster to take intersections. --> <HashDocSet maxSize="3000" loadFactor="0.75"/> <!-- boolToFilterOptimizer converts boolean clauses with zero boost into cached filters if the number of docs selected by the clause exceeds the threshold (represented as a fraction of the total index) --> <boolTofilterOptimizer enabled="true" cacheSize="32" threshold=".05"/> <!-- a newSearcher event is fired whenever a new searcher is being prepared and there is a current searcher handling requests (aka registered). --> <!-- QuerySenderListener takes an array of NamedList and executes a local query request for each NamedList in sequence. --> <!-- <listener event="newSearcher" class="solr.QuerySenderListener"> <arr name="queries"> <lst> <str name="q">solr</str> <str name="start">0</str> <str name="rows">10</str> </lst> <lst> <str name="q">rocks</str> <str name="start">0</str> <str name="rows">10</str> </lst> </arr> </listener> --> <!-- a firstSearcher event is fired whenever a new searcher is being prepared but there is no current registered searcher to handle requests or to gain autowarming data from. --> <!-- <listener event="firstSearcher" class="solr.QuerySenderListener"> <arr name="queries"> <lst> <str name="q">fast_warm</str> <str name="start">0</str> <str name="rows">10</str> </lst> </arr> </listener> --> </query> <!-- requestHandler plugins... incoming queries will be dispatched to the correct handler based on the qt (query type) param matching the name of registered handlers. The "standard" request handler is the default and will be used if qt is not specified in the request. --> <requestHandler name="standard" class="solr.StandardRequestHandler"> <!-- default values for query parameters may optionally be defined here <lst name="defaults"> <int name="rows">10</int> <str name="fl">*</str> <str name="version">2.1</str> <lst> --> </requestHandler> <!-- DisMaxRequestHandler is an example of a request handler that supports optional parameters which are passed to its init() method. --> <requestHandler name="dismax" class="solr.DisMaxRequestHandler" > <lst name="defaults"> <float name="tie">0.01</float> <str name="qf"> text^0.5 features^1.0 name^1.2 sku^1.5 id^10.0 manu^1.1 cat^1.4 </str> <str name="pf"> text^0.2 features^1.1 name^1.5 manu^1.4 manu_exact^1.9 </str> <str name="bf"> ord(poplarity)^0.5 recip(rord(price),1,1000,1000)^0.3 </str> <str name="fl"> id,name,price,score </str> <str name="mm"> 2<-1 5<-2 6<90% </str> <int name="ps">100</int> </lst> </requestHandler> <!-- Note how you can register the same handler multiple times with different names (and different init parameters) --> <requestHandler name="partitioned" class="solr.DisMaxRequestHandler" > <lst name="defaults"> <str name="qf">text^0.5 features^1.0 name^1.2 sku^1.5 id^10.0</str> <str name="mm">2<-1 5<-2 6<90%</str> </lst> <!-- In addition to defaults, "appends" params can be specified to identify values which should be appended to the list of multi-val params from the query (or the existing "defaults"). In this example, the param "fq=instock:true" will be appended to any query time fq params the user may specify, as a mechanism for partitioning the index, independent of any user selected filtering that may also be desired (perhaps as a result of faceted searching). NOTE: there is *absolutely* nothing a client can do to prevent these "appends" values from being used, so don't use this mechanism unless you are sure you always want it. --> <lst name="appends"> <str name="fq">inStock:true</str> </lst> <!-- "invariants" are a way of letting the Solr maintainer lock down the options available to Solr clients. Any params values specified here are used regardless of what values may be specified in either the query, the "defaults", or the "appends" params. In this example, the facet.field and facet.query params are fixed, limiting the facets clients can use. Faceting is not turned on by default - but if the client does specify facet=true in the request, these are the only facets they will be able to see counts for; regardless of what other facet.field or facet.query params they may specify. NOTE: there is *absolutely* nothing a client can do to prevent these "invariants" values from being used, so don't use this mechanism unless you are sure you always want it. --> <lst name="invariants"> <str name="facet.field">cat</str> <str name="facet.field">manu_exact</str> <str name="facet.query">price:[* TO 500]</str> <str name="facet.query">price:[500 TO *]</str> </lst> </requestHandler> <requestHandler name="instock" class="solr.DisMaxRequestHandler" > <!-- for legacy reasons, DisMaxRequestHandler will assume all init params are "defaults" if you don't explicitly specify any defaults. --> <str name="fq"> inStock:true </str> <str name="qf"> text^0.5 features^1.0 name^1.2 sku^1.5 id^10.0 manu^1.1 cat^1.4 </str> <str name="mm"> 2<-1 5<-2 6<90% </str> </requestHandler> <!-- queryResponseWriter plugins... query responses will be written using the writer specified by the 'wt' request parameter matching the name of a registered writer. The "standard" writer is the default and will be used if 'wt' is not specified in the request. XMLResponseWriter will be used if nothing is specified here. The json, python, and ruby writers are also available by default. <queryResponseWriter name="standard" class="org.apache.solr.request.XMLResponseWriter"/> <queryResponseWriter name="json" class="org.apache.solr.request.JSONResponseWriter"/> <queryResponseWriter name="python" class="org.apache.solr.request.PythonResponseWriter"/> <queryResponseWriter name="ruby" class="org.apache.solr.request.RubyResponseWriter"/> <queryResponseWriter name="custom" class="com.example.MyResponseWriter"/> --> <!-- XSLT response writer (SOLR-49) Changes to XSLT transforms are taken into account every xsltCacheLifetimeSeconds at most. --> <queryResponseWriter name="xslt" class="org.apache.solr.request.XSLTResponseWriter" > <int name="xsltCacheLifetimeSeconds">5</int> </queryResponseWriter> <!-- config for the admin interface --> <admin> <defaultQuery>solr</defaultQuery> <gettableFiles>solrconfig.xml schema.xml admin-extra.html</gettableFiles> <pingQuery>q=id:0&start=0&rows=0</pingQuery> <!-- configure a healthcheck file for servers behind a loadbalancer <healthcheck type="file">server-enabled</healthcheck> --> </admin> </config>
<?xml version="1.0" encoding="UTF-8"?> <add> <doc boost="1.5"> <field name="id">76057</field> <field name="unique_id">Article_76057</field> <field name="body">article body</field> <field name="title">article title</field> <field name="site_id">489</field> </doc> </add>
--------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]