charset_id field is related to Charset of the document, not the language, "0" value mean "unknown charset" or this documnet isn't indexed yet. You may also use "GuesserUseMeta yes" command in indexer.conf to allow indexer to take in account META tag values in charset and language detection.
On 8/23/07, Sander Pool <[EMAIL PROTECTED]> wrote: > > > Hi, > > yes, that makes sense. I'm sure Apache is configured correctly but I > doubt all the HTML it serves provides the Language header. > > I grabbed the distinct charset_id values from url along with their counts: > > "1367" "0" > "31" "16" > "10" "23" > "317" "27" > "4" "44" > "1007" "52" > "4435" "54" > > Does this mean 1367 urls have no language indication? > > Thanks, > > Sander > > Maxim Zakharov wrote: > > > > > > Hi, > > > > Stopwords remove from index only if document's language is equal to > > stopword's language, so language missdetection may cause stopwords > > presence in the index. > > Does you web-server supply correct Content-Language header ? What > > language maps do you use ? > > If you use almost all language maps from dpsearch distribution, it's > > resonable to reduce this amount to languages you actually have in your > > pages. > > > > On 8/23/07, * Sander Pool* <[EMAIL PROTECTED]<sander%40tungstentech.com> > > <mailto:[EMAIL PROTECTED] <sander%40tungstentech.com>>> wrote: > > > > > > Hi, > > > > Maybe I don't understand how Stop words work. I use all the English > > stop > > files (the default). Still the word 'of' occurs more than 11000 > > times in > > the dict table. > > > > Is that correct? > > > > Thanks, > > > > Sander > > > > PS 4.47 > > > > > > > > > > -- > > http://www.dataparksearch.org/ <http://www.dataparksearch.org/> > > > > -- http://www.dataparksearch.org/
