Dear Wiki user, You have subscribed to a wiki page or wiki category on "Nutch Wiki" for change notification.
The following page has been changed by JeromeCharron: http://wiki.apache.org/nutch/NewLanguageIdentifier New page: == Architecture == TODO == NGram profile format == TODO == Generating some NGrams profiles == TODO == Open Issues == * ''Labs'' tests are quite good (LanguageIdentifierBenchs), but in ''real life'', they are not. In fact, in its actual version, the NewLanguageIdentifier expects that the provided text to analyze is UTF-8 encoded. However, it is not the case for a lot of fetched documents. So, the NewLanguageIdentifier needs to refer to a {{{content-encoding}}} meta-data. This data must be provided by a (todo) EncodingDetectorPlugin (see [http://issues.apache.org/jira/browse/NUTCH-25 NUTCH-25] issue).
