Hello, I didn't follow discussions about plugins and parsers closely enough - when do plugins get invoked? During fetching, or are they part of some (which?) post-processing step? Concretely, say that I'm fetching a PDF document. Is its text extracted by one of the fetcher threads, or is the PDF stored on disk as-is, and processed later? I think the answer is the former, but I want to double-check this.
Is this also where language identification occurs? I'm asking about this particular plugin because I noticed I'm getting Strings like 'Spanish' or 'English' as identified languages, and I expected to get the 2-letter ISO codes like 'es' or 'en', or 'en-US' and 'en-GB'. I just took a quick look at net.nutch.analysis.lang.HTMLLanguageParser and I see no normalization. This means that I can expect to get back 'english' as well as 'English', and so on. Would it make sense to try and normalize the returned values here? In my particular case this is causing me a bit of trouble because I wanted to store this language information in a CHAR(2) type of a column in a RDBMS... that is, before I saw that I'm getting really variable data back. Thanks, Otis ------------------------------------------------------- This SF.net email is sponsored by: IT Product Guide on ITManagersJournal Use IT products in your business? Tell us what you think of them. Give us Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more http://productguide.itmanagersjournal.com/guidepromo.tmpl _______________________________________________ Nutch-developers mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/nutch-developers
