Thanks Andrzej for mentioning Behemoth! It now lives on http://github.com/jnioche/behemoth
There are indeed quite a few open source NLP framework / resources available that can be used for doing NER, including GATE or Apache UIMA. Apart from the obvious benefits related to their licenses, these resources also have the advantage of being customisable. Behemoth allows to deploy GATE or UIMA applications on Hadoop but it would be doable to embed them in a custom parsing / indexing plugin for Nutch. Some of Behemoth components could be used to that effect. Note that openNLP, stanford or lingpipe components are already available as GATE plugins. GATE also gives you a lot of things e.g. GUIs on top of these resources so it is good to use it regardless of the actual component that you want to get your annotations from. I was (and probably still am) a committer on GATE so feel free to get in touch. <sales_pitch> DigitalPebble do provide support and consultancy services on GATE / UIMA (as well as Nutch of course) </sales_pitch> :-) I recently gave a presentation on Behemoth at Berlin Buzzwords [1] which might be of interest. In particular there is a component that converts Nutch segments into the structure used by Behemoth. HTH, Julien [1] http://berlinbuzzwords.wikidot.com/local--files/links-to-slides/nioche_bbuzz2010.odp On 2 July 2010 17:41, Max Lynch <[email protected]> wrote: > Opennlp is a great java package with working named entity recognition. > I've had a lot of success with it. That said, the Stanford nlp stuff is > always highly acclaimed, though I had trouble figuring it all out the last > time I tried it. > > > On Jul 2, 2010, at 11:12 AM, Claudio Martella <[email protected]> > wrote: > > > I can advice you GATE and ANNIE. Gate is a framework for Text-mining. > > ANNIE is a pipeline of Gate's components for the extraction of Named > > Entities like names of people, locations, companies etc. You can use > > Gate/Annie programmatically throught their Java API. > > > > http://gate.ac.uk/ > > > > > > Alex McLintock wrote: > >> I'm quite interested in OpenCalais - a Reuters/Thompson initiative. It > >> is a web service to take your free text and identify important terms > >> in it like people, businesses, places, and so on. If you are the > >> document owner you can submit your document to their web site and get > >> back important tags saying what this document is about. I'd like to > >> tag this sort of data and feed it into a Lucene style index so that it > >> can be used in searches AND in focussed/topical crawls. > >> > >> Now, here comes the problem. When we crawl the web we don't own the > >> documents we are crawling so we don't really have permission to use > >> Reuters' servers to do this analysis. (Maybe we could cut a deal > >> though if we were a big enough company). > >> > >> So has anyone else looked at alternatives to OpenCalais which takes > >> free text and tries to understand what it is about? I've been looking > >> for software to do this but nothing seems suitable. > >> > >> Alex > >> > >> > > > > > > -- > > Claudio Martella > > Digital Technologies > > Unit Research & Development - Analyst > > > > TIS innovation park > > Via Siemens 19 | Siemensstr. 19 > > 39100 Bolzano | 39100 Bozen > > Tel. +39 0471 068 123 > > Fax +39 0471 068 129 > > [email protected] http://www.tis.bz.it > > > > Short information regarding use of personal data. According to Section 13 > of Italian Legislative Decree no. 196 of 30 June 2003, we inform you that we > process your personal data in order to fulfil contractual and fiscal > obligations and also to send you information regarding our services and > events. Your personal data are processed with and without electronic means > and by respecting data subjects' rights, fundamental freedoms and dignity, > particularly with regard to confidentiality, personal identity and the right > to personal data protection. At any time and without formalities you can > write an e-mail to [email protected] in order to object the processing of > your personal data for the purpose of sending advertising materials and also > to exercise the right to access personal data and other rights referred to > in Section 7 of Decree 196/2003. The data controller is TIS Techno > Innovation Alto Adige, Siemens Street n. 19, Bolzano. You can find the > complete information on the web site www.tis.bz.it. > > > > > -- DigitalPebble Ltd Open Source Solutions for Text Engineering http://www.digitalpebble.com
