solr and uima dictionary annotator
Hi everyone I am using the UIMA DictionaryAnnotator to tag Solr documents. It seems to be working (I do get tags), but I get some strange behavior: 1. I am using the White Space Tokenizer both for the indexed text and for creating the dictionary. Most entries in my dictionary consist of multiple words. From the documentation, it seems that with the default settings, a document must contain all words in order to match the dictionary entry. However, this is not the case in practice. I'm seeing documents being randomly tagged with single words, although my dictionary does not contain an entry for those single words (they only appear as part of multi word entries). This would be fine (even preferable), if it were consistent. But it is not. The tagging happens only for a subset of single words, not for all. What am I doing wrong? 2. If a dictionary word appears multiple times in the analyzed field, it is also added just as many times to the mapped field (i.e. my tags). Is there a way to control/disable this? Thanks! Regards Andreea -- View this message in context: http://lucene.472066.n3.nabble.com/solr-and-uima-dictionary-annotator-tp4208359.html Sent from the Solr - User mailing list archive at Nabble.com.
Solr and UIMA, capturing fields
Hi, I successfully combined Solr and UIMA with the help of https://wiki.apache.org/solr/SolrUIMA and other pages (and am happy to provide some help about how to reach this step). Right now I can run an analysis engine and get some "primitive" feature/fields which I specify in the schema.xml automatically recognized by Solr. But if the features itself are objects, I do not know how to capture them in Solr. I provided the relevant solrconfig.xml in [1], and the schema.xml addition in [2] for the following small example, they are using the AE directly provided by the UIMA example. With the input "This is a sentence with an email at u...@host.com", Solr correctly adds the field: "UIMAname": [ "36" ] since this is the index where the email token starts. I could also successfully capture the feature end to indicate where the found email token ends. However, example.EmailAddress has the features: "begin, end, sofa". sofa is not a primitive feature, but an "object" which itself has features "sofaNum, sofaID, sofaString, ..." How can I access fields in Solr from an annotation like example.EmailAddress that are not simple strings but itself objects? I made an image of the CAS Visual Debugger with this AE and the sentence to show which fields I mean, I hope this makes it more clear: http://tinypic.com/view.php?pic=34rud1s&s=8#.VN5bF7s2cWN Does anyone know how to access such fields with Solr and UIMA? Thanks a lot for any help, Tom [1] /home/toliwa/javalibs/uimaj-2.6.0-bin/apache-uima/examples/descriptors/analysis_engine/UIMA_Analysis_Example.xml false id false text example.EmailAddress begin UIMAname [2]
Solr and UIMA #2
Hi all, just a quick notice to let you know that a new component to consume UIMA objects to a (local or remote) Solr instance is available inside UIMA sandbox [1]. Note that this "writes" to Solr from UIMA pipelines (push) while in SOLR-2129 [2] Solr "asks" UIMA to extract metadata while indexing documents (pull) . I'm going to write some documentation about Solr and UIMA on http://wiki.apache.org/solr/SolrUIMA . Cheers, Tommaso [1] : http://svn.apache.org/repos/asf/uima/sandbox/trunk/Solrcas/ [2] : https://issues.apache.org/jira/browse/SOLR-2129
Re: Solr and UIMA
You can test our UIMA to Solr cas consumer is based on JulieLab Lucas and uses their CAS. but transformed to generate XML which can be saved to a file or posted direcly to solr In the map file you can define which information is generated for each token, and how its concatenaded, allowing the generation of thinks like "the|AD car|NC " which then can be processed using payloads. now you can get it from my page http://www.barcelonamedia.org/personal/joan.codina/en http://www.barcelonamedia.org/personal/joan.codina/en -- View this message in context: http://old.nabble.com/Solr-and-UIMA-tp24567504p27753399.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr and UIMA
Hi Joan, I'm curious to try your Solr-Cas consumer. Do you have news already ;-) Roland On 02/11/2010 03:15 PM, JCodina wrote: > > Things are done :-) > > now we already have done the UIMA CAS consumer for Solr, > we are making it public, more news soon. > > We have also been developing some filters based on payloads > One of the filters is to remove words with the payloads in the list the > other one maintains only these tokens with paylodas in the list. It works > the same way than the stopsFilterFactory > > you can find it at my page: > http://www.barcelonamedia.org/personal/joan.codina/en > http://www.barcelonamedia.org/personal/joan.codina/en >
Re: Solr and UIMA
Things are done :-) now we already have done the UIMA CAS consumer for Solr, we are making it public, more news soon. We have also been developing some filters based on payloads One of the filters is to remove words with the payloads in the list the other one maintains only these tokens with paylodas in the list. It works the same way than the stopsFilterFactory you can find it at my page: http://www.barcelonamedia.org/personal/joan.codina/en http://www.barcelonamedia.org/personal/joan.codina/en -- View this message in context: http://old.nabble.com/Solr-and-UIMA-tp24567504p27544646.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr and UIMA
he set of capabilities to process the xml is defined in XML, >>>> similar to >>>> lucas to define the ouput and in the solr schema to define how this is >>>> processed. >>>> >>>> >>>> I want to use it in order to index something that is common but I >>>> can't get >>>> any tool to do that with sol: indexing a word and coding at the same >>>> position the syntactic and semantic information. I know that in >>>> Lucene this >>>> is evolving and it will be possible to include metadata but for the >>>> moment >>> >>> What does Lucas do with Lucene? Is it putting multiple tokens at the >>> same position or using Payloads? >>> >>> -- >>> Grant Ingersoll >>> http://www.lucidimagination.com/ >>> >>> Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids) >>> using Solr/Lucene: >>> http://www.lucidimagination.com/search >>> >>> >>> >> >> -- >> View this message in context: >> http://www.nabble.com/Solr-and-UIMA-tp24567504p24590509.html >> Sent from the Solr - User mailing list archive at Nabble.com. >> > > -- > Grant Ingersoll > http://www.lucidimagination.com/ > > Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids) using > Solr/Lucene: > http://www.lucidimagination.com/search > >
Re: Solr and UIMA
On Jul 21, 2009, at 11:57 AM, JCodina wrote: Let me sintetize: We (well, I think Grant?) do changes in the DPTFF ( DelimitedPayloadTokenFilterFactory ) so that is able to index at the same position different tokes that may have payloads. 1. token delimiter (#) 2. payload delimiter (|) We (that's me) perform a SolCAS: a UIMA CAS consumer equivalent to LuCAS but that allows indexing using Solr. This SolCAS is able to manage generate different tokens at the same position and maybe payloads, the result is ready for the new DPTFF We (me again) develop some filtering utilities based on the payload that, something like the stopwords filter but instead of rejecting those tokens that are in the stopwords list would reject those that are in the "payloads" list. We will try also to develop an n-gram generator based on the payloads, like for example find the nouns followed by an adjective that are at less than 4 positions. For the moment searches can not be performed based on payloads, not even as a filter... but this is a matter of time. Problems to solve: Perform a nice processing of the N tokens that share the same position, as the tokenizer.Next() will not give them together (which is a pitty) .Write some utility tht would allow the tools that manage multitokens to have a similar front-end and back-end that does multiple Nexts in order to put toguether all the information at the same position, performs the treatment with a multitoken structure and then generates a multitoken that is sent to the backend that has the next again on single tokens... Joan -- View this message in context: http://www.nabble.com/Solr-and-UIMA-tp24567504p24639814.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr and UIMA
On Jul 21, 2009, at 11:57 AM, JCodina wrote: Hello, Grant, there are two ways, to implement this, one is payloads, and the other one is multiple tokens at the same positions. Each of them can be useful, let me explain the way I thick they can be used. Payloads : every token has extra information that can be used in the processing , for example if I can add Part-of-speech then I can develop tokenizers that take into account the POS (or for example I can generate bigrams of Noum Adjective, or Noum prep Noum i can have a better stopwords algorithm) Multiple tokes in one position: If I can have different tokens at the same place, I can have different informations like: "was #verb _be" so I can do a search for "you _be #adjective" to find all the sentences that talk about "you" for example "you were clever" "you are tall" .. This was one of the use cases for payloads as well, but it likely needs more Query support at the moment, as the BoostingTermQuery would only allow you to boost values where it's a verb, not include/exclude. I have not understood the way that the DelimitedPayloadTokenFilterFactory may work in solr, which is the input format? the DPTFF (nice acronym, eh?) allows you to send in your normal Solr XML, but with payloads encoded in the text. For instance: the quick|JJ red|JJ fox|NN jumped|VB over the lazy| JJ brown|JJ dogs|NN The DPTFF will take the value before the delimiter as the Token and the value after the delimiter as the payload. This then allows Solr to add Payloads without modifying a single thing in Solr, at least on the indexing side. so I was thinking in generating an xml where for each token a single string is generated like "was#verb#be" and then there is a tokenfilter that splits by # each white space separated string, in this case in three words and adds the trailing character that allows to search for the right semantic info. But gives them the same increment. Of course the full processing chain must be aware of this. But I must think on multiwords tokens We could likely make a generic TokenFilter that can capture both multiple tokens and payloads all at the same time, simply by allowing it to have to attributes: 1. token delimiter (#) 2. payload delimiter (|) Then, you could do something like: was#be|verb or was#be|0.3 where "was" and "be" are both tokens at the same position and "verb" or "0.3" are payloads on those tokens. This is a nearly trivial variation of the DelimitedPayloadTokenFilter Grant Ingersoll-6 wrote: On Jul 20, 2009, at 6:43 AM, JCodina wrote: D: Break things down. The CAS would only produce XML that solr can process. Then different Tokenizers can be used to deal with the data in the CAS. the main point is that the XML has the doc and field labels of solr. I just committed the DelimitedPayloadTokenFilterFactory, I suspect this is along the lines of what you are thinking, but I haven't done all that much with UIMA. I also suspect the Tee/Sink capabilities of Lucene could be helpful, but they aren't available in Solr yet. E: The set of capabilities to process the xml is defined in XML, similar to lucas to define the ouput and in the solr schema to define how this is processed. I want to use it in order to index something that is common but I can't get any tool to do that with sol: indexing a word and coding at the same position the syntactic and semantic information. I know that in Lucene this is evolving and it will be possible to include metadata but for the moment What does Lucas do with Lucene? Is it putting multiple tokens at the same position or using Payloads? -- Grant Ingersoll http://www.lucidimagination.com/ Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids) using Solr/Lucene: http://www.lucidimagination.com/search -- View this message in context: http://www.nabble.com/Solr-and-UIMA-tp24567504p24590509.html Sent from the Solr - User mailing list archive at Nabble.com. -- Grant Ingersoll http://www.lucidimagination.com/ Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids) using Solr/Lucene: http://www.lucidimagination.com/search
Re: Solr and UIMA
Hello, Grant, there are two ways, to implement this, one is payloads, and the other one is multiple tokens at the same positions. Each of them can be useful, let me explain the way I thick they can be used. Payloads : every token has extra information that can be used in the processing , for example if I can add Part-of-speech then I can develop tokenizers that take into account the POS (or for example I can generate bigrams of Noum Adjective, or Noum prep Noum i can have a better stopwords algorithm) Multiple tokes in one position: If I can have different tokens at the same place, I can have different informations like: "was #verb _be" so I can do a search for "you _be #adjective" to find all the sentences that talk about "you" for example "you were clever" "you are tall" .. I have not understood the way that theDelimitedPayloadTokenFilterFactory may work in solr, which is the input format? so I was thinking in generating an xml where for each token a single string is generated like "was#verb#be" and then there is a tokenfilter that splits by # each white space separated string, in this case in three words and adds the trailing character that allows to search for the right semantic info. But gives them the same increment. Of course the full processing chain must be aware of this. But I must think on multiwords tokens Grant Ingersoll-6 wrote: > > > On Jul 20, 2009, at 6:43 AM, JCodina wrote: > >> D: Break things down. The CAS would only produce XML that solr can >> process. >> Then different Tokenizers can be used to deal with the data in the >> CAS. the >> main point is that the XML has the doc and field labels of solr. > > I just committed the DelimitedPayloadTokenFilterFactory, I suspect > this is along the lines of what you are thinking, but I haven't done > all that much with UIMA. > > I also suspect the Tee/Sink capabilities of Lucene could be helpful, > but they aren't available in Solr yet. > > > > >> E: The set of capabilities to process the xml is defined in XML, >> similar to >> lucas to define the ouput and in the solr schema to define how this is >> processed. >> >> >> I want to use it in order to index something that is common but I >> can't get >> any tool to do that with sol: indexing a word and coding at the same >> position the syntactic and semantic information. I know that in >> Lucene this >> is evolving and it will be possible to include metadata but for the >> moment > > What does Lucas do with Lucene? Is it putting multiple tokens at the > same position or using Payloads? > > -- > Grant Ingersoll > http://www.lucidimagination.com/ > > Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids) > using Solr/Lucene: > http://www.lucidimagination.com/search > > > -- View this message in context: http://www.nabble.com/Solr-and-UIMA-tp24567504p24590509.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr and UIMA
On Jul 20, 2009, at 6:43 AM, JCodina wrote: D: Break things down. The CAS would only produce XML that solr can process. Then different Tokenizers can be used to deal with the data in the CAS. the main point is that the XML has a the doc and field labels of solr. I just committed the DelimitedPayloadTokenFilterFactory, I suspect this is along the lines of what you are thinking, but I haven't done all that much with UIMA. I also suspect the Tee/Sink capabilities of Lucene could be helpful, but they aren't available in Solr yet. E: The set of capabilities to process the xml is defined in XML, similar to lucas to define the ouput and in the solr schema to define how this is processed. I want to use it in order to index something that is common but I can't get any tool to do that with sol: indexing a word and coding at the same position the syntactic and semantic information. I know that in Lucene this is evolving and it will be possible to include metadata but for the moment What does Lucas do with Lucene? Is it putting multiple tokens at the same position or using Payloads? -- Grant Ingersoll http://www.lucidimagination.com/ Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids) using Solr/Lucene: http://www.lucidimagination.com/search
Solr and UIMA
We are starting to use UIMA as a platform to analyze the text. The result of analyzing a document is a UIMA CAS. A Cas is a generic data structure that can contain different data. UIMA processes single documents, They get the documents from a CAS producer, process them using a PIPE that the user defines and finally sends the result to a CAS consumer, that "saves" or "stores" the result. The pipe is then a connection of different tools that annotate the text with different information. Different sets of tools are available out there, each of them deffining it's own data type's that are included in the CAS. To perform a PIPE output and input CAS of the elements to connect need to be compatible There is CAS consumer that feeds a LUCENE index, it's called LUCAS but I was looking to it, and I prefer to use UIMA connected to SOLR, why? A: I know solr ;-) and i like it B: I can configure the fields and their processing in solr using xml. Once done then I have it ready to use with a set of tools that allow me to easily explore the data C: Is easier to use SOLR as a "web service" that may receive docs from different UIMA's (Natural Language processing is CPU intensive ) D: Break things down. The CAS would only produce XML that solr can process. Then different Tokenizers can be used to deal with the data in the CAS. the main point is that the XML has a the doc and field labels of solr. E: The set of capabilities to process the xml is defined in XML, similar to lucas to define the ouput and in the solr schema to define how this is processed. I want to use it in order to index something that is common but I can't get any tool to do that with sol: indexing a word and coding at the same position the syntactic and semantic information. I know that in Lucene this is evolving and it will be possible to include metadata but for the moment So, my idea is first to produce a UIMA CAS consumer that performs the POST of an XML file containing the plain text text of the document to SOLR; then try to modify this in order to include multiple fields and start coding the semantic information. So, before starting, i would like to know your opinions and if anyone is interested to collaborate, or has some code that can be integrated into this. -- View this message in context: http://www.nabble.com/Solr-and-UIMA-tp24567504p24567504.html Sent from the Solr - User mailing list archive at Nabble.com.
