Hi Kaspar,

We took some time to dig into this issue. It turned out that there is a problem in the gazetteer in the particular version you use. The problem exists only in KIM 2.3, so providing you with KIM 2.4 will solve most of the issues. You don't need to use additional JAPE rules to do that.

Thank you very much for reporting this. It is a valuable information and helped us find a bug we weren't aware of. Sorry for the inconvinience. You can get KIM 2.4 from our ftp server : ftp://ftp.ontotext.sirma.bg/kim-platform-2.4-SNAPSHOT-bin-1782.zip using this credentials kimd/asdfvcxz . Please have in mind, that 2.4 is not released yet and it is not in a stable final state. It is generally not our practice to do that, but having in mind the circumstances this is the best way to go. If you encounter some problems feel free to tell us and we will try to be as cooperative as we can.

Best wishes,
Philip

Kaspar Fischer wrote:
Hi Philip,

Some success! After following your explanations -- thanks for the detailed information in your mail! -- and putting the new JAPE files into context/default/resources/grammar/main, I do get different results: The document

 Do you know Nicole Meier, who is Martin Fischer's girlfriend?

now has "Martin Fischer" recognized as an entity. Before, it was just the word "Fischer". However, "Nicole Meier" still does not show up. Also, in the document

Immob Inc., a company owned by Martin Fischer, is based in Basel, Switzerland.

the "Immob Inc." is not recognized. Do you have a tip how to get at least the person recognized?

I understand that you have to add JAPE files to catch NE's. So I could add one to recognise entities with "Inc." as companies. Of course, in general this needs expert linguistics knowledge and I am therefore wondering: What is the state regarding the recognizition of (a) person names and (b) companies/societies/... in English, French, and Spanish?

For person names, I can imagine that name lists (I think these are called "gazetters") can help. I see that person_female.lst in gazetter/ contains "Nicole", so case (a) is probably already supported quite well for Western names (once I get my configuration right). But what about (b)? Are you using the grammar or neighbouring words like "a company"?

I am asking because for our purposes, persons and companies are the most important entities and we need to recognize them without (too much) prior knowledge. (It's not so bad if there are many false positives.)

Again, many thanks for all your feedback and time. I appreciate this a lot.
Kaspar

On 16.02.2009, at 20:03, Philip Alexiev wrote:

Hi Kaspar,

Thank you for your interest.

All the cached data is stored in the $KIM_HOME/context/default/populated folder. It's enough to clear its contents to get rid of the cache. This will affect both Sesame (the semantic repository) and Lucene's (the document repository) stored data.

The idea behind the context folder of KIM is that all the data that may be changed in order to tune KIM to the specific domain will be stored in a centralized location. This location is the context folder. It contains three parts. The 'kb' folder is the place for the ontology schema and the knowledge base (KB). If you decide to extend the default KIM ontology, this is a good place to add the files describing it. The 'resources' folder contains the resources which the NERC pipeline uses for the annotation process. Here is the pipeline description itself, the Jape grammars, the gazetteer lists etc. And of course the 'populated' folder which contains KIM's cached data.

The reason the jape didn't work is that the transducer in the default pipeline you are using reads its rules from another location . The jape file used is $KIM_HOME/context/default/resources/grammar/main/main.jape . This is where you have to include the new grammar. The GATE pipeline configuration file is described in $KIM_HOME/nerc.properties . In the default distribution KIM uses IE-no-keywords.gapp . This is where all the processing resources are configured. Here you can see all the transducers and the japes they use.

I will need more information about the exception. Please change the debug level of Lucene in the log4j configuration file: $KIM_HOME/config/log.prop . Change the line :
log4j.category.LUCENE = , aLUCENE
to :
log4j.category.LUCENE = DEBUG, aLUCENE
Then you should be able to see more details in the LUCENE log file in $KIM_HOME/log .

All the best.
Philip


Kaspar Fischer wrote:
Hi Philip,

Thanks for your response and the JAPE rules you attached. I have done the following:

1. I removed the directory "context/default" and replaced it with the original one. With this, I (hope to have) cleared the repository. Is this correct, or is there an easier way to clear the document and semantic repository? 2. In directory "plugins/ANNIE/resources/NE/" I added your file long_name.jape. 3. I inserted a line "long_name" into file "plugins/ANNIE/resources/NE/main.jape" after the line "name". 4. I added the documents from the previous mail using "./bin/tools/toolPopulate.sh". 5. The Web UI still shows that only "Fischer" is indexed. ("Nicole Meier" is not indexed.)

I noticed however a few things, and maybe these observations help to tackle down the problem:

- My main.jape (see attached) looks very different from the one you sent. Could I be using a very old version of KIM? Mine is called kim-platform-2.3-RC1.

- I first tried with your main.jape (which contains many phases for which there are no JAPE files in the directory, I think). I did not get any errors relating to JAPE rules, neither in KIM's All.txt nor in the Populator's output. Will JAPE/GATE errors be reported?

- I always get an error when I try to index a document "Wealth.txt" which contains the single line "Martin Fischer lives in Switzerland with his mother Elisabeth Fischer. He owns buildings worth $280,30 million.". The error is this:


INFO    8049[Thread-3] - /Users/hbf/downloads/kim-docs/Wealth.txt
INFO    8061[Thread-3] -      Creating ...
INFO    8076[Thread-3] - Creating GATE document ...
INFO    8161[Thread-3] -      Parsing features ...
INFO    8184[Thread-3] -      Annotating ...
INFO    8384[Thread-3] -      Storing ...
ERROR   8434[Thread-3] -      Failed storing the document!
com.ontotext.kim.client.documentrepository.DocumentRepositoryException
at com.ontotext.kim.lucene.LuceneDocumentRepositoryImpl.addDocument(LuceneDocumentRepositoryImpl.java:341)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  at java.lang.reflect.Method.invoke(Method.java:585)
at org.openrdf.util.rmirouting.ChannelIfaceImpl.invoke(ChannelIfaceImpl.java:514)
  at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  at java.lang.reflect.Method.invoke(Method.java:585)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:294)
  at sun.rmi.transport.Transport$1.run(Transport.java:153)
  at java.security.AccessController.doPrivileged(Native Method)
  at sun.rmi.transport.Transport.serviceCall(Transport.java:149)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:466) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:707)
  at java.lang.Thread.run(Thread.java:613)
INFO    8446[Thread-3] -
Completed: 25% ( 1 of 4 files processed )
[Remaining documents indexed without problems...]

I guess the last item is not related to the entity recognition problem; I just mention it to be on the safe side.

Thank you very much for your help!
Kaspar

--
/*
*  main.jape
*
* Copyright (c) 1998-2004, The University of Sheffield.
*
*  This file is part of GATE (see http://gate.ac.uk/), and is free
*  software, licenced under the GNU Library General Public License,
*  Version 2, June 1991 (in the distribution as file licence.html,
*  and also available at http://gate.ac.uk/gate/licence.html).
*
*  Diana Maynard, 02 Aug 2001
*
*  $Id: main.jape 9233 2007-11-23 13:01:52Z dgmaynard $
*/

MultiPhase:    TestTheGrammars
Phases:
first
firstname
name
name_post
date_pre
date
reldate
number
address
url_pre
url
email
identifier
jobtitle
final
unknown
name_context
org_context
loc_context
clean




_______________________________________________
Kim-discussion mailing list
[email protected]
http://ontotext.com/mailman/listinfo/kim-discussion



_______________________________________________
Kim-discussion mailing list
[email protected]
http://ontotext.com/mailman/listinfo/kim-discussion

Reply via email to