Hi Philip,
Some success! After following your explanations -- thanks for the
detailed information in your mail! -- and putting the new JAPE files
into context/default/resources/grammar/main, I do get different
results: The document
Do you know Nicole Meier, who is Martin Fischer's girlfriend?
now has "Martin Fischer" recognized as an entity. Before, it was just
the word "Fischer". However, "Nicole Meier" still does not show up.
Also, in the document
Immob Inc., a company owned by Martin Fischer, is based in Basel,
Switzerland.
the "Immob Inc." is not recognized. Do you have a tip how to get at
least the person recognized?
I understand that you have to add JAPE files to catch NE's. So I could
add one to recognise entities with "Inc." as companies. Of course, in
general this needs expert linguistics knowledge and I am therefore
wondering: What is the state regarding the recognizition of (a) person
names and (b) companies/societies/... in English, French, and Spanish?
For person names, I can imagine that name lists (I think these are
called "gazetters") can help. I see that person_female.lst in
gazetter/ contains "Nicole", so case (a) is probably already supported
quite well for Western names (once I get my configuration right). But
what about (b)? Are you using the grammar or neighbouring words like
"a company"?
I am asking because for our purposes, persons and companies are the
most important entities and we need to recognize them without (too
much) prior knowledge. (It's not so bad if there are many false
positives.)
Again, many thanks for all your feedback and time. I appreciate this a
lot.
Kaspar
On 16.02.2009, at 20:03, Philip Alexiev wrote:
Hi Kaspar,
Thank you for your interest.
All the cached data is stored in the $KIM_HOME/context/default/
populated folder. It's enough to clear its contents to get rid of
the cache. This will affect both Sesame (the semantic repository)
and Lucene's (the document repository) stored data.
The idea behind the context folder of KIM is that all the data that
may be changed in order to tune KIM to the specific domain will be
stored in a centralized location. This location is the context
folder. It contains three parts. The 'kb' folder is the place for
the ontology schema and the knowledge base (KB). If you decide to
extend the default KIM ontology, this is a good place to add the
files describing it. The 'resources' folder contains the resources
which the NERC pipeline uses for the annotation process. Here is the
pipeline description itself, the Jape grammars, the gazetteer lists
etc. And of course the 'populated' folder which contains KIM's
cached data.
The reason the jape didn't work is that the transducer in the
default pipeline you are using reads its rules from another
location . The jape file used is $KIM_HOME/context/default/resources/
grammar/main/main.jape . This is where you have to include the new
grammar. The GATE pipeline configuration file is described in
$KIM_HOME/nerc.properties . In the default distribution KIM uses
IE-no-keywords.gapp . This is where all the processing resources are
configured. Here you can see all the transducers and the japes they
use.
I will need more information about the exception. Please change the
debug level of Lucene in the log4j configuration file: $KIM_HOME/
config/log.prop . Change the line :
log4j.category.LUCENE = , aLUCENE
to :
log4j.category.LUCENE = DEBUG, aLUCENE
Then you should be able to see more details in the LUCENE log file
in $KIM_HOME/log .
All the best.
Philip
Kaspar Fischer wrote:
Hi Philip,
Thanks for your response and the JAPE rules you attached. I have
done the following:
1. I removed the directory "context/default" and replaced it with
the original one. With this, I (hope to have) cleared the
repository. Is this correct, or is there an easier way to clear the
document and semantic repository?
2. In directory "plugins/ANNIE/resources/NE/" I added your file
long_name.jape.
3. I inserted a line "long_name" into file "plugins/ANNIE/resources/
NE/main.jape" after the line "name".
4. I added the documents from the previous mail using "./bin/tools/
toolPopulate.sh".
5. The Web UI still shows that only "Fischer" is indexed. ("Nicole
Meier" is not indexed.)
I noticed however a few things, and maybe these observations help
to tackle down the problem:
- My main.jape (see attached) looks very different from the one you
sent. Could I be using a very old version of KIM? Mine is called
kim-platform-2.3-RC1.
- I first tried with your main.jape (which contains many phases for
which there are no JAPE files in the directory, I think). I did not
get any errors relating to JAPE rules, neither in KIM's All.txt nor
in the Populator's output. Will JAPE/GATE errors be reported?
- I always get an error when I try to index a document "Wealth.txt"
which contains the single line "Martin Fischer lives in Switzerland
with his mother Elisabeth Fischer. He owns buildings worth $280,30
million.". The error is this:
INFO 8049[Thread-3] - /Users/hbf/downloads/kim-docs/Wealth.txt
INFO 8061[Thread-3] - Creating ...
INFO 8076[Thread-3] - Creating GATE document ...
INFO 8161[Thread-3] - Parsing features ...
INFO 8184[Thread-3] - Annotating ...
INFO 8384[Thread-3] - Storing ...
ERROR 8434[Thread-3] - Failed storing the document!
com
.ontotext.kim.client.documentrepository.DocumentRepositoryException
at
com
.ontotext
.kim
.lucene
.LuceneDocumentRepositoryImpl
.addDocument(LuceneDocumentRepositoryImpl.java:341)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun
.reflect
.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun
.reflect
.DelegatingMethodAccessorImpl
.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at
org
.openrdf
.util.rmirouting.ChannelIfaceImpl.invoke(ChannelIfaceImpl.java:514)
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at
sun
.reflect
.DelegatingMethodAccessorImpl
.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:
294)
at sun.rmi.transport.Transport$1.run(Transport.java:153)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:149)
at
sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:
466)
at sun.rmi.transport.tcp.TCPTransport
$ConnectionHandler.run(TCPTransport.java:707)
at java.lang.Thread.run(Thread.java:613)
INFO 8446[Thread-3] -
Completed: 25% ( 1 of 4 files processed )
[Remaining documents indexed without problems...]
I guess the last item is not related to the entity recognition
problem; I just mention it to be on the safe side.
Thank you very much for your help!
Kaspar
--
/*
* main.jape
*
* Copyright (c) 1998-2004, The University of Sheffield.
*
* This file is part of GATE (see http://gate.ac.uk/), and is free
* software, licenced under the GNU Library General Public License,
* Version 2, June 1991 (in the distribution as file licence.html,
* and also available at http://gate.ac.uk/gate/licence.html).
*
* Diana Maynard, 02 Aug 2001
*
* $Id: main.jape 9233 2007-11-23 13:01:52Z dgmaynard $
*/
MultiPhase: TestTheGrammars
Phases:
first
firstname
name
name_post
date_pre
date
reldate
number
address
url_pre
url
email
identifier
jobtitle
final
unknown
name_context
org_context
loc_context
clean
_______________________________________________
Kim-discussion mailing list
[email protected]
http://ontotext.com/mailman/listinfo/kim-discussion