On 3/3/2011 7:12 PM, Dávid Tóth wrote:
> Would it be useful to make a program that would create topic relations for
> each wikipedia article based on the links and the distribution of semantic
> structures?
This would be very useful for me.
I'm thinking about attack this problem by discovering 'low hanging
fruits'.
To some extent you can assume that
:X :wikiLink :Y -> :X skos:related :Y
but the nature and strength of links is hard to estimate. I've
developed a good metric for approximating the importance of topic :X,
but I've yet to get a handle on relationship strength. To take an
example, there's a link from :Metallica to :Yale_University because
:Metallica :Sued :Yale_University
That's not a very strong connection. Now, if Wikipedia mentioned
the "Dead at Cornell" recording which was made when Jerry Garcia had
just gotten hooked on opium and the band was playing at it's best, we
might say
:Grateful_Dead :PlayedAt :Cornell_University
maybe you think that's a stronger connection than the above, maybe
you don't. Then again,
:Rod_Serling :TaughtAt :Ithaca_College
is one of the stronger links involving :Ithaca_College in my opinion.
There are two angles I see for extracting better relationships from
Wikipedia and these are
(i) databases such as Freebase and DBPedia, in particular, these have
certain relationships already semantized and other information that can
be used to infer about possible relationships. For instance,
:Brown_Bear :Sued :Pelican
doesn't make any sense and should be rejected.
(ii) analysis of the text around a link. You could certainly see
certain language patterns that are frequently used, for instance
"A is a B", "C married D", "E was born at F"
you could either find some of these by hand or you could write something
that uses machine learning techniques to discover these. Information
from type (i) could be useful here. For instance, we could find a
bunch of relationships that exist in Freebase and use these as positive
training examples. The trouble I see here is the creation of a good set
of negative training examples, which has a few aspects: one is that
examples that should be positive will slip into a negative sample,
attempts to automatically exclude positives will probably also exclude
'near miss' negatives that would be especially important to include
training set, and generally, the number of negatives would be 1000 or
more times prevalent than positives, which gives most ML methods
Bayesian priors that destroy recall.
Another issue is that you'll see the patterns
"E was born at F"
"[[E]] was born at F"
"E was born at [[F]]"
"[[E]] was born at [[F]]"
all occur (sometimes they make the text describing the subject a link,
sometimes they don't.) Getting good recall then means solving the named
entity extraction problem as well, however, making this part of a
'whole system' might create the kind of feedback control loop that's
necessary for high-performing A.I.
The best attack on this, I think, is to pick one particular
relationship that you want to extract, particularly one that has a bit
of a 'closed world' aspect in that you can presume that that property
ought to exist for all members of a type. For instance, we can say that
"any person was born at some location"
but even there you can get into trouble quick, if you look at
:Joan_of_arc, you see that wikipedia says that she was
"A peasant <http://en.wikipedia.org/wiki/Peasant> girl born in eastern
France"
you note that "A peasant girl" == :Joan_of_arc and that a more specific
birthplace can be found in the infobox.
_______________________________________________
Wikitech-l mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
