[
https://issues.apache.org/jira/browse/OPENNLP-715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14352992#comment-14352992
]
Rodrigo Agerri commented on OPENNLP-715:
----------------------------------------
I have committed the first approach to this and I have tested it training with
brown, word2vec and clark clusters and it works.
First, I have extended, as suggested, the WordClusterFeatureGenerator to
contain a prefix. The prefix is simply the name of the resource provided by the
GeneratorFactory.
Furthermore, I have modified the class W2VClassesDictionary to also read Clark
style cluster lexicons. These are like Word2vec clusters but also contain a
third field which is not used in the feature generator. This way, we can use
the same functions to several types of clusters.
Finally, I think we should changed two things:
1. the name of the feature generator tag to a more general "wordcluster" so
that now the feature descriptor should contain elements such as
<wordcluster dict="word2vec-cluster.txt" />
<wordcluster dict="clark-cluster.txt" />
instead of
<w2vwordcluster dict="word2vec-cluster.txt" />
<w2vwordcluster dict="clark-cluster.txt" />
2. Refactor the W2VClassesDictionary and W2VClassesFeatureGeneratorFactory
classes to a more general WordClusterDictionary and
WordClusterFeatureGeneratorFactory to reflect the neutrality of the classes
with respect to the type of clusters they use.
what do you think?
> Clark clusters NameFinder features
> ----------------------------------
>
> Key: OPENNLP-715
> URL: https://issues.apache.org/jira/browse/OPENNLP-715
> Project: OpenNLP
> Issue Type: New Feature
> Components: Name Finder
> Affects Versions: 1.6.0
> Reporter: Rodrigo Agerri
> Assignee: Rodrigo Agerri
> Priority: Minor
> Fix For: 1.6.1
>
>
> Add token based features from Clark clusters (Clark 2003). This feature is
> actually the same as the one implemented in the WordClusterFeatureGenerator,
> but we should somehow make them separate (perhaps implementing a dynamic
> prefix id for each one, as in the dictionary features) as it has been shown
> that the combination of these clustering-based features improve results.
> Clark clusters can be generated using this tool:
> https://github.com/ninjin/clark_pos_induction
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
