On May 28, 2008, at 2:20 AM, Robin Anil wrote:
Hi Grant,
I have been learning the Hadoop system and the NB
patch you
submitted was of great help. But to implement the CNB classifier
with all
the vector normalization given in Jason Rennies paper. I thought of
modifying the Mapper,Reducer and Model reader to incorporate the
following.
1) Count the Number of Document per Class;
2) Count the Term Document frequency (i.e Number of Documents in
each label
the term occurs)
I thought I already was doing these things, but will brush off my
patch this weekend and take a look.
Since NB, CNB or any other classifier like SVM begins at the same
same
level. ie. counts of Terms and Documents for text. It would be
better if we
put the Model Reader and Mapper under say classifier.common.text and
reuse
them. Once the documents are read then new map reduce tasks can be
run to
build the Corresponding Models with normalizations etc to generate
the final
NB or CNB or SVM Model.
That sounds good, I am hoping to have some time this weekend to work
on this too. So, if you want, put up a patch for me to review, and
I'll take a look.
I felt that there are certain areas like not keeping number of
mappers to
the number labels in a one-one correpondence may help speed up the
reading
of data. I dont have formulation as yet. But hope it becomes clear
when i
understand hadoop better.
Waiting to hear your thoughts
Robin
