Hi Grant,
I have been learning the Hadoop system and the NB patch you
submitted was of great help. But to implement the CNB classifier with all
the vector normalization given in Jason Rennies paper. I thought of
modifying the Mapper,Reducer and Model reader to incorporate the
following.
1) Count the Number of Document per Class;
2) Count the Term Document frequency (i.e Number of Documents in each label
the term occurs)
Since NB, CNB or any other classifier like SVM begins at the same same
level. ie. counts of Terms and Documents for text. It would be better if we
put the Model Reader and Mapper under say classifier.common.text and reuse
them. Once the documents are read then new map reduce tasks can be run to
build the Corresponding Models with normalizations etc to generate the final
NB or CNB or SVM Model.
I felt that there are certain areas like not keeping number of mappers to
the number labels in a one-one correpondence may help speed up the reading
of data. I dont have formulation as yet. But hope it becomes clear when i
understand hadoop better.
Waiting to hear your thoughts
Robin
