Percentage is fine for the train/test thing, but for balancing amongst
categories, it seems like you need a max, no? In other words, given
my sample on mahout-user@, I want History and Science to have the same
number of examples (although an approximation would be fine). In a
non-MR program, I'd just pass in a max and keep a counter. In M/R,
what's the best approach? If I know the number of map tasks
definitively, then I think it is just max/numTasks, right?
-Grant
On Jul 23, 2009, at 1:39 AM, Ted Dunning wrote:
What about specifying percentage to keep or split instead?
On Wed, Jul 22, 2009 at 7:27 PM, Grant Ingersoll
<[email protected]>wrote:
What's the best way, in a M/R program, to keep track of the number
of items
one has produced/consumed? For instance, for the
WikipediaDatasetCreator,
how would you go about putting a maximum number of examples to
include in an
input file (or to split some number between train and test)? In
the old
message passing days, one would have to communicate how many
elements one
has to the other nodes or something like that, right? In theory,
it seems
like each Mapper could just do take max/(num mappers) but I'm not
100%
certain that works as I am not sure that matches Hadoop's
semantics, as I
thought the value was just a suggestion.
-Grant
--
Ted Dunning, CTO
DeepDyve
