[
https://issues.apache.org/jira/browse/MAHOUT-559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12982471#action_12982471
]
Lance Norskog edited comment on MAHOUT-559 at 1/16/11 11:58 PM:
----------------------------------------------------------------
Ignore patch upload.
This is now [MAHOUT-586].
was (Author: lancenorskog):
Total rewrite to a new modular implementation:
Removes old evaluator recommender implementation.
New RecommenderEvaluator interface with PreferenceBased and OrderBased
implementations.
New SamplingDataModel wrapper supplies randomly selected prefs from a delegate
DataModel.
PreferenceBaseRecommenderEvaluator does roughly what the old
Abstract.....Evaluator does, but uses SamplingDataModel to implement hold-outs.
RecommenderEvaluator allows different calculation formula for evaluators. The
different calculations from the first patch are picked with a choosable Enum.
I'm happy with that it does, and was able to analyze my recommender projects
more effectively.
I'm not sure exactly what the old RecommenderEvaluator did with held-out
sampled data. This code from GroupLensRecommenderEvaluatorRunner does the same
thing, I think. The training datamodel holds out the given percentage of both
users and preferences within the remaining users.
RecommenderEvaluator evaluator = new PreferenceBasedRecommenderEvaluator();
File ratingsFile = TasteOptionParser.getRatings(args);
DataModel model = ratingsFile == null ? new GroupLensDataModel() : new
GroupLensDataModel(ratingsFile);
GroupLensRecommenderBuilder recommenderBuilder = new
GroupLensRecommenderBuilder();
DataModel trainingModel = new SamplingDataModel(model, 0.0, 0.9, Mode.USER);
DataModel testModel = glModel;
Recommender trainingRecommender =
recommenderBuilder.buildRecommender(trainingModel);
Recommender testRecommender =
recommenderBuilder.buildRecommender(testModel);
RunningAverage tracker = new CompactRunningAverageAndStdDev();
evaluator.evaluate(trainingRecommender, testRecommender, 50, tracker,
RecommenderEvaluator.Formula.NONE);
double average = tracker.getAverage();
log.info(String.valueOf(average));
> Compare Recommender output by order of recommendations.
> -------------------------------------------------------
>
> Key: MAHOUT-559
> URL: https://issues.apache.org/jira/browse/MAHOUT-559
> Project: Mahout
> Issue Type: New Feature
> Components: Collaborative Filtering
> Reporter: Lance Norskog
> Fix For: 0.5
>
> Attachments: MAHOUT-559.patch, MAHOUT-559.patch,
> OrderBasedRecommenderEvaluator.patch
>
>
> The existing RecommenderEvaluator
> (AverageAbsoluteDifferenceRecommenderEvaluator.java) has a very limited API.
> It evaluates a Recommender's performance on a training v.s. test scenario. It
> does not allow comparing the outputs of different recommenders against the
> same data model. Also, I could not figure out how its comparison criteria.
> OrderBasedRecommenderEvaluator compares the output of two recommenders. It
> only checks the order of the items in the recommendations, ignoring the
> returned preference values.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
