Thank you for your reply.
I think the evaluation process involves randomly choosing the evaluation
proportion. The problem is that I always get the best result when I set
neighbors to 2, which seems unreasonable to me. Since there should be
many test case that the recommender system couldn't predict at all. So
why did I still get a valid result? How does Mahout handle this case?
Sorry I didn't make myself clear for the second question. Here is the
problem: I have a set of inferred preference ranging from 0 to 1000. But
I want to map it to 1 - 5. So there can be many ways for mapping. Let's
take a simple example, if the mapping rule is like the following:
if (inferred_preference < 995) preference = 1;
else preference = inferred_preference - 995.
You can see that this is a really bad mapping algorithms, but if we run
the generated preference to Mahout, it is going to give me a really nice
result because most of the preference is 1. So is there any other metric
to evaluate this?
Any help will be highly appreciated.
Best Regards,
Jimmy
Zhongduo Lin (Jimmy)
MASc candidate in ECE department
University of Toronto
On 2013-05-08 4:44 AM, Sean Owen wrote:
It is true that a process based on user-user similarity only won't be
able to recommend item 4 in this example. This is a drawback of the
algorithm and not something that can be worked around. You could try
not to choose this item in the test set, but then that does not quite
reflect reality in the test.
If you just mean that compressing the range of pref values improves
RMSE in absolute terms, yes it does of course. But not in relative
terms. There is nothing inherently better or worse about a small range
in this example.
RMSE is a fine eval metric, but you can also considered mean average precision.
Sean
On Wed, May 8, 2013 at 4:29 AM, Zhongduo Lin <[email protected]> wrote:
Hi All,
I am using the Mahout to build a user-based recommender system (RS). The
evaluation method I am using is
AverageAbsoluteDifferenceRecommenderEvaluator, which according to the
"Mahout in Action" randomly sets aside some existing preference and
calculate the difference between the predicted value and the real one. The
first question I have is that in a user-based RS, if we choose a small
number of neighbours, then it is quite possible that the prediction is not
available at all. Here is an example:
User 1 rated item 1, 2, 3, 4
neighbour1 of user 1 rated item 1, 2
neighbour2 of user 1 rated item 1, 3
In the case above, the number of neighbours is two, so if we take out the
rating of user 1 to item 4, there is no way to predict it. What will mahout
deal with such a problem?
Also, I am trying to map inferred preferences to a scale of 1-5. But the
problem is that if I simply map all the preference to 1-2, then I will get a
really nice evaluation result (almost 0), but you can easily see that this
is not a right way to do it. So I guess the question is whether there is
another way to evaluate the preference mapping algorithm.
Any help will be highly appreciated.
Best Regards,
Jimmy
