The idea is sound but for a different and I think stronger reason. For this kind of test you need to hold out items that are some of the best recommendations, since that's what the recommender is trying to find. Holding out random items isn't OK since the recommender is not simply trying to parrot back the user's rating set.
In the binary case, all items look the same, so you don't know what's best to hold out. The best you can do is random, and it's not a good thing. Here you have side information about what makes a good recommendation. It's valid to prefer these in your test set, and of course you want to prefer them in the recommender too which you are. On Thu, Jun 21, 2012 at 10:41 PM, Nimrod Priell <[email protected]> wrote: > My rationalization for this was that because 80% of the users have the most > popular item, it is picked somewhat more often to be the hidden item, and > it's very easy for the popular recommender to guess it right. When I ask the > recommender for many items, the effect of the popular ones dampens. Does that > make sense? > > Best, > Nimrod >
