Thanks for the responses.
@Sean we will follow your approach for calculating precision and recall. When you suggest using videos watched most often for most-preferred items for a user do you mean most watched by a particular user or overall? We might also infer a user’s most-preferred items by grouping by show if there are multiple video views from the same show. We will also take a look at the other measures you mention like F1 score, discounted cumulative gain, and mean average precision. @Sebastian thanks for your suggestion. We do have some temporal data like the time of day the video is viewed or relative to a new episode. We will inspect this relationship in more detail. @Lance I hear you on the behaviorial data. Our first release on just Boolean data will definitely leave plenty of room for improvement. For future releases we plan on capturing more implicit viewing data like start and stop time of the video or how many videos watched in a session so we can incorporate more advanced techniques like matrix factorization via Alternating Least Squares. On Sun, Aug 26, 2012 at 1:23 PM, Lance Norskog <[email protected]> wrote: > Behavior is a much better basis for recommendation than user ratings. > Raters are self-selected. If you can track whether someone actually > watches the video, that is gold data. > > There is a meme in machine learning that signal drives out noise as > data increases, so you can get away with simpler algorithms as you get > more data. > > On Sun, Aug 26, 2012 at 8:38 AM, Sebastian Schelter <[email protected]> > wrote: > > If you have temporal information, you should use these to split the data. > > Try to predict later interactions from older ones. > > > > Am 26.08.2012 17:04 schrieb > >> > >> It's the same idea, but yes you'd have to re-implement it for Hadoop. > >> > >> Randomly select a subset of users. Identify a small number of > >> most-preferred items for that user -- perhaps the video(s) watched > >> most often. Hold these data points out as a test set. Run your process > >> on all the rest. > >> > >> Make recommendations for the selected users. You then just see how > >> many in the list were among the test data you held out. The percentage > >> of recs that were in the test list is precision, and the percent of > >> the test list in the recs is recall. > >> > >> Precision and recall are not good tests, but among the only ones you > >> can carry out in the lab. Slightly better are variations on these two > >> metrics, like F1 measure and normalized discounted cumulative gain. > >> Also look up mean average precision. > >> > >> On Sun, Aug 26, 2012 at 10:47 AM, Jonathan Hodges <[email protected]> > >> wrote: > >> > Hi, > >> > > >> > We have been tasked with producing video recommendations for our > users. > >> We > >> > get about 100 million video views per month and track users and the > >> videos > >> > they watch, but currently we don’t collect rating value or preference. > >> > Later we plan on using implicit data like percentage of video watched > to > >> > surmise preferences but for the first release we are stuck with > Boolean > >> > viewing data. To that end we started by using Mahout’s distributed > >> > RecommenderJob with LoglikelihoodSimilarity algorithm to generate 50 > >> video > >> > recommendations for each user. We would like to gauge how well we are > >> doing > >> > by offline measuring precision and recall of these recommendations. We > >> know > >> > we should divide the viewing data into training and test data, but not > >> real > >> > sure what steps to take next. For the non-distributed approach we > would > >> > leverage IRStatistics to get the precision and recall values, but it > >> seems > >> > there isn’t as simple a solution within the Mahout framework for the > >> Hadoop > >> > based calculations. > >> > > >> > Can someone please share/suggest their techniques for evaluating > >> > recommendation accuracy with Mahout’s Hadoop-based distributed > >> algorithms? > >> > > >> > Thanks in advance, > >> > > >> > Jonathan > >> > > > > -- > Lance Norskog > [email protected] >
