Evaluating recommendations with expired items

[Anatoliy Kats]

Hi,

I brought up this question in dev a few weeks ago.  I have a 
recommendation algorithm that learns the similarity matrix relying on 
both current items, and expired ones that should not be recommended.  
However, AverageAbsoluteDifferenceRecommenderEvaluator compares the 
predicted and actual ratings for all items, expired or not.  I believe 
the evaluation would be more realistic if it did not -- it corresponds 
more closely to how the algorithm is normally deployed in production.  
For example, the newer items generally have fewer clicks, so this kind 
of an evaluation emphasizes the cold start problem we would experience 
in production.

The evaluation uses expired items even if if I write a recommender class 
that forces all recommendations to use an IDRescorer that sets their 
scores to NaN.  The reason is that the ...Evaluator calls the 
Recommender::doEstimatePreference function to calculate the predicted 
rating, bypassing the recommend function.  I checked for the presence of 
expired items by running my recommender in the debugger, and checking 
the item IDs when doEstimatePreference is called.

Do I understand the evaluator's behavior correctly?  Do you think this 
is considered a bug?

Thanks,

Anatoliy