On Wed, Sep 11, 2013 at 12:22 AM, Parimi Rohit <rohit.par...@gmail.com>wrote:
> 1. Do we have to follow this setting, to compare algorithms? Can't we > report the parameter combination for which we get highest mean average > precision for the test data, when trained on the train set, with out any > validation set. > As Ted alludes to this would overfit. Think of it as two learning processes. You learn model hyper-parameters like lambda, and you learn model parameters like your matrix decomposition. So there must be two levels of hold-out. > 2. Do we have to tune the "similarityclass" parameter in item-based CF? If > so, do we compare the mean average precision values based on validation > data, and then report the same for the test set? > > Yes you are conceptually looking over the entire hyper-parameter space. If the similarity metric is one of those, you are trying different metrics. Grid search, just brute-force trying combinations, works for 1-2 hyper-parameters. Otherwise I'd try randomly choosing parameters, really, or else it will take way too long to explore. You try to pick hyper-parameters 'nearer' to those that have yielded better scores.
