In a grid search, you won't find a global minima - you enumerate a finite
set of hyperparameters (the carthesian product of the choices you've
picked) and find the one that gives you the lowest value.
Lars was suggesting a clever way of reducing your search space - as long as
the improvement from tuning the hyper parameter is greater than the
variance of your fit, doing exponential leaps across C, then finetuning is
a good idea.
In general, the most important parameter in SVM is C - so focus on that.
If you want to do more sophisticated searches, you can try spearmint or
hyperopt - both excellent tools that do a fantastic job of doing more
sophisticated searches. However, they both take quite a long time to run,
so make sure that it's actually worthwhile to truly optimize what you are
doing.
On Mon, Oct 13, 2014 at 12:40 PM, Adamantios Corais <
adamantios.cor...@gmail.com> wrote:
> Hi Lars,
>
> Thanks for your answer.
>
> I removed tol and I start using per-kernel grids.
>
> It is not clear, though, what's a good range for 'degree' , 'gamma', and
> 'coef0'. In addition, I believe that the "zoom in" -approach may not lead
> to the global maximum but rather to some local ones. Is that correct? What
> should I do with the aforementioned parameters? Should I use step 10, step
> 1, step 0.1, step 0.01 or what?
>
>
> On Mon, Oct 13, 2014 at 1:05 PM, Lars Buitinck <larsm...@gmail.com> wrote:
>
>> 2014-10-13 10:37 GMT+02:00 Adamantios Corais <adamantios.cor...@gmail.com
>> >:
>> > I am running into the problem that the hyperparameters of my svm.SVC()
>> are
>> > too wide such that the GridSearchCV() never gets completed! One idea is
>> to
>> > use RandomizedSearchCV() instead. But again, my dataset is relative big
>> such
>> > that 500 iterations take about 1 hour! My question is, what is a good
>> set-up
>> > (in terms of the range of values for each hyperparameter) in
>> GridSearchCV
>> > (or RandomizedSearchCV) in order to stop wasting resources... In other
>> > words, how to decide whether or not e.g. C values above 100 make sense
>> > and/or step of 1 is neither big not small? Any help is very much
>> > appreciated. This is the set-up am currently using:
>>
>> Start off by not grid-searching tol. It determines when to stop
>> learning, not what the model should look like. You're probably fitting
>> practically the same set of models ten times by using many similar
>> values for tol.
>>
>> Second, use per-kernel grids to prevent searching along irrelevant
>> dimensions:
>>
>> parameters = [
>> {'kernel': ['linear'], 'C': C_values, ...}
>> {'kernel': ['rbf'], 'C': C_values, 'gamma': gamma_values, ...}
>> ]
>>
>> The linear kernel ignores degree, gamma and coef0. The RBF kernel
>> ignores degree and coef0. GridSearchCV doesn't know this, so you're
>> again fitting the same models many times.
>>
>> As for the actual question: C is usually determined by first using a
>> coarse, exponential range like 'C': [1, 10, 100, 1000]. You can then
>> "zoom in" on the optimum in a second grid-search (if it was 10, try
>> [5, 20, 50]).
>>
>>
>> ------------------------------------------------------------------------------
>> Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
>> Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
>> Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
>> Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
>> http://p.sf.net/sfu/Zoho
>> _______________________________________________
>> Scikit-learn-general mailing list
>> Scikit-learn-general@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>>
>
>
>
> ------------------------------------------------------------------------------
> Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
> Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
> Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
> Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
> http://p.sf.net/sfu/Zoho
> _______________________________________________
> Scikit-learn-general mailing list
> Scikit-learn-general@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>
>
------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://p.sf.net/sfu/Zoho
_______________________________________________
Scikit-learn-general mailing list
Scikit-learn-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
