On the contrary, make_scorer, replacing Scorer, was merged into master in
the last couple of days. Try pulling the latest changes.
On Thu, Jul 25, 2013 at 10:33 PM, Josh Wasserstein
<ribonucle...@gmail.com>wrote:
> Got it, I just realized that the
> dev<http://scikit-learn.org/dev/modules/model_evaluation.html#the-scoring-parameter-defining-model-evaluation-rules>documentation
> is outdated (looking at the code I noticed that make_scorer
> has been replaced by Scorer).
>
> Thanks.
>
> Josh
>
>
> On Thu, Jul 25, 2013 at 8:24 AM, Josh Wasserstein
> <ribonucle...@gmail.com>wrote:
>
>> Thanks. I am having problems when using the micro/macro variants for
>> GridSearchCV. I tried creating the corresponding scorer objects, but I got
>> the error:
>>
>> > cannot import name make_scorer
>>
>> This is with 0.14 git (from master) that I checked out about a week ago.
>> Here is the code in more detail
>> ============================
>> from sklearn.metrics import fbeta_score, f1_score, make_scorer
>> f1_micro = make_scorer(f1_score, average='micro')
>> f1_macro = make_scorer(f1_score, average='macro')
>> f1_weighted = make_scorer(f1_score, average='weighted')
>>
>> score_functions = [f1_micro, f1_macro, f1_weighted]
>>
>> for score_func in score_functions:
>> clf = GridSearchCV(SVC(C=1, cache_size=5000),
>> tuned_parameters,
>> scoring=score_func,
>> verbose=1, n_jobs=1, cv=cv_method)
>> clf.fit(X, y)
>> ...
>> ============================
>>
>> Josh
>>
>>
>> On Thu, Jul 25, 2013 at 8:07 AM, Olivier Grisel <olivier.gri...@ensta.org
>> > wrote:
>>
>>> 2013/7/25 Josh Wasserstein <ribonucle...@gmail.com>:
>>> > Thank you Olivier. I went through that paper and I agree, it looks like
>>> > implementing micro-AUC or macro-AUC should not be that hard. I will
>>> try to
>>> > implement within the next week. I have have never contributed to a
>>> project
>>> > in GitHub, so I am not sure to what extent my code would meet the
>>> standards
>>> > but I am happy to try.
>>> >
>>> > In the mean time, is there anything similar to an AUC metric that
>>> scikit
>>> > supports when working with GridSearchCV in a multi-label setting? I am
>>> > looking for some compromise between precision and recall that
>>> indirectly
>>> > optimizes for the AUC score of each label .
>>>
>>> You can try the f1 score that is a balanced score (a tradeoff between
>>> precision and recall) that is a reasonable score for imbalanced
>>> multiclass dataset.
>>>
>>> It supports both micro and macro averaging.
>>>
>>>
>>> --
>>> Olivier
>>> http://twitter.com/ogrisel - http://github.com/ogrisel
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> See everything from the browser to the database with AppDynamics
>>> Get end-to-end visibility with application monitoring from AppDynamics
>>> Isolate bottlenecks and diagnose root cause in seconds.
>>> Start your free trial of AppDynamics Pro today!
>>>
>>> http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
>>> _______________________________________________
>>> Scikit-learn-general mailing list
>>> Scikit-learn-general@lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>>>
>>
>>
>
>
> ------------------------------------------------------------------------------
> See everything from the browser to the database with AppDynamics
> Get end-to-end visibility with application monitoring from AppDynamics
> Isolate bottlenecks and diagnose root cause in seconds.
> Start your free trial of AppDynamics Pro today!
> http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
> _______________________________________________
> Scikit-learn-general mailing list
> Scikit-learn-general@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>
>
------------------------------------------------------------------------------
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
_______________________________________________
Scikit-learn-general mailing list
Scikit-learn-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
