Btw, micro-average is trivial to implement (assuming I understand this
correctly; I've not looked at the paper):
def multilabel_pr_curve(y_true, y_scores):
y_true = LabelBinarizer.fit_transform(y_true)
return precision_recall_curve(y_true.ravel(), y_score.ravel())
Macro-average is trickier to code. (A version I can conceive on based on
the current _binary_clf_curve would flatten things as well, but maintain a
list of labels that is then permuted into score order. You would then
replace cumsum() with cumbincount() being if such a thing existed.)
- Joel
On Wed, Aug 7, 2013 at 10:40 AM, Issam <issamo...@gmail.com> wrote:
> @federico haha! thanks for the motivation!
>
> @Josh, I'm not aware of polling in Github, but it sounds very
> convenient, a polling feature would be a great addition to scikit ;)
>
> @Gael, I also thought that AUC is not suitable for multi-labels, but if
> you check the latest kaggle competitions such as this one
> `http://www.kaggle.com/c/mlsp-2013-birds/forums` they have established
> AUC measure for multi-label classification. I thought of a simple way to
> do it which is to first label Binarize the output so lets say
> y=[[1,2],[1]] which means sample 1 belongs to class 1 and 2 and sample 2
> belongs to class 1, then the binarized form would be
> y=[[0,1,1],[0,1,0]], finally this can be rasterized to form a vector on
> which the predicted probabilities can be evaluated against, using the
> trivial AUC metrics already implemented in scikit, this could be wrong,
> however, the scores achieved were quite as to the leader-board.
>
> There are quite a number of papers that use AUC for multilabels, for
> example, http://www.cse.msu.edu/~rongjin/publications/iccv_camera.pdf
> <http://www.cse.msu.edu/%7Erongjin/publications/iccv_camera.pdf>
>
>
>
> ------------------------------------------------------------------------------
> Get 100% visibility into Java/.NET code with AppDynamics Lite!
> It's a free troubleshooting tool designed for production.
> Get down to code-level detail for bottlenecks, with <2% overhead.
> Download for free and get started troubleshooting in minutes.
> http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
> _______________________________________________
> Scikit-learn-general mailing list
> Scikit-learn-general@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>
------------------------------------------------------------------------------
Get 100% visibility into Java/.NET code with AppDynamics Lite!
It's a free troubleshooting tool designed for production.
Get down to code-level detail for bottlenecks, with <2% overhead.
Download for free and get started troubleshooting in minutes.
http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
_______________________________________________
Scikit-learn-general mailing list
Scikit-learn-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
