I don't think the multiplication by -1 pertains: it happens before the type
check, so it will have already failed!
I think needing a float is validation required for grid search and simply
not required here; even in grid search it would be nice if there were a
distinction between the single objective (must be possible to average and
argmax) and metric evaluations (e.g. multiple classes, P/R/F, positive loss
measures).
On 4 February 2014 21:42, Gael Varoquaux <gael.varoqu...@normalesup.org>wrote:
> OK, I have been rereading the scoring codebase (which it becoming
> frighteningly complicated), and it seems that the reason that we really
> need this check is that the scores can multipled by "-1".
>
> Maybe we should have a "try/except" where the multiplication happens, and
> raise a meaningful error only if that multiplication fails.
>
> I still haven't managed to fit this codebase in my head right, so I am
> not sure how viable my suggestion would be. Anyhow, this is why we need
> this check.
>
> Gaël
>
> On Mon, Feb 03, 2014 at 11:55:17AM -0800, Faraz Mirzaei wrote:
> > Thanks Andy. I'll wait for your input (and will check the bug reports
> myself as
> > well), before creating a new PR.
>
> > In the mean time, I wonder why you would even need to do a type checking
> here?
> > As far as I know, Python strongly encourages duck typing. Why not remove
> the
> > type checking altogether and just let the exception "bubble up" when
> the duck
> > cannot quack (i.e., when the score function output cannot be used by the
> > respective caller)?
>
> > As for cross_val_score being used in GridSearchCV, I quickly skimmed
> through
> > the code and it doesn't look like it's being used there. Can someone else
> > confirm as well?
>
> > Thanks!
>
> > Faraz
>
>
> > On Sun, Feb 2, 2014 at 1:38 PM, Andy <t3k...@gmail.com> wrote:
>
> > On 01/31/2014 09:50 AM, Gael Varoquaux wrote:
> > >> if not isinstance(score, numbers.Number):
> > >> raise ValueError("scoring must return a number, got %s (%s)"
> > >> " instead." % (str(score),
> type(score)))
> > > I am not opposed to making this check more relaxed: we could add
> an 'or
> > > (isinstance(score, np.ndarray) and score.dtype.kind in ('i',
> 'f'))'.
>
> > > PR welcomed
>
> > > G
> > This has come up before. I am not 100% what the trouble was. I put
> this
> > in there as a bug fix.
> > There was a PR, not sure where it went.
> > cross_val_score is not called by GridSearchCV, right?
>
> > I'll try to find out what I was thinking....
>
> > Andy
>
> >
> ------------------------------------------------------------------------------
> > WatchGuard Dimension instantly turns raw network data into actionable
> > security intelligence. It gives you real-time visual feedback on key
> > security issues and trends. Skip the complicated setup - simply
> import
> > a virtual appliance and go from zero to informed in seconds.
> > http://pubads.g.doubleclick.net/gampad/clk?id=123612991&iu=/4140/
> > ostg.clktrk
> > _______________________________________________
> > Scikit-learn-general mailing list
> > Scikit-learn-general@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>
>
>
> >
> ------------------------------------------------------------------------------
> > Managing the Performance of Cloud-Based Applications
> > Take advantage of what the Cloud has to offer - Avoid Common Pitfalls.
> > Read the Whitepaper.
> >
> http://pubads.g.doubleclick.net/gampad/clk?id=121051231&iu=/4140/ostg.clktrk
>
> > _______________________________________________
> > Scikit-learn-general mailing list
> > Scikit-learn-general@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>
>
> --
> Gael Varoquaux
> Researcher, INRIA Parietal
> Laboratoire de Neuro-Imagerie Assistee par Ordinateur
> NeuroSpin/CEA Saclay , Bat 145, 91191 Gif-sur-Yvette France
> Phone: ++ 33-1-69-08-79-68
> http://gael-varoquaux.info http://twitter.com/GaelVaroquaux
>
>
> ------------------------------------------------------------------------------
> Managing the Performance of Cloud-Based Applications
> Take advantage of what the Cloud has to offer - Avoid Common Pitfalls.
> Read the Whitepaper.
>
> http://pubads.g.doubleclick.net/gampad/clk?id=121051231&iu=/4140/ostg.clktrk
> _______________________________________________
> Scikit-learn-general mailing list
> Scikit-learn-general@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>
------------------------------------------------------------------------------
Managing the Performance of Cloud-Based Applications
Take advantage of what the Cloud has to offer - Avoid Common Pitfalls.
Read the Whitepaper.
http://pubads.g.doubleclick.net/gampad/clk?id=121051231&iu=/4140/ostg.clktrk
_______________________________________________
Scikit-learn-general mailing list
Scikit-learn-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
