Re: [Scikit-learn-general] Average Per-Class Accuracy metric

[Joel Nothman]

You mean TP / N, not TP / TN.

And I think the average per-class accuracy does some weird things. Like:

true = [1, 1, 1, 0, 0]
pred = [1, 1, 1, 1, 1]
a.p.c.a = (3 + 3) / 5 / 2

true = [1, 1, 1, 0, 2]
pred = [1, 1, 1, 1, 1]
a.p.c.a = (4 + 4 + 3) / 5 / 3

I don't think that's very useful.

On 9 March 2016 at 13:36, Sebastian Raschka  wrote:

> > Firstly, balanced accuracy is a different thing, and yes, it should be
> supported.
>
> > Secondly, I am correct in thinking you're talking about multiclass (not
> multilabel).
>
>
> Sorry for the confusion, and yes, you are right. I think have mixed the
> terms “average per-class accuracy” with “balanced accuracy” then.
>
> Maybe to clarify, a corrected example to describe what I meant. Given the
> confusion matrix
>
>predicted
>label
>
>[ 3,  0,  0]
>  true[ 7, 50, 12]
>  label   [ 0,  0, 18]
>
>
> I’d compute the accuracy as TP / TN =  (3 + 50 + 18) / 90 = 0.79
>
> and the “average per-class accuracy” as
>
> (83/90 + 71/90 + 78/90) / 3 = (83 + 71 + 78) / (3 * 90) = 0.86
>
> (I hope I got it right this time!)
>
> In any case, I am not finding any literature describing this, and I am
> also not proposing to add it to sickit-learn, just wanted to get some info
> whether this is implemented or not. Thanks! :)
>
>
>
> > On Mar 8, 2016, at 8:29 PM, Joel Nothman  wrote:
> >
> > Firstly, balanced accuracy is a different thing, and yes, it should be
> supported.
> >
> > Secondly, I am correct in thinking you're talking about multiclass (not
> multilabel).
> >
> > However, what you're describing isn't accuracy. It's actually
> micro-averaged recall, except that your dataset is impossible because
> you're allowing there to be fewer predictions than instances. If we assume
> that we're allowed to predict some negative class, that's fine; we can
> nowadays exclude it from micro-averaged recall with the labels parameter to
> recall_score. (If all labels are included in a multiclass problem,
> micro-averaged recall = precision = fscore = accuracy.)
> >
> > I had assumed you meant binarised accuracy, which would add together
> both true positives and true negatives for each class.
> >
> > Either way, if there's no literature on this, I think we'd really best
> not support it.
> >
> > On 9 March 2016 at 11:15, Sebastian Raschka 
> wrote:
> > I haven’t seen this in practice, yet, either. A colleague was looking
> for this in scikit-learn recently, and he asked me if I know whether this
> is implemented or not. I couldn’t find anything in the docs and was just
> curious about your opinion. However, I just found this entry here on
> wikipedia:
> >
> > https://en.wikipedia.org/wiki/Accuracy_and_precision
> > > Another useful performance measure is the balanced accuracy[10] which
> avoids inflated performance estimates on imbalanced datasets. It is defined
> as the arithmetic mean of sensitivity and specificity, or the average
> accuracy obtained on either class:
> >
> > > Am I right in thinking that in the binary case, this is identical to
> accuracy?
> >
> >
> > I think it would only be equal to the “accuracy” if the class labels are
> uniformly distributed.
> >
> > >  I'm not sure what this metric is getting at.
> >
> > I have to think about this more, but I think it may be useful for
> imbalanced datasets where you want to emphasize the minority class. E.g.,
> let’s say we have a dataset of 120 samples and three class labels 1, 2, 3.
> And the classes are distributed like this
> > 10 x 1
> > 50 x 2
> > 60 x 3
> >
> > Now, let’s assume we have a model that makes the following predictions
> >
> > - it gets 0 out of 10 from class 1 right
> > - 45 out of 50 from class 2
> > - 55 out of 60 from class 3
> >
> > So, the accuracy would then be computed as
> >
> > (0 + 45 + 55) / 120 = 0.833
> >
> > But the “balanced accuracy” would be much lower, because the model did
> really badly on class 1, i.e.,
> >
> > (0/10 + 45/50 + 55/60) / 3 = 0.61
> >
> > Hm, if I see this correctly, this is actually very similar to the F1
> score. But instead of computing the harmonic mean between “precision and
> the true positive rate), we compute the harmonic mean between "precision
> and true negative rate"
> >
> > > On Mar 8, 2016, at 6:40 PM, Joel Nothman 
> wrote:
> > >
> > > I've not seen this metric used (references?). Am I right in thinking
> that in the binary case, this is identical to accuracy? If I predict all
> elements to be the majority class, then adding more minority classes into
> the problem increases my score. I'm not sure what this metric is getting at.
> > >
> > > On 8 March 2016 at 11:57, Sebastian Raschka 
> wrote:
> > > Hi,
> > >
> > > I was just wondering why there’s no support for the average per-class
> accuracy in the scorer functions (if I am not overlooking something).
> > > E.g., we have 'f1_macro', 

Re: [Scikit-learn-general] "In-bag" for RandomForest*

[Mathieu Blondel]

If this function is generally useful, it might be a good idea to make it
public.

Mathieu

On Wed, Mar 9, 2016 at 1:29 AM, Ariel Rokem  wrote:

>
> On Mon, Mar 7, 2016 at 8:24 AM, Andreas Mueller  wrote:
>
>> Hi Ariel.
>> We are not storing them any more because of memory issues, but you can
>> recover them using the random state of the tree:
>>
>> https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/ensemble/forest.py#L76
>>
>> > indices = _generate_sample_indices(tree.random_state, n_samples)
>>
>>
> Yes - very helpful - thanks! I have recorded our full solution for
> posterity (and for google-ability) here:
> http://stackoverflow.com/questions/35832786/in-bag-for-randomforest-objects/35872711
>
>
>
>> Hth,
>> Andy
>>
>>
>>
>> On 03/04/2016 07:04 PM, Ariel Rokem wrote:
>>
>> Hi everyone,
>>
>> Is there some way to identify the samples that were used in constructing
>> each tree in a RandomForest* object?
>>
>> I am looking for the equivalent of "keep.inbag" in this R implementation:
>> http://math.furman.edu/~dcs/courses/math47/R/library/randomForest/html/randomForest.html
>>
>> Thanks!
>>
>> Ariel
>>
>>
>> --
>>
>>
>>
>> ___
>> Scikit-learn-general mailing 
>> listScikit-learn-general@lists.sourceforge.nethttps://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>>
>>
>>
>>
>> --
>> Transform Data into Opportunity.
>> Accelerate data analysis in your applications with
>> Intel Data Analytics Acceleration Library.
>> Click to learn more.
>> http://makebettercode.com/inteldaal-eval
>> ___
>> Scikit-learn-general mailing list
>> Scikit-learn-general@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>>
>>
>
>
> --
> Transform Data into Opportunity.
> Accelerate data analysis in your applications with
> Intel Data Analytics Acceleration Library.
> Click to learn more.
> http://makebettercode.com/inteldaal-eval
> ___
> Scikit-learn-general mailing list
> Scikit-learn-general@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>
>
--
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785111=/4140___
Scikit-learn-general mailing list
Scikit-learn-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general

Re: [Scikit-learn-general] Average Per-Class Accuracy metric

[Sebastian Raschka]

> Firstly, balanced accuracy is a different thing, and yes, it should be 
> supported.

> Secondly, I am correct in thinking you're talking about multiclass (not 
> multilabel).


Sorry for the confusion, and yes, you are right. I think have mixed the terms 
“average per-class accuracy” with “balanced accuracy” then.

Maybe to clarify, a corrected example to describe what I meant. Given the 
confusion matrix

   predicted
   label

   [ 3,  0,  0]
 true[ 7, 50, 12]
 label   [ 0,  0, 18]


I’d compute the accuracy as TP / TN =  (3 + 50 + 18) / 90 = 0.79

and the “average per-class accuracy” as 

(83/90 + 71/90 + 78/90) / 3 = (83 + 71 + 78) / (3 * 90) = 0.86 

(I hope I got it right this time!)

In any case, I am not finding any literature describing this, and I am also not 
proposing to add it to sickit-learn, just wanted to get some info whether this 
is implemented or not. Thanks! :)
  
 

> On Mar 8, 2016, at 8:29 PM, Joel Nothman  wrote:
> 
> Firstly, balanced accuracy is a different thing, and yes, it should be 
> supported.
> 
> Secondly, I am correct in thinking you're talking about multiclass (not 
> multilabel).
> 
> However, what you're describing isn't accuracy. It's actually micro-averaged 
> recall, except that your dataset is impossible because you're allowing there 
> to be fewer predictions than instances. If we assume that we're allowed to 
> predict some negative class, that's fine; we can nowadays exclude it from 
> micro-averaged recall with the labels parameter to recall_score. (If all 
> labels are included in a multiclass problem, micro-averaged recall = 
> precision = fscore = accuracy.)
> 
> I had assumed you meant binarised accuracy, which would add together both 
> true positives and true negatives for each class.
> 
> Either way, if there's no literature on this, I think we'd really best not 
> support it.
> 
> On 9 March 2016 at 11:15, Sebastian Raschka  wrote:
> I haven’t seen this in practice, yet, either. A colleague was looking for 
> this in scikit-learn recently, and he asked me if I know whether this is 
> implemented or not. I couldn’t find anything in the docs and was just curious 
> about your opinion. However, I just found this entry here on wikipedia:
> 
> https://en.wikipedia.org/wiki/Accuracy_and_precision
> > Another useful performance measure is the balanced accuracy[10] which 
> > avoids inflated performance estimates on imbalanced datasets. It is defined 
> > as the arithmetic mean of sensitivity and specificity, or the average 
> > accuracy obtained on either class:
> 
> > Am I right in thinking that in the binary case, this is identical to 
> > accuracy?
> 
> 
> I think it would only be equal to the “accuracy” if the class labels are 
> uniformly distributed.
> 
> >  I'm not sure what this metric is getting at.
> 
> I have to think about this more, but I think it may be useful for imbalanced 
> datasets where you want to emphasize the minority class. E.g., let’s say we 
> have a dataset of 120 samples and three class labels 1, 2, 3. And the classes 
> are distributed like this
> 10 x 1
> 50 x 2
> 60 x 3
> 
> Now, let’s assume we have a model that makes the following predictions
> 
> - it gets 0 out of 10 from class 1 right
> - 45 out of 50 from class 2
> - 55 out of 60 from class 3
> 
> So, the accuracy would then be computed as
> 
> (0 + 45 + 55) / 120 = 0.833
> 
> But the “balanced accuracy” would be much lower, because the model did really 
> badly on class 1, i.e.,
> 
> (0/10 + 45/50 + 55/60) / 3 = 0.61
> 
> Hm, if I see this correctly, this is actually very similar to the F1 score. 
> But instead of computing the harmonic mean between “precision and the true 
> positive rate), we compute the harmonic mean between "precision and true 
> negative rate"
> 
> > On Mar 8, 2016, at 6:40 PM, Joel Nothman  wrote:
> >
> > I've not seen this metric used (references?). Am I right in thinking that 
> > in the binary case, this is identical to accuracy? If I predict all 
> > elements to be the majority class, then adding more minority classes into 
> > the problem increases my score. I'm not sure what this metric is getting at.
> >
> > On 8 March 2016 at 11:57, Sebastian Raschka  wrote:
> > Hi,
> >
> > I was just wondering why there’s no support for the average per-class 
> > accuracy in the scorer functions (if I am not overlooking something).
> > E.g., we have 'f1_macro', 'f1_micro', 'f1_samples', ‘f1_weighted’ but I 
> > didn’t see a ‘accuracy_macro’, i.e.,
> > (acc.class_1 + acc.class_2 + … + acc.class_n) / n
> >
> > Would you discourage its usage (in favor of other metrics in imbalanced 
> > class problems) or was it simply not implemented, yet?
> >
> > Best,
> > Sebastian
> > --
> > Transform Data into Opportunity.
> > Accelerate data 

Re: [Scikit-learn-general] Average Per-Class Accuracy metric

[Joel Nothman]

Firstly, balanced accuracy is a different thing, and yes, it should be
supported.

Secondly, I am correct in thinking you're talking about multiclass (not
multilabel).

However, what you're describing isn't accuracy. It's actually
micro-averaged recall, except that your dataset is impossible because
you're allowing there to be fewer predictions than instances. If we assume
that we're allowed to predict some negative class, that's fine; we can
nowadays exclude it from micro-averaged recall with the labels parameter to
recall_score. (If all labels are included in a multiclass problem,
micro-averaged recall = precision = fscore = accuracy.)

I had assumed you meant binarised accuracy, which would add together both
true positives and true negatives for each class.

Either way, if there's no literature on this, I think we'd really best not
support it.

On 9 March 2016 at 11:15, Sebastian Raschka  wrote:

> I haven’t seen this in practice, yet, either. A colleague was looking for
> this in scikit-learn recently, and he asked me if I know whether this is
> implemented or not. I couldn’t find anything in the docs and was just
> curious about your opinion. However, I just found this entry here on
> wikipedia:
>
> https://en.wikipedia.org/wiki/Accuracy_and_precision
> > Another useful performance measure is the balanced accuracy[10] which
> avoids inflated performance estimates on imbalanced datasets. It is defined
> as the arithmetic mean of sensitivity and specificity, or the average
> accuracy obtained on either class:
>
> > Am I right in thinking that in the binary case, this is identical to
> accuracy?
>
>
> I think it would only be equal to the “accuracy” if the class labels are
> uniformly distributed.
>
> >  I'm not sure what this metric is getting at.
>
> I have to think about this more, but I think it may be useful for
> imbalanced datasets where you want to emphasize the minority class. E.g.,
> let’s say we have a dataset of 120 samples and three class labels 1, 2, 3.
> And the classes are distributed like this
> 10 x 1
> 50 x 2
> 60 x 3
>
> Now, let’s assume we have a model that makes the following predictions
>
> - it gets 0 out of 10 from class 1 right
> - 45 out of 50 from class 2
> - 55 out of 60 from class 3
>
> So, the accuracy would then be computed as
>
> (0 + 45 + 55) / 120 = 0.833
>
> But the “balanced accuracy” would be much lower, because the model did
> really badly on class 1, i.e.,
>
> (0/10 + 45/50 + 55/60) / 3 = 0.61
>
> Hm, if I see this correctly, this is actually very similar to the F1
> score. But instead of computing the harmonic mean between “precision and
> the true positive rate), we compute the harmonic mean between "precision
> and true negative rate"
>
> > On Mar 8, 2016, at 6:40 PM, Joel Nothman  wrote:
> >
> > I've not seen this metric used (references?). Am I right in thinking
> that in the binary case, this is identical to accuracy? If I predict all
> elements to be the majority class, then adding more minority classes into
> the problem increases my score. I'm not sure what this metric is getting at.
> >
> > On 8 March 2016 at 11:57, Sebastian Raschka 
> wrote:
> > Hi,
> >
> > I was just wondering why there’s no support for the average per-class
> accuracy in the scorer functions (if I am not overlooking something).
> > E.g., we have 'f1_macro', 'f1_micro', 'f1_samples', ‘f1_weighted’ but I
> didn’t see a ‘accuracy_macro’, i.e.,
> > (acc.class_1 + acc.class_2 + … + acc.class_n) / n
> >
> > Would you discourage its usage (in favor of other metrics in imbalanced
> class problems) or was it simply not implemented, yet?
> >
> > Best,
> > Sebastian
> >
> --
> > Transform Data into Opportunity.
> > Accelerate data analysis in your applications with
> > Intel Data Analytics Acceleration Library.
> > Click to learn more.
> > http://makebettercode.com/inteldaal-eval
> > ___
> > Scikit-learn-general mailing list
> > Scikit-learn-general@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
> >
> >
> --
> > Transform Data into Opportunity.
> > Accelerate data analysis in your applications with
> > Intel Data Analytics Acceleration Library.
> > Click to learn more.
> >
> http://makebettercode.com/inteldaal-eval___
> > Scikit-learn-general mailing list
> > Scikit-learn-general@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>
>
>
> --
> Transform Data into Opportunity.
> Accelerate data analysis in your applications with
> Intel Data Analytics Acceleration Library.
> Click to learn more.
> 

Re: [Scikit-learn-general] Average Per-Class Accuracy metric

[Joel Nothman]

(Although multiloutput accuracy is reasonable to support.)

On 9 March 2016 at 12:29, Joel Nothman  wrote:

> Firstly, balanced accuracy is a different thing, and yes, it should be
> supported.
>
> Secondly, I am correct in thinking you're talking about multiclass (not
> multilabel).
>
> However, what you're describing isn't accuracy. It's actually
> micro-averaged recall, except that your dataset is impossible because
> you're allowing there to be fewer predictions than instances. If we assume
> that we're allowed to predict some negative class, that's fine; we can
> nowadays exclude it from micro-averaged recall with the labels parameter to
> recall_score. (If all labels are included in a multiclass problem,
> micro-averaged recall = precision = fscore = accuracy.)
>
> I had assumed you meant binarised accuracy, which would add together both
> true positives and true negatives for each class.
>
> Either way, if there's no literature on this, I think we'd really best not
> support it.
>
> On 9 March 2016 at 11:15, Sebastian Raschka  wrote:
>
>> I haven’t seen this in practice, yet, either. A colleague was looking for
>> this in scikit-learn recently, and he asked me if I know whether this is
>> implemented or not. I couldn’t find anything in the docs and was just
>> curious about your opinion. However, I just found this entry here on
>> wikipedia:
>>
>> https://en.wikipedia.org/wiki/Accuracy_and_precision
>> > Another useful performance measure is the balanced accuracy[10] which
>> avoids inflated performance estimates on imbalanced datasets. It is defined
>> as the arithmetic mean of sensitivity and specificity, or the average
>> accuracy obtained on either class:
>>
>> > Am I right in thinking that in the binary case, this is identical to
>> accuracy?
>>
>>
>> I think it would only be equal to the “accuracy” if the class labels are
>> uniformly distributed.
>>
>> >  I'm not sure what this metric is getting at.
>>
>> I have to think about this more, but I think it may be useful for
>> imbalanced datasets where you want to emphasize the minority class. E.g.,
>> let’s say we have a dataset of 120 samples and three class labels 1, 2, 3.
>> And the classes are distributed like this
>> 10 x 1
>> 50 x 2
>> 60 x 3
>>
>> Now, let’s assume we have a model that makes the following predictions
>>
>> - it gets 0 out of 10 from class 1 right
>> - 45 out of 50 from class 2
>> - 55 out of 60 from class 3
>>
>> So, the accuracy would then be computed as
>>
>> (0 + 45 + 55) / 120 = 0.833
>>
>> But the “balanced accuracy” would be much lower, because the model did
>> really badly on class 1, i.e.,
>>
>> (0/10 + 45/50 + 55/60) / 3 = 0.61
>>
>> Hm, if I see this correctly, this is actually very similar to the F1
>> score. But instead of computing the harmonic mean between “precision and
>> the true positive rate), we compute the harmonic mean between "precision
>> and true negative rate"
>>
>> > On Mar 8, 2016, at 6:40 PM, Joel Nothman 
>> wrote:
>> >
>> > I've not seen this metric used (references?). Am I right in thinking
>> that in the binary case, this is identical to accuracy? If I predict all
>> elements to be the majority class, then adding more minority classes into
>> the problem increases my score. I'm not sure what this metric is getting at.
>> >
>> > On 8 March 2016 at 11:57, Sebastian Raschka 
>> wrote:
>> > Hi,
>> >
>> > I was just wondering why there’s no support for the average per-class
>> accuracy in the scorer functions (if I am not overlooking something).
>> > E.g., we have 'f1_macro', 'f1_micro', 'f1_samples', ‘f1_weighted’ but I
>> didn’t see a ‘accuracy_macro’, i.e.,
>> > (acc.class_1 + acc.class_2 + … + acc.class_n) / n
>> >
>> > Would you discourage its usage (in favor of other metrics in imbalanced
>> class problems) or was it simply not implemented, yet?
>> >
>> > Best,
>> > Sebastian
>> >
>> --
>> > Transform Data into Opportunity.
>> > Accelerate data analysis in your applications with
>> > Intel Data Analytics Acceleration Library.
>> > Click to learn more.
>> > http://makebettercode.com/inteldaal-eval
>> > ___
>> > Scikit-learn-general mailing list
>> > Scikit-learn-general@lists.sourceforge.net
>> > https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>> >
>> >
>> --
>> > Transform Data into Opportunity.
>> > Accelerate data analysis in your applications with
>> > Intel Data Analytics Acceleration Library.
>> > Click to learn more.
>> >
>> http://makebettercode.com/inteldaal-eval___
>> > Scikit-learn-general mailing list
>> > Scikit-learn-general@lists.sourceforge.net
>> > https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>>
>>
>>
>> 

Re: [Scikit-learn-general] Average Per-Class Accuracy metric

[Sebastian Raschka]

I haven’t seen this in practice, yet, either. A colleague was looking for this 
in scikit-learn recently, and he asked me if I know whether this is implemented 
or not. I couldn’t find anything in the docs and was just curious about your 
opinion. However, I just found this entry here on wikipedia:

https://en.wikipedia.org/wiki/Accuracy_and_precision
> Another useful performance measure is the balanced accuracy[10] which avoids 
> inflated performance estimates on imbalanced datasets. It is defined as the 
> arithmetic mean of sensitivity and specificity, or the average accuracy 
> obtained on either class:

> Am I right in thinking that in the binary case, this is identical to 
> accuracy? 


I think it would only be equal to the “accuracy” if the class labels are 
uniformly distributed.

>  I'm not sure what this metric is getting at.

I have to think about this more, but I think it may be useful for imbalanced 
datasets where you want to emphasize the minority class. E.g., let’s say we 
have a dataset of 120 samples and three class labels 1, 2, 3. And the classes 
are distributed like this
10 x 1
50 x 2
60 x 3

Now, let’s assume we have a model that makes the following predictions

- it gets 0 out of 10 from class 1 right
- 45 out of 50 from class 2
- 55 out of 60 from class 3

So, the accuracy would then be computed as

(0 + 45 + 55) / 120 = 0.833

But the “balanced accuracy” would be much lower, because the model did really 
badly on class 1, i.e., 

(0/10 + 45/50 + 55/60) / 3 = 0.61

Hm, if I see this correctly, this is actually very similar to the F1 score. But 
instead of computing the harmonic mean between “precision and the true positive 
rate), we compute the harmonic mean between "precision and true negative rate"

> On Mar 8, 2016, at 6:40 PM, Joel Nothman  wrote:
> 
> I've not seen this metric used (references?). Am I right in thinking that in 
> the binary case, this is identical to accuracy? If I predict all elements to 
> be the majority class, then adding more minority classes into the problem 
> increases my score. I'm not sure what this metric is getting at.
> 
> On 8 March 2016 at 11:57, Sebastian Raschka  wrote:
> Hi,
> 
> I was just wondering why there’s no support for the average per-class 
> accuracy in the scorer functions (if I am not overlooking something).
> E.g., we have 'f1_macro', 'f1_micro', 'f1_samples', ‘f1_weighted’ but I 
> didn’t see a ‘accuracy_macro’, i.e.,
> (acc.class_1 + acc.class_2 + … + acc.class_n) / n
> 
> Would you discourage its usage (in favor of other metrics in imbalanced class 
> problems) or was it simply not implemented, yet?
> 
> Best,
> Sebastian
> --
> Transform Data into Opportunity.
> Accelerate data analysis in your applications with
> Intel Data Analytics Acceleration Library.
> Click to learn more.
> http://makebettercode.com/inteldaal-eval
> ___
> Scikit-learn-general mailing list
> Scikit-learn-general@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
> 
> --
> Transform Data into Opportunity.
> Accelerate data analysis in your applications with
> Intel Data Analytics Acceleration Library.
> Click to learn more.
> http://makebettercode.com/inteldaal-eval___
> Scikit-learn-general mailing list
> Scikit-learn-general@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general


--
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785111=/4140
___
Scikit-learn-general mailing list
Scikit-learn-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general

Re: [Scikit-learn-general] Average Per-Class Accuracy metric

[Joel Nothman]

I've not seen this metric used (references?). Am I right in thinking that
in the binary case, this is identical to accuracy? If I predict all
elements to be the majority class, then adding more minority classes into
the problem increases my score. I'm not sure what this metric is getting at.

On 8 March 2016 at 11:57, Sebastian Raschka  wrote:

> Hi,
>
> I was just wondering why there’s no support for the average per-class
> accuracy in the scorer functions (if I am not overlooking something).
> E.g., we have 'f1_macro', 'f1_micro', 'f1_samples', ‘f1_weighted’ but I
> didn’t see a ‘accuracy_macro’, i.e.,
> (acc.class_1 + acc.class_2 + … + acc.class_n) / n
>
> Would you discourage its usage (in favor of other metrics in imbalanced
> class problems) or was it simply not implemented, yet?
>
> Best,
> Sebastian
>
> --
> Transform Data into Opportunity.
> Accelerate data analysis in your applications with
> Intel Data Analytics Acceleration Library.
> Click to learn more.
> http://makebettercode.com/inteldaal-eval
> ___
> Scikit-learn-general mailing list
> Scikit-learn-general@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>
--
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://makebettercode.com/inteldaal-eval___
Scikit-learn-general mailing list
Scikit-learn-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general

Re: [Scikit-learn-general] Implementation of Bag-of-Features

[Guillaume Lemaître]

Regarding the MiniBatchKMeans, I use the following parameters

MiniBatchKMeans(n_clusters=nb_words, verbose=1, init='random', batch_size=10
* nb_words, compute_labels=False, reassignment_ratio=0.0, random_state=1,
n_init=3)

With 1000 words. I am not sure about the batch size as well as the
initialisation. Does 'k-means++' should improve the convergence with the
mini-batch?

On 8 March 2016 at 23:23, Guillaume Lemaître  wrote:

> Sorry I was wrong. The MiniBatchKMeans converge after 20 minutes.
> So for one iteration of the CV, I get something like that:
>
> Classification performed
> [[21  2  0]
>  [ 0 20  0]
>  [ 0  0 23]]
> It took 1253.23589396 seconds.
>
> Probably this is not desirable to have a cross-validation. I don't know if
> you consider 20 minutes as reasonable?
>
> On 8 March 2016 at 22:09, Andreas Mueller  wrote:
>
>> Hey Guillaume.
>> If it is a couple of hours, I'm not sure it is worth adding.
>> You can probably aggressively subsample or just do fewer iterations
>> (like, one pass over the data)
>> How do you run MiniBatchKMeans?
>>
>> Cheers,
>> Andy
>>
>>
>> On 03/08/2016 03:21 PM, Guillaume Lemaître wrote:
>>
>> Hi,
>>
>> I made a pull-request with the draft:
>> 
>> https://github.com/scikit-learn/scikit-learn/pull/6509
>> Extracting the feature is taking a honest amount of time (around 30 sec.)
>> The codebook generation through MiniBatchKMeans is more problematic. I am
>> still running it but it could be a couple of hours.
>>
>> Let me know what do you think about it,
>>
>> Cheers,
>>
>> On 24 February 2016 at 00:41, Andy  wrote:
>>
>>> On 02/23/2016 04:32 PM, Guillaume Lemaitre wrote:
>>>
>>> Since that I was working on a cluster I did not realize but loading all
>>> the image in memory will be problematic with a laptop-desktop configuration.
>>>
>>> Or we can learn the PCA projection on a subset and to apply the
>>> dimension reduction right after the patch extraction. However, I am not
>>> sure that all data will fit in memory.
>>>
>>> We have out of core versions for PCA and KMeans.
>>>
>>> I think the way I'd do it is to go over all images, extract only a
>>> couple of patches from each image, store them.
>>> After we have some patches from all images, I'd learn the PCA model.
>>> Then we can go over the data again, transforming the patches. If they
>>> don't fit into memory after dimensionality reduction, we can
>>> use minibatch k-means to do the clustering without loading all the data.
>>> then we need to go over the data one more time to get the cluster
>>> centers and compute the BoW (which will fit in memory)
>>>
>>>
>>> --
>>> Site24x7 APM Insight: Get Deep Visibility into Application Performance
>>> APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
>>> Monitor end-to-end web transactions and take corrective actions now
>>> Troubleshoot faster and improve end-user experience. Signup Now!
>>> http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140
>>> ___
>>> Scikit-learn-general mailing list
>>> Scikit-learn-general@lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>>>
>>>
>>
>>
>> --
>>
>>
>>
>>
>> *LEMAÎTRE Guillaume PhD Candidate MSc Erasmus Mundus ViBOT
>> (Vision-roBOTic) MSc Business Innovation and Technology Management *
>> g.lemaitr...@gmail.com
>>
>> *ViCOROB - Computer Vision and Robotic Team*
>> Universitat de Girona, Campus Montilivi, Edifici P-IV 17071 Girona
>> Tel. +34 972 41 98 12 - Fax. +34 972 41 82 59
>> http://vicorob.udg.es/
>>
>> *LE2I - Le Creusot *IUT Le Creusot, Laboratoire LE2I, 12 rue de la
>> Fonderie, 71200 Le Creusot
>> Tel. +33 3 85 73 10 90 - Fax. +33 3 85 73 10 97
>> http://le2i.cnrs.fr
>>
>> https://sites.google.com/site/glemaitre58/
>> Vice - Chairman of A.S.C. Fours UFOLEP
>> Chairman of A.S.C. Fours FFC
>> Webmaster of http://ascfours.free.fr
>>
>>
>> --
>> Transform Data into Opportunity.
>> Accelerate data analysis in your applications with
>> Intel Data Analytics Acceleration Library.
>> Click to learn more.http://makebettercode.com/inteldaal-eval
>>
>>
>>
>> ___
>> Scikit-learn-general mailing 
>> listScikit-learn-general@lists.sourceforge.nethttps://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>>
>>
>>
>>
>> --
>> Transform Data into Opportunity.
>> Accelerate data analysis in your applications with
>> Intel Data Analytics Acceleration Library.
>> Click to learn more.
>> http://makebettercode.com/inteldaal-eval
>> ___
>> Scikit-learn-general mailing list
>> 

Re: [Scikit-learn-general] Implementation of Bag-of-Features

[Guillaume Lemaître]

Sorry I was wrong. The MiniBatchKMeans converge after 20 minutes.
So for one iteration of the CV, I get something like that:

Classification performed
[[21  2  0]
 [ 0 20  0]
 [ 0  0 23]]
It took 1253.23589396 seconds.

Probably this is not desirable to have a cross-validation. I don't know if
you consider 20 minutes as reasonable?

On 8 March 2016 at 22:09, Andreas Mueller  wrote:

> Hey Guillaume.
> If it is a couple of hours, I'm not sure it is worth adding.
> You can probably aggressively subsample or just do fewer iterations (like,
> one pass over the data)
> How do you run MiniBatchKMeans?
>
> Cheers,
> Andy
>
>
> On 03/08/2016 03:21 PM, Guillaume Lemaître wrote:
>
> Hi,
>
> I made a pull-request with the draft:
> 
> https://github.com/scikit-learn/scikit-learn/pull/6509
> Extracting the feature is taking a honest amount of time (around 30 sec.)
> The codebook generation through MiniBatchKMeans is more problematic. I am
> still running it but it could be a couple of hours.
>
> Let me know what do you think about it,
>
> Cheers,
>
> On 24 February 2016 at 00:41, Andy  wrote:
>
>> On 02/23/2016 04:32 PM, Guillaume Lemaitre wrote:
>>
>> Since that I was working on a cluster I did not realize but loading all
>> the image in memory will be problematic with a laptop-desktop configuration.
>>
>> Or we can learn the PCA projection on a subset and to apply the dimension
>> reduction right after the patch extraction. However, I am not sure that all
>> data will fit in memory.
>>
>> We have out of core versions for PCA and KMeans.
>>
>> I think the way I'd do it is to go over all images, extract only a couple
>> of patches from each image, store them.
>> After we have some patches from all images, I'd learn the PCA model.
>> Then we can go over the data again, transforming the patches. If they
>> don't fit into memory after dimensionality reduction, we can
>> use minibatch k-means to do the clustering without loading all the data.
>> then we need to go over the data one more time to get the cluster centers
>> and compute the BoW (which will fit in memory)
>>
>>
>> --
>> Site24x7 APM Insight: Get Deep Visibility into Application Performance
>> APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
>> Monitor end-to-end web transactions and take corrective actions now
>> Troubleshoot faster and improve end-user experience. Signup Now!
>> http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140
>> ___
>> Scikit-learn-general mailing list
>> Scikit-learn-general@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>>
>>
>
>
> --
>
>
>
>
> *LEMAÎTRE Guillaume PhD Candidate MSc Erasmus Mundus ViBOT
> (Vision-roBOTic) MSc Business Innovation and Technology Management *
> g.lemaitr...@gmail.com
>
> *ViCOROB - Computer Vision and Robotic Team*
> Universitat de Girona, Campus Montilivi, Edifici P-IV 17071 Girona
> Tel. +34 972 41 98 12 - Fax. +34 972 41 82 59
> http://vicorob.udg.es/
>
> *LE2I - Le Creusot *IUT Le Creusot, Laboratoire LE2I, 12 rue de la
> Fonderie, 71200 Le Creusot
> Tel. +33 3 85 73 10 90 - Fax. +33 3 85 73 10 97
> http://le2i.cnrs.fr
>
> https://sites.google.com/site/glemaitre58/
> Vice - Chairman of A.S.C. Fours UFOLEP
> Chairman of A.S.C. Fours FFC
> Webmaster of http://ascfours.free.fr
>
>
> --
> Transform Data into Opportunity.
> Accelerate data analysis in your applications with
> Intel Data Analytics Acceleration Library.
> Click to learn more.http://makebettercode.com/inteldaal-eval
>
>
>
> ___
> Scikit-learn-general mailing 
> listScikit-learn-general@lists.sourceforge.nethttps://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>
>
>
>
> --
> Transform Data into Opportunity.
> Accelerate data analysis in your applications with
> Intel Data Analytics Acceleration Library.
> Click to learn more.
> http://makebettercode.com/inteldaal-eval
> ___
> Scikit-learn-general mailing list
> Scikit-learn-general@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>
>


-- 




*LEMAÎTRE GuillaumePhD CandidateMSc Erasmus Mundus ViBOT
(Vision-roBOTic)MSc Business Innovation and Technology Management*
g.lemaitr...@gmail.com

*ViCOROB - Computer Vision and Robotic Team*
Universitat de Girona, Campus Montilivi, Edifici P-IV 17071 Girona
Tel. +34 972 41 98 12 - Fax. +34 972 41 82 59
http://vicorob.udg.es/

*LE2I - Le Creusot*IUT Le Creusot, Laboratoire LE2I, 12 rue de la Fonderie,
71200 Le Creusot
Tel. +33 3 85 73 10 90 - Fax. +33 3 85 73 10 97

Re: [Scikit-learn-general] Implementation of Bag-of-Features

[Andreas Mueller]

Hey Guillaume.
If it is a couple of hours, I'm not sure it is worth adding.
You can probably aggressively subsample or just do fewer iterations 
(like, one pass over the data)

How do you run MiniBatchKMeans?

Cheers,
Andy

On 03/08/2016 03:21 PM, Guillaume Lemaître wrote:

Hi,

I made a pull-request with the draft: 
https://github.com/scikit-learn/scikit-learn/pull/6509

Extracting the feature is taking a honest amount of time (around 30 sec.)
The codebook generation through MiniBatchKMeans is more problematic. I 
am still running it but it could be a couple of hours.


Let me know what do you think about it,

Cheers,

On 24 February 2016 at 00:41, Andy > wrote:


On 02/23/2016 04:32 PM, Guillaume Lemaitre wrote:

Since that I was working on a cluster I did not realize but
loading all the image in memory will be problematic with a
laptop-desktop configuration.

Or we can learn the PCA projection on a subset and to apply the
dimension reduction right after the patch extraction. However, I
am not sure that all data will fit in memory.


We have out of core versions for PCA and KMeans.

I think the way I'd do it is to go over all images, extract only a
couple of patches from each image, store them.
After we have some patches from all images, I'd learn the PCA model.
Then we can go over the data again, transforming the patches. If
they don't fit into memory after dimensionality reduction, we can
use minibatch k-means to do the clustering without loading all the
data.
then we need to go over the data one more time to get the cluster
centers and compute the BoW (which will fit in memory)


--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140
___
Scikit-learn-general mailing list
Scikit-learn-general@lists.sourceforge.net

https://lists.sourceforge.net/lists/listinfo/scikit-learn-general




--
*LEMAÎTRE Guillaume
PhD Candidate
MSc Erasmus Mundus ViBOT (Vision-roBOTic)
MSc Business Innovation and Technology Management
**
*g.lemaitr...@gmail.com 

*ViCOROB - Computer Vision and Robotic Team*
Universitat de Girona, Campus Montilivi, Edifici P-IV 17071 Girona
Tel. +34 972 41 98 12 - Fax. +34 972 41 82 59
http://vicorob.udg.es/
*LE2I - Le Creusot
*IUT Le Creusot, Laboratoire LE2I, 12 rue de la Fonderie, 71200 Le Creusot
Tel. +33 3 85 73 10 90 - Fax. +33 3 85 73 10 97
http://le2i.cnrs.fr

https://sites.google.com/site/glemaitre58/
Vice - Chairman of A.S.C. Fours UFOLEP
Chairman of A.S.C. Fours FFC
Webmaster of http://ascfours.free.fr


--
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://makebettercode.com/inteldaal-eval


___
Scikit-learn-general mailing list
Scikit-learn-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general


--
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://makebettercode.com/inteldaal-eval___
Scikit-learn-general mailing list
Scikit-learn-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general

Re: [Scikit-learn-general] Implementation of Bag-of-Features

[Guillaume Lemaître]

Hi,

I made a pull-request with the draft:
https://github.com/scikit-learn/scikit-learn/pull/6509
Extracting the feature is taking a honest amount of time (around 30 sec.)
The codebook generation through MiniBatchKMeans is more problematic. I am
still running it but it could be a couple of hours.

Let me know what do you think about it,

Cheers,

On 24 February 2016 at 00:41, Andy  wrote:

> On 02/23/2016 04:32 PM, Guillaume Lemaitre wrote:
>
> Since that I was working on a cluster I did not realize but loading all
> the image in memory will be problematic with a laptop-desktop configuration.
>
> Or we can learn the PCA projection on a subset and to apply the dimension
> reduction right after the patch extraction. However, I am not sure that all
> data will fit in memory.
>
> We have out of core versions for PCA and KMeans.
>
> I think the way I'd do it is to go over all images, extract only a couple
> of patches from each image, store them.
> After we have some patches from all images, I'd learn the PCA model.
> Then we can go over the data again, transforming the patches. If they
> don't fit into memory after dimensionality reduction, we can
> use minibatch k-means to do the clustering without loading all the data.
> then we need to go over the data one more time to get the cluster centers
> and compute the BoW (which will fit in memory)
>
>
> --
> Site24x7 APM Insight: Get Deep Visibility into Application Performance
> APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
> Monitor end-to-end web transactions and take corrective actions now
> Troubleshoot faster and improve end-user experience. Signup Now!
> http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140
> ___
> Scikit-learn-general mailing list
> Scikit-learn-general@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>
>


-- 




*LEMAÎTRE GuillaumePhD CandidateMSc Erasmus Mundus ViBOT
(Vision-roBOTic)MSc Business Innovation and Technology Management*
g.lemaitr...@gmail.com

*ViCOROB - Computer Vision and Robotic Team*
Universitat de Girona, Campus Montilivi, Edifici P-IV 17071 Girona
Tel. +34 972 41 98 12 - Fax. +34 972 41 82 59
http://vicorob.udg.es/

*LE2I - Le Creusot*IUT Le Creusot, Laboratoire LE2I, 12 rue de la Fonderie,
71200 Le Creusot
Tel. +33 3 85 73 10 90 - Fax. +33 3 85 73 10 97
http://le2i.cnrs.fr

https://sites.google.com/site/glemaitre58/
Vice - Chairman of A.S.C. Fours UFOLEP
Chairman of A.S.C. Fours FFC
Webmaster of http://ascfours.free.fr
--
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://makebettercode.com/inteldaal-eval___
Scikit-learn-general mailing list
Scikit-learn-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general

Re: [Scikit-learn-general] scikit-learn in Julia

[Andreas Mueller]

On 03/07/2016 04:47 PM, Cedric St-Jean wrote:
> >> There is already Pandas.jl, Stan.jl, MATLAB.jl and Bokeh.jl following
> >> that trend.
> >That is interesting. Were they done by people associated with the
> >original projects?
>
> As far as I can tell, no, they weren't. Stan.jl and Bokeh.jl are now 
> both recognized (but not explicitly supported) by their parent projects.
>
>  >MATLAB.jl ? And mathworks was fine with that?
>
> I don't know. MATLAB.jl's README has some very clear language that 
> it's not an official MathWorks product. I have some too, but I can 
> step it up.
Pretty sure mathworks just hasn't seen it yet / thinks it's to 
insignificant to sue. IANAL but that seems like a pretty clear trademark 
violation.

--
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://makebettercode.com/inteldaal-eval
___
Scikit-learn-general mailing list
Scikit-learn-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general

Re: [Scikit-learn-general] scikit-learn in Julia

[Cedric St-Jean]

>> There is already Pandas.jl, Stan.jl, MATLAB.jl and Bokeh.jl following
>> that trend.
>That is interesting. Were they done by people associated with the
>original projects?

As far as I can tell, no, they weren't. Stan.jl and Bokeh.jl are now both
recognized (but not explicitly supported) by their parent projects.

 >MATLAB.jl ? And mathworks was fine with that?

I don't know. MATLAB.jl's README has some very clear language that it's not
an official MathWorks product. I have some too, but I can step it up.
--
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://makebettercode.com/inteldaal-eval___
Scikit-learn-general mailing list
Scikit-learn-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general

Re: [Scikit-learn-general] [Matplotlib-users] Scipy2016: call for proposals

[Kyle Kastner]

I am on the fence still - internship this summer so I need to check on
timing/vacation expectation

On Mon, Mar 7, 2016 at 3:09 PM, Jacob Vanderplas 
wrote:

> I'm not going to be able to make it this year, unfortunately.
>   Jake
>
>  Jake VanderPlas
>  Senior Data Science Fellow
>  Director of Research in Physical Sciences
>  University of Washington eScience Institute
>
> On Mon, Mar 7, 2016 at 9:31 AM, Andreas Mueller  wrote:
>
>> Are any more core devs planning to attend?
>> Jake? Kyle? Olivier? Gael? Vlad?
>>
>>
>> On 02/22/2016 05:48 PM, Andreas Mueller wrote:
>>
>> Hi Nelson.
>>
>> There will be a scikit-learn sprint :)
>> Not sure how many other core-devs will be there, though.
>>
>> Cheers,
>> Andy
>>
>> On 02/22/2016 05:35 PM, Nelson Liu wrote:
>>
>> Hi all,
>> I might be attending, is there going to be a scikit-learn sprint? I'd
>> also be interested in helping put together a tutorial :)
>>
>> Nelson Liu
>>
>> On Mon, Feb 22, 2016, 9:20 AM Sebastian Raschka 
>> wrote:
>>
>>> After missing all the fun last year, I am also planning on attending —
>>> I’d also be happy to help if there’s a shortage in core devs for the
>>> tutorials ;)
>>>
>>> Cheers,
>>> Sebastian
>>>
>>> > On Feb 22, 2016, at 12:11 PM, Manoj Kumar <
>>> manojkumarsivaraj...@gmail.com> wrote:
>>> >
>>> > Hi everyone.
>>> >
>>> > I'll definitely be happy to help on the tutorial!
>>> >
>>> > On Mon, Feb 22, 2016 at 11:41 AM, Andreas Mueller 
>>> wrote:
>>> > Who's going?
>>> > I'll definitely be there and am happy to do a tutorial.
>>> > Who's in?
>>> >
>>> >
>>> >
>>> > On 02/22/2016 04:15 AM, Nelle Varoquaux wrote:
>>> >>
>>> >> Dear all,
>>> >>
>>> >> SciPy 2016, the Fifteenth Annual Conference on Python in Science,
>>> takes place in Austin, TX on July, 11th to 17th. The conference features
>>> two days of tutorials by followed by three days of presentations, and
>>> concludes with two days of developer sprints on projects of interest to
>>> attendees. .
>>> >>
>>> >> The topics presented at SciPy are very diverse, with a focus on
>>> advanced software engineering and original uses of Python and its
>>> scientific libraries, either in theoretical or experimental research, from
>>> both academia and the industry. This year we are happy to announce two
>>> specialized tracks that run in parallel to the general conference (Data
>>> Science , High Performance Computing) and 8 mini-symposia (Earth and Space
>>> Science, Biology and Medicine, Engineering, Social Sciences, Special
>>> Purpose Databases, Case Studies in Industry, Education, Reproducibility)
>>> >>
>>> >> Submissions for talks and posters are welcome on our website (
>>> http://scipy2016.scipy.org). In your abstract, please provide details
>>> on what Python tools are being employed, and how. The talk and poster
>>> submission deadline is March 25th, 2016, while the tutorial submission
>>> deadline is March, 21st, 2016.
>>> >>
>>> >>
>>> >> Important dates:
>>> >>
>>> >> Mar 21: Tutorial Proposals Due
>>> >> Mar 25: Talk and Poster Proposals Due
>>> >> May 11: Plotting Contest Submissions Due
>>> >> Apr 22: Tutorials Announced
>>> >> Apr 22: Financial Aid Submissions Due
>>> >> May 4: Talk and Posters Announced
>>> >> May 11: Financial Aid Recipients Notified
>>> >> May 22: Early Bird Registration Deadline
>>> >> Jul 11-12: SciPy 2016 Tutorials
>>> >> Jul 13-15: SciPy 2016 General Conference
>>> >> Jul 16-17: SciPy 2016 Sprints
>>> >>
>>> >> We look forward to an exciting conference and hope to see you in
>>> Austin in July!
>>> >>
>>> >>
>>> >> The Scipy 2016
>>> >> http://scipy2016.scipy.org/
>>> >>
>>> >> Conference Chairs: Aric Hagberg, Prabhu Ramachandran
>>> >> Tutorial Chairs: Justin Vincent, Ben Root
>>> >> Program Chair: Serge Rey, Nelle Varoquaux
>>> >> Proceeding Chairs: Sebastian Benthall
>>> >>
>>> >>
>>> >>
>>> >>
>>> --
>>> >> Site24x7 APM Insight: Get Deep Visibility into Application Performance
>>> >> APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
>>> >> Monitor end-to-end web transactions and take corrective actions now
>>> >> Troubleshoot faster and improve end-user experience. Signup Now!
>>> >>
>>> >> http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140
>>> >>
>>> >>
>>> >> ___
>>> >> Matplotlib-users mailing list
>>> >>
>>> >> matplotlib-us...@lists.sourceforge.net
>>> >> https://lists.sourceforge.net/lists/listinfo/matplotlib-users
>>> >
>>> >
>>> >
>>> --
>>> > Site24x7 APM Insight: Get Deep Visibility into Application Performance
>>> > APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
>>> > Monitor end-to-end web transactions and take corrective actions now
>>> > Troubleshoot faster and improve