Re: [Numpy-discussion] The process I intend to follow for any proposed changes to NumPy

[Ondřej Čertík]

Hi Travis,

On Sun, Sep 13, 2015 at 4:51 PM, Travis Oliphant  wrote:
> Hey all,
>
> I just wanted to clarify, that I am very excited about a few ideas I have
> --- but I don't have time myself to engage in the community process to get
> these changes into NumPy. However, those are real processes --- I've
> been coaching a few people in those processes for the past several years
> already.
>
> So, rather than do nothing, what I'm looking to do is to work with a few
> people who I can share my ideas with, get excited about the ideas, and then
> who will work with the community to get them implemented.   That's what I
> was announcing and talking about yesterday --- looking for interested people
> who want to work on NumPy *with* the NumPy community.
>
> In my enthusiasm, I realize that some may have mis-understood my intention.
> There is no 'imminent' fork, nor am I planning on doing some crazy amount of
> work that I then try to force on other developers of NumPy.
>
> What I'm planning to do is find people to train on NumPy code base (people
> to increase the diversity of the developers would be ideal -- but hard to
> accomplish).  I plan to train them on NumPy based on my experience, and on
> what I think should be done --- and then have *them* work through the
> community process and engage with others to get consensus (hopefully not
> losing too much in translation in the process --- but instead getting even
> better).
>
> During that process I will engage as a member of the community and help
> write NEPs and other documents and help clarify where it makes sense as I
> can.   I will be filtering for people that actually want to see NumPy get
> better.Until I identify the people and work with them, it will be hard
> to tell how this will best work.   So, stay tuned.
>
> If all goes well, what you should see in a few weeks time are specific
> proposals, a branch or two, and the beginnings of some pull requests.If
> you don't see that, then I will not have found the right people to help me,
> and we will all continue to go back to searching.
>
> While I'm expecting the best, in the worst case, we get additional people
> who know the NumPy code base and can help squash bugs as well as implement
> changes that are desired.Three things are needed if you want to
> participate in this:  1) A willingness to work with the open source
> community, 2) a deep knowledge of C and in-particular CPython's brand of C,
> and 3) a willingness to engage with me, do a mind-meld and dump around the
> NumPy code base, and then improve on what is in my head with the rest of the
> community.


I don't have time to do the work myself, but I'll be happy to help in
terms of the community. Travis has paid me in the past to do the NumPy
1.7 release, and implement/fix some things that Travis had in mind. I
had no prior involvement with the NumPy codebase / community, and
within few months I had to get myself familiar with the code, fix
release critical bugs, get my PRs accepted by the community,
eventually get push access (from the core developers, not from Travis)
so that I can merge other people's patches, and most importantly, make
the community allow me to actually do the release, even though I came
from outside. As far as I know, Travis didn't have to pull any strings
for me, and as far as I know, there was no friction in the community
either --- there was a job to be done, we got it done, and that was
it.

So I'll be happy to share my experience and help the person (if coming
from outside) how to do this. In this case a push access is not
necessary, so it's just about sending high quality PRs and good
communication.

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] [RFC] should we argue for a matrix power operator, @@?

[Ondřej Čertík]

On Mon, Mar 17, 2014 at 11:30 AM, Fernando Perez fperez@gmail.com wrote:
 On Mon, Mar 17, 2014 at 10:01 AM, Aron Ahmadia a...@ahmadia.net wrote:


 On Mon, Mar 17, 2014 at 7:53 AM, Nathaniel Smith n...@pobox.com wrote:

 The thread so far, it sounds like the consensus answer is meh,
 whatever. So I'm thinking we should just drop @@ from the PEP, and if
 it turns out that this is a problem we can always revisit it in the
 ~3.6/3.7 timeframe.


 +1 from here.


 +1 too. Absent *clear* enthusiasm and support for new syntax/operators, I
 think being conservative and slow is the right approach. Just having @ will
 give us data and experience with this space, and it may become clear after
 one more cycle that we really need/want @@, or not, as the case may be.  But
 it's easier to add it later if we really need it than to remove it if it
 proves to be a bad idea, so +1 for moving slowly on this.

+1. Thanks Nathan for pushing this!

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] argsort speed

[Ondřej Čertík]

On Fri, Feb 21, 2014 at 11:09 PM, Charles R Harris
charlesr.har...@gmail.com wrote:



 On Fri, Feb 21, 2014 at 10:35 PM, Ondřej Čertík ondrej.cer...@gmail.com
 wrote:

 On Mon, Feb 17, 2014 at 11:40 AM, Charles R Harris
 charlesr.har...@gmail.com wrote:
 
 
 
  On Mon, Feb 17, 2014 at 11:32 AM, Julian Taylor
  jtaylor.deb...@googlemail.com wrote:
 
  On 17.02.2014 15:18, Francesc Alted wrote:
   On 2/17/14, 1:08 AM, josef.p...@gmail.com wrote:
   On Sun, Feb 16, 2014 at 6:12 PM, Daπid davidmen...@gmail.com
   wrote:
   On 16 February 2014 23:43, josef.p...@gmail.com wrote:
   What's the fastest argsort for a 1d array with around 28 Million
   elements, roughly uniformly distributed, random order?
  
   On numpy latest version:
  
   for kind in ['quicksort', 'mergesort', 'heapsort']:
print kind
%timeit np.sort(data, kind=kind)
%timeit np.argsort(data, kind=kind)
  
  
   quicksort
   1 loops, best of 3: 3.55 s per loop
   1 loops, best of 3: 10.3 s per loop
   mergesort
   1 loops, best of 3: 4.84 s per loop
   1 loops, best of 3: 9.49 s per loop
   heapsort
   1 loops, best of 3: 12.1 s per loop
   1 loops, best of 3: 39.3 s per loop
  
  
   It looks quicksort is quicker sorting, but mergesort is marginally
   faster
   sorting args. The diference is slim, but upon repetition, it
   remains
   significant.
  
   Why is that? Probably part of the reason is what Eelco said, and
   part
   is
   that in sort comparison are done accessing the array elements
   directly, but
   in argsort you have to index the array, introducing some overhead.
   Thanks, both.
  
   I also gain a second with mergesort.
  
   matlab would be nicer in my case, it returns both.
   I still need to use the argsort to index into the array to also get
   the sorted array.
  
   Many years ago I needed something similar, so I made some functions
   for
   sorting and argsorting in one single shot.  Maybe you want to reuse
   them.  Here it is an example of the C implementation:
  
   https://github.com/PyTables/PyTables/blob/develop/src/idx-opt.c#L619
  
   and here the Cython wrapper for all of them:
  
  
  
   https://github.com/PyTables/PyTables/blob/develop/tables/indexesextension.pyx#L129
  
   Francesc
  
 
  that doesn't really make a big difference if the data is randomly
  distributed.
  the sorting operation is normally much more expensive than latter
  applying the indices:
 
  In [1]: d = np.arange(1000)
 
  In [2]: np.random.shuffle(d)
 
  In [3]: %timeit  np.argsort(d)
  1 loops, best of 3: 1.99 s per loop
 
  In [4]: idx = np.argsort(d)
 
  In [5]: %timeit d[idx]
  1 loops, best of 3: 213 ms per loop
 
 
 
  But if your data is not random it can make a difference as even
  quicksort can be a lot faster then.
  timsort would be a nice addition to numpy, it performs very well for
  partially sorted data. Unfortunately its quite complicated to
  implement.
 
 
  Quicksort and shellsort gain speed by having simple inner loops. I have
  the
  impression that timsort is optimal when compares and memory access are
  expensive, but I haven't seen any benchmarks for native types in
  contiguous
  memory.

 I found some benchmarks for continuous memory here:

 https://github.com/swenson/sort/
 https://github.com/gfx/cpp-TimSort

 The first one seems the best, it probably can be directly reused in numpy.
 The only issue is that it only sorts the array, but does not provide
 argsort.


 I'm impressed by the heapsort time. Heapsort is the slowest of the numpy
 sorts. So either the heapsort implementation is better than ours or the
 other sort are worse ;)

 Partially sorted sequence are pretty common, so timsort might be a worthy
 addition. Last time I looked, JDK was using timsort for sorting objects, and
 quicksort for native types. Another sort is dual pivot quicksort that I've
 heard some good things about.

 Adding indirect sorts isn't so difficult once the basic sort is available.
 Since the memory access tends to be larger as it gets randomly accessed,
 timsort might be a good choice for that.

Indeed, I think one has to be very careful about these benchmarks, since
it highly depends on the structure of the arrays being sorted.

I've been looking into this a bit, since I need some fast algorithm in Fortran,
that returns indices that sort the array. So far I use quicksort, but
this Timsort
might perform better for partially sorted arrays, which is typically
my use case.

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] argsort speed

[Ondřej Čertík]

On Mon, Feb 17, 2014 at 11:40 AM, Charles R Harris
charlesr.har...@gmail.com wrote:



 On Mon, Feb 17, 2014 at 11:32 AM, Julian Taylor
 jtaylor.deb...@googlemail.com wrote:

 On 17.02.2014 15:18, Francesc Alted wrote:
  On 2/17/14, 1:08 AM, josef.p...@gmail.com wrote:
  On Sun, Feb 16, 2014 at 6:12 PM, Daπid davidmen...@gmail.com wrote:
  On 16 February 2014 23:43, josef.p...@gmail.com wrote:
  What's the fastest argsort for a 1d array with around 28 Million
  elements, roughly uniformly distributed, random order?
 
  On numpy latest version:
 
  for kind in ['quicksort', 'mergesort', 'heapsort']:
   print kind
   %timeit np.sort(data, kind=kind)
   %timeit np.argsort(data, kind=kind)
 
 
  quicksort
  1 loops, best of 3: 3.55 s per loop
  1 loops, best of 3: 10.3 s per loop
  mergesort
  1 loops, best of 3: 4.84 s per loop
  1 loops, best of 3: 9.49 s per loop
  heapsort
  1 loops, best of 3: 12.1 s per loop
  1 loops, best of 3: 39.3 s per loop
 
 
  It looks quicksort is quicker sorting, but mergesort is marginally
  faster
  sorting args. The diference is slim, but upon repetition, it remains
  significant.
 
  Why is that? Probably part of the reason is what Eelco said, and part
  is
  that in sort comparison are done accessing the array elements
  directly, but
  in argsort you have to index the array, introducing some overhead.
  Thanks, both.
 
  I also gain a second with mergesort.
 
  matlab would be nicer in my case, it returns both.
  I still need to use the argsort to index into the array to also get
  the sorted array.
 
  Many years ago I needed something similar, so I made some functions for
  sorting and argsorting in one single shot.  Maybe you want to reuse
  them.  Here it is an example of the C implementation:
 
  https://github.com/PyTables/PyTables/blob/develop/src/idx-opt.c#L619
 
  and here the Cython wrapper for all of them:
 
 
  https://github.com/PyTables/PyTables/blob/develop/tables/indexesextension.pyx#L129
 
  Francesc
 

 that doesn't really make a big difference if the data is randomly
 distributed.
 the sorting operation is normally much more expensive than latter
 applying the indices:

 In [1]: d = np.arange(1000)

 In [2]: np.random.shuffle(d)

 In [3]: %timeit  np.argsort(d)
 1 loops, best of 3: 1.99 s per loop

 In [4]: idx = np.argsort(d)

 In [5]: %timeit d[idx]
 1 loops, best of 3: 213 ms per loop



 But if your data is not random it can make a difference as even
 quicksort can be a lot faster then.
 timsort would be a nice addition to numpy, it performs very well for
 partially sorted data. Unfortunately its quite complicated to implement.


 Quicksort and shellsort gain speed by having simple inner loops. I have the
 impression that timsort is optimal when compares and memory access are
 expensive, but I haven't seen any benchmarks for native types in contiguous
 memory.

I found some benchmarks for continuous memory here:

https://github.com/swenson/sort/
https://github.com/gfx/cpp-TimSort

The first one seems the best, it probably can be directly reused in numpy.
The only issue is that it only sorts the array, but does not provide argsort.

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Masked arrays: Rationale for False convention

[Ondřej Čertík]

On Tue, Oct 1, 2013 at 4:29 AM, Robert Kern robert.k...@gmail.com wrote:
 On Tue, Oct 1, 2013 at 3:57 AM, Ondřej Čertík ondrej.cer...@gmail.com
 wrote:

 I see, that makes sense. So to remember this, the rule is:

 Specify elements that you want to get masked using True in 'mask'.

 Yes. This convention dates back at least to the original MA package in
 Numeric; I don't know if Paul Dubois stole it from any previous software.

I see, thanks.


 One way to motivate the convention is to think about doing a binary
 operation on masked arrays, which is really the most common kind of thing
 one does with masked arrays. The mask of the result is the logical OR of the
 two operand masks (barring additional masked elements from domain
 violations, 0/0, etc.).

In the other convention, you just use logical AND, so that seams equally
simple, unless I am missing something.

 I assume that the convention was decided mostly on
 what was most convenient and efficient for the common internal operations
 for *implementing* the masked arrays and not necessarily matching any
 particular intuitions when putting data *into* the masked arrays.

That makes sense.


On Mon, Sep 30, 2013 at 9:05 PM, Eric Firing efir...@hawaii.edu wrote:
 On 2013/09/30 4:57 PM, Ondřej Čertík wrote:

 But why do I need to invert the mask when I want to see the valid elements:

 In [1]: from numpy import ma

 In [2]: a = ma.array([1, 2, 3, 4], mask=[False, False, True, False])

 In [3]: a
 Out[3]:
 masked_array(data = [1 2 -- 4],
   mask = [False False  True False],
 fill_value = 99)


 In [4]: a[~a.mask]
 Out[4]:
 masked_array(data = [1 2 4],
   mask = [False False False],
 fill_value = 99)


 I would find natural to write [4] as a[a.mask]. This is when it gets 
 confusing.

 There is no getting around it; each of the two possible conventions has
 its advantages.  But try this instead:

 In [2]: a = ma.array([1, 2, 3, 4], mask=[False, False, True, False])

 In [3]: a.compressed()
 Out[3]: array([1, 2, 4])


 I do occasionally need a goodmask which is the inverse of a.mask, but
 not very often; and when I do, needing to invert a.mask doesn't bother me.

a.compressed() works for getting data out --- but I also use it to
assign data in,
e.g.:

a[~a.mask] = 1




Thanks everybody for the discussion. It sheds some light onto the current
convention.

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Masked arrays: Rationale for False convention

[Ondřej Čertík]

Hi,

What is the rationale for using False in 'mask' for elements that
should be included?

http://docs.scipy.org/doc/numpy/reference/maskedarray.generic.html

As opposed to using True for elements that should be included, which
is what I was intuitively expecting when I started using the masked
arrays. This True convention also happens to be the one used in
Fortran, see e.g.:

http://gcc.gnu.org/onlinedocs/gfortran/SUM.html

So it's confusing why NumPy would chose a False convention. Could it
be, that NumPy views 'mask' as opacity? Then it would make sense to
use True to make a value 'opaque'.

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Masked arrays: Rationale for False convention

[Ondřej Čertík]

On Mon, Sep 30, 2013 at 8:29 PM, Eric Firing efir...@hawaii.edu wrote:
 On 2013/09/30 4:05 PM, josef.p...@gmail.com wrote:
 On Mon, Sep 30, 2013 at 9:38 PM, Charles R Harris
 charlesr.har...@gmail.com wrote:



 On Mon, Sep 30, 2013 at 7:05 PM, Ondřej Čertík ondrej.cer...@gmail.com
 wrote:

 Hi,

 What is the rationale for using False in 'mask' for elements that
 should be included?

 http://docs.scipy.org/doc/numpy/reference/maskedarray.generic.html

 As opposed to using True for elements that should be included, which
 is what I was intuitively expecting when I started using the masked
 arrays. This True convention also happens to be the one used in
 Fortran, see e.g.:

 http://gcc.gnu.org/onlinedocs/gfortran/SUM.html

 So it's confusing why NumPy would chose a False convention. Could it
 be, that NumPy views 'mask' as opacity? Then it would make sense to
 use True to make a value 'opaque'.


 There was a lengthy discussion of this point back when the NA work was done.
 You might be able to find the thread with a search.

 As to why it is as it is, I suspect it is historical consistency. Pierre
 wrote the masked array package for numpy, but it may very well go back to
 the masked array package implemented for Numeric.

 I don't know ancient history, but I thought it's natural. (Actually,
 I never thought about it.)

 I always thought `mask` indicates the masked (invalid, hidden)
 values, and masked arrays mask the missing values.

 Exactly.  It is also consistent with the C and Unix convention of
 returning 0 on success and 1, or a non-zero error code on failure.  In a
 similar vein, it works nicely with bit-mapped quality control flags,
 etc.  When nothing is flagged, the value is good, and consequently not
 masked out.

I see, that makes sense. So to remember this, the rule is:

Specify elements that you want to get masked using True in 'mask'.

But why do I need to invert the mask when I want to see the valid elements:

In [1]: from numpy import ma

In [2]: a = ma.array([1, 2, 3, 4], mask=[False, False, True, False])

In [3]: a
Out[3]:
masked_array(data = [1 2 -- 4],
 mask = [False False  True False],
   fill_value = 99)


In [4]: a[~a.mask]
Out[4]:
masked_array(data = [1 2 4],
 mask = [False False False],
   fill_value = 99)


I would find natural to write [4] as a[a.mask]. This is when it gets confusing.

For example in Fortran, one does:

integer :: a(4) = [1, 2, 3, 4]
logical :: mask(4) = [.true., .true., .false., .true.]
print *, a
print *, pack(a, mask)

and it prints:

   1   2   3   4
   1   2   4

So the behavior of mask when used as an index to select elements from
an array is identical to NumPy --- True means include the element,
False means exclude it.

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] ANN: NumPy 1.7.1 release

[Ondřej Čertík]

On Tue, Apr 23, 2013 at 12:10 PM, Frédéric Bastien no...@nouiz.org wrote:
 Hi,

 A big thanks for that release.

 I also think it would be useful to do a release candidate about this. This
 release changed the behavior releated to python long and broke a test in
 Theano. Nothing important, but we could have fixed this before the release.

 The numpy change is that a python long that don't fit in an int64, but fit
 in an uint64, was throwing an overflow exception. Now it return an uint64.

My apologies for this. There was a release candidate here:

http://mail.scipy.org/pipermail/numpy-discussion/2013-March/065948.html

and I don't see any offending patch between the 1.7.1rc1 and 1.7.1.

If the bugs are in numpy, would you please report it into issues? So
that we can fix it.

Thanks,
Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] ANN: NumPy 1.7.1 release

[Ondřej Čertík]

On Sun, Apr 7, 2013 at 2:09 AM, Ondřej Čertík ondrej.cer...@gmail.com wrote:
 Hi,

 I'm pleased to announce the availability of the final NumPy 1.7.1 release.

 Sources and binary installers can be found at
 https://sourceforge.net/projects/numpy/files/NumPy/1.7.1/

 Only three simple bugs were fixed since 1.7.1rc1 (#3166, #3179, #3187).

 I would like to thank everybody who contributed patches since 1.7.1rc1:
 Eric Fode, Nathaniel J. Smith and Charles Harris.

 Cheers,
 Ondrej

 P.S. I'll create the Mac binary installers in a few days. Pypi is updated.

I've uploaded Mac binaries:

https://sourceforge.net/projects/numpy/files/NumPy/1.7.1/

Ralf, if you have a minute, would you mind uploading the Mac 10.6 binary?
I don't have access to such a mac.

Thanks
Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] datetime64 1970 issue

[Ondřej Čertík]

On Wed, Apr 17, 2013 at 10:11 AM, Sebastian Berg
sebast...@sipsolutions.net wrote:
 On Wed, 2013-04-17 at 09:07 -0700, Chris Barker - NOAA Federal wrote:
 On Wed, Apr 17, 2013 at 9:04 AM, Chris Barker - NOAA Federal
 chris.bar...@noaa.gov wrote:
  On Tue, Apr 16, 2013 at 8:23 PM, Zachary Ploskey zplos...@gmail.com 
  wrote:
  I'd say we need some more unit-tests!

 speaking of which, where are the tests? I just did a quick poke at
 github, and found:

 https://github.com/numpy/numpy/tree/master/numpy/testing
 and
 https://github.com/numpy/numpy/tree/master/numpy/test

 but there's very little in there.


 The datetime is implemented in the core, so the tests are here:

 https://github.com/numpy/numpy/blob/master/numpy/core/tests/test_datetime.py


I wonder if the problem has something to do with these functions here:

https://github.com/numpy/numpy/blob/master/numpy/core/src/multiarray/datetime_strings.c#L189

https://github.com/numpy/numpy/blob/master/numpy/core/src/multiarray/datetime_strings.c#L265

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] datetime64 1970 issue

[Ondřej Čertík]

On Wed, Apr 17, 2013 at 2:09 PM, Bob Nnamtrop bob.nnamt...@gmail.com wrote:
 It would seem that before 1970 the dates do not include the time zone
 adjustment while after 1970 they do. This is the source of the extra 7
 hours.

 In [21]: np.datetime64('1970-01-01 00')
 Out[21]: numpy.datetime64('1970-01-01T00:00-0700','h')

 In [22]: np.datetime64('1969-12-31 00')
 Out[22]: numpy.datetime64('1969-12-31T00:00Z','h')

 I saw the other thread about the time zone issues and I think getting rid of
 timezones (perhaps unless they are explicitly requested) is the right thing
 to do.

I think you are right. The timezone conversion only happens for years = 1970:

https://github.com/numpy/numpy/blob/master/numpy/core/src/multiarray/datetime_strings.c#L299

Which explains the problem.

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] datetime64 1970 issue

[Ondřej Čertík]

On Tue, Apr 16, 2013 at 4:55 PM, Bob Nnamtrop bob.nnamt...@gmail.com wrote:
 I am curious if others have noticed an issue with datetime64 at the
 beginning of 1970. First:

 In [144]: (np.datetime64('1970-01-01') - np.datetime64('1969-12-31'))
 Out[144]: numpy.timedelta64(1,'D')

 OK this look fine, they are one day apart. But look at this:

 In [145]: (np.datetime64('1970-01-01 00') - np.datetime64('1969-12-31 00'))
 Out[145]: numpy.timedelta64(31,'h')

 Hmmm, seems like there are 7 extra hours? Am I missing something? I don't
 see this at any other year. This discontinuity makes it hard to use the
 datetime64 object without special adjustment in ones code. I assume this a
 bug?

Indeed, this looks like a bug, I can reproduce it on linux as well:

In [1]: import numpy as np

In [2]: np.datetime64('1970-01-01') - np.datetime64('1969-12-31')
Out[2]: numpy.timedelta64(1,'D')

In [3]: np.datetime64('1970-01-01 00') - np.datetime64('1969-12-31 00')
Out[3]: numpy.timedelta64(31,'h')

In [4]: np.__version__
Out[4]: '1.7.1'


We need to look into the sources to see what is going on.

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] ANN: NumPy 1.7.1 release

[Ondřej Čertík]

Hi,

I'm pleased to announce the availability of the final NumPy 1.7.1 release.

Sources and binary installers can be found at
https://sourceforge.net/projects/numpy/files/NumPy/1.7.1/

Only three simple bugs were fixed since 1.7.1rc1 (#3166, #3179, #3187).

I would like to thank everybody who contributed patches since 1.7.1rc1:
Eric Fode, Nathaniel J. Smith and Charles Harris.

Cheers,
Ondrej

P.S. I'll create the Mac binary installers in a few days. Pypi is updated.


=
NumPy 1.7.1 Release Notes
=

This is a bugfix only release in the 1.7.x series.


Issues fixed


gh-2973   Fix `1` is printed during numpy.test()
gh-2983   BUG: gh-2969: Backport memory leak fix 80b3a34.
gh-3007   Backport gh-3006
gh-2984   Backport fix complex polynomial fit
gh-2982   BUG: Make nansum work with booleans.
gh-2985   Backport large sort fixes
gh-3039   Backport object take
gh-3105   Backport nditer fix op axes initialization
gh-3108   BUG: npy-pkg-config ini files were missing after Bento build.
gh-3124   BUG: PyArray_LexSort allocates too much temporary memory.
gh-3131   BUG: Exported f2py_size symbol prevents linking multiple f2py
modules.
gh-3117   Backport gh-2992
gh-3135   DOC: Add mention of PyArray_SetBaseObject stealing a reference
gh-3134   DOC: Fix typo in fft docs (the indexing variable is 'm', not 'n').
gh-3136   Backport #3128

Checksums
=

9e369a96b94b107bf3fab7e07fef8557
release/installers/numpy-1.7.1-win32-superpack-python2.6.exe
0ab72b3b83528a7ae79c6df9042d61c6  release/installers/numpy-1.7.1.tar.gz
bb0d30de007d649757a2d6d2e1c59c9a
release/installers/numpy-1.7.1-win32-superpack-python3.2.exe
9a72db3cad7a6286c0d22ee43ad9bc6c  release/installers/numpy-1.7.1.zip
0842258fad82060800b8d1f0896cb83b
release/installers/numpy-1.7.1-win32-superpack-python3.1.exe
1b8f29b1fa89a801f83f551adc13aaf5
release/installers/numpy-1.7.1-win32-superpack-python2.7.exe
9ca22df942e5d5362cf7154217cb4b69
release/installers/numpy-1.7.1-win32-superpack-python2.5.exe
2fd475b893d8427e26153e03ad7d5b69
release/installers/numpy-1.7.1-win32-superpack-python3.3.exe
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] ANN: NumPy 1.7.1rc1 release

[Ondřej Čertík]

On Tue, Mar 26, 2013 at 6:32 PM, Ondřej Čertík ondrej.cer...@gmail.com wrote:
[...]
 Yes. I created an issue here for them to test it:

 https://github.com/scikit-learn/scikit-learn/issues/1809

 Just to make sure.

There doesn't seem to be any more problems, so I am releasing 1.7.1 now.

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Any plans for windows 64-bit installer for 1.7?

[Ondřej Čertík]

Hi Matthew,

On Tue, Mar 26, 2013 at 5:48 PM, Matthew Brett matthew.br...@gmail.com wrote:
 Hi Ondrej,

 On Thu, Feb 7, 2013 at 3:18 PM, Ondřej Čertík ondrej.cer...@gmail.com wrote:
 On Thu, Feb 7, 2013 at 12:29 PM, Chris Barker - NOAA Federal
 chris.bar...@noaa.gov wrote:
 On Thu, Feb 7, 2013 at 11:38 AM, Matthew Brett matthew.br...@gmail.com 
 wrote:
 a) If we cannot build Scipy now, it may or may not be acceptable to
 release numpy now.  I think it is, you (Ralf) think it isn't, we
 haven't discussed that.  It may not come up.

 Is anyone suggesting we hold the whole release for this? I fnot, then

 Just to make it clear, I do not plan to hold the whole release because of 
 this.
 Previous releases also didn't have this official 64bit Windows binary,
 so there is
 no regression.

 Once we figure out how to create 64bit binaries, then we'll start
 uploading them.

 Did you make any progress with this?  Worth making some notes?
 Anything we can do to help?

Unfortunately I've been too busy the last month to push this through,
so right now I am just concentrating on getting 1.7.1 out of the door,
as that is higher priority.

I am starting a new job on Monday, so once things settle down, I
should be able to get back to this. I will post notes once I get to
this again.

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] ANN: NumPy 1.7.1rc1 release

[Ondřej Čertík]

On Sun, Mar 24, 2013 at 8:00 PM, Charles R Harris
charlesr.har...@gmail.com wrote:


 On Sun, Mar 24, 2013 at 3:02 PM, Ondřej Čertík ondrej.cer...@gmail.com
 wrote:

 Hi,

 I'm pleased to announce the availability of the first release candidate of
 NumPy 1.7.1rc1.

 Sources and binary installers can be found at
 https://sourceforge.net/projects/numpy/files/NumPy/1.7.1rc1/

 Please test it and report any bugs. It fixes a few bugs, listed below.

 I would like to thank everybody who contributed patches to this release:
 Nathaniel J. Smith, Sebastian Berg, Charles Harris, Bradley M. Froehle,
 Ralf Gommers, Christoph Gohlke, Mark Wiebe and Maximilian Albert.

 Cheers,
 Ondrej



 =
 NumPy 1.7.1 Release Notes
 =

 This is a bugfix only release in the 1.7.x series.


 Issues fixed
 

 gh-2973   Fix `1` is printed during numpy.test()
 gh-2983   BUG: gh-2969: Backport memory leak fix 80b3a34.
 gh-3007   Backport gh-3006
 gh-2984   Backport fix complex polynomial fit
 gh-2982   BUG: Make nansum work with booleans.
 gh-2985   Backport large sort fixes
 gh-3039   Backport object take
 gh-3105   Backport nditer fix op axes initialization
 gh-3108   BUG: npy-pkg-config ini files were missing after Bento build.
 gh-3124   BUG: PyArray_LexSort allocates too much temporary memory.
 gh-3131   BUG: Exported f2py_size symbol prevents linking multiple f2py
 modules.
 gh-3117   Backport gh-2992
 gh-3135   DOC: Add mention of PyArray_SetBaseObject stealing a reference
 gh-3134   DOC: Fix typo in fft docs (the indexing variable is 'm', not
 'n').
 gh-3136   Backport #3128

 Checksums
 =

 28c3f3e71b5eaa6bfab6e8340dbd35e7  release/installers/numpy-1.7.1rc1.tar.gz
 436f416dee10d157314bd9da7ab95c9c
 release/installers/numpy-1.7.1rc1-win32-superpack-python2.7.exe
 a543c8cf69f66ff2b4c9565646105863
 release/installers/numpy-1.7.1rc1-win32-superpack-python2.6.exe
 6dfcbbd449b7fe4e841c5fd1bfa7af7c
 release/installers/numpy-1.7.1rc1-win32-superpack-python2.5.exe
 22912792a1b6155ae2bdbc30bee8fadc
 release/installers/numpy-1.7.1rc1-win32-superpack-python3.2.exe
 95bc5a5fcce9fcbc2717a774dccae31b
 release/installers/numpy-1.7.1rc1-win32-superpack-python3.3.exe
 33cf283765a148846b49b89fb96d67d5
 release/installers/numpy-1.7.1rc1-win32-superpack-python3.1.exe
 9761de4b35493fed38c5d177da9c3b37  release/installers/numpy-1.7.1rc1.zip
 __


 Great. The fix for the memory leak should make some folks happy.

Yes. I created an issue here for them to test it:

https://github.com/scikit-learn/scikit-learn/issues/1809

Just to make sure.

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Numpy 1.7.1

[Ondřej Čertík]

On Fri, Mar 22, 2013 at 1:02 AM, Charles R Harris
charlesr.har...@gmail.com wrote:
 The Numpy 1.7.1 release process seems to have stalled.

My apologies for that.

 What do we need to
 finish up to get it going again? I think it would be nice to shoot for a
 release maybe the weekend after next.


I think just the release notes need to be written, which I am doing
right now. Then I release 1.7.1rc1. If more things need to be merged,
I can do 1.7.1rc2. Or we can later release 1.7.2, depending on the
result of the discussion of the plan here:

https://github.com/numpy/numpy/issues/3158

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] ANN: NumPy 1.7.1rc1 release

[Ondřej Čertík]

Hi,

I'm pleased to announce the availability of the first release candidate of
NumPy 1.7.1rc1.

Sources and binary installers can be found at
https://sourceforge.net/projects/numpy/files/NumPy/1.7.1rc1/

Please test it and report any bugs. It fixes a few bugs, listed below.

I would like to thank everybody who contributed patches to this release:
Nathaniel J. Smith, Sebastian Berg, Charles Harris, Bradley M. Froehle,
Ralf Gommers, Christoph Gohlke, Mark Wiebe and Maximilian Albert.

Cheers,
Ondrej



=
NumPy 1.7.1 Release Notes
=

This is a bugfix only release in the 1.7.x series.


Issues fixed


gh-2973   Fix `1` is printed during numpy.test()
gh-2983   BUG: gh-2969: Backport memory leak fix 80b3a34.
gh-3007   Backport gh-3006
gh-2984   Backport fix complex polynomial fit
gh-2982   BUG: Make nansum work with booleans.
gh-2985   Backport large sort fixes
gh-3039   Backport object take
gh-3105   Backport nditer fix op axes initialization
gh-3108   BUG: npy-pkg-config ini files were missing after Bento build.
gh-3124   BUG: PyArray_LexSort allocates too much temporary memory.
gh-3131   BUG: Exported f2py_size symbol prevents linking multiple f2py
modules.
gh-3117   Backport gh-2992
gh-3135   DOC: Add mention of PyArray_SetBaseObject stealing a reference
gh-3134   DOC: Fix typo in fft docs (the indexing variable is 'm', not 'n').
gh-3136   Backport #3128

Checksums
=

28c3f3e71b5eaa6bfab6e8340dbd35e7  release/installers/numpy-1.7.1rc1.tar.gz
436f416dee10d157314bd9da7ab95c9c
release/installers/numpy-1.7.1rc1-win32-superpack-python2.7.exe
a543c8cf69f66ff2b4c9565646105863
release/installers/numpy-1.7.1rc1-win32-superpack-python2.6.exe
6dfcbbd449b7fe4e841c5fd1bfa7af7c
release/installers/numpy-1.7.1rc1-win32-superpack-python2.5.exe
22912792a1b6155ae2bdbc30bee8fadc
release/installers/numpy-1.7.1rc1-win32-superpack-python3.2.exe
95bc5a5fcce9fcbc2717a774dccae31b
release/installers/numpy-1.7.1rc1-win32-superpack-python3.3.exe
33cf283765a148846b49b89fb96d67d5
release/installers/numpy-1.7.1rc1-win32-superpack-python3.1.exe
9761de4b35493fed38c5d177da9c3b37  release/installers/numpy-1.7.1rc1.zip
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Scheduling the 1.7.1 and 1.8 releases

[Ondřej Čertík]

On Wed, Mar 6, 2013 at 9:52 PM, Ralf Gommers ralf.gomm...@gmail.com wrote:



 On Wed, Mar 6, 2013 at 9:06 PM, Nathaniel Smith n...@pobox.com wrote:

 On Wed, Mar 6, 2013 at 6:43 PM, Charles R Harris
 charlesr.har...@gmail.com wrote:
  Hi All,
 
  There are now some 14 non-merge commits in the 1.7.x branch including
  the
  critical diagonal leak fix. I think there is maybe one more critical
  backport and perhaps several low priority fixes, documentation and such,
  but
  I think we should start up the release process with a goal of getting
  1.7.1
  out by the middle of April.

 What's the critical backport you're thinking of? This last shows just
 two backport PRs waiting to be merged, one trivial one that I just
 submitted, the other that needs a tweak but won't take long:
   https://github.com/numpy/numpy/issues?milestone=27page=1state=open
 But I agree, basically we should merge those two (today?) and then
 release the first RC as soon as Ondrej has a moment to do so...


 I added issue 2999, which I think should be taken along. Other than that, +1
 for a quick release.



  The development branch has been accumulating stuff since last summer, I
  suggest we look to get it out in May, branching at the end of this
  month.

 I would say let's fix the blockers and then branch as soon as Ondrej
 has time to do it, but in practice I suspect this comes out the same
 as what you just said :-). I just pruned the list of blockers; here's
 what we've got:
   https://github.com/numpy/numpy/issues?milestone=1page=1state=open


 It looks like we're not doing so well with setting Milestones correctly.
 Only 4 closed issues for 1.8

 Release quickly after 1.7.1 sounds good.

I hope to finish the rest of issues for 1.7.1 today or tomorrow.
Should I release 1.7.1rc1 first? I think that makes sense, just to be
sure, right?

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] 1.7.1

[Ondřej Čertík]

On Tue, Feb 26, 2013 at 1:21 PM, Charles R Harris
charlesr.har...@gmail.com wrote:


 On Tue, Feb 26, 2013 at 2:03 PM, Ralf Gommers ralf.gomm...@gmail.com
 wrote:




 On Tue, Feb 26, 2013 at 7:47 PM, Charles R Harris
 charlesr.har...@gmail.com wrote:

 When should we put out 1.7.1? Discuss ;)


 When we have X times more fixes in maintenance/1.7.x than the one commit
 with a one-liner fix that we have now. Where X is = 5 at least, unless
 there's a very high prio fix that needs releasing asap?

 Having at least 2 months between bugfix releases unless something very
 urgent comes up would also make sense to me.

 I think we do need to be diligent in backporting fixes quickly after
 they're merged into master, and not leaving that till right before the
 release candidate is scheduled.


 Ralph, the backports are PR's marked with the backport tag and there are
 more than one. It is up to Ondrej to decide whether to include them or not.

I can't see the backport tag. How can I find it at github?

Otherwise I think we should release sooner, I think this bug is quite annoying
for scikit-learn:

https://github.com/scikit-learn/scikit-learn/issues/1715

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] a question about freeze on numpy 1.7.0

[Ondřej Čertík]

Hi Gelin,

On Sun, Feb 24, 2013 at 12:08 AM, Gelin Yan dynami...@gmail.com wrote:
 Hi All

  When I used numpy 1.7.0 with cx_freeze 4.3.1 on windows, I quickly
 found out even a simple import numpy may lead to program failed with
 following exception:

 AttributeError: 'module' object has no attribute 'sys'

 After a poking around some codes I noticed /numpy/core/__init__.py has a
 line 'del sys' at the bottom. After I commented this line, and repacked the
 whole program, It ran fine.
 I also noticed this 'del sys' didn't exist on numpy 1.6.2

 I am curious why this 'del sys' should be here and whether it is safe to
 omit it. Thanks.

The del sys line was introduced in the commit:

https://github.com/numpy/numpy/commit/4c0576fe9947ef2af8351405e0990cebd83ccbb6

and it seems to me that it is needed so that the numpy.core namespace is not
cluttered by it.

Can you post the full stacktrace of your program (and preferably some
instructions
how to reproduce the problem)? It should become clear where the problem is.

Thanks,
Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Fwd: Package: scipy-0.11.0-0.1.rc2.fc18 Tag: f18-updates-candidate Status: failed Built by: orion

[Ondřej Čertík]

Orion,

On Wed, Feb 13, 2013 at 4:06 PM, Orion Poplawski or...@cora.nwra.com wrote:
 On 09/21/2012 11:41 AM, Ondřej Čertík wrote:

 Hi Orion,

 On Thu, Sep 20, 2012 at 2:56 PM, Orion Poplawski or...@cora.nwra.com
 wrote:

 This is a plea for some help.  We've been having trouble getting scipy to
 pass all of the tests in the Fedora 18 build with python 3.3 (although it
 seems to build okay in Fedora 19).  Below are the logs of the build.
 There
 appears to be some kind of memory corruption that manifests itself a
 little
 differently on 32-bit vs. 64-bit.  I really have no idea myself how to
 pursue debugging this, though I'm happy to provide any more needed
 information.


 Thanks for testing the latest beta2 release.

 Task 4509077 on buildvm-35.phx2.fedoraproject.org
 Task Type: buildArch (scipy-0.11.0-0.1.rc2.fc18.src.rpm, i686)
 logs:

 http://koji.fedoraproject.org/koji/getfile?taskID=4509077name=build.log


 This link has the following stacktrace:

 /lib/libpython3.3m.so.1.0(PyMem_Free+0x1c)[0xbf044c]

 /usr/lib/python3.3/site-packages/numpy/core/multiarray.cpython-33m.so(+0x4d52b)[0x42252b]

 /usr/lib/python3.3/site-packages/numpy/core/multiarray.cpython-33m.so(+0xcb7c5)[0x4a07c5]

 /usr/lib/python3.3/site-packages/numpy/core/multiarray.cpython-33m.so(+0xcbc5e)[0x4a0c5e]

 Which indeed looks like in NumPy. Would you be able to obtain full
 stacktrace?

 There has certainly been segfaults in Python 3.3 with NumPy, but we've
 fixed all that we could reproduce. That doesn't mean there couldn't be
 more. If you could nail it down a little bit more, that would be
 great. I'll help once I can reproduce it somehow.

 Ondrej
 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion


 Trying to get back to this as we still see it with numpy 1.7.0 and scipy
 0.11.

 I'm seeing a segfault in malloc_consolidate(), which seems like would only
 occur if there were problems earlier, so I'm not sure a stack trace is all
 that useful.

 Starting program: /usr/bin/python3
 /export/home/orion/redhat/BUILDROOT/scipy-0.11.0-3.fc19.x86_64/usr/lib64/python3.3/site-packages/scipy/linalg/tests/test_decomp.py
 [Thread debugging using libthread_db enabled]
 Using host libthread_db library /lib64/libthread_db.so.1.
 ..
 Program received signal SIGSEGV, Segmentation fault.
 0x003d8d67bdad in malloc_consolidate (av=av@entry=0x3d8d9b1740
 main_arena)
 at malloc.c:4151
 4151  unlink(av, nextchunk, bck, fwd);

 Here's some:
 #0  0x003d8d67bdad in malloc_consolidate (av=av@entry=0x3d8d9b1740
 main_arena)
 at malloc.c:4151
 #1  0x003d8d67d09e in _int_malloc (av=0x3d8d9b1740 main_arena,
 bytes=optimized out)
 at malloc.c:3422
 #2  0x003d8d67f443 in __GI___libc_malloc (bytes=2632) at malloc.c:2862
 #3  0x7121816c in PyArray_IterNew (obj=numpy.ndarray at remote
 0xe02fd0)
 at numpy/core/src/multiarray/iterators.c:385
 #4  0x71218201 in PyArray_IterAllButAxis (obj=obj@entry=
 numpy.ndarray at remote 0xe02fd0, inaxis=inaxis@entry=0x7fff873c)
 at numpy/core/src/multiarray/iterators.c:488
 #5  0x71257970 in _new_argsort (which=NPY_QUICKSORT, axis=0,
 op=0xe02fd0)
 at numpy/core/src/multiarray/item_selection.c:815
 #6  PyArray_ArgSort (op=op@entry=0xe02fd0, axis=0, which=NPY_QUICKSORT)
 at numpy/core/src/multiarray/item_selection.c:1104
 #7  0x7125873a in array_argsort (self=0xe02fd0, args=optimized
 out,
 kwds=optimized out) at numpy/core/src/multiarray/methods.c:1213
 #8  0x003b74d0cc8e in call_function (oparg=optimized out,
 pp_stack=0x7fff8998)
 at /usr/src/debug/Python-3.3.0/Python/ceval.c:4091
 #9  PyEval_EvalFrameEx (f=f@entry=
 Frame 0xd3ecb0, for file
 /usr/lib64/python3.3/site-packages/numpy/core/fromnumeric.py, line 681, in
 argsort (a=numpy.ndarray at remote 0xe02fd0, axis=-1, kind='quicksort',
 order=None, argsort=built-in method argsort of numpy.ndarray object at
 remote 0xe02fd0),
 throwflag=throwflag@entry=0) at
 /usr/src/debug/Python-3.3.0/Python/ceval.c:2703
 #10 0x003b74d0de63 in PyEval_EvalCodeEx (_co=_co@entry=code at remote
 0x71a8ac00,
 globals=optimized out, locals=locals@entry=0x0, args=optimized out,
 argcount=argcount@entry=1, kws=0xe23ab8, kwcount=kwcount@entry=0,
 defs=0x71a965b8,
 defcount=3, kwdefs=0x0, closure=0x0) at
 /usr/src/debug/Python-3.3.0/Python/ceval.c:3462
 #11 0x003b74d0c707 in fast_function (nk=0, na=1, n=optimized out,
 pp_stack=
 0x7fff8c88, func=function at remote 0x71160320)
 at /usr/src/debug/Python-3.3.0/Python/ceval.c:4189
 #12 call_function (oparg=optimized out, pp_stack=0x7fff8c88)
 at /usr/src/debug/Python-3.3.0/Python/ceval.c:4112


 (gdb) up 3
 #3  0x7121816c in PyArray_IterNew (obj=numpy.ndarray at remote
 0xe02fd0)
 at numpy/core/src/multiarray/iterators.c:385
 385

Re: [Numpy-discussion] How to upload to pypi

[Ondřej Čertík]

Hi Robert,

On Tue, Feb 12, 2013 at 2:40 AM, Robert Kern robert.k...@gmail.com wrote:
 On Tue, Feb 12, 2013 at 5:49 AM, Ondřej Čertík ondrej.cer...@gmail.com 
 wrote:
 Hi,

 I have uploaded the NumPy 1.7.0 source distribution to pypi:

 http://pypi.python.org/pypi/numpy/1.7.0

 I did it by uploading the file PKG-INFO from numpy-1.7.0.tar.gz. It
 said Error processing form. Form Failure; reset form submission
 about 3x times,
 but on the 4th try it went through. I reported the issue here:

 https://sourceforge.net/tracker/?func=detailaid=3604194group_id=66150atid=513504

 I then attached the numpy-1.7.0.tar.gz and numpy-1.7.0.zip source files.

 Now I am having trouble attaching the windows installers, just like
 they are here:

 http://pypi.python.org/pypi/numpy/1.6.2

 but whenever I upload the file
 numpy-1.7.0-win32-superpack-python2.5.exe (and set it as MS Windows
 installer),
 it uploads and then I get a blank page with the text:

 
 Error processing form

 invalid distribution file
 

 PyPI does some validation on the files that are uploaded. .exe files
 must be created by bdist_wininst.

 https://bitbucket.org/loewis/pypi/src/fc588bcd668aba643e2e7f9bd6901a7a4296dddb/verify_filetype.py?at=default#cl-15

 I am guessing that the superpack installer is manually built through
 another mechanism.

Ha, I see --- thanks for pointing me to the source code. All is clear now.


 That's it Not very useful. Do you know if the sources of pypi are
 somewhere online? (I didn't find them, only a similar package
 https://github.com/schmir/pypiserver, but that doesn't seem to be it.)

 http://wiki.python.org/moin/CheeseShopDev

 You can get help with PyPI on Catalog-SIG:

 http://mail.python.org/mailman/listinfo/catalog-sig

Thanks again.

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] How to upload to pypi

[Ondřej Čertík]

David,

On Tue, Feb 12, 2013 at 6:46 AM, David Cournapeau courn...@gmail.com wrote:
 On Tue, Feb 12, 2013 at 5:49 AM, Ondřej Čertík ondrej.cer...@gmail.com 
 wrote:
 Hi,

 I have uploaded the NumPy 1.7.0 source distribution to pypi:

 http://pypi.python.org/pypi/numpy/1.7.0

 I did it by uploading the file PKG-INFO from numpy-1.7.0.tar.gz. It
 said Error processing form. Form Failure; reset form submission
 about 3x times,
 but on the 4th try it went through. I reported the issue here:

 https://sourceforge.net/tracker/?func=detailaid=3604194group_id=66150atid=513504

 I then attached the numpy-1.7.0.tar.gz and numpy-1.7.0.zip source files.

 Now I am having trouble attaching the windows installers, just like
 they are here:

 http://pypi.python.org/pypi/numpy/1.6.2

 Those installers are ones built through bdist_wininst. You should
 *not* upload superpack installers there, as most python tools will not
 know what to do with it. For example, easy_install will not work with
 those, even though it does with simple installers from bdist_wininst.

 So ideally, one should build simple (== bdist_wininst-generated)
 installers using the lowest common denominator for architecture (i.e.
 pure blas/lapack, not atlas), and the superpack installer on
 sourceforge. Incidentally, that's why the super pack installer uses a
 different filename, to avoid confusion.

I see. I looked into my scripts and it turns out that actually I do
build the bdist_wininst versions as well, I just didn't know what they
are for, so I ignored them. Now I can see that those will get uploaded
to pypi, so I did that now and it works!

http://pypi.python.org/pypi/numpy/1.7.0

I learned something new today, thanks for the explanation. So pypi
should be done.

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] How to upload to pypi

[Ondřej Čertík]

Hi,

I have uploaded the NumPy 1.7.0 source distribution to pypi:

http://pypi.python.org/pypi/numpy/1.7.0

I did it by uploading the file PKG-INFO from numpy-1.7.0.tar.gz. It
said Error processing form. Form Failure; reset form submission
about 3x times,
but on the 4th try it went through. I reported the issue here:

https://sourceforge.net/tracker/?func=detailaid=3604194group_id=66150atid=513504

I then attached the numpy-1.7.0.tar.gz and numpy-1.7.0.zip source files.

Now I am having trouble attaching the windows installers, just like
they are here:

http://pypi.python.org/pypi/numpy/1.6.2

but whenever I upload the file
numpy-1.7.0-win32-superpack-python2.5.exe (and set it as MS Windows
installer),
it uploads and then I get a blank page with the text:


Error processing form

invalid distribution file


That's it Not very useful. Do you know if the sources of pypi are
somewhere online? (I didn't find them, only a similar package
https://github.com/schmir/pypiserver, but that doesn't seem to be it.)
Anyway, so I at least reported the bug here:

http://sourceforge.net/tracker/?func=detailaid=3604193group_id=66150atid=513504

Finally, should we try to upload the Windows installers to pypi at all?
I am probably doing something wrong if I am hitting these pypi bugs.

Many thanks,
Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Any plans for windows 64-bit installer for 1.7?

[Ondřej Čertík]

On Thu, Feb 7, 2013 at 10:59 AM, Matthew Brett matthew.br...@gmail.com wrote:
 Hi,

 On Thu, Feb 7, 2013 at 10:47 AM, Ralf Gommers ralf.gomm...@gmail.com wrote:



 On Thu, Feb 7, 2013 at 7:41 AM, Matthew Brett matthew.br...@gmail.com
 wrote:


 I think you will need a developer's license for MKL for Numpy.  Ralf -
 any ETA for those?


 No, I'll have to ask again.


 I think I'm right in thinking you'll need a Fortran compiler for Scipy
 but not Numpy?


 Correct.


 Can we defer the Scipy build until after the Numpy build?


 That doesn't sound like a good idea to me.

 I must say I'm a little confused as to how we're going to make the
 decisions here.

 I'm sure you agree that there's an opposite argument to be made, and I
 would make it if I thought it would make a difference, but I'm losing
 faith in my ability to keep the discussion on track, and I don't know
 what to do about that.

Matthew I don't see any problem here. I agree with Ralf that we need
to figure out how to build scipy with Fortran pretty much at the same time,
to see if the solution is a viable solution. With Christoph help and experience,
I am sure it can get done in a satisfactory way.

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Any plans for windows 64-bit installer for 1.7?

[Ondřej Čertík]

On Thu, Feb 7, 2013 at 11:05 AM, Ralf Gommers ralf.gomm...@gmail.com wrote:



 On Thu, Feb 7, 2013 at 7:59 PM, Matthew Brett matthew.br...@gmail.com
 wrote:


  Can we defer the Scipy build until after the Numpy build?
 
 
  That doesn't sound like a good idea to me.

 I must say I'm a little confused as to how we're going to make the
 decisions here.


 How about: attempt to reach consensus? David's concern on DLLs hasn't been
 addressed yet, nor has mine on packages being unavailable. I was actually
 still answering another of your emails, but I can't seem to reply fast
 enough.

Yep, we will need to address those.



 I'm sure you agree that there's an opposite argument to be made, and I
 would make it if I thought it would make a difference, but I'm losing
 faith in my ability to keep the discussion on track, and I don't know
 what to do about that.


 I don't see the problem. Before you offered to put in work. Ondrej is
 willing to help, so is Christoph. So why is it impossible to do Scipy
 builds?

 I can see us getting to a solution here, but offering Numpy installers
 without Scipy ones is not a solution in my book.

Exactly. There is no problem here. Fortran needs to be working as
a first class citizen. I personally use modern Fortran a lot. I've
setup this page:

http://fortran90.org/

with a relevant FAQ about binary compatibility:

http://fortran90.org/src/faq.html#are-fortran-compilers-abi-compatible

and based on how things work on Windows, I'll be happy to extend the
information there.

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Modern Fortran vs NumPy syntax

[Ondřej Čertík]

Hi,

I have recently setup a page about modern Fortran:

http://fortran90.org/

and in particular, it has a long section with side by side syntax
examples of Python/NumPy vs Fortran:

http://fortran90.org/src/rosetta.html

I would be very interested if some NumPy gurus would provide me
feedback. I personally knew
NumPy long before I learned Fortran, and I was amazed that the modern
Fortran pretty much
allows 1:1 syntax with NumPy, including most of all the fancy indexing etc.

Is there some NumPy feature that is not covered there? I would like it
to be a nice resource
for people who know NumPy to feel like at home with Fortran, and vice
versa. I personally
use both every day (Fortran a bit more than NumPy).

Or of you have any other comments or tips for the site, please let me
know. Eventually I'd like
to also put there C++ way of doing the same things, but at the moment
I want to get Fortran
and Python/NumPy done first.

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Any plans for windows 64-bit installer for 1.7?

[Ondřej Čertík]

On Thu, Feb 7, 2013 at 12:29 PM, Chris Barker - NOAA Federal
chris.bar...@noaa.gov wrote:
 On Thu, Feb 7, 2013 at 11:38 AM, Matthew Brett matthew.br...@gmail.com 
 wrote:
 a) If we cannot build Scipy now, it may or may not be acceptable to
 release numpy now.  I think it is, you (Ralf) think it isn't, we
 haven't discussed that.  It may not come up.

 Is anyone suggesting we hold the whole release for this? I fnot, then

Just to make it clear, I do not plan to hold the whole release because of this.
Previous releases also didn't have this official 64bit Windows binary,
so there is
no regression.

Once we figure out how to create 64bit binaries, then we'll start
uploading them.

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Modern Fortran vs NumPy syntax

[Ondřej Čertík]

Frédéric,

On Thu, Feb 7, 2013 at 11:31 AM, Frédéric Bastien no...@nouiz.org wrote:
 Hi,

 I just read a paper[1] that compare python with numpy or pypy vs c++ and
 fortran from a code, memory and speed point of view. The python code was
 still better as you can't have list of ndarray in fortran and some other
 stuff was harder to do. The fastest was fortran, then C++, but pypy around
 2x slower then c++. That isn't bad for a more productive development
 language.

 Maybe you can check that article to find more case to compare.

Yes, I know about this article --- I've been in touch with Sylwester about it,
as I found his code on github a few months ago, so we discussed it.
I also CCed him if he wants to add some comments.

The article is well balanced. To my taste, they use way too much OOP
in the Fortran version,
in fact I am bit surprised that Fortran was still the fastest, even
with the OOP. But Sylwester was interested in comparing OOP
approaches, so that's fair.

If I have time (which I don't see likely soon, but who knows), I will
see if I can write a simple direct non-OOP version of their Fortran
code:

https://github.com/slayoo/mpdata

Possibly just by understanding the original reference [2] and see what
datastructures/arrays I would use to implement it.
In general however, I like their approach, that they took a real world
method and not some artificial benchmark.
So it's a very good contribution.

Ondrej

[2] Smolarkiewicz, P. K. (1984). A Fully Multidimensional Positive
Definite Advection Transport Algorithm with Small Implicit Diffusion.
Journal of Computational Physics, 54(2), 325–362.
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] ANN: pandas 0.10.1 is released

[Ondřej Čertík]

Hi Wes,

On Tue, Jan 22, 2013 at 8:32 AM, Wes McKinney wesmck...@gmail.com wrote:
 hi all,

 We've released pandas 0.10.1 which includes many bug fixes from
 0.10.0 (including a number of issues with the new file parser,
 e.g. reading multiple files in separate threads), various
 performance improvements, and major new PyTables/HDF5-based
 functionality contributed by Jeff Reback. I strongly recommend
 that all users upgrade.

 Thanks to all who contributed to this release, especially Chang
 She, Jeff Reback, and Yoval P.

 As always source archives and Windows installers are on PyPI.

 What's new: http://pandas.pydata.org/pandas-docs/stable/whatsnew.html
 Installers: http://pypi.python.org/pypi/pandas

 $ git log v0.10.0..v0.10.1 --pretty=format:%aN | sort | uniq -c | sort -rn
  66 jreback
  59 Wes McKinney
  43 Chang She
  12 y-p
   5 Vincent Arel-Bundock
   4 Damien Garaud
   3 Christopher Whelan
   3 Andy Hayden
   2 Jay Parlar
   2 Dan Allan
   1 Thouis (Ray) Jones
   1 svaksha
   1 herrfz
   1 Garrett Drapala
   1 elpres
   1 Dieter Vandenbussche
   1 Anton I. Sipos

In case you were interested, you can get this easily by just:

$ git shortlog -ns v0.10.0..v0.10.1
66  jreback
59  Wes McKinney
43  Chang She
12  y-p
 5  Vincent Arel-Bundock
 4  Damien Garaud
 3  Andy Hayden
 3  Christopher Whelan
 2  Dan Allan
 2  Jay Parlar
 1  Anton I. Sipos
 1  Dieter Vandenbussche
 1  Garrett Drapala
 1  Thouis (Ray) Jones
 1  elpres
 1  herrfz
 1  svaksha

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] ANN: NumPy 1.7.0rc2 release

[Ondřej Čertík]

Hi,

I'm pleased to announce the availability of the second release candidate of
NumPy 1.7.0rc2.

Sources and binary installers can be found at
https://sourceforge.net/projects/numpy/files/NumPy/1.7.0rc2/

We have fixed all issues known to us since the 1.7.0rc1 release.

Please test this release and report any issues on the numpy-discussion
mailing list. If there are no further problems, I plan to release the final
version in a few days.

I would like to thank Sandro Tosi, Sebastian Berg, Charles Harris,
Marcin Juszkiewicz, Mark Wiebe, Ralf Gommers and Nathaniel J. Smith
for sending patches, fixes and helping with reviews for this release since
1.7.0rc1, and Vincent Davis for providing the Mac build machine.

Cheers,
Ondrej




=
NumPy 1.7.0 Release Notes
=

This release includes several new features as well as numerous bug fixes and
refactorings. It supports Python 2.4 - 2.7 and 3.1 - 3.3 and is the last
release that supports Python 2.4 - 2.5.

Highlights
==

* ``where=`` parameter to ufuncs (allows the use of boolean arrays to choose
  where a computation should be done)
* ``vectorize`` improvements (added 'excluded' and 'cache' keyword, general
  cleanup and bug fixes)
* ``numpy.random.choice`` (random sample generating function)


Compatibility notes
===

In a future version of numpy, the functions np.diag, np.diagonal, and the
diagonal method of ndarrays will return a view onto the original array,
instead of producing a copy as they do now. This makes a difference if you
write to the array returned by any of these functions. To facilitate this
transition, numpy 1.7 produces a FutureWarning if it detects that you may
be attempting to write to such an array. See the documentation for
np.diagonal for details.

Similar to np.diagonal above, in a future version of numpy, indexing a
record array by a list of field names will return a view onto the original
array, instead of producing a copy as they do now. As with np.diagonal,
numpy 1.7 produces a FutureWarning if it detects that you may be attempting
to write to such an array. See the documentation for array indexing for
details.

In a future version of numpy, the default casting rule for UFunc out=
parameters will be changed from 'unsafe' to 'same_kind'. (This also applies
to in-place operations like a += b, which is equivalent to np.add(a, b,
out=a).) Most usages which violate the 'same_kind' rule are likely bugs, so
this change may expose previously undetected errors in projects that depend
on NumPy. In this version of numpy, such usages will continue to succeed,
but will raise a DeprecationWarning.

Full-array boolean indexing has been optimized to use a different,
optimized code path.   This code path should produce the same results,
but any feedback about changes to your code would be appreciated.

Attempting to write to a read-only array (one with ``arr.flags.writeable``
set to ``False``) used to raise either a RuntimeError, ValueError, or
TypeError inconsistently, depending on which code path was taken. It now
consistently raises a ValueError.

The ufunc.reduce functions evaluate some reductions in a different order
than in previous versions of NumPy, generally providing higher performance.
Because of the nature of floating-point arithmetic, this may subtly change
some results, just as linking NumPy to a different BLAS implementations
such as MKL can.

If upgrading from 1.5, then generally in 1.6 and 1.7 there have been
substantial code added and some code paths altered, particularly in the
areas of type resolution and buffered iteration over universal functions.
This might have an impact on your code particularly if you relied on
accidental behavior in the past.

New features


Reduction UFuncs Generalize axis= Parameter
---

Any ufunc.reduce function call, as well as other reductions like sum, prod,
any, all, max and min support the ability to choose a subset of the axes to
reduce over. Previously, one could say axis=None to mean all the axes or
axis=# to pick a single axis.  Now, one can also say axis=(#,#) to pick a
list of axes for reduction.

Reduction UFuncs New keepdims= Parameter


There is a new keepdims= parameter, which if set to True, doesn't throw
away the reduction axes but instead sets them to have size one.  When this
option is set, the reduction result will broadcast correctly to the
original operand which was reduced.

Datetime support


.. note:: The datetime API is *experimental* in 1.7.0, and may undergo changes
   in future versions of NumPy.

There have been a lot of fixes and enhancements to datetime64 compared
to NumPy 1.6:

* the parser is quite strict about only accepting ISO 8601 dates, with a few
  convenience extensions
* converts between units correctly
* datetime arithmetic works correctly
* business day functionality (allows the datetime 

Re: [Numpy-discussion] Any plans for windows 64-bit installer for 1.7?

[Ondřej Čertík]

On Wed, Feb 6, 2013 at 9:20 PM, Dag Sverre Seljebotn
d.s.seljeb...@astro.uio.no wrote:
 On 02/07/2013 12:16 AM, Matthew Brett wrote:
[...]
 Can you clarify the people you think will get stuck?  I think I'm
 right in saying that anyone with a C extension should be able to build
 them against numpy, by installing the free (as-in-beer) MS tools?  So
 do you just mean people needing a Fortran compiler?  That's a small
 constituency, I think.

 Off the top of my head there's SciPy and pymc...

 Anyway, I'm butting in because I wish this discussion could separate
 between the user perspective and the developer perspective.

 FWIW,

 1) From a user's perspective, I don't understand this either. If you are
 already using a closed source, not-free-as-in-beer operating system, why
 would you not use (or buy!) a closed source, not-free-as-in-beer Fortran
 compiler?

Indeed. Though I really have no clue on the Windows use cases. Maybe
most Windows users don't want to compile anything, just
use numpy and scipy from Python?


 2) BUT, the argument I've seen that I can at least understand is that
 the release manager should be able to do a release using only open
 source tools (even using Wine instead of Windows) and not rely on a
 limited number of licenses. And that the release manager must be able to
 perform all the official builds directly.

As the release manager, I really only have two requirements:

* I want to ssh in there from my Ubuntu
* I want to automate the whole process

For Mac, linux and Wine I can do that. So I have just spend few hours
browsing the net and it looks like that the combination of Windows
PowerShell 2.0:

http://en.wikipedia.org/wiki/Windows_PowerShell

and some SSH server, there are quite a few, one commercial but free
for one user one connection (perfect for me!):

http://www.powershellinside.com/powershell/ssh/

So if I understand the pages correctly, I can login there from linux,
and then I use the PowerShell commands to script anything. It looks
like I can even use my Fabric fabfiles with powershell:

https://gist.github.com/diyan/2850866

I can also use git with PowerShell:

http://windows.github.com/
http://haacked.com/archive/2011/12/13/better-git-with-powershell.aspx


So the final problem is how to execute MSVC and Fortran from Power
Shell on Windows. These links might help for MSVC:

http://stackoverflow.com/questions/4398136/use-powershell-for-visual-studio-command-prompt
http://geekswithblogs.net/dwdii/archive/2011/05/20/automating-a-visual-studio-build-with-powershell---part-1.aspx

Finally, for Intel Fortran + powershell:

http://software.intel.com/en-us/forums/topic/284425


So I think it is all possible. If somebody can provide a machine with
Windows, MSVC, PowerShell2.0, SSH server and some Fortran compiler, it
should be possible for me to automate everything from Ubuntu using my
Fabric files (https://github.com/certik/numpy-vendor).

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Any plans for windows 64-bit installer for 1.7?

[Ondřej Čertík]

Christoph,

On Tue, Feb 5, 2013 at 3:04 PM, Christoph Gohlke cgoh...@uci.edu wrote:
[...]
 In order not to leave this discussion without a resolution:

 Christophe - would you allow us to distribute your numpy binaries for
 1.7 from the numpy sourceforge page?

 Cheers,

 Matthew


 I am OK with providing 64 bit numpy-MKL binaries (that is numpy
 compiled with MSVC compilers and linked to Intel's MKL) for official
 numpy releases.

 However:

 1) There seems to be no real consensus and urge for doing this. Using a
 free toolchain capable of building the whole scipy-stack would be much
 preferred. Several 64 bit Python distributions containing numpy-MKL are
 already available, some for free.

 2) Releasing 64 bit numpy without matching scipy binaries would make
 little sense to me.

 3) Please do not just redistribute the binaries from my website and
 declare them official. They might contain unreleased fixes from git
 master and pull requests that are needed for my work and other packages.

 4) Numpy-MKL requires the Intel runtime DLLs (MKL is linked statically
 btw). I ship those with the installers and append the directory
 containing the DLLs to os.environ['PATH'] in numpy/__init__.py. This is
 a big no-no according to numpy developers. I don't agree. Anyway, those
 changes are not in the numpy source repositories.

 5) My numpy-MKL installers are Python distutils bdist_wininst
 installers. That means if Python was installed for all users, installing
 numpy-MKL on Windows 6.0 will prompt for UAC elevation. Another no-no?

I think that all these things should be possible to fix so that the
binary is acceptable
for the official NumPy binary.

How exactly do you build the binaries? I wasn't able to find the info at:

http://www.lfd.uci.edu/~gohlke/pythonlibs/

Do you have some scripts to do that? Do you use PowerShell? Or you do
it by hand by mouse and clicks in Visual Studio somehow? If I can
figure out how to do these builds, I'll be happy to figure out how to
automate it and then we can try to figure out a solution that works
for NumPy.

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Will numpy 1.7.0 final be binary compatible with the rc?

[Ondřej Čertík]

On Wed, Feb 6, 2013 at 2:37 AM, Peter Cock p.j.a.c...@googlemail.com wrote:
 On Wed, Feb 6, 2013 at 3:46 AM, Ondřej Čertík ondrej.cer...@gmail.com wrote:
 On Tue, Feb 5, 2013 at 12:22 PM, Ralf Gommers ralf.gomm...@gmail.com wrote:
 On Tue, Feb 5, 2013 at 3:01 PM, Peter Cock p.j.a.c...@googlemail.com
 wrote:

 Hello all,

 Will the numpy 1.7.0 'final' be binary compatible with the release
 candidate(s)? i.e. Would it be safe for me to release a Windows
 installer for a package using the NumPy C API compiled against
 the NumPy 1.7.0rc?


 Yes, that should be safe.

 Yes. I plan to release rc2 immediately once

 https://github.com/numpy/numpy/pull/2964

 is merged (e.g. I am hoping for today). The final should then be
 identical to rc2.

 Ondrej

 Great - in that case I'll wait a couple of days and use rc2 for this
 (just in case there is a subtle difference from rc1).

Awesome. The rc2 is out.

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Will numpy 1.7.0 final be binary compatible with the rc?

[Ondřej Čertík]

On Tue, Feb 5, 2013 at 12:22 PM, Ralf Gommers ralf.gomm...@gmail.com wrote:



 On Tue, Feb 5, 2013 at 3:01 PM, Peter Cock p.j.a.c...@googlemail.com
 wrote:

 Hello all,

 Will the numpy 1.7.0 'final' be binary compatible with the release
 candidate(s)? i.e. Would it be safe for me to release a Windows
 installer for a package using the NumPy C API compiled against
 the NumPy 1.7.0rc?


 Yes, that should be safe.

Yes. I plan to release rc2 immediately once

https://github.com/numpy/numpy/pull/2964

is merged (e.g. I am hoping for today). The final should then be
identical to rc2.

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Any plans for windows 64-bit installer for 1.7?

[Ondřej Čertík]

On Sun, Feb 3, 2013 at 2:57 AM, David Cournapeau courn...@gmail.com wrote:
 On Sun, Feb 3, 2013 at 12:28 AM,  josef.p...@gmail.com wrote:
 On Sat, Feb 2, 2013 at 6:14 PM, Matthew Brett matthew.br...@gmail.com 
 wrote:
 Hi,

 I see there is no Windows 64 bit installer for the 1.7 rc1.

 related:
 Is there any chance to get newer mingw or mingw-w64 support soonish?

 The problem has no solution until we can restrict support to windows 7
 and above. Otherwise, any acceptable solution would require user to be
 an admin.

The installer is built with this VM/scripts:

https://github.com/certik/numpy-vendor

currently the VM itself is 32 bit. I think that might be upgraded to 64bit,
and maybe it's possible to use 64 bit Wine:

http://wiki.winehq.org/Wine64

but then we would need to figure out how to use Mingw with 64 bits.

I would be very happy to accept patches to the above repository.

Alternatively, if the actual Windows 64bit machine would have to be used,
is there any way to automate the process? Would you compile it from command line
(cmd.exe), just like I do in Wine? I would much prefer if we can figure out
how to do this in Wine, so that the process can be automated and other people
can easily reproduce it.

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Any plans for windows 64-bit installer for 1.7?

[Ondřej Čertík]

On Mon, Feb 4, 2013 at 12:36 PM, Matthew Brett matthew.br...@gmail.com wrote:
 Hi,

 On Mon, Feb 4, 2013 at 12:27 PM, Ondřej Čertík ondrej.cer...@gmail.com 
 wrote:
 On Sun, Feb 3, 2013 at 2:57 AM, David Cournapeau courn...@gmail.com wrote:
 On Sun, Feb 3, 2013 at 12:28 AM,  josef.p...@gmail.com wrote:
 On Sat, Feb 2, 2013 at 6:14 PM, Matthew Brett matthew.br...@gmail.com 
 wrote:
 Hi,

 I see there is no Windows 64 bit installer for the 1.7 rc1.

 related:
 Is there any chance to get newer mingw or mingw-w64 support soonish?

 The problem has no solution until we can restrict support to windows 7
 and above. Otherwise, any acceptable solution would require user to be
 an admin.

 The installer is built with this VM/scripts:

 https://github.com/certik/numpy-vendor

 currently the VM itself is 32 bit. I think that might be upgraded to 64bit,
 and maybe it's possible to use 64 bit Wine:

 http://wiki.winehq.org/Wine64

 but then we would need to figure out how to use Mingw with 64 bits.

 I would be very happy to accept patches to the above repository.

 Alternatively, if the actual Windows 64bit machine would have to be used,
 is there any way to automate the process? Would you compile it from command 
 line
 (cmd.exe), just like I do in Wine? I would much prefer if we can figure out
 how to do this in Wine, so that the process can be automated and other people
 can easily reproduce it.

 I wonder whether getting ming64 to work on 64 bit Wine is too hard to
 get working before the release?  I often can't get 32-bit Wine
 working,

Yep, it took me over a week of work to figure out exactly how to get
32-bit Wine working
with mingw and numpy. However, the work is done and you can either run
my scripts
in the VM, or you can easily reproduce it yourself by consulting the scripts:

https://github.com/certik/numpy-vendor/blob/master/setup-wine.sh
https://github.com/certik/numpy-vendor/blob/master/fabfile.py

There are a lot of little tricks involved.

However, as Ralf says, apparently we would need to use MSVC + MKL anyway
on 64bits.

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Any plans for windows 64-bit installer for 1.7?

[Ondřej Čertík]

On Mon, Feb 4, 2013 at 3:49 PM, Christoph Gohlke cgoh...@uci.edu wrote:
 On 2/4/2013 12:59 PM, David Cournapeau wrote:
 On Mon, Feb 4, 2013 at 8:27 PM, Ondřej Čertík ondrej.cer...@gmail.com 
 wrote:
 On Sun, Feb 3, 2013 at 2:57 AM, David Cournapeau courn...@gmail.com wrote:
 On Sun, Feb 3, 2013 at 12:28 AM,  josef.p...@gmail.com wrote:
 On Sat, Feb 2, 2013 at 6:14 PM, Matthew Brett matthew.br...@gmail.com 
 wrote:
 Hi,

 I see there is no Windows 64 bit installer for the 1.7 rc1.

 related:
 Is there any chance to get newer mingw or mingw-w64 support soonish?

 The problem has no solution until we can restrict support to windows 7
 and above. Otherwise, any acceptable solution would require user to be
 an admin.

 The installer is built with this VM/scripts:

 I am not sure whether you're replying to my observation or just giving
 a status update: with mingw-w64 (or recent mingw), the built installer

I was just giving a general status, sorry about not being clear.

 will depend on several .dll (libgcc_s_sjil.dll) that we can't easily
 distribute. The only place we can realistically put them is in
 C:\Python$VERSION (or wherever python happens to be installed), and I
 think it is a very bad idea to install dll from NumPy there. In
 Windows 2008 and above, one can refer in .pyd where to look for dlls
 in another directory which is private to numpy.

Yes.


 David

 If I understand correctly the problem is distributing dependency/runtime
 DLLs with a package and ensuring the DLLs are found by Windows when the
 pyd extensions are imported?
 For numpy-MKL and other packages I include/install the extra DLLs in the
 package directories and, if necessary, (i) append the package directory
 to os.environ['PATH'] or (ii) pre-load the DLLs into the process using
 Ctypes, both early in the package's main __init__.py. No admin rights
 are required.

So that seems to be the only option. Is there any other solution?

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Travis failures with no errors

[Ondřej Čertík]

On Thu, Dec 20, 2012 at 6:32 PM, Charles R Harris
charlesr.har...@gmail.com wrote:


 On Thu, Dec 20, 2012 at 6:25 PM, Ondřej Čertík ondrej.cer...@gmail.com
 wrote:

 On Thu, Dec 13, 2012 at 4:39 PM, Ondřej Čertík ondrej.cer...@gmail.com
 wrote:
  Hi,
 
  I found these recent weird failures in Travis, but I can't find any
  problem with the log and all tests pass. Any ideas what is going on?
 
  https://travis-ci.org/numpy/numpy/jobs/3570123
  https://travis-ci.org/numpy/numpy/jobs/3539549
  https://travis-ci.org/numpy/numpy/jobs/3369629

 And here is another one:

 https://travis-ci.org/numpy/numpy/jobs/3768782


 Hmm, that is strange indeed. The first three are old, = 12 days, but the
 last is new, although the run time was getting up there. Might try running
 the last one again. I don't know if the is an easy way to do that.

And another one from 3 days ago:

https://travis-ci.org/numpy/numpy/jobs/4118113

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Which Python to use for Mac binaries

[Ondřej Čertík]

On Wed, Jan 9, 2013 at 11:21 PM, Ralf Gommers ralf.gomm...@gmail.com wrote:
 Sure, no problem. For the part that needs to be built on 10.6 that is.
 Vincent's box still has 10.5, right?

Yes.

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Which Python to use for Mac binaries

[Ondřej Čertík]

On Thu, Jan 10, 2013 at 9:26 AM, Chris Barker - NOAA Federal
chris.bar...@noaa.gov wrote:
 Ondřej, Vincent, and Ralf (and others..)

 Thank you so much for doing all this -- it's a great service to the
 MacPython community.

Chris, thank you for your help as well!

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Building Numpy 1.6.2 for Python 3.3 on Windows

[Ondřej Čertík]

On Thu, Jan 10, 2013 at 12:20 PM, klo klo...@gmail.com wrote:
 Actually, this isn't over. It builds fine, but when I try to import
 numpy I get error:

 
 ...

 Sorry for the noise, after re-reading tracelog, I realized that I accidentally
 removed c:\python33\libs\libpython33.a while removing previous non-working
 numpy build

Cool, I am glad to hear that 1.7.0rc1 works great. Thanks for letting us know.

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Which Python to use for Mac binaries

[Ondřej Čertík]

On Tue, Jan 8, 2013 at 8:45 AM, Chris Barker - NOAA Federal
chris.bar...@noaa.gov wrote:
 On Mon, Jan 7, 2013 at 10:23 PM, Ondřej Čertík ondrej.cer...@gmail.com 
 wrote:
 http://www.commandlinefu.com/commands/view/2031/install-an-mpkg-from-the-command-line-on-osx

 This requires root access. Without sudo, I get:

 $ installer -pkg /Volumes/Python\ 2.7.3/Python.mpkg/ -target ondrej
 installer: This package requires authentication to install.

 and since I don't have root access, it doesn't work.

 So one way around it would be to install python from source, that
 shouldn't require root access.

 hmm -- this all may be a trick -- both the *.mpkg and the standard
 build put everything in /Library/Frameworks/Python -- which is where
 it belongs. Bu tif you need root access to write there, then there is
 a problem. I'm sure a non-root build could put everything in the
 users' home directory, then packages built against that would have
 their paths messed up.

Right.


 What's odd is that I'm pretty sure I've been able to point+click
 install those without sudo...(I could recall incorrectly).

 This would be a good question for the pythonmac list -- low traffic,
 but there are some very smart and helpful folks there:

 http://mail.python.org/mailman/listinfo/pythonmac-sig


 But I am not currently sure what to do with it. The Python.mpkg
 directory seems to contain the sources.

 It should be possible to unpack a mpkg by hand, but it contains both
 the contents, and various instal scripts, so that seems like a really
 ugly solution.

Yep.

In the meantime, the hard drive on Vincent's box failed, so he
reinstalled the box completely.
Also he explained to me a lot of Mac things over the phone, so I think
I now understand what is going on with the dmg.

As such, I have updated my instructions in my release helper repo:

https://github.com/certik/numpy-vendor

by the following paragraph:


First prepare the Mac build box as follows:

* Install Python 2.5, 2.6, 2.7 from python.org using the dmg disk image
* Install setuptools and bdist_mpkg into all these Pythons
* Install Paver into the default Python

Tip: Add the /Library/Frameworks/Python.framework directory into git
and commit after each installation of any package or Python. That way
you can easily remove temporary installations.


And you need sudo access to do those. If your user is an admin, then
it can do it, otherwise it can't.
So one can only use a Mac, which has the above setup installed. With
that, my Fabfile can then do the rest.

So I just built the following binaries:

numpy-1.7.0rc1-py2.5-python.org-macosx10.3.dmg
numpy-1.7.0rc1-py2.6-python.org-macosx10.3.dmg
numpy-1.7.0rc1-py2.7-python.org-macosx10.3.dmg

and uploaded to:

https://sourceforge.net/projects/numpy/files/NumPy/1.7.0rc1/


So I think we are all set here. Ralf, would you be willing to build
the final binary on 10.6? I don't think you have to do it for this
rc1, but I am going to release rc2 now and for that it would be nice
to have it.

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Which Python to use for Mac binaries

[Ondřej Čertík]

On Mon, Jan 7, 2013 at 11:36 PM, Ralf Gommers ralf.gomm...@gmail.com wrote:



 On Tue, Jan 8, 2013 at 3:12 AM, Ondřej Čertík ondrej.cer...@gmail.com
 wrote:

 On Sun, Jan 6, 2013 at 2:04 AM, Ralf Gommers ralf.gomm...@gmail.com
 wrote:
 
 
 
  On Sun, Jan 6, 2013 at 3:21 AM, Ondřej Čertík ondrej.cer...@gmail.com
  wrote:
 
  Hi,
 
  Currently the NumPy binaries are built using the pavement.py script,
  which uses the following Pythons:
 
  MPKG_PYTHON = {
  2.5:
  [/Library/Frameworks/Python.framework/Versions/2.5/bin/python],
  2.6:
  [/Library/Frameworks/Python.framework/Versions/2.6/bin/python],
  2.7:
  [/Library/Frameworks/Python.framework/Versions/2.7/bin/python],
  3.1:
  [/Library/Frameworks/Python.framework/Versions/3.1/bin/python3],
  3.2:
  [/Library/Frameworks/Python.framework/Versions/3.2/bin/python3],
  3.3:
  [/Library/Frameworks/Python.framework/Versions/3.3/bin/python3],
  }
 
  So for example I can easily create the 2.6 binary if that Python is
  pre-installed on the Mac box that I am using.
  On one of the Mac boxes that I am using, the 2.7 is missing, so are
  3.1, 3.2 and 3.3. So I was thinking
  of updating my Fabric fab file to automatically install all Pythons
  from source and build against that, just like I do for Wine.
 
  Which exact Python do we need to use on Mac? Do we need to use the
  binary installer from python.org?
 
 
  Yes, the one from python.org.
 
 
  Or can I install it from source? Finally, for which Python versions
  should we provide binary installers for Mac?
  For reference, the 1.6.2 had installers for 2.5, 2.6 and 2.7 only for
  OS X 10.3. There is only 2.7 version for OS X 10.6.
 
 
  The provided installers and naming scheme should match what's done for
  Python itself on python.org.
 
  The 10.3 installers for 2.5, 2.6 and 2.7 should be compiled on OS X
  10.5.
  This is kind of hard to come by these days, but Vincent Davis maintains
  a
  build machine for numpy and scipy. That's already set up correctly, so
  all
  you have to do is connect to it via ssh, check out v.17.0 in
  ~/Code/numpy,
  check in release.sh that the section for OS X 10.6 is disabled and for
  10.5
  enabled and run it.
 
  OS X 10.6 broke support for previous versions in some subtle ways, so
  even
  when using the 10.4 SDK numpy compiled on 10.6 won't run on 10.5. As
  long as
  we're supporting 10.5 you therefore need to compile on it.
 
  The 10.7 -- 10.6 support hasn't been checked, but I wouldn't trust it.
  I
  have a 10.6 machine, so I can compile those binaries if needed.
 
 
  Also, what is the meaning of the following piece of code in
  pavement.py:
 
  def _build_mpkg(pyver):
  # account for differences between Python 2.7.1 versions from
  python.org
  if os.environ.get('MACOSX_DEPLOYMENT_TARGET', None) == 10.6:
  ldflags = -undefined dynamic_lookup -bundle -arch i386 -arch
  x86_64 -Wl,-search_paths_first
  else:
  ldflags = -undefined dynamic_lookup -bundle -arch i386 -arch
  ppc -Wl,-search_paths_first
  ldflags +=  -L%s % os.path.join(os.path.dirname(__file__),
  build)
 
 
  The 10.6 binaries support only Intel Macs, both 32-bit and 64-bit. The
  10.3
  binaries support PPC Macs and 32-bit Intel. That's what the above does.
  Note
  that we simply follow the choice made by the Python release managers
  here.
 
 
  if pyver == 2.5:
  sh(CC=gcc-4.0 LDFLAGS='%s' %s setupegg.py bdist_mpkg %
  (ldflags,  .join(MPKG_PYTHON[pyver])))
  else:
  sh(LDFLAGS='%s' %s setupegg.py bdist_mpkg % (ldflags, 
  .join(MPKG_PYTHON[pyver])))
 
 
  This is necessary because in Python 2.5, distutils asks for gcc
  instead of
  gcc-4.0, so you may get the wrong one without CC=gcc-4.0. From Python
  2.6
  on this was fixed.
 
 
  In particular, the last line gets executed and it then fails with:
 
  paver dmg -p 2.6
  --- pavement.dmg
  --- pavement.clean
  LDFLAGS='-undefined dynamic_lookup -bundle -arch i386 -arch ppc
  -Wl,-search_paths_first -Lbuild'
  /Library/Frameworks/Python.framework/Versions/2.6/bin/python
  setupegg.py bdist_mpkg
  Traceback (most recent call last):
File setupegg.py, line 17, in module
  from setuptools import setup
  ImportError: No module named setuptools
 
 
  The reason is (I think) that if the Python binary is called explicitly
  with /Library/Frameworks/Python.framework/Versions/2.6/bin/python,
  then the paths are not setup properly in virtualenv, and thus
  setuptools (which is only installed in virtualenv, but not in system
  Python) fails to import. The solution is to simply apply this patch:
 
 
  Avoid using system Python for anything. The first thing to do on any new
  OS
  X system is install Python some other way, preferably from python.org.
 
 
  diff --git a/pavement.py b/pavement.py
  index e693016..0c637f8 100644
  --- a/pavement.py
  +++ b/pavement.py
  @@ -449,7 +449,7 @@ def _build_mpkg(pyver):
   if pyver == 2.5:
   sh

Re: [Numpy-discussion] Which Python to use for Mac binaries

[Ondřej Čertík]

On Sun, Jan 6, 2013 at 2:40 PM, Chris Barker - NOAA Federal
chris.bar...@noaa.gov wrote:
 On Sun, Jan 6, 2013 at 2:04 AM, Ralf Gommers ralf.gomm...@gmail.com wrote:
 Which exact Python do we need to use on Mac? Do we need to use the
 binary installer from python.org?

 Yes, the one from python.org.

 Or can I install it from source?

 you could install from source using the same method that the
 python.org binaries are built -- there is a script with the source to
 do that, though I'm not sure what the point of that would be.

Is it possible to install the dmg images without root access from the
command line?

I know how to access the contents:

$ hdiutil attach python-2.7.3-macosx10.6.dmg
$ ls /Volumes/Python\ 2.7.3/
Build.txt   License.txt Python.mpkg ReadMe.txt

But I am not currently sure what to do with it. The Python.mpkg
directory seems to contain the sources.


I have access to Vincent's computer, as suggested by Ralf and it is
already setup, so I am using it.
But I am not able (so far) to replicate the setup there so that I can
create the binaries on any other Mac
computer, which makes me feel really uneasy.

By replicating the setup, at least once (preferably automated) would
make me understand things much better.
If possible, I would prefer to just use a command line (ssh) to do all
that. (So that's maybe building from source
is the only option.)

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Which Python to use for Mac binaries

[Ondřej Čertík]

On Sun, Jan 6, 2013 at 2:04 AM, Ralf Gommers ralf.gomm...@gmail.com wrote:



 On Sun, Jan 6, 2013 at 3:21 AM, Ondřej Čertík ondrej.cer...@gmail.com
 wrote:

 Hi,

 Currently the NumPy binaries are built using the pavement.py script,
 which uses the following Pythons:

 MPKG_PYTHON = {
 2.5:
 [/Library/Frameworks/Python.framework/Versions/2.5/bin/python],
 2.6:
 [/Library/Frameworks/Python.framework/Versions/2.6/bin/python],
 2.7:
 [/Library/Frameworks/Python.framework/Versions/2.7/bin/python],
 3.1:
 [/Library/Frameworks/Python.framework/Versions/3.1/bin/python3],
 3.2:
 [/Library/Frameworks/Python.framework/Versions/3.2/bin/python3],
 3.3:
 [/Library/Frameworks/Python.framework/Versions/3.3/bin/python3],
 }

 So for example I can easily create the 2.6 binary if that Python is
 pre-installed on the Mac box that I am using.
 On one of the Mac boxes that I am using, the 2.7 is missing, so are
 3.1, 3.2 and 3.3. So I was thinking
 of updating my Fabric fab file to automatically install all Pythons
 from source and build against that, just like I do for Wine.

 Which exact Python do we need to use on Mac? Do we need to use the
 binary installer from python.org?


 Yes, the one from python.org.


 Or can I install it from source? Finally, for which Python versions
 should we provide binary installers for Mac?
 For reference, the 1.6.2 had installers for 2.5, 2.6 and 2.7 only for
 OS X 10.3. There is only 2.7 version for OS X 10.6.


 The provided installers and naming scheme should match what's done for
 Python itself on python.org.

 The 10.3 installers for 2.5, 2.6 and 2.7 should be compiled on OS X 10.5.
 This is kind of hard to come by these days, but Vincent Davis maintains a
 build machine for numpy and scipy. That's already set up correctly, so all
 you have to do is connect to it via ssh, check out v.17.0 in ~/Code/numpy,
 check in release.sh that the section for OS X 10.6 is disabled and for 10.5
 enabled and run it.

 OS X 10.6 broke support for previous versions in some subtle ways, so even
 when using the 10.4 SDK numpy compiled on 10.6 won't run on 10.5. As long as
 we're supporting 10.5 you therefore need to compile on it.

 The 10.7 -- 10.6 support hasn't been checked, but I wouldn't trust it. I
 have a 10.6 machine, so I can compile those binaries if needed.


 Also, what is the meaning of the following piece of code in pavement.py:

 def _build_mpkg(pyver):
 # account for differences between Python 2.7.1 versions from
 python.org
 if os.environ.get('MACOSX_DEPLOYMENT_TARGET', None) == 10.6:
 ldflags = -undefined dynamic_lookup -bundle -arch i386 -arch
 x86_64 -Wl,-search_paths_first
 else:
 ldflags = -undefined dynamic_lookup -bundle -arch i386 -arch
 ppc -Wl,-search_paths_first
 ldflags +=  -L%s % os.path.join(os.path.dirname(__file__), build)


 The 10.6 binaries support only Intel Macs, both 32-bit and 64-bit. The 10.3
 binaries support PPC Macs and 32-bit Intel. That's what the above does. Note
 that we simply follow the choice made by the Python release managers here.


 if pyver == 2.5:
 sh(CC=gcc-4.0 LDFLAGS='%s' %s setupegg.py bdist_mpkg %
 (ldflags,  .join(MPKG_PYTHON[pyver])))
 else:
 sh(LDFLAGS='%s' %s setupegg.py bdist_mpkg % (ldflags, 
 .join(MPKG_PYTHON[pyver])))


 This is necessary because in Python 2.5, distutils asks for gcc instead of
 gcc-4.0, so you may get the wrong one without CC=gcc-4.0. From Python 2.6
 on this was fixed.


 In particular, the last line gets executed and it then fails with:

 paver dmg -p 2.6
 --- pavement.dmg
 --- pavement.clean
 LDFLAGS='-undefined dynamic_lookup -bundle -arch i386 -arch ppc
 -Wl,-search_paths_first -Lbuild'
 /Library/Frameworks/Python.framework/Versions/2.6/bin/python
 setupegg.py bdist_mpkg
 Traceback (most recent call last):
   File setupegg.py, line 17, in module
 from setuptools import setup
 ImportError: No module named setuptools


 The reason is (I think) that if the Python binary is called explicitly
 with /Library/Frameworks/Python.framework/Versions/2.6/bin/python,
 then the paths are not setup properly in virtualenv, and thus
 setuptools (which is only installed in virtualenv, but not in system
 Python) fails to import. The solution is to simply apply this patch:


 Avoid using system Python for anything. The first thing to do on any new OS
 X system is install Python some other way, preferably from python.org.


 diff --git a/pavement.py b/pavement.py
 index e693016..0c637f8 100644
 --- a/pavement.py
 +++ b/pavement.py
 @@ -449,7 +449,7 @@ def _build_mpkg(pyver):
  if pyver == 2.5:
  sh(CC=gcc-4.0 LDFLAGS='%s' %s setupegg.py bdist_mpkg %
 (ldflags,  .join(MPKG_PYTHON[pyver])))
  else:
 -sh(LDFLAGS='%s' %s setupegg.py bdist_mpkg % (ldflags, 
 .join(MPKG_PYTHON[pyver])))
 +sh(python setupegg.py bdist_mpkg)


 This doesn't work unless using virtualenvs, you're just throwing away

Re: [Numpy-discussion] Which Python to use for Mac binaries

[Ondřej Čertík]

On Mon, Jan 7, 2013 at 8:41 PM, Chris Barker - NOAA Federal
chris.bar...@noaa.gov wrote:
 On Mon, Jan 7, 2013 at 6:09 PM, Ondřej Čertík ondrej.cer...@gmail.com wrote:
 Is it possible to install the dmg images without root access from the
 command line?

 I've never tried, but it looks like you can:

 http://www.commandlinefu.com/commands/view/2031/install-an-mpkg-from-the-command-line-on-osx

This requires root access. Without sudo, I get:

$ installer -pkg /Volumes/Python\ 2.7.3/Python.mpkg/ -target ondrej
installer: This package requires authentication to install.

and since I don't have root access, it doesn't work.

So one way around it would be to install python from source, that
shouldn't require root access.



 But I am not currently sure what to do with it. The Python.mpkg
 directory seems to contain the sources.

 yup -- that's where everything is. the installer command should be
 able to unpack it.

Ok.


 By replicating the setup, at least once (preferably automated) would
 make me understand things much better.
 If possible, I would prefer to just use a command line (ssh) to do all
 that. (So that's maybe building from source
 is the only option.)

 If you ndo need to build from source, see this message for a bit more info:

 http://mail.python.org/pipermail/pythonmac-sig/2012-October/023742.html

 there are a few prerequisites you need to install first...

 Either way, you should be able to build a start-to-finish build script.

Yes, that would be my goal eventually. Without root access.
But right now, I am not even sure it's possible.

So for now I'll simply use already pre-configured box.

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Which Python to use for Mac binaries

[Ondřej Čertík]

Hi,

Currently the NumPy binaries are built using the pavement.py script,
which uses the following Pythons:

MPKG_PYTHON = {
2.5: [/Library/Frameworks/Python.framework/Versions/2.5/bin/python],
2.6: [/Library/Frameworks/Python.framework/Versions/2.6/bin/python],
2.7: [/Library/Frameworks/Python.framework/Versions/2.7/bin/python],
3.1: 
[/Library/Frameworks/Python.framework/Versions/3.1/bin/python3],
3.2: 
[/Library/Frameworks/Python.framework/Versions/3.2/bin/python3],
3.3: 
[/Library/Frameworks/Python.framework/Versions/3.3/bin/python3],
}

So for example I can easily create the 2.6 binary if that Python is
pre-installed on the Mac box that I am using.
On one of the Mac boxes that I am using, the 2.7 is missing, so are
3.1, 3.2 and 3.3. So I was thinking
of updating my Fabric fab file to automatically install all Pythons
from source and build against that, just like
I do for Wine.

Which exact Python do we need to use on Mac? Do we need to use the
binary installer from python.org?
Or can I install it from source? Finally, for which Python versions
should we provide binary installers for Mac?
For reference, the 1.6.2 had installers for 2.5, 2.6 and 2.7 only for
OS X 10.3. There is only 2.7 version for OS X 10.6.


Also, what is the meaning of the following piece of code in pavement.py:

def _build_mpkg(pyver):
# account for differences between Python 2.7.1 versions from python.org
if os.environ.get('MACOSX_DEPLOYMENT_TARGET', None) == 10.6:
ldflags = -undefined dynamic_lookup -bundle -arch i386 -arch
x86_64 -Wl,-search_paths_first
else:
ldflags = -undefined dynamic_lookup -bundle -arch i386 -arch
ppc -Wl,-search_paths_first
ldflags +=  -L%s % os.path.join(os.path.dirname(__file__), build)

if pyver == 2.5:
sh(CC=gcc-4.0 LDFLAGS='%s' %s setupegg.py bdist_mpkg %
(ldflags,  .join(MPKG_PYTHON[pyver])))
else:
sh(LDFLAGS='%s' %s setupegg.py bdist_mpkg % (ldflags, 
.join(MPKG_PYTHON[pyver])))


In particular, the last line gets executed and it then fails with:

paver dmg -p 2.6
--- pavement.dmg
--- pavement.clean
LDFLAGS='-undefined dynamic_lookup -bundle -arch i386 -arch ppc
-Wl,-search_paths_first -Lbuild'
/Library/Frameworks/Python.framework/Versions/2.6/bin/python
setupegg.py bdist_mpkg
Traceback (most recent call last):
  File setupegg.py, line 17, in module
from setuptools import setup
ImportError: No module named setuptools


The reason is (I think) that if the Python binary is called explicitly
with /Library/Frameworks/Python.framework/Versions/2.6/bin/python,
then the paths are not setup properly in virtualenv, and thus
setuptools (which is only installed in virtualenv, but not in system
Python) fails to import. The solution is to simply apply this patch:

diff --git a/pavement.py b/pavement.py
index e693016..0c637f8 100644
--- a/pavement.py
+++ b/pavement.py
@@ -449,7 +449,7 @@ def _build_mpkg(pyver):
 if pyver == 2.5:
 sh(CC=gcc-4.0 LDFLAGS='%s' %s setupegg.py bdist_mpkg %
(ldflags,  .join(MPKG_PYTHON[pyver])))
 else:
-sh(LDFLAGS='%s' %s setupegg.py bdist_mpkg % (ldflags, 
.join(MPKG_PYTHON[pyver])))
+sh(python setupegg.py bdist_mpkg)

 @task
 def simple_dmg():


and then things work. So an obvious question is --- why do we need to
fiddle with LDFLAGS and paths to the exact Python version? Here is a
proposed simpler version of the build_mpkg() function:

def _build_mpkg(pyver):
sh(python setupegg.py bdist_mpkg)

Thanks for any tips.

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] test failures when embedded (in matlab)

[Ondřej Čertík]

On Thu, Jan 3, 2013 at 7:54 AM, Robin robi...@gmail.com wrote:
 Hi All,

 When using Numpy from an embedded Python (Python embedded in a Matlab
 mex function) I get a lot of test failures (see attached log).

 I am using CentOS 6.3, distribution packaged Python (2.6) and Numpy
 (1.4.1). Running numpy tests from a normal Python interpreter results
 in no errors or failures.

 Most of the failures look to be to do with errors calling Fortran
 functions - is it possible there is some linking / ABI problem? Would
 there be any way to overcome it?

 I get similar errors using EPD 7.3.

 Any advice appreciated,

In your log I can see failures of the type:

==
ERROR: test_cdouble (test_linalg.TestDet)
--
Traceback (most recent call last):
  File /usr/lib64/python2.6/site-packages/numpy/linalg/tests/test_linalg.py,
line 44, in test_cdouble
self.do(a, b)
  File /usr/lib64/python2.6/site-packages/numpy/linalg/tests/test_linalg.py,
line 129, in do
d = linalg.det(a)
  File /usr/lib64/python2.6/site-packages/numpy/linalg/linalg.py,
line 1503, in det
raise TypeError, Illegal input to Fortran routine
TypeError: Illegal input to Fortran routine


So I can only offer a general advice, that I learned while fixing
release critical bugs in NumPy:
I would look into the source file numpy/linalg/linalg.py,  line 1503
and start debugging
to figure out why the TypeError is raised. Which exact numpy do you use?

In the latest master, the line numbers are different and the det()
routine seems to be reworked. But in general,
you can see there a code like this:

results = lapack_routine(n, n, a, n, pivots, 0)
info = results['info']
if (info  0):
raise TypeError(Illegal input to Fortran routine)

so that typically means that some wrong argument are being passed to
the Lapack routine. Try to print the info variable and then lookup
the Lapack documentation, it should say more (e.g. which exact
argument is wrong). Then you can go from there, e.g. I would put some
debug print statements into the code which gets called in
lapack_routine(), i.e. is it lapack_lite from NumPy, or some other
Lapack implementation? And so on.

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] ANN: NumPy 1.7.0rc1 release

[Ondřej Čertík]

Hi Neal,

On Sat, Dec 29, 2012 at 9:35 AM, Neal Becker ndbeck...@gmail.com wrote:
 Are release notes available?

Yes. There are here:

http://sourceforge.net/projects/numpy/files/NumPy/1.7.0rc1/

if you slide the page down a little bit (sf.net just shows the file README.txt).
I am posting them here as well for reference (I forgot to do it in my
release email).

Ondrej

---

=
NumPy 1.7.0 Release Notes
=

This release includes several new features as well as numerous bug fixes and
refactorings. It supports Python 2.4 - 2.7 and 3.1 - 3.3 and is the last
release that supports Python 2.4 - 2.5.

Highlights
==

* ``where=`` parameter to ufuncs (allows the use of boolean arrays to choose
  where a computation should be done)
* ``vectorize`` improvements (added 'excluded' and 'cache' keyword, general
  cleanup and bug fixes)
* ``numpy.random.choice`` (random sample generating function)


Compatibility notes
===

In a future version of numpy, the functions np.diag, np.diagonal, and the
diagonal method of ndarrays will return a view onto the original array,
instead of producing a copy as they do now. This makes a difference if you
write to the array returned by any of these functions. To facilitate this
transition, numpy 1.7 produces a FutureWarning if it detects that you may
be attempting to write to such an array. See the documentation for
np.diagonal for details.

Similar to np.diagonal above, in a future version of numpy, indexing a
record array by a list of field names will return a view onto the original
array, instead of producing a copy as they do now. As with np.diagonal,
numpy 1.7 produces a FutureWarning if it detects that you may be attempting
to write to such an array. See the documentation for array indexing for
details.

In a future version of numpy, the default casting rule for UFunc out=
parameters will be changed from 'unsafe' to 'same_kind'. (This also applies
to in-place operations like a += b, which is equivalent to np.add(a, b,
out=a).) Most usages which violate the 'same_kind' rule are likely bugs, so
this change may expose previously undetected errors in projects that depend
on NumPy. In this version of numpy, such usages will continue to succeed,
but will raise a DeprecationWarning.

Full-array boolean indexing has been optimized to use a different,
optimized code path.   This code path should produce the same results,
but any feedback about changes to your code would be appreciated.

Attempting to write to a read-only array (one with ``arr.flags.writeable``
set to ``False``) used to raise either a RuntimeError, ValueError, or
TypeError inconsistently, depending on which code path was taken. It now
consistently raises a ValueError.

The ufunc.reduce functions evaluate some reductions in a different order
than in previous versions of NumPy, generally providing higher performance.
Because of the nature of floating-point arithmetic, this may subtly change
some results, just as linking NumPy to a different BLAS implementations
such as MKL can.

If upgrading from 1.5, then generally in 1.6 and 1.7 there have been
substantial code added and some code paths altered, particularly in the
areas of type resolution and buffered iteration over universal functions.
This might have an impact on your code particularly if you relied on
accidental behavior in the past.

New features


Reduction UFuncs Generalize axis= Parameter
---

Any ufunc.reduce function call, as well as other reductions like sum, prod,
any, all, max and min support the ability to choose a subset of the axes to
reduce over. Previously, one could say axis=None to mean all the axes or
axis=# to pick a single axis.  Now, one can also say axis=(#,#) to pick a
list of axes for reduction.

Reduction UFuncs New keepdims= Parameter


There is a new keepdims= parameter, which if set to True, doesn't throw
away the reduction axes but instead sets them to have size one.  When this
option is set, the reduction result will broadcast correctly to the
original operand which was reduced.

Datetime support


.. note:: The datetime API is *experimental* in 1.7.0, and may undergo changes
   in future versions of NumPy.

There have been a lot of fixes and enhancements to datetime64 compared
to NumPy 1.6:

* the parser is quite strict about only accepting ISO 8601 dates, with a few
  convenience extensions
* converts between units correctly
* datetime arithmetic works correctly
* business day functionality (allows the datetime to be used in contexts where
  only certain days of the week are valid)

The notes in `doc/source/reference/arrays.datetime.rst
https://github.com/numpy/numpy/blob/maintenance/1.7.x/doc/source/reference/arrays.datetime.rst`_
(also available in the online docs at `arrays.datetime.html

Re: [Numpy-discussion] ANN: NumPy 1.7.0rc1 release

[Ondřej Čertík]

Hi Christoph,

On Sat, Dec 29, 2012 at 5:46 AM, Christoph Gohlke cgoh...@uci.edu wrote:
 Looks good so far.

 I tested numpy-MKL-1.7.0rc1.win-amd64-py2.7 with some packages that were
 compiled with numpy 1.6.x
 http://www.lfd.uci.edu/~gohlke/pythonlibs/tests/20121228-win-amd64-py2.7-numpy-1.7.0rc1/.
 There are a few additional test failures in bottleneck and Cython, but
 they don't look serious.

 The rc works well on Python 3.3 too
 http://www.lfd.uci.edu/~gohlke/pythonlibs/tests/20121229-win-amd64-py3.3/.

Thanks! I created an issue for it here:

https://github.com/numpy/numpy/issues/2870

Ondrej

P.S. Would you mind adding your name to your github profile
(https://github.com/cgohlke) please? I was trying to figure out
your full name so that I could thank you in the release email, but I
could only find your handle cgohlke both at github and in the commit
history. My apologies for that. Now I'll remember to google it in my
gmail. :) But if you could also update it at github, that'd be the
easiest. I can see that in master we have an updated .mailmap which
fixes precisely this issue. I should have used it -- I was running it
on the release branch which does not have it yet.
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] ANN: NumPy 1.7.0rc1 release

[Ondřej Čertík]

Hi,

I'm pleased to announce the availability of the first release candidate of
NumPy 1.7.0rc1.

Sources and binary installers can be found at
https://sourceforge.net/projects/numpy/files/NumPy/1.7.0rc1/

We have fixed all issues known to us since the 1.7.0b2 release.
The only remaining issue is a documentation improvement:

https://github.com/numpy/numpy/issues/561

Please test this release and report any issues on the numpy-discussion
mailing list. If there are no more problems, we'll release the final
version soon. I'll wait at least a week and please write me an email
if you need more time for testing.

I would like to thank Sebastian Berg, Ralf Gommers, Han Genuit,
Nathaniel J. Smith, Jay Bourque, Gael Varoquaux, Mark Wiebe,
Matthew Brett, Skipper Seabold, Peter Cock, Charles Harris, Frederic,
Gabriel, Luis Pedro Coelho, Pauli Virtanen, Travis E. Oliphant
and cgohlke for sending patches and fixes for this release since
1.7.0b2.

Cheers,
Ondrej

P.S. Source code is uploaded to sourceforge, and I'll upload the
rest of the Windows and Mac binaries in a few hours as they finish building.
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Many failing doctests - release blocker? Enable for default test runs?

[Ondřej Čertík]

Hi Matthew,

On Sat, Dec 22, 2012 at 9:40 PM, Matthew Brett matthew.br...@gmail.com wrote:
 Hi,

 I noticed that enabling the doctests on the 1.7.x maintenance branch
 caused lots and lots of doctest failures.

 (np-devel)[mb312@blair ~/dev_trees]$ python -c 'import numpy as np;
 np.test(doctests=True)'
 1.7.0rc1.dev-1e8fcdf
 Running unit tests and doctests for numpy
 NumPy version 1.7.0rc1.dev-1e8fcdf
 NumPy is installed in
 /Users/mb312/.virtualenvs/np-devel/lib/python2.6/site-packages/numpy
 Python version 2.6.6 (r266:84374, Aug 31 2010, 11:00:51) [GCC 4.0.1
 (Apple Inc. build 5493)]
 nose version 1.1.2
 ...
 Ran 3839 tests in 59.928s

 FAILED (KNOWNFAIL=4, SKIP=4, errors=23, failures=175)

 The doctests also throw up somewhere round 10 matplotlib plots, so
 presumably those would fail as well on a machine without a display
 without forcing the import of an 'Agg' backend or similar.

 I have never checked the doctests on Python 3.  Has anyone run those recently?

 For the projects I work on most, we enable doctests for the default
 test run - as in 'doctests=True' by default in the numpy testing
 machinery.   Do ya'll see any disadvantage in doing that for numpy?

 In case someone gets to this before I do, we've also got some logic
 for doing conditional skips of doctests when optional packages are not
 available such as matplotlib, inspired by something similar in
 IPython:

 https://github.com/nipy/nipy/blob/master/nipy/testing/doctester.py#L193

 If Christmas allows I'll send a pull request with something like that
 in the next few days.

Thanks for pointing this out. I think in the long term, we should definitely
run doctests as part of the test suite on Travis-CI. Because what use
is a doctest if it doesn't work?

Matthew, do you know if doctests fail for the 1.6 release as well?

I am swamped with other bugs for the 1.7 release and since I assume
they also fail for 1.6, I want to get the release out as soon as we fix our
current issues.

However, I think it's a good idea to run doctests automatically on Travis,
once they are all fixed.

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] help with f2py

[Ondřej Čertík]

On Sun, Dec 23, 2012 at 11:48 AM, Aronne Merrelli
aronne.merre...@gmail.com wrote:
 Hi,

 I'm trying to run f2py and running into some trouble. Starting from
 http://www.scipy.org/Cookbook/F2Py, and the very simple 'Wrapping Hermite
 Polynomial' example, I can get the pyf file created with no issues. The
 system I am using is RedHat linux, and has several Fortran compilers:

 $ f2py -c --help-fcompiler
 snip
 Fortran compilers found:
   --fcompiler=g95  G95 Fortran Compiler (0.92)
   --fcompiler=gnu  GNU Fortran 77 compiler (3.4.6)
   --fcompiler=gnu95GNU Fortran 95 compiler (4.1.2)
   --fcompiler=intelem  Intel Fortran Compiler for 64-bit apps (11.1)

 All of these will successfully create the .so file except for g95, but when
 I try to import into python I get this ImportError for any of the other
 three compilers:

 In [5]: import hermite
 ImportError: ./hermite.so: undefined symbol: c06ebf_

 If I look at the shared object I find that symbol here:

 $ nm hermite.so
 snip
 43c0 T array_from_pyobj
  U c06eaf_
  U c06ebf_

This U here means undefined (see man nm). I don't know if this
symbol is something that f2py introduces, or if it is present in the
original Fortran sources. One way to know for sure is to examine all
.f90 and .c files generated by f2py and search for this symbol and
make sure that these subroutines/functions are linked in.

 And that about hits my limit of compiler knowledge, as I am pretty much a
 novice with these things. Any ideas on what is going wrong here, or
 suggestions of things to try?

I personally wrap Fortran like any other C code using the
iso_c_binding Fortran module. I use Cython, but you can also use
ctypes or any other method. See here:

http://fortran90.org/src/best-practices.html#interfacing-with-c
http://fortran90.org/src/best-practices.html#interfacing-with-python

That way it is easy to see what is going on under the hood.

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Travis-CI stopped supporting Python 3.1, but added 3.3

[Ondřej Čertík]

On Fri, Dec 21, 2012 at 12:05 PM, Nathaniel Smith n...@pobox.com wrote:
 On Fri, Dec 21, 2012 at 1:23 AM, Ondřej Čertík ondrej.cer...@gmail.com 
 wrote:
 Hi,

 I noticed that the 3.1 tests are now failing. After clarification with
 the Travis guys:

 https://groups.google.com/d/topic/travis-ci/02iRu6kmwY8/discussion

 I've submitted a fix to our .travis.yml (and backported to 1.7):

 https://github.com/numpy/numpy/pull/2850
 https://github.com/numpy/numpy/pull/2851

 In case you were wondering. Do we need to support Python 3.1?
 We could in principle test 3.1 just like we test 2.4. I don't know if
 it is worth the pain.

 It probably isn't much pain, since Python is easy to compile, and I
 don't think we've been running into many cases where supporting 3.1
 required workarounds yet? (As compared to 2.4, where we get
 compatibility-breaking patches constantly.)

Yes, I was going to suggest that it really is not a big deal to support 3.1,
as far as I see it by my experience with Travis tests on numpy PRs
in the last half a year.

We can compile it like we do for 2.4, or we can even provide a prebuild
binary and just install it in the Travis VM each time.

So I'll try to provide a PR which implements testing of 3.1, unless
someone beats me to it.

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Travis-CI stopped supporting Python 3.1, but added 3.3

[Ondřej Čertík]

Hi,

I noticed that the 3.1 tests are now failing. After clarification with
the Travis guys:

https://groups.google.com/d/topic/travis-ci/02iRu6kmwY8/discussion

I've submitted a fix to our .travis.yml (and backported to 1.7):

https://github.com/numpy/numpy/pull/2850
https://github.com/numpy/numpy/pull/2851

In case you were wondering. Do we need to support Python 3.1?
We could in principle test 3.1 just like we test 2.4. I don't know if
it is worth the pain.

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Travis failures with no errors

[Ondřej Čertík]

On Thu, Dec 13, 2012 at 4:39 PM, Ondřej Čertík ondrej.cer...@gmail.com wrote:
 Hi,

 I found these recent weird failures in Travis, but I can't find any
 problem with the log and all tests pass. Any ideas what is going on?

 https://travis-ci.org/numpy/numpy/jobs/3570123
 https://travis-ci.org/numpy/numpy/jobs/3539549
 https://travis-ci.org/numpy/numpy/jobs/3369629

And here is another one:

https://travis-ci.org/numpy/numpy/jobs/3768782

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] DARPA funding for Blaze and passing the NumPy torch

[Ondřej Čertík]

Hi Travis,

On Sun, Dec 16, 2012 at 10:07 PM, Travis Oliphant tra...@continuum.io wrote:
 Hello all,

 There is a lot happening in my life right now and I am spread quite thin 
 among the various projects that I take an interest in. In particular, I 
 am thrilled to publicly announce on this list that Continuum Analytics has 
 received DARPA funding (to the tune of at least $3 million) for Blaze, Numba, 
 and Bokeh which we are writing to take NumPy, SciPy, and visualization into 
 the domain of very large data sets.This is part of the XDATA program, and 
 I will be taking an active role in it.You can read more about Blaze here: 
  http://blaze.pydata.org.   You can read more about XDATA here:  
 http://www.darpa.mil/Our_Work/I2O/Programs/XDATA.aspx

First of all, congratulations!


 I personally think Blaze is the future of array-oriented computing in Python. 
   I will be putting efforts and resources next year behind making that case.  
  How it interacts with future incarnations of NumPy, Pandas, or other 
 projects is an interesting and open question.  I have no doubt the future 
 will be a rich ecosystem of interoperating array-oriented data-structures.
  I invite anyone interested in Blaze to participate in the discussions and 
 development at 
 https://groups.google.com/a/continuum.io/forum/#!forum/blaze-dev or watch the 
 project on our public GitHub repo:  https://github.com/ContinuumIO/blaze.  
 Blaze is being incubated under the ContinuumIO GitHub project for now, but 
 eventually I hope it will receive its own GitHub project page later next 
 year.   Development of Blaze is early but we are moving rapidly with it (and 
 have deliverable deadlines --- thus while we will welcome input and pull 
 requests we won't have a ton of time to respond to simple queries unt
 il
   at least May or June).There is more that we are working on behind the 
 scenes with respect to Blaze that will be coming out next year as well but 
 isn't quite ready to show yet.

 As I look at the coming months and years, my time for direct involvement in 
 NumPy development is therefore only going to get smaller.  As a result it is 
 not appropriate that I remain as head steward of the NumPy project (a term 
 I prefer to BFD12 or anything else).   I'm sure that it is apparent that 
 while I've tried to help personally where I can this year on the NumPy 
 project, my role has been more one of coordination, seeking funding, and 
 providing expert advice on certain sections of code.I fundamentally agree 
 with Fernando Perez that the responsibility of care-taking open source 
 projects is one of stewardship --- something akin to public service.I 
 have tried to emulate that belief this year --- even while not always 
 succeeding.

 It is time for me to make official what is already becoming apparent to 
 observers of this community, namely, that I am stepping down as someone who 
 might be considered head steward for the NumPy project and officially 
 leaving the development of the project in the hands of others in the 
 community.   I don't think the project actually needs a new head steward 
 --- especially from a development perspective. Instead I see a lot of 
 strong developers offering key opinions for the project as well as a great 
 set of new developers offering pull requests.

 My strong suggestion is that development discussions of the project continue 
 on this list with consensus among the active participants being the goal for 
 development.  I don't think 100% consensus is a rigid requirement --- but 
 certainly a super-majority should be the goal, and serious changes should not 
 be made with out a clear consensus. I would pay special attention to 
 under-represented people (users with intense usage of NumPy but small voices 
 on this list).   There are many of them.If you push me for specifics then 
 at this point in NumPy's history, I would say that if Chuck, Nathaniel, and 
 Ralf agree on a course of action, it will likely be a good thing for the 
 project.   I suspect that even if only 2 of the 3 agree at one time it
might still be a good thing (but I would expect more detail and
discussion).There are others whose opinion should be sought as
well:  Ondrej Certik, Perry Greenfield, Robert Kern, David Cournapeau,
Francesc Alted, and Mark Wiebe to
  name a few.For some questions, I might even seek input from people like 
 Konrad Hinsen and Paul Dubois --- if they have time to give it.   I will 
 still be willing to offer my view from time to time and if I am asked.


After being involved with the numpy development/release lately, Chuck,
Nathaniel, and Ralf would be my (independent) choice as well, in fact
I have always sought their (and yours) approval to any PR that I had
doubts merging, and I have been treating them as the de-facto leaders
of the project... So it's good that you made it official.



 Greg Wilson (of Software Carpentry fame) asked me recently 

Re: [Numpy-discussion] Future of numpy (was: DARPA funding for Blaze and passing the NumPy torch)

[Ondřej Čertík]

Hi Matthew,

On Thu, Dec 20, 2012 at 3:46 PM, Matthew Brett matthew.br...@gmail.com wrote:
 Hi,

 [Travis wrote...]
 My strong suggestion is that development discussions of the project continue 
 on
 this list with consensus among the active participants being the goal for
 development.  I don't think 100% consensus is a rigid requirement --- but
 certainly a super-majority should be the goal, and serious changes should 
 not be
 made with out a clear consensus. I would pay special attention to
 under-represented people (users with intense usage of NumPy but small voices 
 on
 this list).   There are many of them.If you push me for specifics then at
 this point in NumPy's history, I would say that if Chuck, Nathaniel, and Ralf
 agree on a course of action, it will likely be a good thing for the project. 
   I
 suspect that even if only 2 of the 3 agree at one time it might still be a 
 good
 thing (but I would expect more detail and discussion).There are others 
 whose
 opinion should be sought as well:  Ondrej Certik, Perry Greenfield, Robert 
 Kern,
 David Cournapeau, Francesc Alted, and Mark Wiebe to name a few.For some
 questions, I might even seek input from people like Konrad Hinsen and Paul
 Dubois --- if they have time to give it.   I will still be willing to offer 
 my
 view from time to time and if I am asked.

 Thank you for starting this discussion.

 I am more or less offline at the moment in Cuba and flying, but I hope
 very much this will be an opportunity for a good discussion on the
 best way forward for numpy.

 Travis - I think you are suggesting that there should be no  one
 person in charge of numpy, and I think this is very unlikely to work
 well.   Perhaps there are good examples of well-led projects where
 there is not a clear leader, but I can't think of any myself at the
 moment.  My worry would be that, without a clear leader, it will be
 unclear how decisions are made, and that will make it very hard to
 take strategic decisions.

 I would like to humbly suggest the following in the hope that it spurs
 discussion.

 As first pass, Ralf, Chuck and Nathaniel decide on a core group of
 people that will form the succession committee.  Maybe this could be
 the list of people you listed above.

 Ralf, Chuck and Nathaniel then ask for people to wish to propose
 themselves as the leader of numpy.  Anyone proposing themselves to
 lead numpy would remove themselves from the succession committee.

 The proposed leaders of numpy write a short manifesto saying why they
 are the right choice for the job, and what they intend to do if
 elected.  The manifestos and issues arising are discussed in public on
 the mailing list - the equivalent of an online presidential debate.

 In due course - say after 2 weeks or after the discussion seems to be
 dying out - the succession committee votes on the leader.  I propose
 that these votes should be public, but I can see the opposite
 argument.

Travis has very clearly made Chuck, Nathaniel, and Ralf as the leaders
of the project. But as always  he didn't pick them because he
needed to create leaders. They were already the de-facto leaders
of the project due to their actions, involvements and respect in the community,
so he just made it official.


 How does that sound?

To me that sounds like a bad idea. That being said, if Chuck,
Nathaniel, and Ralf
agree that it would be a good idea, that's fine with me too.

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Future of numpy (was: DARPA funding for Blaze and passing the NumPy torch)

[Ondřej Čertík]

On Thu, Dec 20, 2012 at 5:48 PM, Ondřej Čertík ondrej.cer...@gmail.com wrote:
 Hi Matthew,

 On Thu, Dec 20, 2012 at 3:46 PM, Matthew Brett matthew.br...@gmail.com 
 wrote:
 Hi,

 [Travis wrote...]
 My strong suggestion is that development discussions of the project 
 continue on
 this list with consensus among the active participants being the goal for
 development.  I don't think 100% consensus is a rigid requirement --- but
 certainly a super-majority should be the goal, and serious changes should 
 not be
 made with out a clear consensus. I would pay special attention to
 under-represented people (users with intense usage of NumPy but small 
 voices on
 this list).   There are many of them.If you push me for specifics then 
 at
 this point in NumPy's history, I would say that if Chuck, Nathaniel, and 
 Ralf
 agree on a course of action, it will likely be a good thing for the 
 project.   I
 suspect that even if only 2 of the 3 agree at one time it might still be a 
 good
 thing (but I would expect more detail and discussion).There are others 
 whose
 opinion should be sought as well:  Ondrej Certik, Perry Greenfield, Robert 
 Kern,
 David Cournapeau, Francesc Alted, and Mark Wiebe to name a few.For some
 questions, I might even seek input from people like Konrad Hinsen and Paul
 Dubois --- if they have time to give it.   I will still be willing to offer 
 my
 view from time to time and if I am asked.

 Thank you for starting this discussion.

 I am more or less offline at the moment in Cuba and flying, but I hope
 very much this will be an opportunity for a good discussion on the
 best way forward for numpy.

 Travis - I think you are suggesting that there should be no  one
 person in charge of numpy, and I think this is very unlikely to work
 well.   Perhaps there are good examples of well-led projects where
 there is not a clear leader, but I can't think of any myself at the
 moment.  My worry would be that, without a clear leader, it will be
 unclear how decisions are made, and that will make it very hard to
 take strategic decisions.

 I would like to humbly suggest the following in the hope that it spurs
 discussion.

 As first pass, Ralf, Chuck and Nathaniel decide on a core group of
 people that will form the succession committee.  Maybe this could be
 the list of people you listed above.

 Ralf, Chuck and Nathaniel then ask for people to wish to propose
 themselves as the leader of numpy.  Anyone proposing themselves to
 lead numpy would remove themselves from the succession committee.

 The proposed leaders of numpy write a short manifesto saying why they
 are the right choice for the job, and what they intend to do if
 elected.  The manifestos and issues arising are discussed in public on
 the mailing list - the equivalent of an online presidential debate.

 In due course - say after 2 weeks or after the discussion seems to be
 dying out - the succession committee votes on the leader.  I propose
 that these votes should be public, but I can see the opposite
 argument.

 Travis has very clearly made Chuck, Nathaniel, and Ralf as the leaders
 of the project. But as always  he didn't pick them because he
 needed to create leaders. They were already the de-facto leaders
 of the project due to their actions, involvements and respect in the 
 community,
 so he just made it official.


 How does that sound?

 To me that sounds like a bad idea. That being said, if Chuck,
 Nathaniel, and Ralf
 agree that it would be a good idea, that's fine with me too.

I forgot to add --- Matthew, I'll be happy to discuss this over phone
once you get back to the US. I know we had discussions over this
in the past, but I couldn't offer much advice, as I didn't understand
the inner working of the numpy development. I now understand
it much better.

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Status of the 1.7 release

[Ondřej Čertík]

Thanks Ralf and Nathan,

I have put high priority on the issues that need to be fixed before the rc1.
There are now 4 issues:

https://github.com/numpy/numpy/issues?labels=priority%3A+highmilestone=3page=1state=open

I am working on the mingw one, as that one is the most difficult.
Ralf (or anyone else), do you know how to fix this one:

https://github.com/numpy/numpy/issues/438

I am not very familiar with this part of numpy, so maybe you know how
to document it well.

The sooner we can fix these 4 issues, the sooner we can release.

Ondrej

On Sun, Dec 16, 2012 at 1:49 AM, Ralf Gommers ralf.gomm...@gmail.com wrote:



 On Sun, Dec 16, 2012 at 3:17 AM, Nathaniel Smith n...@pobox.com wrote:

 #294 is a regression, so probably should be considered release critical. I
 can't tell if #2750 is a real problem or not. #378 looks serious, but afaict
 has actually been fixed even though the bug is still marked open? At least
 fixed in 1.7.x?

 On 15 Dec 2012 23:52, Ondřej Čertík ondrej.cer...@gmail.com wrote:

 Hi,

 If you go to the issues for 1.7 and click high priority:


 https://github.com/numpy/numpy/issues?labels=priority%3A+highmilestone=3state=open

 you will see 3 issues as of right now. Two of those have PR attached.
 It's been a lot of work
 to get to this point and I'd like to thank all of you for helping out
 with the issues.


 In particular, I have just fixed a very annoying segfault (#2738) in the
 PR:

 https://github.com/numpy/numpy/pull/2831

 If you can review that one carefully, that would be highly
 appreciated. The more people the better,
 it's a reference counting issue and since this would go into the 1.7
 release and it's in the core of numpy,
 I want to make sure that it's correct.

 So the last high priority issue is:

 https://github.com/numpy/numpy/issues/568

 and that's the one I will be concentrating on now. After it's fixed, I
 think we are ready to release the rc1.

 There are more open issues (that are not high priority):


 https://github.com/numpy/numpy/issues?labels=milestone=3page=1state=open

 But I don't think we should delay the release any longer because of
 them. Let me know if there
 are any objections. Of course, if you attach a PR fixing any of those,
 we'll merge it.


 Properly documenting .base (gh-2737) and casting rules (gh-561) changes
 should be finished before rc1. I agree that the Debian issues all shouldn't
 block the release.

 Ralf


 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Status of the 1.7 release

[Ondřej Čertík]

Hi,

If you go to the issues for 1.7 and click high priority:

https://github.com/numpy/numpy/issues?labels=priority%3A+highmilestone=3state=open

you will see 3 issues as of right now. Two of those have PR attached.
It's been a lot of work
to get to this point and I'd like to thank all of you for helping out
with the issues.


In particular, I have just fixed a very annoying segfault (#2738) in the PR:

https://github.com/numpy/numpy/pull/2831

If you can review that one carefully, that would be highly
appreciated. The more people the better,
it's a reference counting issue and since this would go into the 1.7
release and it's in the core of numpy,
I want to make sure that it's correct.

So the last high priority issue is:

https://github.com/numpy/numpy/issues/568

and that's the one I will be concentrating on now. After it's fixed, I
think we are ready to release the rc1.

There are more open issues (that are not high priority):

https://github.com/numpy/numpy/issues?labels=milestone=3page=1state=open

But I don't think we should delay the release any longer because of
them. Let me know if there
are any objections. Of course, if you attach a PR fixing any of those,
we'll merge it.

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Failure in test_iterator.py at Travis

[Ondřej Čertík]

On Fri, Dec 14, 2012 at 12:32 AM, Nathaniel Smith n...@pobox.com wrote:
 I only checked this build:
 https://secure.travis-ci.org/#!/certik/numpy/jobs/3656960
 But that log clearly shows 'python setup.py install' being used instead of
 'pip install'. How certain are you that your branch actually has my fix?

Right. So is this one (using python setup.py install):

https://travis-ci.org/numpy/numpy/jobs/3330235

No wonder that it fails. Since I rebased this on top of master and the master
has this fix, and so does the 1.7 branch, I can't explain it. I know that
the master at certik/numpy github does not have that fix (since I didn't push in
there lately), but I don't see how that could matter, unless there is
a bug at Travis.
The branches that I created the pull requests from should have that fix.


Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Failure in test_iterator.py at Travis

[Ondřej Čertík]

On Fri, Dec 14, 2012 at 9:49 AM, Nathaniel Smith n...@pobox.com wrote:
 The top of the build log has the actual git command they used to check out
 the source - it's some clever GitHub thing that gives the same thing as
 pressing the green button would iirc. You could copy the commands from the
 log to check that out locally though and see what the .travis.tml actually
 says. And if it's wrong poke around in the git history to figure out why.

You are right! I followed the one:

https://travis-ci.org/numpy/numpy/jobs/3330235

and .travis.yml indeed contains the old python setup.py install!
I just don't understand why. My strong suspicion is that the actual
commands are *not* equivalent to the github green merge button.

It took the latest patches from the PR (which do not contain the .travis.yml
fix), and merged into:

commit 089bfa5865cd39e2b40099755e8563d8f0d04f5f
Merge: 1065cc5 df2958e
Author: Ralf Gommers ralf.gomm...@googlemail.com
Date:   Sat Nov 24 02:54:48 2012 -0800

Merge pull request #2766 from g2p/master

Assume we can use sys.stdout.fileno() and friends.


I have no idea why. Unless --- it is merging into the master,
which was current at the time the PR was created. That would explain
it this particular failure.


Ok. And finally, the one at certik/numpy fails, because that one is
checking out the sources
from certik/numpy, but those do not have the improvement. It is then
the Travis's bug, that it links to certik/numpy log
for PRs at numpy/numpy.

So I think that all is explained now.

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Travis failures with no errors

[Ondřej Čertík]

Hi,

I found these recent weird failures in Travis, but I can't find any
problem with the log and all tests pass. Any ideas what is going on?

https://travis-ci.org/numpy/numpy/jobs/3570123
https://travis-ci.org/numpy/numpy/jobs/3539549
https://travis-ci.org/numpy/numpy/jobs/3369629

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Failure in test_iterator.py at Travis

[Ondřej Čertík]

Hi,

Another weird bug sometimes happen in
numpy/core/tests/test_iterator.py, it looks like this:

==
FAIL: test_iterator.test_iter_array_cast
--
Traceback (most recent call last):
  File 
/home/travis/virtualenv/python2.5/lib/python2.5/site-packages/nose/case.py,
line 197, in runTest
self.test(*self.arg)
  File 
/home/travis/virtualenv/python2.5/lib/python2.5/site-packages/numpy/core/tests/test_iterator.py,
line 836, in test_iter_array_cast
assert_equal(i.operands[0].strides, (-96,8,-32))
  File 
/home/travis/virtualenv/python2.5/lib/python2.5/site-packages/numpy/testing/utils.py,
line 252, in assert_equal
assert_equal(actual[k], desired[k], 'item=%r\n%s' % (k,err_msg), verbose)
  File 
/home/travis/virtualenv/python2.5/lib/python2.5/site-packages/numpy/testing/utils.py,
line 314, in assert_equal
raise AssertionError(msg)
AssertionError:
Items are not equal:
item=0

 ACTUAL: 96
 DESIRED: -96

--


But the problem is that there is no numpy/core/tests/test_iterator.py
file in current branches This error was triggered for example by
these PRs:

https://github.com/numpy/numpy/pull/2765
https://github.com/numpy/numpy/pull/2815

and here are links to the failing Travis tests:

https://travis-ci.org/certik/numpy/builds/3656959
https://travis-ci.org/numpy/numpy/builds/3330234


Any idea what is happening here and how to fix it?

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Failure in test_iterator.py at Travis

[Ondřej Čertík]

On Thu, Dec 13, 2012 at 7:16 PM, Charles R Harris
charlesr.har...@gmail.com wrote:


 On Thu, Dec 13, 2012 at 8:04 PM, Charles R Harris
 charlesr.har...@gmail.com wrote:



 On Thu, Dec 13, 2012 at 7:23 PM, Ondřej Čertík ondrej.cer...@gmail.com
 wrote:

 Hi,

 Another weird bug sometimes happen in
 numpy/core/tests/test_iterator.py, it looks like this:

 ==
 FAIL: test_iterator.test_iter_array_cast
 --
 Traceback (most recent call last):
   File
 /home/travis/virtualenv/python2.5/lib/python2.5/site-packages/nose/case.py,
 line 197, in runTest
 self.test(*self.arg)
   File
 /home/travis/virtualenv/python2.5/lib/python2.5/site-packages/numpy/core/tests/test_iterator.py,
 line 836, in test_iter_array_cast
 assert_equal(i.operands[0].strides, (-96,8,-32))
   File
 /home/travis/virtualenv/python2.5/lib/python2.5/site-packages/numpy/testing/utils.py,
 line 252, in assert_equal
 assert_equal(actual[k], desired[k], 'item=%r\n%s' % (k,err_msg),
 verbose)
   File
 /home/travis/virtualenv/python2.5/lib/python2.5/site-packages/numpy/testing/utils.py,
 line 314, in assert_equal
 raise AssertionError(msg)
 AssertionError:
 Items are not equal:
 item=0

  ACTUAL: 96
  DESIRED: -96

 --


 But the problem is that there is no numpy/core/tests/test_iterator.py
 file in current branches This error was triggered for example by
 these PRs:

 https://github.com/numpy/numpy/pull/2765
 https://github.com/numpy/numpy/pull/2815

 and here are links to the failing Travis tests:

 https://travis-ci.org/certik/numpy/builds/3656959
 https://travis-ci.org/numpy/numpy/builds/3330234


 Any idea what is happening here and how to fix it?


 That should have been fixed by Nathaniel's travis fix. Hmm...


 And a quick check here shows pip not removing a previous 1.6.2 install.
 Maybe it is a pip version problem, pip 1.0.2 here.

That's what I thought as well. I think we need to update the Travis
script to manually remove the installed numpy.

Btw, Travis seems to be using pip 1.2.1, at least according to:

https://travis-ci.org/numpy/numpy/jobs/3330236

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Numpy Trac migration

[Ondřej Čertík]

On Wed, Dec 5, 2012 at 4:45 PM, Pauli Virtanen p...@iki.fi wrote:
 Hi,

 For those whom it may concern: Since the Numpy Trac - Github migration
 is complete, I went ahead and added redirects

 projects.scipy.org/numpy/register - github.com/numpy/numpy/issues
 projects.scipy.org/numpy/newticket - github.com/numpy/numpy/issues

 plus an ugly warning bar on top to direct potential users towards Github.

Thanks, that is excellent.

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Weird Travis-CI bugs in the release 1.7.x branch

[Ondřej Čertík]

On Mon, Dec 3, 2012 at 7:10 PM, Nathaniel Smith n...@pobox.com wrote:
 On 4 Dec 2012 02:27, Ondřej Čertík ondrej.cer...@gmail.com wrote:

 Hi,

 I started to work on the release again and noticed weird failures at
 Travis-CI:
 […]
   File
 /home/travis/virtualenv/python2.5/lib/python2.5/site-packages/numpy/core/tests/test_iterator.py,

 The problem is that Travis started installing numpy in all python
 virtualenvs by default, and our Travis build script just runs setup.py
 install, which is too dumb to notice that there is a numpy already installed
 and just overwrites it. The file mentioned above doesn't even exist in 1.7,
 it's left over from the 1.6 install.

 I did a PR to fix this in master a few days ago, you want to back port that.
 (Sorry for lack of link, I'm on my phone.)

Thanks! I backported it in:

https://github.com/numpy/numpy/pull/2786

Nice, I was not aware of the fact that pip install . fixes this
problem with setup.py ---
I've burned myself with this so many times already and I always forget
about this bug.


 P.S. My thesis was finally approved by the grad school today,
 doing some final changes took more time than expected, but
 I think that I am done now.

 Congratulations Dr. Čertík!

Thanks. I am glad it's over.

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Weird Travis-CI bugs in the release 1.7.x branch

[Ondřej Čertík]

On Tue, Dec 4, 2012 at 8:14 AM, Ondřej Čertík ondrej.cer...@gmail.com wrote:
 On Mon, Dec 3, 2012 at 7:10 PM, Nathaniel Smith n...@pobox.com wrote:
 On 4 Dec 2012 02:27, Ondřej Čertík ondrej.cer...@gmail.com wrote:

 Hi,

 I started to work on the release again and noticed weird failures at
 Travis-CI:
 […]
   File
 /home/travis/virtualenv/python2.5/lib/python2.5/site-packages/numpy/core/tests/test_iterator.py,

 The problem is that Travis started installing numpy in all python
 virtualenvs by default, and our Travis build script just runs setup.py
 install, which is too dumb to notice that there is a numpy already installed
 and just overwrites it. The file mentioned above doesn't even exist in 1.7,
 it's left over from the 1.6 install.

 I did a PR to fix this in master a few days ago, you want to back port that.
 (Sorry for lack of link, I'm on my phone.)

 Thanks! I backported it in:

 https://github.com/numpy/numpy/pull/2786

 Nice, I was not aware of the fact that pip install . fixes this
 problem with setup.py ---
 I've burned myself with this so many times already and I always forget
 about this bug.

It's fixed in the release branch now. So both master and the release branch
pass all tests on Travis again.

Thanks for your help.

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Weird Travis-CI bugs in the release 1.7.x branch

[Ondřej Čertík]

Hi,

I started to work on the release again and noticed weird failures at Travis-CI:

https://github.com/numpy/numpy/pull/2782

The first commit (8a18fc7) should not trigger this failure:

==
FAIL: test_iterator.test_iter_array_cast
--
Traceback (most recent call last):
  File 
/home/travis/virtualenv/python2.5/lib/python2.5/site-packages/nose/case.py,
line 197, in runTest
self.test(*self.arg)
  File 
/home/travis/virtualenv/python2.5/lib/python2.5/site-packages/numpy/core/tests/test_iterator.py,
line 836, in test_iter_array_cast
assert_equal(i.operands[0].strides, (-96,8,-32))
  File 
/home/travis/virtualenv/python2.5/lib/python2.5/site-packages/numpy/testing/utils.py,
line 252, in assert_equal
assert_equal(actual[k], desired[k], 'item=%r\n%s' % (k,err_msg), verbose)
  File 
/home/travis/virtualenv/python2.5/lib/python2.5/site-packages/numpy/testing/utils.py,
line 314, in assert_equal
raise AssertionError(msg)
AssertionError:
Items are not equal:
item=0

 ACTUAL: 96
 DESIRED: -96

So I pushed a whitespace commit into the PR (516b478) yet it has the
same failure. So it's there, it's
not some random fluke at Travis. I created this testing PR:

https://github.com/numpy/numpy/pull/2783

to try to nail it down. But I can't see what could have caused this,
because the release branch was passing all tests
last time I worked on it.

Any ideas?

Btw, I managed to reproduce the SPARC64 bug:

https://github.com/numpy/numpy/issues/2668

so that's good. Now I just need to debug it.

Ondrej

P.S. My thesis was finally approved by the grad school today,
doing some final changes took more time than expected, but
I think that I am done now.
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] 1.7.0 release

[Ondřej Čertík]

 Hi all,

 Having looked at the README.txt and INSTALL.txt files on the
 branch, I see no mention of which Python 3.x versions are supported:

 https://github.com/numpy/numpy/blob/maintenance/1.7.x/README.txt
 https://github.com/numpy/numpy/blob/maintenance/1.7.x/INSTALL.txt

 Is NumPy 1.7 intended to support Python 3.3?

 My impression from this thread is probably yes:
 http://mail.scipy.org/pipermail/numpy-discussion/2012-July/063483.html
 ...
 http://mail.scipy.org/pipermail/numpy-discussion/2012-August/063597.html

 If so, then under Windows (32 bit at least) where the Python.org
 provided Python 3.3 is compiled with MSCV 2010 there are some
 problems - see: https://github.com/numpy/numpy/pull/2726

Thanks Peter for the email, yes I think we should support Python 3.3.

 Should an issue be filed for this?

I just did:

https://github.com/numpy/numpy/issues/2743


Btw, I also bumbed this issue as priority high:

https://github.com/numpy/numpy/issues/2738

Because it segfaults numpy.

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Should abs([nan]) be supported?

[Ondřej Čertík]

On Fri, Sep 7, 2012 at 7:54 AM, Ralf Gommers ralf.gomm...@gmail.com wrote:


 On Wed, Sep 5, 2012 at 7:06 AM, Travis Oliphant tra...@continuum.io wrote:

 The framework for catching errors relies on hardware flags getting set and
 our C code making the right calls to detect those flags.

 This has usually worked correctly in the past --- but it is an area where
 changes in compilers or platforms could create problems.


 I don't think it ever did, for less common platforms at least. See all the
 Debian test issues that were filed by Sandro this week. And even between
 Windows and Linux, there are some inconsistencies.



 We should test to be sure that the correct warnings are issued, I would
 think.Perhaps using a catch_warnings context would be helpful (from
 http://docs.python.org/library/warnings.html)


 There are some tests for that already, in core/test_numeric.py. For example:

 ==
 FAIL: test_default (test_numeric.TestSeterr)
 --
 Traceback (most recent call last):
   File /Users/rgommers/Code/numpy/numpy/core/tests/test_numeric.py, line
 231, in test_default
 under='ignore',
 AssertionError: {'over': 'ignore', 'divide': 'ignore', 'invalid': 'ignore',
 'under': 'ignore'} != {'over': 'warn', 'divide': 'warn', 'invalid': 'warn',
 'under': 'ignore'}

 --

 They're not exhaustive though.



 import warnings

 def fxn():
 warnings.warn(deprecated, DeprecationWarning)

 with warnings.catch_warnings(record=True) as w:
 # Cause all warnings to always be triggered.
 warnings.simplefilter(always)
 # Trigger a warning.
 fxn()
 # Verify some things
 assert len(w) == 1
 assert issubclass(w[-1].category, DeprecationWarning)
 assert deprecated in str(w[-1].message)




 Use ``from numpy.testing import WarningManager`` for a 2.4-compatible
 version of catch_warnings (with explicitly calling its __enter__ and
 __exit__ methods).


In the release 1.7.x branch these warnings are suppressed, but in
master they frequently make the Travis tests fail. That's extremely
annoying to me, so I started working on a fix here:

https://github.com/numpy/numpy/pull/2745

using the recommendation from Travis above.

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] 1.7.0 release

[Ondřej Čertík]

On Tue, Nov 13, 2012 at 4:18 AM, Gael Varoquaux
gael.varoqu...@normalesup.org wrote:
 On Mon, Nov 12, 2012 at 02:27:02PM -0800, Ondřej Čertík wrote:
 I successfully defended my Ph.D. thesis last Thursday, I just need to
 do some changes to it and submit it and I am done.

 Yey! Tag and release (the thesis, I mean)!

 Congratulation.

 G

 PS: It seems to me that it was yesterday that you moved to the state. You
 didn't loose time with your PhD. Very impressive!

Thanks everybody. Gael, it feels like yesterday, but it's been 4 years ago.
And I had an M.S. degree already from Prague which took 2 years, so the total
time is 6, and I had to wait 0.5 years after finishing my M.S. and before coming
to the US (paperwork), so this gives 6.5 years since finishing my B.C.
degree. The average
time for physics Ph.D. seems to be 5.5 years, at least according to [1], so
I am 1 year late, but I had to overcome a few bumps along the way ---
maybe I'll write
a blog post later.

Ondrej

[1] http://physics.uchicago.edu/prospective/graduate/index.html
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] 1.7.0 release

[Ondřej Čertík]

On Mon, Nov 12, 2012 at 2:27 PM, Ondřej Čertík ondrej.cer...@gmail.com wrote:
[...]
 Here is a list of issues that need to be fixed before the release:

 https://github.com/numpy/numpy/issues?milestone=3state=open

 If anyone wants to help, we just need to get through them and submit a
 PR for each, or close it if it doesn't apply anymore.
 This is what I am doing now.

Ok, I went over all the issues, closed fixed issues and sent PRs for about
6 issues. Just go to the link above to see them all (or go to issues and
click on NumPy 1.7 milestone) and left comments on most issues.

Most of the minor problems are fixed. There are only 3 big issues that
need to be fixed so I set them priority high:

https://github.com/numpy/numpy/issues?labels=priority%3A+highmilestone=3page=1state=open

in particular:

https://github.com/numpy/numpy/issues/568
https://github.com/numpy/numpy/issues/2668
https://github.com/numpy/numpy/issues/606

after that I think we should be good to go. If you have some spare
cycles, just concentrate on these 3.

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] 1.7.0 release

[Ondřej Čertík]

Hi,

On Mon, Nov 5, 2012 at 11:33 PM, Travis Oliphant tra...@continuum.io wrote:
 Hey all,

 Ondrej has been tied up finishing his PhD for the past several weeks.  He is 
 defending his work shortly and should be available to continue to help with 
 the 1.7.0 release around the first of December.He and I have been in 
 contact during this process, and I've been helping where I can.   
 Fortunately, other NumPy developers have been active closing tickets and 
 reviewing pull requests which has helped the process substantially.

 The release has taken us longer than we expected, but I'm really glad that 
 we've received the bug-reports and issues that we have seen because it will 
 help the 1.7.0 release be a more stable series.   Also, the merging of the 
 Trac issues with Git has exposed over-looked problems as well and will 
 hopefully encourage more Git-focused participation by users.

 We are targeting getting the final release of 1.7.0 out by mid December 
 (based on Ondrej's availability).   But, I would like to find out which 
 issues are seen as blockers by people on this list.   I think most of the 
 issues that I had as blockers have been resolved.If there are no more 
 remaining blockers, then we may be able to accelerate the final release of 
 1.7.0 to just after Thanksgiving.



I successfully defended my Ph.D. thesis last Thursday, I just need to
do some changes to it and submit it and I am done.
So I started to work on the release again (my apologies that it got
delayed, I had to devote my full attention to finishing my school
first).

Here is a list of issues that need to be fixed before the release:

https://github.com/numpy/numpy/issues?milestone=3state=open

If anyone wants to help, we just need to get through them and submit a
PR for each, or close it if it doesn't apply anymore.
This is what I am doing now.

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] 1.7.0 release

[Ondřej Čertík]

On Mon, Nov 12, 2012 at 3:47 PM, Charles R Harris
charlesr.har...@gmail.com wrote:


 On Mon, Nov 12, 2012 at 3:27 PM, Ondřej Čertík ondrej.cer...@gmail.com
 wrote:

 Hi,

 On Mon, Nov 5, 2012 at 11:33 PM, Travis Oliphant tra...@continuum.io
 wrote:
  Hey all,
 
  Ondrej has been tied up finishing his PhD for the past several weeks.
  He is defending his work shortly and should be available to continue to 
  help
  with the 1.7.0 release around the first of December.He and I have been
  in contact during this process, and I've been helping where I can.
  Fortunately, other NumPy developers have been active closing tickets and
  reviewing pull requests which has helped the process substantially.
 
  The release has taken us longer than we expected, but I'm really glad
  that we've received the bug-reports and issues that we have seen because it
  will help the 1.7.0 release be a more stable series.   Also, the merging of
  the Trac issues with Git has exposed over-looked problems as well and will
  hopefully encourage more Git-focused participation by users.
 
  We are targeting getting the final release of 1.7.0 out by mid December
  (based on Ondrej's availability).   But, I would like to find out which
  issues are seen as blockers by people on this list.   I think most of the
  issues that I had as blockers have been resolved.If there are no more
  remaining blockers, then we may be able to accelerate the final release of
  1.7.0 to just after Thanksgiving.



 I successfully defended my Ph.D. thesis last Thursday, I just need to
 do some changes to it and submit it and I am done.
 So I started to work on the release again (my apologies that it got
 delayed, I had to devote my full attention to finishing my school
 first).

 Here is a list of issues that need to be fixed before the release:

 https://github.com/numpy/numpy/issues?milestone=3state=open

 If anyone wants to help, we just need to get through them and submit a
 PR for each, or close it if it doesn't apply anymore.
 This is what I am doing now.


 Concrats. There are some commits already due for backports. Do you want to
 do that, or will it be OK for others to do it? Another possibility would be
 to open new PR's for the backports and tag them as 1.7.0.

I'll be happy to do that. If you know which commits are those, let me know.
I actually like to open a PR just for the backports as well, to see if tests
pass and to have more eyes look on it to minimize mistakes.

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Making numpy sensible: backward compatibility please

[Ondřej Čertík]

On Sat, Sep 29, 2012 at 2:16 AM, Gael Varoquaux
gael.varoqu...@normalesup.org wrote:
 On Fri, Sep 28, 2012 at 07:37:35PM -0600, Charles R Harris wrote:
 +1, I think we should endeavor to have a respectful and welcoming
 community.

 With a bit of humour now and then among the old timers, no? Look, I've been
 outright insulted on the list and I don't recall anyone weighing in about
 civility at the time. But nor am I a tender flower. But we are going way off
 topic, no?

 I think that Chuck and I know each other well enough to understand that
 this pique is not intended to be disrespectful and is full of humour. I
 bystander might feel different, but I am actually laughing and smiling at
 Chuck's answer. I know that he is serious and means it, and respect his
 point of view I somehow feel that this is just a very opinionated way of
 expressing his enthusiasm about numpy, almost like a Frenchman would do
 :}.

 Chuck: point taken. I agree with some of your points, and I am certainly
 not taking it badly. I know you well enough to know that you are an
 enthusiastic, and very friendly person who invests a lot of time in
 numpy. Next time I see you, I owe you a beer for making you cross :). I
 should be at scipy next year. I'll settle my debt in beer hopefully, if I
 have the chance to see you.

I agree with what Fernando wrote above. If possible, please let's keep
the emails
respectful. I am glad that you didn't get offended Gael and I am sure
that you Chuck wouldn't be either with such a response. However, for me,
as a newcomer to the numpy development, I didn't know that you Chuck and Gael
know each other well. And even if you know each other well and while
it's ok with
either of you, it might not be for other people, and it definitely
creates an unwelcoming
atmosphere on the mailinglist.

If I can offer a suggestion that works for me --- I personally love to
say what I frankly think,
and I am not against swear words at all, I believe it's better to
express what I think in one
simple word, than in two paragraphs --- but it only works over a beer
if I know that nobody
will get offended. Over private email it's harder and on public
mailinglist, where a lot of
people will read it who have never met me, it's counter productive.



Chuck, Gael, here is my todo list for the 1.7.0 release:

https://github.com/numpy/numpy/issues/396

Is there anything missing in terms of backwards compatibility? Gael,
if you have a minute, would you mind creating issues for things that
broke since the 1.6.2 release that are not mentioned in #396 yet? It's
my top priority to make sure that as little things break as possible
by the new 1.7.0 release, that's also why we decided to release two
beta versions to give people more chance to test it well.

My apologies that it has taken me longer than it should have to get it
all done.

Ondrej

P.S. For beers I would love to join both of you anytime. :)
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] ANN: NumPy 1.7.0b2 release

[Ondřej Čertík]

On Mon, Sep 24, 2012 at 3:49 PM, Nathaniel Smith n...@pobox.com wrote:
 On Mon, Sep 24, 2012 at 10:47 PM, Charles R Harris
 charlesr.har...@gmail.com wrote:


 On Mon, Sep 24, 2012 at 2:25 PM, Frédéric Bastien no...@nouiz.org wrote:

 Hi,

 I tested this new beta on Theano and discovered an interface change
 that was not there in the beta 1.

 New behavior:
 numpy.ndindex().next()
 (0,)

 Old behavior:
 numpy.ndindex().next()
 ()

 This break some Theano code that look like this:

 import numpy
 shape=()
 out_shape=[12]
 random_state=numpy.random.RandomState()

 out = numpy.zeros(out_shape, int)
 for i in numpy.ndindex(*shape):
 out[i] = random_state.permutation(5)


 I suppose this is an regression as the only mention of ndindex in the
 first email of this change is that it is faster.


 I think this problem has been brought up on the list. It is interesting that
 it turned up after the first beta. Could you do a bisection to discover
 which commit is responsible?

 No need, the problem is already known. It was introduced by that
 ndindex speed up patch, PR #393, which was backported into the first
 beta as well. There's a follow-up patch in PR #445 that fixes both of
 these issues, though it also exposes some more fundamental issues with
 the nditer API, so there's lots of discussion there about if we want
 some more changes... this is a good summary:
 https://github.com/numpy/numpy/pull/445#issuecomment-8740982

 For 1.7 purposes though the bottom line is that we already have
 multiple acceptable solutions, so both the issues reported here should
 definitely be fixed.

Should we just remove (revert) this PR #393 patch from the release branch?
It shouldn't have been there in the first place, the only reason I included it
is because other patches depended on it and I would have to fix collisions,
and we thought it would be harmless to just include it. Which turned out
to be a mistake, for which I apologize.

That way we'll feel confident that the branch works, and we can get the right
solution into master and test it there.

So I am actually convinced I should simply revert this patch in the
release branch.
Let me know what you think.

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Fwd: Package: scipy-0.11.0-0.1.rc2.fc18 Tag: f18-updates-candidate Status: failed Built by: orion

[Ondřej Čertík]

Hi Orion,

On Thu, Sep 20, 2012 at 2:56 PM, Orion Poplawski or...@cora.nwra.com wrote:
 This is a plea for some help.  We've been having trouble getting scipy to
 pass all of the tests in the Fedora 18 build with python 3.3 (although it
 seems to build okay in Fedora 19).  Below are the logs of the build.  There
 appears to be some kind of memory corruption that manifests itself a little
 differently on 32-bit vs. 64-bit.  I really have no idea myself how to
 pursue debugging this, though I'm happy to provide any more needed
 information.

Thanks for testing the latest beta2 release.

 Task 4509077 on buildvm-35.phx2.fedoraproject.org
 Task Type: buildArch (scipy-0.11.0-0.1.rc2.fc18.src.rpm, i686)
 logs:
   http://koji.fedoraproject.org/koji/getfile?taskID=4509077name=build.log

This link has the following stacktrace:

/lib/libpython3.3m.so.1.0(PyMem_Free+0x1c)[0xbf044c]
/usr/lib/python3.3/site-packages/numpy/core/multiarray.cpython-33m.so(+0x4d52b)[0x42252b]
/usr/lib/python3.3/site-packages/numpy/core/multiarray.cpython-33m.so(+0xcb7c5)[0x4a07c5]
/usr/lib/python3.3/site-packages/numpy/core/multiarray.cpython-33m.so(+0xcbc5e)[0x4a0c5e]

Which indeed looks like in NumPy. Would you be able to obtain full stacktrace?

There has certainly been segfaults in Python 3.3 with NumPy, but we've
fixed all that we could reproduce. That doesn't mean there couldn't be
more. If you could nail it down a little bit more, that would be
great. I'll help once I can reproduce it somehow.

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] ANN: NumPy 1.7.0b2 release

[Ondřej Čertík]

Hi,

I'm pleased to announce the availability of the second beta release of
NumPy 1.7.0b2.

Sources and binary installers can be found at
https://sourceforge.net/projects/numpy/files/NumPy/1.7.0b2/

Please test this release and report any issues on the numpy-discussion
mailing list. Since beta1, we've fixed most of the known (back then)
issues, except:

http://projects.scipy.org/numpy/ticket/2076
http://projects.scipy.org/numpy/ticket/2101
http://projects.scipy.org/numpy/ticket/2108
http://projects.scipy.org/numpy/ticket/2150

And many other issues that were reported since the beta1 release. The
log of changes is attached. The full list of issues that we still need
to work on is at:

https://github.com/numpy/numpy/issues/396

Any help is welcome, the best is to send a PR fixing any of the issues
-- against master, and I'll then back-port it to the release branch
(unless it is something release specific, in which case just send the
PR against the release branch).

Cheers,
Ondrej


* f217517 Release 1.7.0b2
* 50f71cb MAINT: silence Cython warnings about changes dtype/ufunc size.
* fcacdcc FIX: use py24-compatible version of virtualenv on Travis
* d01354e FIX: loosen numerical tolerance in test_pareto()
* 65ec87e TST: Add test for boolean insert
* 9ee9984 TST: Add extra test for multidimensional inserts.
* 8460514 BUG: Fix for issues #378 and #392 This should fix the
problems with numpy.insert(), where the input values were not checked
for all scalar types and where values did not get inserted properly,
but got duplicated by default.
* 07e02d0 BUG: fix npymath install location.
* 6da087e BUG: fix custom post_check.
* 095a3ab BUG: forgot to build _dotblas in bento build.
* cb0de72 REF: remove unused imports in bscript.
* 6e3e289 FIX: Regenerate mtrand.c with Cython 0.17
* 3dc3b1b Retain backward compatibility. Enforce C order.
* 5a471b5 Improve ndindex execution speed.
* 2f28db6 FIX: Add a test for Ticket #2066
* ca29849 BUG: Add a test for Ticket #2189
* 1ee4a00 BUG: Add a test for Ticket #1588
* 7b5dba0 BUG: Fix ticket #1588/gh issue #398, refcount error in clip
* f65ff87 FIX: simplify the import statement
* 124a608 Fix returned copy
* 996a9fb FIX: bug in np.where and recarray swapping
* 7583adc MAINT: silence DeprecationWarning in np.safe_eval().
* 416af9a pavement.py: rename yop to atlas
* 3930881 BUG: fix bento build.
* fbad4a7 Remove test_recarray_from_long_formats
* 5cb80f8 Add test for long number in shape specifier of dtype string
* 24da7f6 Add test for long numbers in numpy.rec.array formats string
* 77da3f8 Allow long numbers in numpy.rec.array formats string
* 99c9397 Use PyUnicode_DecodeUTF32()
* 31660d0 Follow the C guidelines
* d5d6894 Fix memory leak in concatenate.
* 8141e1e FIX: Make sure the tests produce valid unicode
* d67785b FIX: Fixes the PyUnicodeObject problem in py-3.3
* a022015 Re-enable unpickling optimization for large py3k bytes objects.
* 470486b Copy bytes object when unpickling an array
* d72280f Fix tests for empty shape, strides and suboffsets on Python 3.3
* a1561c2 [FIX] Add missing header so separate compilation works again
* ea23de8 TST: set raise-on-warning behavior of NoseTester to release mode.
* 28ffac7 REL: set version number to 1.7.0rc1-dev.
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] ANN: NumPy 1.7.0b2 release

[Ondřej Čertík]

On Thu, Sep 20, 2012 at 4:50 AM, Richard Hattersley
rhatters...@gmail.com wrote:
 Hi,

 [First of all - thanks to everyone involved in the 1.7 release. Especially
 Ondřej - it takes a lot of time  energy to coordinate something like this.]

 Is there an up to date release schedule anywhere? The trac milestone still
 references June.

Well, originally we were supposed to release about a month ago, but it
turned out there are more things to fix.
Currently, we just need to fix all the issues here:

https://github.com/numpy/numpy/issues/396

it looks like a lot, but many of them are really easy to fix, so my
hope is that it will not take long. The hardest one is this:

http://projects.scipy.org/numpy/ticket/2108

if anyone wants to help with this one, that'd be very much appreciated.
After these are fixed, the rc1 (possibly rc2) and the final release
should go quickly, as I already know how to make the binaries easily.

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] ANN: NumPy 1.7.0b2 release

[Ondřej Čertík]

On Thu, Sep 20, 2012 at 12:00 PM, Nathaniel Smith n...@pobox.com wrote:
 On Thu, Sep 20, 2012 at 3:33 PM, Ondřej Čertík ondrej.cer...@gmail.com 
 wrote:
 On Thu, Sep 20, 2012 at 4:50 AM, Richard Hattersley
 rhatters...@gmail.com wrote:
 Hi,

 [First of all - thanks to everyone involved in the 1.7 release. Especially
 Ondřej - it takes a lot of time  energy to coordinate something like this.]

 Is there an up to date release schedule anywhere? The trac milestone still
 references June.

 Well, originally we were supposed to release about a month ago, but it
 turned out there are more things to fix.
 Currently, we just need to fix all the issues here:

 https://github.com/numpy/numpy/issues/396

 it looks like a lot, but many of them are really easy to fix, so my
 hope is that it will not take long. The hardest one is this:

 http://projects.scipy.org/numpy/ticket/2108

 if anyone wants to help with this one, that'd be very much appreciated.

 This particular bug should actually be pretty trivial to fix if anyone
 is looking for something to do (esp. if you have a working win32 build
 environment to test your work):
   http://thread.gmane.org/gmane.comp.python.numeric.general/50950/focus=50980

Ah, that looks easy. I'll try to give it a shot. See my repo here how
to get a working win32 environment:

https://github.com/certik/numpy-vendor

However, I don't have access to MSVC, but I am sure somebody else can
test it there, once the PR is ready.

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Status of fixing bugs for the 1.7.0rc1 release

[Ondřej Čertík]

Hi,

I've finally finished review of

https://github.com/numpy/numpy/pull/439

which back-ports all the PRs from master into the release branch and
pushed it in. Here is the current status of bugs for the 1.7.0
release:

https://github.com/numpy/numpy/issues/396

I believe that for example a lot of the Debian based bugs were fixed
by now (in the 1.7.0 branch). Can I release 1.7.0b2? So that others
can try it out,
while we work on the rest of the issues. I don't think it's ready for
rc1 yet, but we've done a lot of work since beta1 I think.

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Contiguity of result of astype changed - intentional?

[Ondřej Čertík]

Hi Matt,

On Wed, Sep 12, 2012 at 1:27 PM, Travis Oliphant tra...@continuum.io wrote:

 Is this intended?  Is there a performance reason to keep the same
 strides in 1.7.0?

 I believe that this could be because in 1.7.0, NumPy was changed so that 
 copying does not always default to C-order but to Keep-order.So, in 
 1.7.0, the strides of b is governed by the strides of a, while in 1.6.1, 
 the strides of b is C-order (because of the copy).


 Thanks for the reply.

 So maybe the bottom line is that the user should not assume any
 contiguity from ``astype``?   If that's the case I'll submit a
 docstring PR to say that.


 Yes, that would be a great addition to the docstring.   Mark, can you confirm 
 this is the desired behavior?  Ondrej, this would be something to put in the 
 release notes, if it isn't already.

If you could submit the PR with the docs, that'd be awesome. In the
meantime, I've created an issue for it:

https://github.com/numpy/numpy/issues/437

and put it into my TODO list:

https://github.com/numpy/numpy/issues/396

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Numpy 1.7b1 API change cause big trouble

[Ondřej Čertík]

Hi Fred,

On Wed, Sep 5, 2012 at 10:56 AM, Nathaniel Smith n...@pobox.com wrote:
 On Wed, Sep 5, 2012 at 6:36 PM, Frédéric Bastien no...@nouiz.org wrote:
 Hi,

 I spent up to now 2 or 3 days making change to Theano to support numpy
 1.7b1. But now, I just find an interface change that will need
 recoding a function, not just small code change.

 My understanding was that 1.7 is not supposed to require any code
 changes... so, separate from your actual question about assigning to
 the data field can I ask: are the changes you're talking about just to
 avoid *deprecated* APIs, or did you have actual problems running
 Theano against 1.7b1? And if you had actual problems, could you say
 what? (Or just post a diff of the changes you found you had to make,
 which should amount to the same thing?)


Thank you for trying the beta version, that was the purpose to put it out there
and see if it breaks things.

As others said, if you can give us more details, that'd be great. Let's get it
fixed before rc1.

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Mysterious test_pareto failure on Travis

[Ondřej Čertík]

On Tue, Sep 4, 2012 at 3:15 AM, Nathaniel Smith n...@pobox.com wrote:
 The last two Travis builds of master have failed consistently with the
 same error:
   http://travis-ci.org/#!/numpy/numpy/builds
 It looks like a real failure -- we're getting the same error on every
 build variant, some sort of problem in test_pareto. Example:
   http://travis-ci.org/#!/numpy/numpy/jobs/2328823

 The obvious culprit would be the previous commit, which regenerated
 mtrand.c with Cython 0.17:
   
 http://github.com/numpy/numpy/commit/cd9092aa71d23359b33e89d938c55fb14b9bf606

 What's weird, though, is that that commit passed just fine on Travis:
   http://travis-ci.org/#!/numpy/numpy/builds/2313124

 It's just the two commits since then that failed. But these commits
 have been 1-line docstring changes, so I don't see how they could have
 possibly created the problem.

 Also, the test passes fine with python 2.7 on my laptop with current master.

 Can anyone reproduce this failure? Any ideas what might be going on?

I made this:

https://github.com/numpy/numpy/issues/424

It was me who updated the Cython file. It seemed to be working. I've
added the issue
to the release TODO.

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Mysterious test_pareto failure on Travis

[Ondřej Čertík]

On Tue, Sep 4, 2012 at 12:31 PM, Ondřej Čertík ondrej.cer...@gmail.com wrote:
 On Tue, Sep 4, 2012 at 3:15 AM, Nathaniel Smith n...@pobox.com wrote:
 The last two Travis builds of master have failed consistently with the
 same error:
   http://travis-ci.org/#!/numpy/numpy/builds
 It looks like a real failure -- we're getting the same error on every
 build variant, some sort of problem in test_pareto. Example:
   http://travis-ci.org/#!/numpy/numpy/jobs/2328823

 The obvious culprit would be the previous commit, which regenerated
 mtrand.c with Cython 0.17:
   
 http://github.com/numpy/numpy/commit/cd9092aa71d23359b33e89d938c55fb14b9bf606

 What's weird, though, is that that commit passed just fine on Travis:
   http://travis-ci.org/#!/numpy/numpy/builds/2313124

 It's just the two commits since then that failed. But these commits
 have been 1-line docstring changes, so I don't see how they could have
 possibly created the problem.

 Also, the test passes fine with python 2.7 on my laptop with current master.

 Can anyone reproduce this failure? Any ideas what might be going on?

 I made this:

 https://github.com/numpy/numpy/issues/424

 It was me who updated the Cython file. It seemed to be working. I've
 added the issue
 to the release TODO.

Ok, here is how to reproduce the problem:

1) install my numpy-vendor vagrant image (32 bit Ubuntu), as directed
in the README:

https://github.com/certik/numpy-vendor

2) run tests, you'll get:

https://gist.github.com/3625509

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Mysterious test_pareto failure on Travis

[Ondřej Čertík]

On Tue, Sep 4, 2012 at 12:41 PM, Ondřej Čertík ondrej.cer...@gmail.com wrote:
 On Tue, Sep 4, 2012 at 12:31 PM, Ondřej Čertík ondrej.cer...@gmail.com 
 wrote:
 On Tue, Sep 4, 2012 at 3:15 AM, Nathaniel Smith n...@pobox.com wrote:
 The last two Travis builds of master have failed consistently with the
 same error:
   http://travis-ci.org/#!/numpy/numpy/builds
 It looks like a real failure -- we're getting the same error on every
 build variant, some sort of problem in test_pareto. Example:
   http://travis-ci.org/#!/numpy/numpy/jobs/2328823

 The obvious culprit would be the previous commit, which regenerated
 mtrand.c with Cython 0.17:
   
 http://github.com/numpy/numpy/commit/cd9092aa71d23359b33e89d938c55fb14b9bf606

 What's weird, though, is that that commit passed just fine on Travis:
   http://travis-ci.org/#!/numpy/numpy/builds/2313124

 It's just the two commits since then that failed. But these commits
 have been 1-line docstring changes, so I don't see how they could have
 possibly created the problem.

 Also, the test passes fine with python 2.7 on my laptop with current master.

 Can anyone reproduce this failure? Any ideas what might be going on?

 I made this:

 https://github.com/numpy/numpy/issues/424

 It was me who updated the Cython file. It seemed to be working. I've
 added the issue
 to the release TODO.

 Ok, here is how to reproduce the problem:

 1) install my numpy-vendor vagrant image (32 bit Ubuntu), as directed
 in the README:

 https://github.com/certik/numpy-vendor

 2) run tests, you'll get:

 https://gist.github.com/3625509

So the problem was actually introduced much earlier. Most probably it
has never worked
in 32bit Ubuntu 12.04. I tried for example this old commit:

https://github.com/numpy/numpy/commit/3882d65c42acf6d5fff8cc9b3f410bb3e49c8af8

and it still fails:

https://gist.github.com/3625943

I tried to test even the first commit that introduced the problem:

https://github.com/numpy/numpy/commit/898e6bdc625cdd3c97865ef99f8d51c5f43eafff

but while it compiled, I got some import error:


(py)vagrant@precise32:~/repos/numpy/tools$ nosetests
/home/vagrant/repos/numpy/py/local/lib/python2.7/site-packages/numpy/random/tests/test_random.py
RuntimeError: module compiled against API version 6 but this version
of numpy is 5
E
==
ERROR: Failure: ImportError (numpy.core.multiarray failed to import)
--
Traceback (most recent call last):
  File 
/home/vagrant/repos/numpy/py/local/lib/python2.7/site-packages/nose/loader.py,
line 390, in loadTestsFromName
addr.filename, addr.module)
  File 
/home/vagrant/repos/numpy/py/local/lib/python2.7/site-packages/nose/importer.py,
line 39, in importFromPath
return self.importFromDir(dir_path, fqname)
  File 
/home/vagrant/repos/numpy/py/local/lib/python2.7/site-packages/nose/importer.py,
line 86, in importFromDir
mod = load_module(part_fqname, fh, filename, desc)
  File 
/home/vagrant/repos/numpy/py/local/lib/python2.7/site-packages/numpy/random/tests/test_random.py,
line 1, in module
from numpy.testing import TestCase, run_module_suite, assert_
  File 
/home/vagrant/repos/numpy/py/local/lib/python2.7/site-packages/numpy/__init__.py,
line 137, in module
import add_newdocs
  File 
/home/vagrant/repos/numpy/py/local/lib/python2.7/site-packages/numpy/add_newdocs.py,
line 9, in module
from numpy.lib import add_newdoc
  File 
/home/vagrant/repos/numpy/py/local/lib/python2.7/site-packages/numpy/lib/__init__.py,
line 4, in module
from type_check import *
  File 
/home/vagrant/repos/numpy/py/local/lib/python2.7/site-packages/numpy/lib/type_check.py,
line 8, in module
import numpy.core.numeric as _nx
  File 
/home/vagrant/repos/numpy/py/local/lib/python2.7/site-packages/numpy/core/__init__.py,
line 10, in module
import _sort
ImportError: numpy.core.multiarray failed to import

--
Ran 1 test in 0.002s

FAILED (errors=1)


But in any case, this seems to be a problem with the actual 32bit
Ubuntu 12.04 itself. So maybe something in gcc
has changed that now triggers the problem.

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Mysterious test_pareto failure on Travis

[Ondřej Čertík]

On Tue, Sep 4, 2012 at 1:48 PM,  josef.p...@gmail.com wrote:
 On Tue, Sep 4, 2012 at 4:37 PM, Nathaniel Smith n...@pobox.com wrote:
 On Tue, Sep 4, 2012 at 9:17 PM, Ondřej Čertík ondrej.cer...@gmail.com 
 wrote:
 On Tue, Sep 4, 2012 at 12:41 PM, Ondřej Čertík ondrej.cer...@gmail.com 
 wrote:
 On Tue, Sep 4, 2012 at 12:31 PM, Ondřej Čertík ondrej.cer...@gmail.com 
 wrote:
 On Tue, Sep 4, 2012 at 3:15 AM, Nathaniel Smith n...@pobox.com wrote:
 The last two Travis builds of master have failed consistently with the
 same error:
   http://travis-ci.org/#!/numpy/numpy/builds
 It looks like a real failure -- we're getting the same error on every
 build variant, some sort of problem in test_pareto. Example:
   http://travis-ci.org/#!/numpy/numpy/jobs/2328823

 The obvious culprit would be the previous commit, which regenerated
 mtrand.c with Cython 0.17:
   
 http://github.com/numpy/numpy/commit/cd9092aa71d23359b33e89d938c55fb14b9bf606

 What's weird, though, is that that commit passed just fine on Travis:
   http://travis-ci.org/#!/numpy/numpy/builds/2313124

 It's just the two commits since then that failed. But these commits
 have been 1-line docstring changes, so I don't see how they could have
 possibly created the problem.

 Also, the test passes fine with python 2.7 on my laptop with current 
 master.

 Can anyone reproduce this failure? Any ideas what might be going on?

 I made this:

 https://github.com/numpy/numpy/issues/424

 It was me who updated the Cython file. It seemed to be working. I've
 added the issue
 to the release TODO.

 Ok, here is how to reproduce the problem:

 1) install my numpy-vendor vagrant image (32 bit Ubuntu), as directed
 in the README:

 https://github.com/certik/numpy-vendor

 2) run tests, you'll get:

 https://gist.github.com/3625509

 So the problem was actually introduced much earlier. Most probably it
 has never worked
 in 32bit Ubuntu 12.04. I tried for example this old commit:

 https://github.com/numpy/numpy/commit/3882d65c42acf6d5fff8cc9b3f410bb3e49c8af8

 and it still fails:
 [...]
 But in any case, this seems to be a problem with the actual 32bit
 Ubuntu 12.04 itself. So maybe something in gcc
 has changed that now triggers the problem.

 To be clear, the mismatching value is:

 /home/njs/numpy/.tox/py27/local/lib/python2.7/site-packages/numpy/random/tests/test_random.py(363)test_pareto()
 - np.testing.assert_array_almost_equal(actual, desired, decimal=15)
 (Pdb) actual[1, 0]
 52828779.702948704
 (Pdb) desired[1, 0]
 52828779.702948518

 So they do match to 14 decimal points, and it's entirely possible that
 this is just a problem of the test being too stringent in requiring 15
 decimal points of match. Maybe the 32-bit GCC is spilling registers
 differently, tripping the famous x86 idiosyncrasy where register
 spilling triggers rounding. But I'd feel better if someone familiar
 with the pareto code could confirm.

 I don't understand this. Isn't assert_almost_equal testing decimals
 not significant digits?

Yes it is, see the docs:

The test is equivalent to ``abs(desired-actual)  0.5 * 10**(-decimal)``.

and the unit test is badly written, because that floating point
number has maybe 14 or 15 good significant digits, but it simply does not have
15 digits after the decimal point, so in particular, that test is
testing that the two
floating point numbers are exactly equal.


Here is a fix:

https://github.com/numpy/numpy/pull/425

Let me know if it is ok to merge it, so that we can work on other
issues and have a working test suite.

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Mysterious test_pareto failure on Travis

[Ondřej Čertík]

On Tue, Sep 4, 2012 at 2:58 PM,  josef.p...@gmail.com wrote:
 On Tue, Sep 4, 2012 at 5:49 PM, Ondřej Čertík ondrej.cer...@gmail.com wrote:
 On Tue, Sep 4, 2012 at 1:48 PM,  josef.p...@gmail.com wrote:
 On Tue, Sep 4, 2012 at 4:37 PM, Nathaniel Smith n...@pobox.com wrote:
 On Tue, Sep 4, 2012 at 9:17 PM, Ondřej Čertík ondrej.cer...@gmail.com 
 wrote:
 On Tue, Sep 4, 2012 at 12:41 PM, Ondřej Čertík ondrej.cer...@gmail.com 
 wrote:
 On Tue, Sep 4, 2012 at 12:31 PM, Ondřej Čertík ondrej.cer...@gmail.com 
 wrote:
 On Tue, Sep 4, 2012 at 3:15 AM, Nathaniel Smith n...@pobox.com wrote:
 The last two Travis builds of master have failed consistently with the
 same error:
   http://travis-ci.org/#!/numpy/numpy/builds
 It looks like a real failure -- we're getting the same error on every
 build variant, some sort of problem in test_pareto. Example:
   http://travis-ci.org/#!/numpy/numpy/jobs/2328823

 The obvious culprit would be the previous commit, which regenerated
 mtrand.c with Cython 0.17:
   
 http://github.com/numpy/numpy/commit/cd9092aa71d23359b33e89d938c55fb14b9bf606

 What's weird, though, is that that commit passed just fine on Travis:
   http://travis-ci.org/#!/numpy/numpy/builds/2313124

 It's just the two commits since then that failed. But these commits
 have been 1-line docstring changes, so I don't see how they could have
 possibly created the problem.

 Also, the test passes fine with python 2.7 on my laptop with current 
 master.

 Can anyone reproduce this failure? Any ideas what might be going on?

 I made this:

 https://github.com/numpy/numpy/issues/424

 It was me who updated the Cython file. It seemed to be working. I've
 added the issue
 to the release TODO.

 Ok, here is how to reproduce the problem:

 1) install my numpy-vendor vagrant image (32 bit Ubuntu), as directed
 in the README:

 https://github.com/certik/numpy-vendor

 2) run tests, you'll get:

 https://gist.github.com/3625509

 So the problem was actually introduced much earlier. Most probably it
 has never worked
 in 32bit Ubuntu 12.04. I tried for example this old commit:

 https://github.com/numpy/numpy/commit/3882d65c42acf6d5fff8cc9b3f410bb3e49c8af8

 and it still fails:
 [...]
 But in any case, this seems to be a problem with the actual 32bit
 Ubuntu 12.04 itself. So maybe something in gcc
 has changed that now triggers the problem.

 To be clear, the mismatching value is:

 /home/njs/numpy/.tox/py27/local/lib/python2.7/site-packages/numpy/random/tests/test_random.py(363)test_pareto()
 - np.testing.assert_array_almost_equal(actual, desired, decimal=15)
 (Pdb) actual[1, 0]
 52828779.702948704
 (Pdb) desired[1, 0]
 52828779.702948518

 So they do match to 14 decimal points, and it's entirely possible that
 this is just a problem of the test being too stringent in requiring 15
 decimal points of match. Maybe the 32-bit GCC is spilling registers
 differently, tripping the famous x86 idiosyncrasy where register
 spilling triggers rounding. But I'd feel better if someone familiar
 with the pareto code could confirm.

 I don't understand this. Isn't assert_almost_equal testing decimals
 not significant digits?

 Yes it is, see the docs:

 The test is equivalent to ``abs(desired-actual)  0.5 * 10**(-decimal)``.

 and the unit test is badly written, because that floating point
 number has maybe 14 or 15 good significant digits, but it simply does not 
 have
 15 digits after the decimal point, so in particular, that test is
 testing that the two
 floating point numbers are exactly equal.


 Here is a fix:

 https://github.com/numpy/numpy/pull/425

 Let me know if it is ok to merge it, so that we can work on other
 issues and have a working test suite.

 I think  all the tests should be changed to use the new assert with
 relative tolerance (similar to approx equal)
 https://gist.github.com/3625943  shows also the other distributions
 with 15 decimals when the values are larger than 10 for example

That sounds reasonable. Josef, would you mind sending a PR with this better fix?
The PR #425 fixes it for now and for me it's now very important to get
all the tests pass again on Travis due to their upgrade.

I am now working on other failures at Travis:

https://github.com/numpy/numpy/issues/394
https://github.com/numpy/numpy/issues/426

As those seem to be causing all the Debian buildbots test failures
reported by Sandro.

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] ANN: NumPy 1.7.0b1 release

[Ondřej Čertík]

On Sat, Sep 1, 2012 at 2:19 AM, Sandro Tosi mo...@debian.org wrote:
 On Fri, Aug 31, 2012 at 8:07 PM, Sandro Tosi mo...@debian.org wrote:
 On Fri, Aug 31, 2012 at 7:17 PM, Ondřej Čertík ondrej.cer...@gmail.com 
 wrote:
 If you could create issues at github: https://github.com/numpy/numpy/issues
 that would be great. If you have time, also with some info about the 
 platform
 and how to reproduce it. Or at least a link to the build logs.

 I've reported it here: https://github.com/numpy/numpy/issues/402

 I've just spammed the issue tracker with additional issues, reporting
 all the test suite failures on Debian architectures; issues are 406 -
 414 .

 Don't hesitate to contact me if you need any support or clarification.

Thanks Sandro for reporting it! I put all of them into my release issue:

https://github.com/numpy/numpy/issues/396

most of the failures seem to be caused by these two issues:

https://github.com/numpy/numpy/issues/394
https://github.com/numpy/numpy/issues/426

so I am looking into this now.

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Should abs([nan]) be supported?

[Ondřej Čertík]

Hi,

When running the test suite, there are problems of this kind:

https://github.com/numpy/numpy/issues/394

which then causes for example the Debian buildbots tests to fail
(https://github.com/numpy/numpy/issues/406).
The problem is really simple:


 from numpy import array, abs, nan
 a = array([1, nan, 3])
 a
array([  1.,  nan,   3.])
 abs(a)
__main__:1: RuntimeWarning: invalid value encountered in absolute
array([  1.,  nan,   3.])


See the issue #394 for detailed explanation why nan is being passed
to abs(). Now the question is, what should the right fix be?

1) Should the runtime warning be disabled?

2) Should the tests be reworked, so that nan is not tested in allclose()?

3) Should abs() be fixed to not emit the warning?

4) Should the test suite be somehow fixed not to fail if there are
runtime warnings?

Let me know which direction we should go.

Thanks,
Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Should abs([nan]) be supported?

[Ondřej Čertík]

On Tue, Sep 4, 2012 at 8:38 PM, Travis Oliphant tra...@continuum.io wrote:

 There is an error context that controls how floating point signals are 
 handled.   There is a separate control for underflow, overflow, divide by 
 zero, and invalid.   IIRC, it was decided on this list a while ago to make 
 the default ignore for underflow and warning for  overflow, invalid and 
 divide by zero.

 However, an oversight pushed versions of NumPy where all the error handlers 
 where set to ignore and this test was probably written then.I think the 
 test should be changed to check for RuntimeWarning on some of the cases.   
 This might take a little work as it looks like the code uses generators 
 across multiple tests and would have to be changed to handle expecting 
 warnings.

 Alternatively, the error context can be set before the test runs and then 
 restored afterwords:

 olderr = np.seterr(invalid='ignore')
 abs(a)
 np.seterr(**olderr)


 or, using an errstate context ---

 with np.errstate(invalid='ignore'):
   abs(a)

I see --- so abs([nan]) should emit a warning, but in the test we
should suppress it.
I'll work on that.

The only thing that I don't understand is why it only happens on some
platforms and doesn't on some other platforms (apparently). But it's
clear how to fix it now.

Thanks for the information.

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] How to debug reference counting errors

[Ondřej Čertík]

Hi,

There is segfault reported here:

http://projects.scipy.org/numpy/ticket/1588

I've managed to isolate the problem and even provide a simple patch,
that fixes it here:

https://github.com/numpy/numpy/issues/398

however the patch simply doesn't decrease the proper reference, so it
might leak. I've used
bisection (took the whole evening unfortunately...) but the good news
is that I've isolated commits
that actually broke it. See the github issue #398 for details, diffs etc.

Unfortunately, it's 12 commits from Mark and the individual commits
raise exception on the segfaulting code,
so I can't pin point the problem further.

In general, how can I debug this sort of problem? I tried to use
valgrind, with a debugging build of numpy,
but it provides tons of false (?) positives: https://gist.github.com/3549063

Mark, by looking at the changes that broke it, as well as at my fix,
do you see where the problem could be?

I suspect it is something with the changes in PyArray_FromAny() or
PyArray_FromArray() in ctors.c.
But I don't see anything so far that could cause it.

Thanks for any help. This is one of the issues blocking the 1.7.0 release.

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] view of recarray issue

[Ondřej Čertík]

On Fri, Aug 31, 2012 at 6:15 AM, Jay Bourque jay.bour...@continuum.io wrote:
 Ondrej,

 Sorry for the delay in getting back to this. I have some free time today to
 get this resolved if you haven't already fixed it.

I haven't. If you can look at it, that would be absolutely awesome.
If you don't manage to fix it, if you can give me some hints what's
going on, that would also be a huge help.

Many thanks!
Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Issues for 1.7.0

[Ondřej Čertík]

On Fri, Aug 31, 2012 at 9:27 AM, Charles R Harris
charlesr.har...@gmail.com wrote:


 On Thu, Aug 30, 2012 at 10:47 PM, Ondřej Čertík ondrej.cer...@gmail.com
 wrote:

 Hi,

 I am keeping track of all issues that need to be done for the 1.7.0
 release here:

 https://github.com/numpy/numpy/issues/396

 If you have trac and github push access, here is how you can help (by
 closing/merging):

 Issues that need clarification:

 http://projects.scipy.org/numpy/ticket/2150
 http://projects.scipy.org/numpy/ticket/2101

 Issues fixed (should be closed):

 http://projects.scipy.org/numpy/ticket/2185
 http://projects.scipy.org/numpy/ticket/2066
 http://projects.scipy.org/numpy/ticket/2189

 PRs that need merging:

 https://github.com/numpy/numpy/pull/395
 https://github.com/numpy/numpy/pull/397


 There are still a few more (see my github issue above), that I am
 working on right now.


 Ondrej,

 It looks like you don't have commit rights. Is that the case? If you are the
 release manager I think you need both commit rights and the right to close
 tickets.

Yes, I don't have commit rights nor the rights to close tickets.

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Temporary error accessing NumPy tickets

[Ondřej Čertík]

On Fri, Aug 31, 2012 at 9:35 AM, Pauli Virtanen p...@iki.fi wrote:
 Ondřej Čertík ondrej.certik at gmail.com writes:
 When I access tickets, for example:

 http://projects.scipy.org/numpy/ticket/2185

 then sometimes I get:

 Trac detected an internal error:
 OperationalError: database is locked

 For example yesterday. A refresh in about a minute fixed the problem.
 Today it still lasts at the moment.

 The failures are probably partly triggered by the machine running out of 
 memory.
 It runs services on mod_python, which apparently slowly leaks. Someone (who?)
 with root access on the machine needs to restart Apache. (Note: apachectl
 graceful is not enough to correct this, it needs a real restart of the 
 process.)

I see.


 Longer term solution is to move out of mod_python (mod_wsgi likely, going to 
 CGI
 will create other performance problems), or to transition the stuff there to a
 more beefy server.

Or move the tickets to github.

Yesterday it was very unreliable (I had to wait a long time before a
comment was posted, and about 50%
of the time it was not posted due to the database error). So I just
created a github issue for the same thing and posted
my comments there. Then I could work fast.

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] ANN: NumPy 1.7.0b1 release

[Ondřej Čertík]

Hi Sandro,

On Fri, Aug 31, 2012 at 6:18 AM, Sandro Tosi mo...@debian.org wrote:
 Hello,

 On Tue, Aug 21, 2012 at 6:24 PM, Ondřej Čertík ondrej.cer...@gmail.com 
 wrote:
 Hi,

 I'm pleased to announce the availability of the first beta release of
 NumPy 1.7.0b1.

 I've just uploaded it to Debian experimental, so we can give it a run
 while in freeze. Some of the buildds are already building[1] the
 package, so we should get results asap (either failures or successes).

This is awesome, thanks you so much for doing this. This should
reveal some bugs.


 [1] 
 https://buildd.debian.org/status/package.php?p=python-numpysuite=experimental

 If tests fail, it won't stop the build, and indeed I got at least 2
 errors (actually 1 error and 1 crash), when running tests for python
 2.7 and 3.2 with debug enabled:

 2.7 dbg

 ==
 ERROR: test_power_zero (test_umath.TestPower)
 --
 Traceback (most recent call last):
   File 
 /tmp/buildd/python-numpy-1.7.0~b1/debian/tmp/usr/lib/python2.7/dist-packages/numpy/core/tests/test_umath.py,
 line 139, in test_power_zero
 assert_complex_equal(np.power(zero, 0+1j), cnan)
 RuntimeWarning: invalid value encountered in power

 --

 3.2 dbg

 python3.2-dbg: numpy/core/src/multiarray/common.c:161:
 PyArray_DTypeFromObjectHelper: Assertion
 `((PyObject*)(temp))-ob_type))-tp_flags  ((1L27))) != 0)'
 failed.
 Aborted

 I'm reporting them here since you asked so, dunno if you want an issue
 on github to track them. I'll look at the buildds logs and report
 additional failures if they come up.

If you could create issues at github: https://github.com/numpy/numpy/issues
that would be great. If you have time, also with some info about the platform
and how to reproduce it. Or at least a link to the build logs.

I'll add it to the release TODO and try to fix it.

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion